A biotech company · 2026 · Biotech, AI infrastructure

Multi-tenant GPU inference platform

A biotech company ran its LLM inference on fixed, on-premises GPU servers, where new capacity took 12 to 20 weeks to arrive. I designed and led the hands-on build of a multi-tenant GPU platform on AWS EKS, then migrated every tenant's workloads onto it.

Role
Technical Consultant (Stratgy): platform design and hands-on build
Team
Led the client's engineers through the build and migration
Scope
Multi-tenant vLLM inference · NVIDIA H200 with MIG · HIPAA, SOC 2, ISO 27001

The challenge

Inference ran on a fixed set of on-premises GPU servers. Adding capacity meant buying hardware, with lead times of 12 to 20 weeks. Several teams shared the same machines, and the environment had to meet HIPAA, SOC 2 and ISO 27001 requirements throughout.

What I did

  • Designed the platform and built it with the client’s engineers: Terraform, EKS Auto Mode, and GitOps delivery through ArgoCD and Helm.
  • Added GPU autoscaling with KEDA, including scale to zero, across Spot and On-Demand node pools.
  • Partitioned NVIDIA H200 GPUs with MIG, using profiles sized to each workload to raise density and cut cost.
  • Built tenant isolation into the platform:
    • per-tenant namespaces and quotas
    • NetworkPolicies
    • Istio Ambient mTLS and AuthorizationPolicies
    • Pod Security Standards
    • least-privilege IRSA
    • KMS encryption
    • audit logging
  • Managed secrets with External Secrets, set up observability with Prometheus and Grafana, and wrote operational runbooks.
  • Validated the platform with load, chaos (node and GPU failure) and DR testing, then migrated all tenant workloads to production.

Outcome

  • All tenantsmigrated to the new platform in production
  • On demandGPU capacity, replacing 12–20 week hardware lead times
  • Scale to zeroidle GPU capacity released automatically
  • Testedload, chaos and DR testing completed before go-live

Stack

  • AWS EKS Auto Mode
  • Terraform
  • vLLM
  • NVIDIA H200 MIG
  • KEDA
  • Istio Ambient
  • External Secrets
  • ArgoCD
  • Helm
  • Prometheus
  • Grafana
  • KMS
  • IRSA
Next projectCloud-native EV charging platform →