Multi-tenant GPU inference platform
A biotech company ran its LLM inference on fixed, on-premises GPU servers, where new capacity took 12 to 20 weeks to arrive. I designed and led the hands-on build of a multi-tenant GPU platform on AWS EKS, then migrated every tenant's workloads onto it.
- Role
- Technical Consultant (Stratgy): platform design and hands-on build
- Team
- Led the client's engineers through the build and migration
- Scope
- Multi-tenant vLLM inference · NVIDIA H200 with MIG · HIPAA, SOC 2, ISO 27001
The challenge
Inference ran on a fixed set of on-premises GPU servers. Adding capacity meant buying hardware, with lead times of 12 to 20 weeks. Several teams shared the same machines, and the environment had to meet HIPAA, SOC 2 and ISO 27001 requirements throughout.
What I did
- Designed the platform and built it with the client’s engineers: Terraform, EKS Auto Mode, and GitOps delivery through ArgoCD and Helm.
- Added GPU autoscaling with KEDA, including scale to zero, across Spot and On-Demand node pools.
- Partitioned NVIDIA H200 GPUs with MIG, using profiles sized to each workload to raise density and cut cost.
- Built tenant isolation into the platform:
- per-tenant namespaces and quotas
- NetworkPolicies
- Istio Ambient mTLS and AuthorizationPolicies
- Pod Security Standards
- least-privilege IRSA
- KMS encryption
- audit logging
- Managed secrets with External Secrets, set up observability with Prometheus and Grafana, and wrote operational runbooks.
- Validated the platform with load, chaos (node and GPU failure) and DR testing, then migrated all tenant workloads to production.
Outcome
- All tenantsmigrated to the new platform in production
- On demandGPU capacity, replacing 12–20 week hardware lead times
- Scale to zeroidle GPU capacity released automatically
- Testedload, chaos and DR testing completed before go-live
Stack
- AWS EKS Auto Mode
- Terraform
- vLLM
- NVIDIA H200 MIG
- KEDA
- Istio Ambient
- External Secrets
- ArgoCD
- Helm
- Prometheus
- Grafana
- KMS
- IRSA