Experimenting with GPUs: GKE managed DRANET and Inference Gateway AI Deployment

Building and serving models on infrastructure is a strong use case for businesses. In Google Cloud, you have the ability to design your AI infrastructure to suit your workloads. Recently, I experimented with Google Kubernetes Engine (GKE) managed DRANET while deploying a model for inference with NVIDIA B200 GPUs on GKE. In this blog, we will explore this setup in easy to follow steps. What is DRANET Dynamic Resource Allocation (DRA) is a feature that lets you request and share resources among Pods. DRANET allows you to request and allocate networking resources for your Pods, including network interfaces that support TPUs & Remote Direct Memory Access (RDMA). In my case, the use of high-end GPUs. How GPU RDMA VPC works The RDMA network is set up as an isolated VPC, which is regional and assigned a network profile type. In this case, the network profile type is RoCEv2. This VPC is dedicated for GPU-to-GPU communication. The GPU VM families have RDMA capable NICs that connect to the RDMA VPC. The GPUs communicate between multiple nodes via this low latency, high speed rail aligned setup. Design pattern example Our aim was to deploy a LLM model (Deepseek) onto a GKE cluster with A4 nodes that support 8 B200 GPUs and serve it via GKE Inference gateway privately. To set up an AI Hypercomputer GKE cluster, you can use the Cluster Toolkit, but in my case, I wanted to test the GKE managed DRANET dynamic setup of the networking that supports RDMA for the GPU communication. This design utilizes the following services to provide an end-to-end solution: VPC: Total of 3 VPC. One VPC manually created, two created automatically by GKE managed DRANET, one standard and one for RDMA. GKE: To deploy the workload. GKE Inference gateway: To expose the workload internally using a regional internal Application Load Balancers type gke-l7-rilb. A4 VM’s: These support RoCEv2 with NVIDIA B200 GPU. Putting it together To get access to the A4 VM a future reservation was used. This is linked to a specific zone. Begin: Set up the environment Create a standard VPC, with firewall rules and subnet in the same zone as the reservation. Create a proxy-only subnet this will be used with the Internal regional application load balancer attached to the GKE inference gateway Next: Create a standard GKE cluster node and default node pool. code_block <ListValue: [StructValue([('code', 'gcloud container clusters create CLUSTER_NAME rn --location=ZONE rn --num-nodes=1 rn --machine-type=e2-standard-16 rn --network={GVNIC_NETWORK_PREFIX}-main rn --subnetwork={GVNIC_NETWORK_PREFIX}-sub rn --release-channel rapid rn --enable-dataplane-v2 rn --enable-ip-alias rn --addons=HttpLoadBalancing,RayOperator rn --gateway-api=standard rn --enable-ray-cluster-logging rn --enable-ray-cluster-monitoring rn --enable-managed-prometheus rn --enable-dataplane-v2-metrics rn --monitoring=SYSTEM'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da5fa0>)])]> Once that is complete you can connect to your cluster: code_block <ListValue: [StructValue([('code', 'gcloud container clusters get-credentials CLUSTER_NAME --zone ZONE --project PROJECT'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da5130>)])]> Create a GPU node pool (this example uses, A4 VM with reservation) and additionals flags: ---accelerator-network-profile=auto (GKE automatically adds the gke.networks.io/accelerator-network-profile: auto label to the nodes) --node-labels=cloud.google.com/gke-networking-dra-driver=true (Enables DRA for high-performance networking) code_block <ListValue: [StructValue([('code', 'gcloud beta container node-pools create NODE_POOL_NAME rn --cluster CLUSTER_NAME rn --location ZONE rn --node-locations ZONE rn --machine-type a4-highgpu-8g rn --accelerator type=nvidia-b200,count=8,gpu-driver-version=latest rn --enable-autoscaling --num-nodes=1 --total-min-nodes=1 --total-max-nodes=3 rn --reservation-affinity=specific rn--reservation=projects/PROJECT/reservations/RESERVATION_NAME/reservationBlocks/BLOCK_NAME rn --accelerator-network-profile=auto rn--node-labels=cloud.google.com/gke-networking-dra-driver=true'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da5970>)])]> Next: Create a ResourceClaimTemplate, which will be used to attach the networking resources to your deployments. The deviceClassName: mrdma.google.com is used for GPU workloads: code_block <ListValue: [StructValue([('code', 'apiVersion: resource.k8s.io/v1rnkind: ResourceClaimTemplaternmetadata:rn name: all-mrdmarnspec:rn spec:rn devices:rn requests:rn - name: req-mrdmarn exactly:rn deviceClassName: mrdma.google.comrn allocationMode: All'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da56d0>)])]> Deploy model and inference Now that a cluster and node pool is setup, we can deploy a model and serve it via Inference gateway. In my experiment I used DeepSeek but this could be any model. Deploy model and services The nodeSelector: gke.networks.io/accelerator-network-profile: auto is used to assign to the GPU node The resourceClaims: attaches the resource we defined for networking Create a secret (I used Hugging Face token): code_block <ListValue: [StructValue([('code', 'kubectl create secret generic hf-secret rn --from-literal=hf_token={HF_TOKEN}'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da5070>)])]> Deployment code_block <ListValue: [StructValue([('code', 'apiVersion: apps/v1rnkind: Deploymentrnmetadata:rn name: deepseek-v3-1-deployrnspec:rn replicas: 1rn selector:rn matchLabels:rn app: deepseek-v3-1rn template:rn metadata:rn labels:rn app: deepseek-v3-1rn ai.gke.io/model: deepseek-v3-1rn ai.gke.io/inference-server: vllmrn examples.ai.gke.io/source: user-guidern spec:rn containers:rn - name: vllm-inferencern image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250819_0916_RC01rn resources:rn requests:rn cpu: "190"rn memory: "1800Gi"rn ephemeral-storage: "1Ti"rn nvidia.com/gpu: "8"rn limits:rn cpu: "190"rn memory: "1800Gi"rn ephemeral-storage: "1Ti"rn nvidia.com/gpu: "8"rn claims:rn - name: rdma-claimrn command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]rn args:rn - --model=(MODEL_ID)rn - --tensor-parallel-size=8rn - --host=0.0.0.0rn - --port=8000rn - --max-model-len=32768rn - --max-num-seqs=32rn - --gpu-memory-utilization=0.90rn - --enable-chunked-prefillrn - --enforce-eagerrn - --trust-remote-codern env:rn - name: MODEL_IDrn value: deepseek-ai/DeepSeek-V3.1rn - name: HUGGING_FACE_HUB_TOKENrn valueFrom:rn secretKeyRef:rn name: hf-secretrn key: hf_tokenrn volumeMounts:rn - mountPath: /dev/shmrn name: dshmrn livenessProbe:rn httpGet:rn path: /healthrn port: 8000rn initialDelaySeconds: 1800rn periodSeconds: 10rn readinessProbe:rn httpGet:rn path: /healthrn port: 8000rn initialDelaySeconds: 1800rn periodSeconds: 5rn volumes:rn - name: dshmrn emptyDir:rn medium: Memoryrn nodeSelector:rn gke.networks.io/accelerator-network-profile: autorn resourceClaims:rn - name: rdma-claimrn resourceClaimTemplateName: all-mrdmarn---rnapiVersion: v1rnkind: Servicernmetadata:rn name: deepseek-v3-1-servicernspec:rn selector:rn app: deepseek-v3-1rn type: ClusterIPrn ports:rn - protocol: TCPrn port: 8000rn targetPort: 8000rn---rnapiVersion: monitoring.googleapis.com/v1rnkind: PodMonitoringrnmetadata:rn name: deepseek-v3-1-monitoringrnspec:rn selector:rn matchLabels:rn app: deepseek-v3-1rn endpoints:rn - port: 8000rn path: /metricsrn interval: 30s'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da5f70>)])]> Deploy GKE Inference Gateway This install needed Custom Resource Definitions (CRDs) in your GKE cluster: For GKE versions 1.34.0-gke.1626000 or later, install only the alpha InferenceObjective CRD: code_block <ListValue: [StructValue([('code', 'kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/v1.0.0/config/crd/bases/inference.networking.x-k8s.io_inferenceobjectives.yaml'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da55e0>)])]> Create Inference pool code_block <ListValue: [StructValue([('code', 'helm install deepseek-v3-pool rn oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool rn --version v1.0.1 rn --set inferencePool.modelServers.matchLabels.app=deepseek-v3-1 rn --set provider.name=gke rn --set inferenceExtension.monitoring.gke.enabled=true'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da5520>)])]> Create the Gateway, HTTPRoute and InferenceObjective code_block <ListValue: [StructValue([('code', '# 1. The Regional Internal Gateway (ILB)rnapiVersion: gateway.networking.k8s.io/v1rnkind: Gatewayrnmetadata:rn name: deepseek-v3-gatewayrn namespace: defaultrnspec:rn gatewayClassName: gke-l7-rilbrn listeners:rn - name: httprn protocol: HTTPrn port: 80rn allowedRoutes:rn namespaces:rn from: Samern---rn# 2. The HTTPRoute (Routing to the Pool)rnapiVersion: gateway.networking.k8s.io/v1rnkind: HTTPRouternmetadata:rn name: deepseek-v3-routern namespace: defaultrnspec:rn parentRefs:rn - name: deepseek-v3-gatewayrn rules:rn - matches:rn - path:rn type: PathPrefixrn value: /rn backendRefs:rn - group: inference.networking.k8s.iorn kind: InferencePoolrn name: deepseek-v3-poolrn---rn# 3. The Inference Objective (Performance Logic)rnapiVersion: inference.networking.x-k8s.io/v1alpha2rnkind: InferenceObjectivernmetadata:rn name: deepseek-v3-objectivern namespace: defaultrnspec:rn poolRef:rn name: deepseek-v3-pool'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da5370>)])]> Once complete, you can create a test VM in your main VPC and make a call to the IP address of the GKE Inference Gateway: code_block <ListValue: [StructValue([('code', 'curl -N -s -X POST "http://$GATEWAY_IP/v1/chat/completions" rn -H "Content-Type: application/json" rn -d '{rn "model": "deepseek-ai/DeepSeek-V3.1",rn "messages": [{"role": "user", "content": "Box A: red. Box B: blue. Box C: empty. Move A to C, Move B to A, Swap B and C. Where is red?"}],rn "stream": truern }' | stdbuf -oL grep "data: " | sed -u s/^data: //' | grep -v " [DONE ]" | rn jq --unbuffered -rj '.choices[0].delta | (.reasoning_content // .reasoning // .content // empty)''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7f3aa9da57c0>)])]> Next Steps Take a deeper dive into GKE managed DRANET and GKE Inference Gateway, review the following. Blog: DRA: A new era of Kubernetes device management with Dynamic Resource Allocation Document set: DRANET Documentation: AI Hypercomputer Want to ask a question, find out more or share a thought? Please connect with me on Linkedin.

Source: Google Cloud AI | 2026-04-08

Related