NVIDIA Software Engineer Case Interview: Designing and Optimizing a GPU-Accelerated Inference Service

Overview This case simulates a common NVIDIA problem space: designing and performance-tuning a GPU-accelerated microservice for real-time ML inference. It emphasizes NVIDIA’s culture of deep technical rigor, performance-first thinking, and end-to-end ownership. Candidates are expected to quantify trade-offs, reason about CUDA/C++ details, and leverage NVIDIA’s platform stack (e.g., CUDA, TensorRT, Triton Inference Server, NCCL, Nsight Systems/Compute, DCGM) while making pragmatic system design choices. Candidate Prompt (give verbatim) “You are building a low-latency vision inference service to power a global AR application. Initial target: 6,000 RPS at p99 ≤ 35 ms end-to-end per request, with models deployed on NVIDIA GPUs in a Kubernetes-based cluster. The model is a ResNet50-like CNN initially in FP32. You will run on nodes with 8x NVIDIA GPUs connected via NVLink; some regions offer MIG-capable GPUs. Network is 100 Gbps with GPUDirect RDMA available. Your task is to propose an end-to-end design, then identify and implementable optimizations to reliably meet the SLO across regions while keeping cost per 1,000 requests low. Expect to handle traffic bursts (2x for 5 minutes), A/B model rollouts, and graceful degradation when a GPU fails or thermally throttles.” Key Requirements and Constraints to Share - SLOs: p50 ≤ 15 ms, p95 ≤ 25 ms, p99 ≤ 35 ms; availability ≥ 99.9%. - Hardware: 8x data center GPUs/node, NVLink, NICs with GPUDirect; some pools with MIG enabled. - Software stack candidates may use: CUDA, TensorRT, Triton, NCCL, Nsight Systems/Compute, CUDA Graphs, DCGM + Prometheus, Kubernetes device plugin. - Model traffic mix: 80% standard requests, 20% high-priority (AR UI) with stricter p99 (≤ 25 ms) and preemption allowed. - Compliance: zero plaintext PII at rest; encrypted in transit. What the Candidate Should Produce (during the interview) - A high-level architecture: request path from NIC → preprocessing → inference → postprocessing → response. - A GPU-aware performance plan: batch sizing, concurrency, streams, memory movement strategies, and telemetry. - A scaling plan: multi-GPU and multi-node strategy (replication vs. sharding), MIG usage, and bin-packing on K8s. - A reliability plan: failure modes, admission control, canaries, graceful degradation, and SLO protection. - Concrete optimization steps prioritized by expected latency/throughput impact and measurement strategy. Suggested Agenda (interviewer pacing) - 0–10 min: Clarify goals, metrics, and constraints; candidate proposes initial design. - 10–25 min: GPU pipeline deep dive (CUDA/TensorRT/Triton choices, memory hierarchy, streams, CUDA Graphs, overlapping H2D/D2H transfer with compute). - 25–40 min: Scaling and scheduling (multi-GPU with NVLink/NCCL, MIG partitioning, K8s scheduling and QoS, burst handling, priority traffic isolation). - 40–55 min: Reliability/observability (DCGM metrics, Nsight profiling plan, nvidia-smi integration, circuit breakers, backpressure, SLO-aware admission control). - 55–70 min: Trade-offs and back-of-the-envelope sizing; cost/SLA calculations; security considerations; rollout plan. - 70–75 min: Wrap-up: recap priorities and next steps. Deep-Dive Prompts (use as follow-ups) - Batching vs. latency: How do you pick dynamic batch sizes per priority class? Where would you set max queue delay? How do you use Triton’s dynamic batching and concurrency groups? - Memory strategy: Pinned host memory, zero-copy opportunities, staging buffers; how to avoid pageable faults. Shared memory vs. L2 vs. HBM trade-offs; mitigating warp divergence. - Concurrency: Streams per model instance, CUDA events for synchronization, overlapping preprocess/inference/postprocess; when to apply CUDA Graphs. - Precision and optimization: FP32 → FP16/INT8 via TensorRT; calibration strategy; expected latency/throughput gains and accuracy impact. - Multi-GPU: Model replication vs. tensor/pipe parallelism; NVLink bandwidth considerations; when to use NCCL; spillover between GPUs. - MIG: When to partition; sizing strategy for mixed workload isolation; pros/cons versus full-GPU scheduling. - Reliability: Handling GPU ECC errors, thermal throttling, driver resets; health checks using DCGM; fast failover strategies. - Observability: What metrics matter (GPU util, SM occupancy, DRAM bw, L2 hit rate, achieved occupancy, queue depth, p50/p95/p99, tail at scale)? How to instrument with Nsight Systems/Compute and production telemetry. - Security: TLS termination, memory cleanup on context destruction, multi-tenant isolation. Evaluation Rubric (NVIDIA-specific emphasis) - Problem framing and SLO ownership (15%): Clarifies targets, proposes measurable success criteria; pushes for data before choosing defaults. - GPU-aware system design (25%): Correctly uses CUDA/TensorRT/Triton; demonstrates understanding of streams, memory transfers, occupancy, divergence, and how these affect p99. - Performance engineering (25%): Prioritizes high-impact optimizations; proposes a sound profiling plan (Nsight Systems/Compute); quantifies expected gains (e.g., FP16 ~1.5–2.5× latency improvement depending on model/batch; INT8 further if accuracy holds). - Distributed scaling and scheduling (15%): Sound reasoning on replication vs. sharding, NVLink/NCCL use, MIG partitioning, K8s bin-packing, and priority isolation. - Reliability and production readiness (10%): Failure modes, health checks, admission control, canaries, rollback, and SLO protection. - Communication and NVIDIA culture fit (10%): Direct, data-driven, detail-oriented; balances low-level depth with system view; demonstrates ownership and collaboration. What Good Looks Like (signals) - Anchors decisions to p99 and throughput targets; uses back-of-the-envelope math to justify batch/concurrency. - Chooses FP16 or INT8 with a calibration/validation plan; uses CUDA Graphs to reduce launch overhead at high QPS. - Overlaps H2D/D2H with compute using pinned memory + multiple streams; proposes dynamic batching with per-class queue caps. - Plans Nsight profiling runs that isolate kernel vs. transfer bottlenecks; ties findings to specific TensorRT/Triton settings. - Presents a simple, testable MVP and a roadmap of iterative optimizations with clear metrics and rollback criteria.