Datadog AI Engineer case interview: Design a Bits AI–powered incident triage and code‑fix agent

This Datadog-style case centers on scoping, designing, and operationalizing an AI agent that uses Datadog telemetry (logs, metrics, traces, RUM, security signals) to triage a live incident, explain likely root cause, and propose a safe fix—optionally generating a PR via the Dev Agent workflow. You’ll be asked to reason about data flows, model choices, inference/serving, evaluation, and safe rollout in a multi-tenant, high-scale observability platform. The scenario reflects Datadog’s real AI surface area (Bits AI and the Dev Agent) and emphasis on pragmatic incident response and developer workflows. ([datadoghq.com](https://www.datadoghq.com/product/platform/bits-ai/?utm_source=chatgpt.com)) Format (approx. 75 minutes): - 10 min: Prompt and clarifying questions (define the customer impact, SLOs—e.g., single‑digit‑second interactive responses—and constraints like data residency, RBAC, and multi-tenant safety). - 30 min: System and data design (ingestion and aggregation from APM/logs/metrics; schema/tags such as service and version; feature/RAG stores; retrieval vs. fine‑tuning choices; latency/cost tradeoffs; caching; fallbacks; multi‑region HA; guardrails for prompt injection/PII). ([docs.datadoghq.com](https://docs.datadoghq.com/bits_ai/chat_with_bits_ai?utm_source=chatgpt.com)) - 15 min: Modeling and serving (LLM selection, policy flows, tool use/agents, Ray- or microservice‑based inference, observability for LLMs, shadow/A‑B testing, rollback). ([careers.datadoghq.com](https://careers.datadoghq.com/detail/7043352/?utm_source=chatgpt.com)) - 10 min: Evaluation strategy (offline metrics, golden sets from past incidents, online KPIs like MTTR/false-action rate, safety checks, token/latency cost budgets) and success criteria. - 10 min: Productionization & culture (feature‑flag rollouts, on‑call readiness, incident comms, postmortems, and user feedback loops—aligned with Datadog’s ship‑fast, talk‑to‑users ethos; expect a collaborative, conversational interview dynamic). ([careers.datadoghq.com](https://careers.datadoghq.com/detail/6628140/?utm_source=chatgpt.com), [reddit.com](https://www.reddit.com/r/cscareerquestionsEU/comments/1dl6eh9?utm_source=chatgpt.com)) What we probe (focus areas): - Problem framing: clear restatement of the incident and the AI agent’s objective and guardrails. - Architecture depth: end‑to‑end data paths from telemetry to action interface; tenancy boundaries; failure modes; backpressure/queuing; cost controls. - AI/ML rigor: retrieval vs. fine‑tune rationale; evaluation design; safety; drift detection; LLMObs signals you’d collect (latency, token usage, hallucination/error classes). ([careers.datadoghq.com](https://careers.datadoghq.com/detail/7043352/?utm_source=chatgpt.com)) - MLOps/serving: GPU/CPU mix, autoscaling, model registry, shadow/A‑B, progressive delivery, and rollback. ([careers.datadoghq.com](https://careers.datadoghq.com/detail/7050510/?utm_source=chatgpt.com)) - Operability: dashboards/alerts for the agent, SLOs, incident playbooks, and postmortem automation using Bits AI. ([datadoghq.com](https://www.datadoghq.com/blog/bits-ai-sre/?utm_source=chatgpt.com)) - Datadog fit: show ownership, pragmatism, and teamwork; you may encounter one in‑person step in the overall process. ([careers.datadoghq.com](https://careers.datadoghq.com/candidate-experience/?utm_source=chatgpt.com))