Meta AI Engineer Case Interview: Large-Scale Recommendation & Integrity ML System Design

Format overview (Meta-specific): - Signal-based case focused on impact, pragmatism, and safety. Interviewers probe product sense, ML depth, and systems thinking under real Meta constraints (planet-scale data, tight latency budgets, online experimentation culture, and Community Standards). Expect frequent: "What trade-off are you making and why?" and "How would you know it worked?" Follow-ups become progressively harder. Case prompt (given at start): "You are an AI Engineer joining the Reels/Feed ranking team. Leadership asks for a 0.5–1.0% lift in long-term value while reducing policy-violating content exposure. Design an end-to-end ML solution to: (a) retrieve and rank content for each user session, (b) proactively down-rank harmful/low-quality items, and (c) ship safely via online experiments. You have strict P99 inference latency of ≤100 ms for ranking on production hardware and must respect privacy and fairness constraints." What the candidate should cover: 1) Problem framing and goals - Clarify north-star and guardrail metrics: examples include long-term user value/retention, MSI (meaningful social interactions), session quality, creator health, integrity prevalence, reports/hides, and time spent quality (not just raw engagement). Define objective boundaries: "Lift engagement without increasing violation exposure or teen risk metrics." - Identify stakeholders: users, creators, integrity ops/policy, privacy, infra, and product. 2) Data and labeling strategy - Signals: graph/social features, user-action sequences, dwell time, hides/reports/surveys, creator reputation, lightweight NLP/vision/audio signals, country/age/regulatory flags. - Labels: mix of explicit actions, human-rating programs, weak supervision, model-generated pseudo-labels; discuss label noise, drift, and rater bias controls. - Privacy-by-design: data minimization, regional storage/processing, aggregation, differential privacy or noise where appropriate. 3) Candidate generation and ranking architecture - Two-stage or three-stage design: (a) candidate retrieval (e.g., two-tower or ANN over embeddings), (b) mid-rank filters for integrity/quality, (c) final ranker. - Model choices: multi-task learning with calibrated heads (click, re-share, MSI proxy, satisfaction/survey), plus integrity risk head; consider mixture-of-experts or DLRM-style models for personalization; optionally multimodal encoders for content understanding (vision/text/audio). Discuss distillation/quantization to meet latency and memory budgets. - Objective: composite loss L = w1*engagement + w2*satisfaction + w3*MSI - w4*integrity_risk - w5*bad_experience; justify weights and regularization; ensure calibration for decision thresholds. 4) Integrity and safety integration - Real-time safety filters: policy classifiers, hash/blocklists, language/age safety, link trust, adversarial/spam detection. - Strategy: pre-filter high-risk, then down-rank via calibrated risk scores; hard blocks for certain categories. Explain appeals/override pathways and creator feedback loops to avoid over-suppression. - Fairness checks: compare impact across demographics/regions; define bias metrics and mitigations. 5) Evaluation plan (Meta-style experimentation) - Offline: AUC/PR for heads, NDCG for ranking, calibration (ECE), confusion cost analysis for integrity. - Online A/B: staged rollout (1% → 10% → 50%), holdouts, sequential testing, guardrails (reports, hides, policy-violation exposure, creator impact), pre-defined stop conditions and kill switches. - Long-term and novelty effects: ghost/shadow experiments, interleaving, switchback where applicable. 6) Productionization and reliability - Latency/throughput: caching hot features, precomputed embeddings, feature stores, batch vs streaming updates, approximate retrieval (ANN), inference batching with mixed precision, model partitioning across CPU/GPU. - Monitoring: real-time dashboards for integrity prevalence, calibration drift, data freshness, and feature integrity; on-call playbooks. - Rollback/gradual rollout, dry-runs in shadow and canary, autoscaling and budget awareness. 7) Extensions and deeper probes (used for follow-ups) - Cold start handling for new users/creators; onboarding exploration budget. - Multimodal quality: using LLM/VLM embeddings for semantics and safety; distillation for edge/server efficiency. - Region-specific policy differences and regulatory constraints. - Adversarial behavior: creator gaming, coordinated inauthentic activity; red-teaming and continuous retraining. Time plan (used by interviewer): - 5 min: Clarify goals/metrics and constraints. - 25–30 min: Architecture deep dive (data → models → ranking/integrity → latency/cost). - 10–15 min: Evaluation/experimentation plan and rollout. - 10 min: Safety, fairness, and abuse scenarios. - 5 min: Extensions/Q&A. What good looks like (signals Meta calibrates on): - Product sense: frames measurable, user-centric objectives and explicit guardrails. - ML depth: chooses appropriate architectures, explains trade-offs, calibration, and label quality. - System design at scale: clear latency/cost plan, caching/feature freshness, reliability and rollback. - Experimentation rigor: concrete offline→online path, guardrails, and interpretation of conflicting metrics. - Integrity mindset: proactive safety with fairness considerations and appeals. - Communication: structured, concise, decisions justified with data and trade-offs. Common pitfalls (red flags): - Chasing raw engagement without safety guardrails; vague metrics; ignoring latency/cost. - Hand-wavy "use a bigger model" answers without a deployable plan. - No calibration, no rollback strategy, or missing experiment guardrails. - Over-filtering that harms creators or minority communities without fairness analysis. Interviewer materials: whiteboard/diagramming, scratchpad for metric math, and probing prompts such as "If MSI lifts but reports rise 10%, ship or not?" and "You miss P99 by 20 ms—what do you cut first?" Candidate deliverables by end of case: - A crisp problem statement with target and guardrails, a block diagram of retrieval→ranking→integrity, latency budget table, composite objective design, offline/online experiment plan with stop conditions, and a safe rollout strategy aligned with Meta's values of moving fast while protecting people and communities.