Morgan Stanley AI Engineer Case: LLM/RAG platform for Wealth Management with model risk controls

You’ll design an LLM-enabled assistant for Morgan Stanley Wealth Management financial advisors that drafts client-ready communications, answers product and portfolio questions using internal research, and summarizes meetings—while satisfying bank-grade controls. The case probes your ability to balance AI capability with regulatory, security, and operational constraints common at Morgan Stanley. Scope and context: - Users: Financial Advisors (FAs), Supervisory Principals, and Client Service Associates within Wealth Management; read access must respect entitlements and information barriers from Institutional Securities. - Data: Internal research notes, product term sheets, CRM/portfolio data, archived communications, and licensed market data (e.g., Bloomberg/Refinitiv)—with strict licensing and PII constraints. - Objective: Grounded, auditable answers with citations; latency suitable for live client prep; demonstrable reduction in FA prep time without increasing compliance risk. What you’ll do in-session: 1) Architecture: Propose an LLM/RAG design (doc ingestion, chunking/embeddings, vector store, entitlement-aware retrieval, prompt assembly) and justify on-prem/VPC vs vendor API, key management, and network segregation. Define SLAs (p95 latency), throughput, cost controls, and fallback modes. 2) Risk and compliance: Build guardrails for hallucinations (attribution, confidence gating), prompt injection defenses, PII redaction, communication archiving, immutable audit logs, and feature flags for supervised rollout. Address model governance (approval, versioning, canary tests), explainability for supervision/audit, and SR 11-7–style model risk management lifecycle (development, validation, monitoring, change control). 3) Evaluation plan: Offline/online evals with domain-specific question sets, reference-grounded factuality scoring, red-team scenarios (e.g., restricted research, client PII leakage), and safety metrics. Define acceptance gates to promote from pilot to production. 4) Systems design details: API/contract for a “/answer” endpoint, retrieval middleware pseudocode, caching strategy, content filters, and observability (telemetry, drift detection, cost/latency/error dashboards). Discuss multi-tenant data isolation and entitlement checks before retrieval. 5) Trade-offs: RAG vs fine-tuning, document freshness vs index cost, quality vs latency, central vs desk-specific models, and how to scale usage while keeping spend predictable. Consider disaster recovery, incident playbooks, and vendor lock-in. Deep-dive prompts you should expect: - How do you enforce information barriers and entitlements at retrieval time? - What’s your plan for citation integrity and auditor-friendly traceability? - How do you prevent proprietary research or client data from training external models? - What’s your rollout plan (pilot desks, feature flags, kill switches) and success metrics for business adoption? - How do you measure and mitigate toxic or investment-advice compliance risks in generated text? Evaluation rubric (what interviewers look for): clarity of architecture under constraints; practical risk/compliance integration; sound metrics and experiments; crisp trade-off reasoning; stakeholder awareness (FA, Compliance, Model Risk, Cyber, Legal, SRE); and communication tailored to a regulated enterprise.