Pharma and life sciences · Playbook 52

Manufacturing deviations and CAPA investigation

Every new deviation starts with the similar past cases, the likely root-cause areas and a draft outline already in front of the investigator, and recurring problems are spotted before the next one. QA decides every classification, root cause, CAPA and batch disposition.

  • Retrieval (RAG)
  • Predictive ML
  • Human in the loop
  • UK GMP (Orange Guide)
  • MHRA GxP data integrity
  • ICH Q9(R1) and Q10
  • Draft EU GMP Annex 22
  • UK GDPR

At a glance

The problem
Investigations start from a blank page, run late, and miss that the same failure happened before
Typical volume
Around 120 deviations a month on one site, about a quarter needing a full investigation
Who's involved
Head of QA, Qualified Persons, investigators in production, engineering and QC, validation, IT and OT, data protection
Pilot
8–10 weeks on one production area, after a 3–4 week discovery
Platforms
AWS, Azure, or a sovereign or on-prem stack

An illustrative scenario. The numbers are typical examples, not a client case study. The cloud services and UK rules were checked on 9 November 2026.

The business problem

On a busy manufacturing site, deviations are a daily event: a granulation endpoint outside its range, a label reconciliation that doesn't add up, a balance found out of calibration, an environmental monitoring excursion. Each is logged in the eQMS, classified by QA, and the larger ones need a full investigation, a root cause and corrective and preventive actions.

The investigation is where time goes. An investigator gathers the batch record, MES events, LIMS results, maintenance history and training records, then searches the eQMS for anything similar. Search is by keyword, so a past case worded differently is missed. Under time pressure, 'human error' becomes the root cause and retraining becomes the CAPA. The same failure comes back a few months later on a different shift, and nobody connects the two until an annual review or an inspection.

Batches can wait on this. A deviation that may affect a batch must be investigated before the Qualified Person can certify it, so late investigations hold stock and, for some medicines, supply. Quality systems, including deviation and CAPA handling, have long been among the areas most often cited in the MHRA's published GMP inspection findings. Faster is only useful if the investigations are also better.

What AI changes

  • When a deviation is raised, the investigator sees the most similar past deviations and their outcomes, matched by meaning across product, equipment, process step and failure mode, not just keywords.
  • The assistant suggests likely root-cause areas, each linked to the past cases and data behind it, and prompts for the evidence that would confirm or rule each one out.
  • A draft outline of the investigation is assembled: the event description, a timeline from MES and LIMS data, and the impact questions the site's SOP requires. The investigator writes the analysis and the conclusions.
  • A trend model looks across all deviations each week and flags rising or recurring patterns by equipment, material, area or process step, for the quality review meeting.
  • When a CAPA is closed, the same failure is watched for. If it recurs, the CAPA's effectiveness check is flagged for review.
  • The AI is advisory throughout. QA classifies, approves the root cause and the CAPA, and decides batch disposition. The Qualified Person certifies. Nothing the AI produces sits in the batch release path.

How success is measured

Agree these, and measure the baseline, before anything is built. Example targets are typical, not promises.

Investigation cycle time

Days from a deviation being raised to its investigation being approved.

ExampleDown by a quarter, without lowering the approval bar

Investigations closed on time

Share closed within the site's own target, often 30 days.

ExampleUp, measured against last year

Recurrence rate

Deviations with the same root cause on the same equipment or process within 12 months.

ExampleFalling quarter on quarter

'Human error' as root cause

How often the conclusion is human error alone. A falling share usually means deeper investigations.

ExampleDown, with QA review of each one

Similar-case usefulness

How often investigators mark the suggested past cases as relevant.

ExampleStable, and reviewed monthly

Batches waiting on investigations

Batches whose certification is held by an open deviation.

ExampleFewer, and for less time

UK regulation and controls

Cross-sector rules are on the UK regulation map.

  • MHRAUK GMP: the Human Medicines Regulations 2012 and the 'Orange Guide'

    The manufacturing licence holder runs a pharmaceutical quality system that investigates deviations, finds the root cause and takes corrective and preventive action. Those are QA's decisions. The assistant helps the investigator find and organise evidence; it never classifies, approves or closes a deviation or a CAPA.

  • MHRAThe Qualified Person

    Each batch is certified by a Qualified Person, who must be satisfied that deviations affecting it have been investigated. The assistant has no connection to certification, and nothing in the release path waits for, or depends on, its output.

  • ICHQ9(R1) quality risk management and Q10 pharmaceutical quality system

    Q9(R1) asks for risk assessments that manage subjectivity. Suggestions from past cases can anchor investigators on old answers, so the assistant shows evidence for and against, and the investigator records their own reasoning. Q10 expects CAPA effectiveness to be checked and reviewed by management; the recurrence watch feeds that review, it doesn't replace it.

  • MHRAGxP data integrity guidance (2018) and computerised system validation

    Records must be attributable, legible, contemporaneous, original and accurate (ALCOA+). The investigation report is the investigator's record, signed in the eQMS. The assistant's suggestions are logged with their sources and model version, and the tool is validated for its intended use, using a risk-based approach such as ISPE's GAMP 5 (second edition) and its GAMP guide on AI (July 2025) as industry guidance.

  • European Commission and PIC/SEU GMP Annex 11 revision and draft Annex 22 on AI

    Consulted on from July to October 2025 and still in draft in November 2026. The draft Annex 22 covers static, deterministic models in critical GMP uses and says generative AI and large language models shouldn't be used in critical GMP applications; in non-critical uses, trained people are responsible for the output. It applies directly to sites supplying the EU and is a strong signal for everyone else, so this design keeps the language model in a non-critical, advisory role.

  • ICOUK GDPR and the Data Protection Act 2018

    Deviation records name operators and investigators. The trend model looks at processes, equipment and materials, not individual people's performance, and it isn't used for disciplinary decisions. That needs a DPIA, a clear purpose and a retention period, and staff should be told how the tool is used.

The solution

Showing AWSAzuresovereign / on-prem services. Switch with the toggle; every diagram and table on this page follows it.

Manufacturing deviations and CAPA investigation: reference architecture
  1. 01Sources

    • Deviation raised on the shop floor or in QC
    • Batch records and MES events
    • LIMS results and maintenance history
  2. 02Intake

    • API GatewayAPI ManagementKong, or your existing API gatewayeQMS events
    • AWS GlueAzure Data FactoryApache AirflowRead-only data extracts
    • Amazon S3Blob StorageCeph, or IONOS S3 Object StorageInvestigation data store
  3. 03Intelligence

    • Bedrock Knowledge BasesFoundry IQ (Azure AI Search)OpenSearch, or PostgreSQL with pgvectorSimilar-deviation search
    • SageMaker AIAzure Machine LearningMLflow and Kubeflow on KubernetesRecurrence and trend model
    • Agent
      Bedrock AgentCoreFoundry Agent ServiceLangGraph on Kubernetes, with open-weight models via vLLMInvestigation assistant
  4. 04Orchestration

    • Step FunctionsLogic Apps / Durable FunctionsTemporalInvestigation workflow
    • EventBridge + SQSEvent Grid + Service BusApache Kafka or RabbitMQEvents and queues
  5. 05Systems of record

    • eQMS: deviations, CAPA, change control
    • MES and electronic batch records
    • LIMS
  6. 06People

    • Person
      Investigator: writes the analysis
    • Person
      QA: classifies, approves root cause and CAPA
    • Person
      Qualified Person: certifies batches
  • Cloud or platform service (AWSAzuresovereign / on-prem)
  • AI agent
  • A person decides
  • Your existing systems
  • Outside your organisation

Components, side by side

Manufacturing deviations and CAPA investigation: components on AWS, Azure and a sovereign or on-prem stack
LayerAWSAzureSovereign / on-prem
Intake: events and extractsRead-only. Nothing writes back to MES or LIMS.API Gateway, AWS Glue, S3API Management, Azure Data Factory, Blob StorageKong or your existing API gateway; Apache Airflow; Ceph or IONOS S3 Object Storage
Similar-deviation search (retrieval)Index closed, approved investigations only, with their CAPAs and outcomes, so the assistant learns from conclusions QA signed off.Bedrock Knowledge BasesFoundry IQ, built on Azure AI SearchOpenSearch, or PostgreSQL with pgvector
Root-cause suggestions and draft outlineBoth clouds offer a choice of models, including Claude. Every suggestion links to the past cases and data behind it.Amazon BedrockMicrosoft FoundryOpen-weight models served with vLLM on your own GPUs, or IONOS AI Model Hub
Recurrence and trend modelA fixed, versioned model retrained only under change control. Open-source anomaly libraries (e.g. PyOD) suit the trend flags; SageMaker Clarify and Model Monitor are closed to new customers, so monitor drift with your own checks.SageMaker AIAzure Machine LearningMLflow and Kubeflow on Kubernetes
Agent runtimeThe original Bedrock Agents closed to new customers on 30 July 2026; new builds use AgentCore.Amazon Bedrock AgentCoreFoundry Agent ServiceAn open-source agent framework (e.g. LangGraph) on Kubernetes
WorkflowAWS Step FunctionsLogic Apps or Durable FunctionsTemporal, or n8n
Middleware and eventsAmazon EventBridge, SQSEvent Grid, Service BusApache Kafka or RabbitMQ
Link to the eQMSAgree in discovery what the supplier allows. Suggestions usually appear as a linked, clearly labelled attachment, not as text in the record.Through the eQMS supplier's published APIThrough the eQMS supplier's published APIThrough the eQMS supplier's published API
Human reviewInvestigators and QA work where they already work.In the existing eQMSIn the existing eQMSIn the existing eQMS
GuardrailsBlocks the assistant from proposing a classification, a batch decision or a conclusion.Bedrock GuardrailsFoundry guardrails (Azure AI Content Safety)Open-source guardrails (e.g. NeMo Guardrails, Llama Guard)
Identity, keys, data protectionIAM Identity Center, KMS, MacieEntra ID, Key Vault, PurviewKeycloak or your existing Active Directory; OpenBao or HashiCorp Vault; an HSM
Monitoring and auditLog every suggestion with its sources, model version and index version, and what the investigator did with it.CloudWatch, CloudTrailAzure Monitor, Log AnalyticsPrometheus, Grafana, OpenTelemetry
UK data residencyIONOS AI Model Hub runs in German data centres: EU, not UK, residency.eu-west-2 (London)UK South / UK WestYour own UK data centre, or a UK or EU provider

On a phone, the table shows the platform you pick. Sovereign / on-prem means open, portable components you can run in your own data centre or with a UK or European provider; IONOS is shown as one example, and others exist. What sovereign cloud means →

Hybrid and on-prem

Many organisations keep core systems and sensitive data on their own servers. This is how the same design works across both.

Stays on-prem

MES, electronic batch records, LIMS and the plant's OT network stay exactly where they are, and so does the eQMS if it's hosted on site. Nothing in this design touches equipment or the release path.

Uses cloud AI

The similar-deviation index, the drafting assistant and the trend model can run in a UK cloud region, or on your own GPUs with open-weight models if GMP data must not leave the site. They work on read-only extracts of closed investigations and the data for the open one.

How they connect

Read-only extracts leave through the site's DMZ, never directly from the OT network, over a private link such as AWS Direct Connect or Azure ExpressRoute. Suggestions come back to the eQMS through its API as labelled attachments.

If the link drops

Investigations carry on exactly as they did before, with keyword search in the eQMS. No batch, deviation or CAPA waits for the AI, and the trend report simply runs when the link returns.

The agents

What each agent may do through its tools, what it may not, and where a person signs off. The limits are enforced by the platform, not the prompt.

Investigation assistant

May do

  • Read the new deviation and its linked batch, MES and LIMS data
  • Find similar past deviations and show their root causes, CAPAs and outcomes
  • Suggest likely root-cause areas, each with its evidence and what would rule it out
  • Assemble a draft timeline and outline for the investigator
  • Attach its suggestions to the deviation as a labelled, read-only document

May not do

  • Classify a deviation as minor, major or critical
  • Write or change the investigation's conclusions or root cause
  • Create, approve or close a CAPA or a deviation
  • Suggest or record a batch disposition
  • Write to MES, LIMS or any batch record

Where a person signs offThe investigator writes and signs the analysis. QA approves classification, root cause and CAPA, and decides batch disposition. The Qualified Person certifies. The agent's permissions are enforced by the platform's identity and policy controls, not by instructions in a prompt.

Trend and recurrence watch

May do

  • Analyse all deviations and CAPAs each week
  • Flag rising trends and repeat failures by equipment, material, area or step
  • Flag closed CAPAs whose failure has recurred
  • Prepare a trend summary for the quality review meeting

May not do

  • Open a deviation or CAPA
  • Rank or report on individual operators
  • Change a CAPA's status or effectiveness result

Where a person signs offQA reviews every flag at the quality review meeting and decides whether to act. Decisions are recorded in the eQMS as now.

Middleware and integration

The eQMS stays the record. The assistant uses the supplier's published API with an identity that can read deviations and CAPAs and add a labelled attachment, and nothing else. If your eQMS supplier offers its own AI features, compare them in discovery; the controls in this design apply either way.

MES, batch records and LIMS are read through scheduled, read-only extracts from a reporting copy, never from the live systems on the OT network. That keeps the AI out of the manufacturing control path and out of the scope of the systems' own validation.

Events keep the AI decoupled. A new or updated deviation becomes a message on a queue. If the AI is slow or down, the investigation simply starts without suggestions, and nothing is lost or created twice.

Process and people

  • The investigator owns the investigation. Suggestions are evidence to consider, and the report records which were used and why.
  • Only closed, QA-approved investigations go into the index, so the assistant never learns from drafts.
  • QA reviews every investigation that concludes 'human error' alone, whether or not the AI was involved.
  • A named owner for the model, the index and the prompts, with change control and re-validation like any other GMP computerised system.
  • A monthly review of suggestion usefulness and of trend flags acted on or dismissed.
  • Investigators trained before go-live, including what the assistant can't see: anything that was never written down.

Delivery plan

  1. 3–4 weeks

    Discovery

    Map today's deviation process, measure the baseline, check the quality of past investigation data, agree the eQMS route with its supplier, and agree the intended use and risk level with QA.

  2. Gate · must be true to passBaseline agreed, intended use written down, validation approach agreed with QA, DPIA started.

  3. 8–10 weeks

    Pilot

    One production area in shadow mode: suggestions are produced, investigators work as normal, and QA compares the two.

  4. Gate · must be true to passSuggestion relevance within agreed limits, no influence on classification or disposition, validation evidence complete, head of QA sign-off.

  5. 4–6 weeks

    Production

    Suggestions go live for the pilot area, with monitoring, support and a rollback plan. The trend report joins the quality review meeting.

  6. Gate · must be true to passMonitoring, support model, rollback plan and investigator training in place.

  7. Ongoing

    Scale

    Add areas and QC laboratories one at a time, then reuse the pattern for complaints and change control.

  8. Gate · must be true to passEach new area passes the same pilot gate.

Roles and RACI

Who is responsible, accountable, consulted and informed, by activity
ActivityHead of QAProduct ownerSolutions architectI can fill thisTechnical PMI can fill thisEngineersI can fill thisValidation & CSVDPOSecurityI can fill this
Business case and benefitsAccountableResponsibleConsultedConsultedInformedConsultedInformedInformed
Target architecture and eQMS integrationConsultedConsultedAccountable and ResponsibleConsultedConsultedConsultedConsultedConsulted
Intended use, risk assessment and validationAccountableConsultedConsultedConsultedResponsibleResponsibleInformedConsulted
DPIA and staff communicationConsultedConsultedConsultedInformedInformedInformedAccountable and ResponsibleConsulted
Build and integrationInformedConsultedAccountableResponsibleResponsibleConsultedInformedConsulted
Testing on past investigationsAccountableResponsibleResponsibleConsultedResponsibleResponsibleInformedInformed
Go-live decisionAccountableResponsibleConsultedResponsibleInformedConsultedConsultedConsulted
Run, monitor and improveAccountableResponsibleConsultedConsultedResponsibleConsultedConsultedConsulted

Business case and benefits

Head of QA
A · Accountable
Product owner
R · Responsible
Solutions architect
C · Consulted
Technical PM
C · Consulted
Engineers
I · Informed
Validation & CSV
C · Consulted
DPO
I · Informed
Security
I · Informed

Target architecture and eQMS integration

Head of QA
C · Consulted
Product owner
C · Consulted
Solutions architect
A/R · Accountable and Responsible
Technical PM
C · Consulted
Engineers
C · Consulted
Validation & CSV
C · Consulted
DPO
C · Consulted
Security
C · Consulted

Intended use, risk assessment and validation

Head of QA
A · Accountable
Product owner
C · Consulted
Solutions architect
C · Consulted
Technical PM
C · Consulted
Engineers
R · Responsible
Validation & CSV
R · Responsible
DPO
I · Informed
Security
C · Consulted

DPIA and staff communication

Head of QA
C · Consulted
Product owner
C · Consulted
Solutions architect
C · Consulted
Technical PM
I · Informed
Engineers
I · Informed
Validation & CSV
I · Informed
DPO
A/R · Accountable and Responsible
Security
C · Consulted

Build and integration

Head of QA
I · Informed
Product owner
C · Consulted
Solutions architect
A · Accountable
Technical PM
R · Responsible
Engineers
R · Responsible
Validation & CSV
C · Consulted
DPO
I · Informed
Security
C · Consulted

Testing on past investigations

Head of QA
A · Accountable
Product owner
R · Responsible
Solutions architect
R · Responsible
Technical PM
C · Consulted
Engineers
R · Responsible
Validation & CSV
R · Responsible
DPO
I · Informed
Security
I · Informed

Go-live decision

Head of QA
A · Accountable
Product owner
R · Responsible
Solutions architect
C · Consulted
Technical PM
R · Responsible
Engineers
I · Informed
Validation & CSV
C · Consulted
DPO
C · Consulted
Security
C · Consulted

Run, monitor and improve

Head of QA
A · Accountable
Product owner
R · Responsible
Solutions architect
C · Consulted
Technical PM
C · Consulted
Engineers
R · Responsible
Validation & CSV
C · Consulted
DPO
C · Consulted
Security
C · Consulted

R ResponsibleA AccountableC ConsultedI Informed· not involved

Where I'd fit

Option A

Lead the architecture

If there's no architect on the programme, I'd own the design end to end: the read-only data flows from MES and LIMS, the eQMS integration, the retrieval and trend models, and the gates. I'd work closely with your head of QA and validation lead, and report to your programme board.

Option B

Support your architect

If your partner, your eQMS supplier or your own team already has an architect, I'd take on the retrieval, the trend model and the integration under their lead, and give them a second senior opinion on OT boundaries, security and the validation evidence.

Where this goes wrong

  1. Goes wrong

    Investigators copy the suggested root cause because it matched last time.

    How the design avoids it

    Evidence for and against each suggestion, the investigator's own reasoning recorded, and QA checking how often suggestions were adopted unchanged.

  2. Goes wrong

    The index is full of old 'human error' conclusions, so the assistant suggests more of them.

    How the design avoids it

    Index only QA-approved investigations, weight recent ones, and review the root-cause categories before go-live.

  3. Goes wrong

    The tool drifts into the release decision because a QP starts relying on it.

    How the design avoids it

    No link to certification, guardrails against disposition language, and the intended use written down and trained.

  4. Goes wrong

    Someone connects the AI straight to MES on the OT network.

    How the design avoids it

    Read-only extracts from a reporting copy, through the DMZ, agreed with OT security in discovery.

  5. Goes wrong

    Validation is left until the end, or skipped because 'it's only advisory'.

    How the design avoids it

    Agree the intended use and risk level in discovery and validate proportionately from the start.

On the horizon

Changes already announced or under way that affect this design. More in the UK horizon scan.

  1. Q4 2026 target
    Final texts of EU GMP Annex 22 on AI, the revised Annex 11 and Chapter 4, due to go to the European Commission. Regulators held a workshop in mid-2026 on a risk-based approach to generative AI, so the language model position may change.

    Design to the draft now: language models advisory only, static and versioned models where it matters, and the intended use written down. Review against the final text when it's published.

  2. 2026–2030
    The MHRA's Strategy 2030 (October 2026), including a Centre for AI and Digital Health to set out expectations for AI used across the medicines lifecycle.

    Keep the validation pack, the decision logs and the intended use current, so you can show how the AI is controlled when asked.

  3. 2027
    The MHRA's AI Medicines Development Action Plan, promised in the strategy.

    Watch it for any expectations on AI in manufacturing and quality, and map them to what you already do under the draft Annex 22.

  4. 2028
    The NCSC's post-quantum cryptography milestone: a full migration plan by 2028.

    GMP records are kept for years. Record which encryption the new data flows use, and keep it easy to change.

Questions to ask your partner or vendor

Good answers are specific. Vague answers to any of these are worth following up.

Copy them into your next supplier meeting

  1. What exactly is the intended use, and who agreed it is non-critical?
  2. Which past investigations go into the index, and who approved them?
  3. How is the tool validated, and what triggers re-validation?
  4. How do we make sure nothing the AI produces reaches batch certification?
  5. How do data extracts leave the OT network, and who signed that off?
  6. What does our eQMS supplier allow us to write into a deviation, and how?

Planning something like this?

Whether you need someone to lead the design or to support the architect you already have, let's talk it through.

Sam Naseem
Sam NaseemBirmingham, UK · UK & Europe
Available now