qsteadgroup@yahoo.com (619) 718-1853

Case Study · CAPA & Laboratory Investigations

The Oven That Only Failed Sometimes

A model OOS investigation using Kepner-Tregoe and Shainin Red X®-style analysis: how a QC lab traced intermittent, severely low assay results to a hydrolysis oven, proved it, and closed the CAPA the right way.

Based on real-world events. To protect confidentiality, the names of companies, products, people, and places, along with any proprietary or confidential details, have been changed or removed. Some details have been combined or simplified for teaching, and any data shown are illustrative. Any resemblance to a specific company, product, or person is coincidental.

At a glance

Product / testProtein-content assay on a finished product: acid hydrolysis, pre-column derivatization, and UPLC
Specification4.0–6.0% (process target 5.0%)
The problemSeveral lots tested at about 3.3%, roughly a third below target and far below anything in the product’s history
Root causeIntermittent heating failure in the older of two qualified hydrolysis ovens, causing incomplete hydrolysis during some overnight cycles
How it was provenRed X®-style contrast, an oven swap, extended multi-probe temperature logging, and a side-by-side confirmation run
OutcomeAssignable cause documented; OOS invalidated with QA approval; oven retired; CAPA effective

1. The signal

A QC lab tests every lot of a finished product for protein content. The method has four stages: samples are hydrolyzed overnight in 6 N hydrochloric acid in sealed vacuum vessels, the freed amino acids are derivatized so the detector can see them, the samples are run on UPLC, and the result is calculated against a reference standard.

For years, results sat comfortably near the 5.0% target. Then several lots came back at about 3.3%: out of specification (OOS) and far lower than anything in the product’s history. Failures appeared in some runs but not others.

2. First actions: contain and define

  • Contain. Affected lots were placed on hold, and QA was notified the same day, as the OOS procedure requires.
  • Preserve evidence. Retained samples, prepared solutions, and hydrolysates from the failing runs were kept, labeled, and stored, not discarded.
  • Define the problem precisely. The team wrote a problem statement and a Kepner-Tregoe Is / Is-Not before testing anything, including the questions it could not yet answer.
IsIs notWhat it suggests
WhatResults ~3.3% on some lotsNot every lot; the same sample can pass on repeatThe test failed, not necessarily the product
WhereAll routine runs use one hydrolysis oven (Oven A)Unknown: the second qualified oven (Oven B) has not been used for this testLocation can’t be judged yet; a comparison is needed
WhenIntermittent, recentNot a steady, every-run failureSomething that switches on and off
WhoSeveral analystsNot tied to one personOperator unlikely
ExtentAbout one-third lowNot a small biasA step that can lose a lot of signal

3. Phase I laboratory investigation

Size of the miss

A result one-third low narrows the search. Balance and pipetting errors of that size would be obvious in recorded weights and routine checks, and a bad reference standard would shift every sample in a run, not just some. The team reviewed the records for the failing runs, confirmed that weights, pipette verifications, system suitability, and standard results were all normal, and documented why these suspects were ruled out.

Map the method into families

StepPossible causes of low results
WeighingBalance error, sample non-homogeneity
Acid hydrolysis (overnight, sealed vacuum vessel)Temperature too low or uneven, vessel seal failure, time too short
Reconstitution and mixingIncomplete redissolution, mixing time not defined in the SOP
DerivatizationReagent lot, reaction time, pH
UPLC and calculationInjection, integration, standard

Retained material from a failing sample was then split to isolate the stage. Re-injecting the original hydrolysate gave the same low value (UPLC ruled out), and re-derivatizing it also stayed low (derivatization ruled out). The signal was already missing before derivatization, which pointed to hydrolysis.

Phase I conclusion

No laboratory error in weighing, derivatization, chromatography, or calculation. The evidence points to the hydrolysis stage. The case was escalated to a Phase II investigation led by QA, with a short documented manufacturing review that found no process deviations for the affected lots.

4. Testing the suspects under protocol

Because each test cycle takes one to two days, the team wrote a single QA-approved protocol that tested several hypotheses in parallel, with a control sample of known value in every run and every oven position.

HypothesisTestResultConclusion
Derivatization reagent lotFresh reagent lot in runs where failures occurredFailures continuedRuled out
Vessel seal / vacuumSeal checks; deliberately loosened lid on control vesselsControls normalNot a primary cause
Mixing after hydrolysisShort vs. extended mixing timesNo differenceNot a cause; time defined in SOP anyway

With the common suspects eliminated and the hydrolysis stage implicated, the team switched to a Red X®-style approach: instead of testing causes one by one, let the contrast between the best and worst results point to the cause.

5. Finding the contrast: the Red X® approach

Shainin’s Red X® method starts from the result and works backward (“Y to X” thinking). It assumes one dominant cause, the Red X, explains most of the difference between good and bad outcomes, and finds it by comparing extremes and eliminating whole families of causes at a time.

Define the Green Y

The Green Y is the measurable result being investigated. Here it was the protein result (%) for a homogenized control sample of known value, run in every hydrolysis batch. Using a control instead of production lots removes product variation from the comparison.

Check the measurement first

Before hunting for a cause, the team confirmed the measurement could see the problem clearly (the idea behind Shainin’s Isoplot). Repeat injections of the same hydrolysate varied by about ±0.05%, while good and bad runs differed by about 1.7%. Measurement noise was tiny compared with the effect, so the difference was real.

Best of Best vs. Worst of Worst

The team pulled the three best and three worst control results from recent runs and compared everything about them.

RunControl resultOvenAnalystReagent lot
BOB 15.03%A1R1
BOB 25.01%A2R2
BOB 34.99%A3R2
WOW 13.28%A2R1
WOW 23.31%A3R2
WOW 33.36%A1R2

Analysts and reagent lots appear on both sides, so neither separates good from bad. Every run used Oven A, so the oven couldn’t be judged yet. The difference had to be something that varied between cycles of the same equipment.

Multi-vari: where does the variation live?

Family of variationWhat the data showedVerdict
Within a run (position to position)Normal runs: controls within ±0.03% across positions. Failing runs: lower-rear positions lowest.Secondary; appears only in bad cycles
Run to run (cycle to cycle)About 5.0% in some runs, about 3.3% in othersDominant family
Lot to lotThe same lot gave both low and normal resultsEliminated
Analyst to analystEvery analyst had both good and bad runsEliminated

The variation lived cycle to cycle inside the hydrolysis step. That narrowed the search to whatever changes from one overnight cycle to the next: the oven’s behavior.

The swap

The lab had a second qualified oven (Oven B) that had never been used for this test. Keeping everything else the same (samples, reagents, analysts, method), the team hydrolyzed the next three batches in Oven B. All three passed. Changing one variable made the failures disappear, the equivalent of Shainin’s component swap. But three passes are evidence, not proof.

6. Proving the cause

Extended temperature logging

Oven A had passed its routine calibration, a single-point check in an empty chamber. That check couldn’t catch a problem that only happened sometimes, somewhere in the chamber, during a long run. So the team placed a calibrated multi-probe data logger (front, center, back; upper and lower shelves) in Oven A with a full sample load, and logged five overnight cycles.

CycleTemperature behaviorControl sample
1Stable at setpoint all nightNormal
2Dropped 8–10 °C below setpoint for about 3 hours; worst at lower-rear positionsLow
3Stable at setpointNormal
4Repeated drops of 6–9 °C over about 2 hoursLow
5Stable at setpointNormal

The pattern matched exactly: low results occurred only in cycles with temperature drops, and the oven’s own display never showed the dips. The heating was intermittently cutting out overnight. This also explained the multi-vari finding: when the heating failed, the lower-rear positions cooled the most.

Confirmation: “take the Red X® to court”

The final test turned the failure on and off. Matched aliquots of the control were split between Oven B (the better condition, “B”) and Oven A (the current condition, “C”) in the same overnight run, with the logger running in Oven A. Oven A logged a temperature drop that night.

ConditionAliquot 1Aliquot 2Aliquot 3
B: Oven B (stable)5.01%4.98%5.03%
C: Oven A (logged drop)3.31%3.45%3.28%

Every B result beat every C result. With three of each, complete separation like this has only about a 1-in-20 chance of happening by luck, which is roughly 95% confidence.

Cause demonstrated

Intermittent loss of heating in Oven A during overnight hydrolysis leads to incomplete hydrolysis and low results. This was confirmed by the best-vs-worst contrast, the oven swap, temperature logging correlated with results, and a side-by-side B vs. C run.

7. Root cause statement

Technical causeIntermittent heating failure in Oven A caused temperature drops during overnight hydrolysis, leading to incomplete hydrolysis and low protein results.
Why it wasn’t detectedRoutine calibration was a single-point, empty-chamber check that couldn’t detect intermittent or position-specific behavior during long cycles, and the oven had no independent temperature monitoring or alarm.
Contributing factorThe SOP didn’t define the mixing time after hydrolysis, leaving room for run-to-run variation (not causal here, but corrected).

The cause statement stops at what was demonstrated. A component-level teardown of Oven A (heater, relay, or controller) was judged unnecessary: the oven was being removed from service, so knowing which part failed wouldn’t change any decision. This risk-based choice was documented in the investigation.

8. Disposition of the OOS results

  • With an assignable cause demonstrated, QA approved invalidation of the original OOS results, with the evidence attached to the investigation record.
  • Affected lots were retested per the OOS procedure in qualified Oven B, passed, and were released by QA.
  • Look-back: results for lots hydrolyzed in Oven A over the relevant period were reviewed for possible low bias. No other lots were affected.

9. CAPA

TypeAction
CorrectionOven A removed from GMP use and retired; hydrolysis moved to qualified Oven B.
Corrective actionContinuous temperature logging with low-temperature alarms added for hydrolysis ovens; each run’s temperature record reviewed before results are approved.
Corrective actionOven qualification upgraded to include loaded, multi-position, full-cycle mapping; calibration program updated to include periodic extended checks.
Preventive actionSOP revised through change control to define mixing time and to require a control sample of known value in every hydrolysis run.
Preventive actionRead-across: other long-cycle heated equipment (ovens, heating blocks, incubators) assessed for the same detection gap.

10. Effectiveness check

Criteria were set before closing: over the next 30 hydrolysis runs, no OOS results attributable to hydrolysis, control samples within their limits in every run, and no unreviewed temperature excursions. All criteria were met, and the CAPA was closed as effective.

11. How the tools worked together

StageToolWhat it contributedIts limit
DefineKepner-Tregoe Is / Is-NotSeparated a test failure from a product failure; exposed the open question about the ovenOrganizes facts; doesn’t prove cause
NarrowProcess map, size-of-the-miss analysisGrouped suspects into families; eliminated steps that couldn’t cause a one-third lossNeeds good knowledge of the method
IsolateSplit testing of retained materialPinned the loss to the hydrolysis stageRequires preserved evidence
TestHypothesis protocol with controlsEliminated reagent, seal, and mixing suspects in parallelWeak when the failure isn’t active
ContrastRed X®: Green Y, BOB/WOW, multi-variShowed the variation lived cycle to cycle inside hydrolysisNeeds a measurable result and real extremes
SwapOven A vs. Oven BChanged one variable and made the failure disappearEvidence, not yet proof
ProveData logging, B vs. C runShowed the mechanism and turned the failure on and offNeeds enough cycles to catch the fault
CloseCAPA with effectiveness criteriaFixed the cause and the detection gap, and verified it workedOnly as good as the criteria set up front

When Red X® fits, and when it doesn’t

Good fit (as here): a measurable result, clear good and bad extremes, one dominant cause, and the ability to compare or swap conditions. Intermittent equipment problems are a classic example.

Poor fit: many small causes of similar size, failures too rare to find extremes, evidence that no longer exists, or causes that can’t safely be recreated (sterility, for example). Red X® also finds the technical cause, not why the quality system missed it. That’s why this investigation paired it with a look at calibration and monitoring before closing the CAPA.

Key takeaways

  1. Let the size of the miss guide the search. A one-third loss rules out many small-error suspects immediately.
  2. A passing retest is not an explanation. The investigation continued until the cause was shown.
  3. Split the method into families and use retained material to find where the signal is lost.
  4. Compare the extremes. Best-vs-worst runs and a multi-vari study pointed to cycle-to-cycle variation faster than testing suspects one at a time.
  5. Change one thing at a time. The oven swap worked because nothing else changed.
  6. Snapshots can’t catch intermittent problems. Extended, loaded, multi-position logging found what a single calibration point missed.
  7. Prove it by turning it on and off. A side-by-side B vs. C run turned a strong suspect into a demonstrated cause.
  8. Fix the detection gap, not just the equipment. The CAPA addressed why the problem wasn’t caught, so it won’t recur elsewhere.
  9. Match investigation depth to the decisions that depend on it. Proving the oven was the cause mattered; finding the failed component did not.

Questions to think about

  1. The same sample failed once and passed on repeat. Why isn’t that enough to invalidate the first result?
  2. Why use a control sample, rather than production lots, as the Green Y?
  3. In the best-vs-worst table, every run used Oven A. What could and couldn’t that comparison tell the team?
  4. The oven passed its routine calibration. What would you change in a calibration program to catch problems like this?
  5. Each test cycle took one to two days. How would you design experiments to reach a conclusion faster?
  6. When is it reasonable to stop investigating before identifying the failed component?

Related reading: Advanced Root Cause Analysis and Pipette OOS: When Two Things Change at Once.

Red X® is a registered trademark of Shainin, and Isoplot and the other Shainin tools named here are part of its proprietary methodology. Kepner-Tregoe is a proprietary method of its owner. These methods are described here for educational purposes only; QStead Group is not affiliated with either. This case study is for educational purposes only.

Working through an OOS or CAPA investigation that won’t close? Get in touch.

← Back to Case Studies