Based on real-world events. To protect confidentiality, the names of companies, products, people, and places, along with any proprietary or confidential details, have been changed or removed. Some details have been combined or simplified for teaching, and any data shown are illustrative. Any resemblance to a specific company, product, or person is coincidental.
At a glance
| Product / test | Protein-content assay on a finished product: acid hydrolysis, pre-column derivatization, and UPLC |
|---|---|
| Specification | 4.0–6.0% (process target 5.0%) |
| The problem | Several lots tested at about 3.3%, roughly a third below target and far below anything in the product’s history |
| Root cause | Intermittent heating failure in the older of two qualified hydrolysis ovens, causing incomplete hydrolysis during some overnight cycles |
| How it was proven | Red X®-style contrast, an oven swap, extended multi-probe temperature logging, and a side-by-side confirmation run |
| Outcome | Assignable cause documented; OOS invalidated with QA approval; oven retired; CAPA effective |
1. The signal
A QC lab tests every lot of a finished product for protein content. The method has four stages: samples are hydrolyzed overnight in 6 N hydrochloric acid in sealed vacuum vessels, the freed amino acids are derivatized so the detector can see them, the samples are run on UPLC, and the result is calculated against a reference standard.
For years, results sat comfortably near the 5.0% target. Then several lots came back at about 3.3%: out of specification (OOS) and far lower than anything in the product’s history. Failures appeared in some runs but not others.
2. First actions: contain and define
- Contain. Affected lots were placed on hold, and QA was notified the same day, as the OOS procedure requires.
- Preserve evidence. Retained samples, prepared solutions, and hydrolysates from the failing runs were kept, labeled, and stored, not discarded.
- Define the problem precisely. The team wrote a problem statement and a Kepner-Tregoe Is / Is-Not before testing anything, including the questions it could not yet answer.
| Is | Is not | What it suggests | |
|---|---|---|---|
| What | Results ~3.3% on some lots | Not every lot; the same sample can pass on repeat | The test failed, not necessarily the product |
| Where | All routine runs use one hydrolysis oven (Oven A) | Unknown: the second qualified oven (Oven B) has not been used for this test | Location can’t be judged yet; a comparison is needed |
| When | Intermittent, recent | Not a steady, every-run failure | Something that switches on and off |
| Who | Several analysts | Not tied to one person | Operator unlikely |
| Extent | About one-third low | Not a small bias | A step that can lose a lot of signal |
3. Phase I laboratory investigation
Size of the miss
A result one-third low narrows the search. Balance and pipetting errors of that size would be obvious in recorded weights and routine checks, and a bad reference standard would shift every sample in a run, not just some. The team reviewed the records for the failing runs, confirmed that weights, pipette verifications, system suitability, and standard results were all normal, and documented why these suspects were ruled out.
Map the method into families
| Step | Possible causes of low results |
|---|---|
| Weighing | Balance error, sample non-homogeneity |
| Acid hydrolysis (overnight, sealed vacuum vessel) | Temperature too low or uneven, vessel seal failure, time too short |
| Reconstitution and mixing | Incomplete redissolution, mixing time not defined in the SOP |
| Derivatization | Reagent lot, reaction time, pH |
| UPLC and calculation | Injection, integration, standard |
Retained material from a failing sample was then split to isolate the stage. Re-injecting the original hydrolysate gave the same low value (UPLC ruled out), and re-derivatizing it also stayed low (derivatization ruled out). The signal was already missing before derivatization, which pointed to hydrolysis.
Phase I conclusion
No laboratory error in weighing, derivatization, chromatography, or calculation. The evidence points to the hydrolysis stage. The case was escalated to a Phase II investigation led by QA, with a short documented manufacturing review that found no process deviations for the affected lots.
4. Testing the suspects under protocol
Because each test cycle takes one to two days, the team wrote a single QA-approved protocol that tested several hypotheses in parallel, with a control sample of known value in every run and every oven position.
| Hypothesis | Test | Result | Conclusion |
|---|---|---|---|
| Derivatization reagent lot | Fresh reagent lot in runs where failures occurred | Failures continued | Ruled out |
| Vessel seal / vacuum | Seal checks; deliberately loosened lid on control vessels | Controls normal | Not a primary cause |
| Mixing after hydrolysis | Short vs. extended mixing times | No difference | Not a cause; time defined in SOP anyway |
With the common suspects eliminated and the hydrolysis stage implicated, the team switched to a Red X®-style approach: instead of testing causes one by one, let the contrast between the best and worst results point to the cause.
5. Finding the contrast: the Red X® approach
Shainin’s Red X® method starts from the result and works backward (“Y to X” thinking). It assumes one dominant cause, the Red X, explains most of the difference between good and bad outcomes, and finds it by comparing extremes and eliminating whole families of causes at a time.
Define the Green Y
The Green Y is the measurable result being investigated. Here it was the protein result (%) for a homogenized control sample of known value, run in every hydrolysis batch. Using a control instead of production lots removes product variation from the comparison.
Check the measurement first
Before hunting for a cause, the team confirmed the measurement could see the problem clearly (the idea behind Shainin’s Isoplot). Repeat injections of the same hydrolysate varied by about ±0.05%, while good and bad runs differed by about 1.7%. Measurement noise was tiny compared with the effect, so the difference was real.
Best of Best vs. Worst of Worst
The team pulled the three best and three worst control results from recent runs and compared everything about them.
| Run | Control result | Oven | Analyst | Reagent lot |
|---|---|---|---|---|
| BOB 1 | 5.03% | A | 1 | R1 |
| BOB 2 | 5.01% | A | 2 | R2 |
| BOB 3 | 4.99% | A | 3 | R2 |
| WOW 1 | 3.28% | A | 2 | R1 |
| WOW 2 | 3.31% | A | 3 | R2 |
| WOW 3 | 3.36% | A | 1 | R2 |
Analysts and reagent lots appear on both sides, so neither separates good from bad. Every run used Oven A, so the oven couldn’t be judged yet. The difference had to be something that varied between cycles of the same equipment.
Multi-vari: where does the variation live?
| Family of variation | What the data showed | Verdict |
|---|---|---|
| Within a run (position to position) | Normal runs: controls within ±0.03% across positions. Failing runs: lower-rear positions lowest. | Secondary; appears only in bad cycles |
| Run to run (cycle to cycle) | About 5.0% in some runs, about 3.3% in others | Dominant family |
| Lot to lot | The same lot gave both low and normal results | Eliminated |
| Analyst to analyst | Every analyst had both good and bad runs | Eliminated |
The variation lived cycle to cycle inside the hydrolysis step. That narrowed the search to whatever changes from one overnight cycle to the next: the oven’s behavior.
The swap
The lab had a second qualified oven (Oven B) that had never been used for this test. Keeping everything else the same (samples, reagents, analysts, method), the team hydrolyzed the next three batches in Oven B. All three passed. Changing one variable made the failures disappear, the equivalent of Shainin’s component swap. But three passes are evidence, not proof.
6. Proving the cause
Extended temperature logging
Oven A had passed its routine calibration, a single-point check in an empty chamber. That check couldn’t catch a problem that only happened sometimes, somewhere in the chamber, during a long run. So the team placed a calibrated multi-probe data logger (front, center, back; upper and lower shelves) in Oven A with a full sample load, and logged five overnight cycles.
| Cycle | Temperature behavior | Control sample |
|---|---|---|
| 1 | Stable at setpoint all night | Normal |
| 2 | Dropped 8–10 °C below setpoint for about 3 hours; worst at lower-rear positions | Low |
| 3 | Stable at setpoint | Normal |
| 4 | Repeated drops of 6–9 °C over about 2 hours | Low |
| 5 | Stable at setpoint | Normal |
The pattern matched exactly: low results occurred only in cycles with temperature drops, and the oven’s own display never showed the dips. The heating was intermittently cutting out overnight. This also explained the multi-vari finding: when the heating failed, the lower-rear positions cooled the most.
Confirmation: “take the Red X® to court”
The final test turned the failure on and off. Matched aliquots of the control were split between Oven B (the better condition, “B”) and Oven A (the current condition, “C”) in the same overnight run, with the logger running in Oven A. Oven A logged a temperature drop that night.
| Condition | Aliquot 1 | Aliquot 2 | Aliquot 3 |
|---|---|---|---|
| B: Oven B (stable) | 5.01% | 4.98% | 5.03% |
| C: Oven A (logged drop) | 3.31% | 3.45% | 3.28% |
Every B result beat every C result. With three of each, complete separation like this has only about a 1-in-20 chance of happening by luck, which is roughly 95% confidence.
Cause demonstrated
Intermittent loss of heating in Oven A during overnight hydrolysis leads to incomplete hydrolysis and low results. This was confirmed by the best-vs-worst contrast, the oven swap, temperature logging correlated with results, and a side-by-side B vs. C run.
7. Root cause statement
| Technical cause | Intermittent heating failure in Oven A caused temperature drops during overnight hydrolysis, leading to incomplete hydrolysis and low protein results. |
|---|---|
| Why it wasn’t detected | Routine calibration was a single-point, empty-chamber check that couldn’t detect intermittent or position-specific behavior during long cycles, and the oven had no independent temperature monitoring or alarm. |
| Contributing factor | The SOP didn’t define the mixing time after hydrolysis, leaving room for run-to-run variation (not causal here, but corrected). |
The cause statement stops at what was demonstrated. A component-level teardown of Oven A (heater, relay, or controller) was judged unnecessary: the oven was being removed from service, so knowing which part failed wouldn’t change any decision. This risk-based choice was documented in the investigation.
8. Disposition of the OOS results
- With an assignable cause demonstrated, QA approved invalidation of the original OOS results, with the evidence attached to the investigation record.
- Affected lots were retested per the OOS procedure in qualified Oven B, passed, and were released by QA.
- Look-back: results for lots hydrolyzed in Oven A over the relevant period were reviewed for possible low bias. No other lots were affected.
9. CAPA
| Type | Action |
|---|---|
| Correction | Oven A removed from GMP use and retired; hydrolysis moved to qualified Oven B. |
| Corrective action | Continuous temperature logging with low-temperature alarms added for hydrolysis ovens; each run’s temperature record reviewed before results are approved. |
| Corrective action | Oven qualification upgraded to include loaded, multi-position, full-cycle mapping; calibration program updated to include periodic extended checks. |
| Preventive action | SOP revised through change control to define mixing time and to require a control sample of known value in every hydrolysis run. |
| Preventive action | Read-across: other long-cycle heated equipment (ovens, heating blocks, incubators) assessed for the same detection gap. |
10. Effectiveness check
Criteria were set before closing: over the next 30 hydrolysis runs, no OOS results attributable to hydrolysis, control samples within their limits in every run, and no unreviewed temperature excursions. All criteria were met, and the CAPA was closed as effective.
11. How the tools worked together
| Stage | Tool | What it contributed | Its limit |
|---|---|---|---|
| Define | Kepner-Tregoe Is / Is-Not | Separated a test failure from a product failure; exposed the open question about the oven | Organizes facts; doesn’t prove cause |
| Narrow | Process map, size-of-the-miss analysis | Grouped suspects into families; eliminated steps that couldn’t cause a one-third loss | Needs good knowledge of the method |
| Isolate | Split testing of retained material | Pinned the loss to the hydrolysis stage | Requires preserved evidence |
| Test | Hypothesis protocol with controls | Eliminated reagent, seal, and mixing suspects in parallel | Weak when the failure isn’t active |
| Contrast | Red X®: Green Y, BOB/WOW, multi-vari | Showed the variation lived cycle to cycle inside hydrolysis | Needs a measurable result and real extremes |
| Swap | Oven A vs. Oven B | Changed one variable and made the failure disappear | Evidence, not yet proof |
| Prove | Data logging, B vs. C run | Showed the mechanism and turned the failure on and off | Needs enough cycles to catch the fault |
| Close | CAPA with effectiveness criteria | Fixed the cause and the detection gap, and verified it worked | Only as good as the criteria set up front |
When Red X® fits, and when it doesn’t
Good fit (as here): a measurable result, clear good and bad extremes, one dominant cause, and the ability to compare or swap conditions. Intermittent equipment problems are a classic example.
Poor fit: many small causes of similar size, failures too rare to find extremes, evidence that no longer exists, or causes that can’t safely be recreated (sterility, for example). Red X® also finds the technical cause, not why the quality system missed it. That’s why this investigation paired it with a look at calibration and monitoring before closing the CAPA.
Key takeaways
- Let the size of the miss guide the search. A one-third loss rules out many small-error suspects immediately.
- A passing retest is not an explanation. The investigation continued until the cause was shown.
- Split the method into families and use retained material to find where the signal is lost.
- Compare the extremes. Best-vs-worst runs and a multi-vari study pointed to cycle-to-cycle variation faster than testing suspects one at a time.
- Change one thing at a time. The oven swap worked because nothing else changed.
- Snapshots can’t catch intermittent problems. Extended, loaded, multi-position logging found what a single calibration point missed.
- Prove it by turning it on and off. A side-by-side B vs. C run turned a strong suspect into a demonstrated cause.
- Fix the detection gap, not just the equipment. The CAPA addressed why the problem wasn’t caught, so it won’t recur elsewhere.
- Match investigation depth to the decisions that depend on it. Proving the oven was the cause mattered; finding the failed component did not.
Questions to think about
- The same sample failed once and passed on repeat. Why isn’t that enough to invalidate the first result?
- Why use a control sample, rather than production lots, as the Green Y?
- In the best-vs-worst table, every run used Oven A. What could and couldn’t that comparison tell the team?
- The oven passed its routine calibration. What would you change in a calibration program to catch problems like this?
- Each test cycle took one to two days. How would you design experiments to reach a conclusion faster?
- When is it reasonable to stop investigating before identifying the failed component?
Related reading: Advanced Root Cause Analysis and Pipette OOS: When Two Things Change at Once.
Red X® is a registered trademark of Shainin, and Isoplot and the other Shainin tools named here are part of its proprietary methodology. Kepner-Tregoe is a proprietary method of its owner. These methods are described here for educational purposes only; QStead Group is not affiliated with either. This case study is for educational purposes only.
Working through an OOS or CAPA investigation that won’t close? Get in touch.