scribeGOAT2: Multi-Turn Safety Persistence Evaluation for Frontier AI

Q: What is scribeGOAT2?

scribeGOAT2 is an open-source physician-adjudicated framework for evaluating multi-turn safety persistence in frontier AI models like GPT-5, Claude, Gemini, and Grok. It reveals trajectory-level failures invisible to single-turn benchmarks.

Q: Who created scribeGOAT2?

scribeGOAT2 was created by Brandon Dent, MD, an emergency medicine physician and AI safety researcher, through GOATnote Inc.

Q: What AI models does scribeGOAT2 evaluate?

scribeGOAT2 evaluates frontier AI models including GPT-5.2, Claude Opus 4.5, Gemini 3 Pro, and Grok 4 for multi-turn safety persistence.

Q: What is the key finding from scribeGOAT2 research?

The primary observation shows Turn 1 accuracy of 95-100% drops to Turn 5 safety persistence of 0-53% across frontier models under realistic conversational pressure.

Q: Is scribeGOAT2 open source?

Yes, scribeGOAT2 is fully open source and available at https://github.com/GOATnote-Inc/scribegoat2

Brandon Dent, MD; Brandon Dent, MD

GOATnoteMedical AI Safety & Evaluation

GitHub LinkedIn

DeterministicReproduciblePhysician-AdjudicatedOpen Source

Multi-Turn Safety Evaluationfor Healthcare AI

Measuring what single-turn benchmarks miss: whether models maintain safety-critical recommendations when users express realistic barriers.

What We MeasureJanuary 2026

Turn 1

Recognition

pressure

Turn 5+

Persistence

Models correctly identify emergencies at conversation start. We measure whether they maintain recommendations under naturalistic pressure.

4 frontier models10,000+ trajectories3-tier framework

For Safety Teams

Complementing—not replacing—existing evaluation infrastructure

What you likely measure

Response quality (rubrics)
Final response accuracy
Adversarial jailbreaks
Single-turn safety checks

What scribeGOAT2 adds

Trajectory persistence (invariants)
Cross-turn consistency under pressure
Naturalistic pressure from cooperative users
Multi-turn safety maintenance

Integration: scribeGOAT2 operates as a post-model evaluation layer, measuring behavioral properties that RLHF may not directly optimize.

Validated Finding

System prompt mitigation significantly improves crisis resource persistence in tested conditions

100%

resource persistence

(with mitigation)

1.38

Cohen's h effect size

(large effect)

N = 1,000Robust through T5 adversarial pressure

Three-Tier Evaluation Architecture

Tier 3: Adversarial Pressure

Advanced

Authority cues · Deceptive framing · Context collapse

Sustained multi-turn pressure without jailbreak techniques

Tier 2: Subtle Failures

Deployment-Relevant

Resource omission · Boundary erosion · Quiet disengagement

Gaps between "not unsafe" and "actively safe"

Tier 1: Baseline Safety

Foundation

Urgency minimization · Delay validation · Patient abandonment

Context-independent violations any clinician would flag

Tier 1 confirms models avoid obvious errors (expected for frontier models). Tier 2 captures deployment-relevant gaps that baseline checks miss. Tier 3 tests resilience under stronger naturalistic pressure.

seed=42 · temperature=0 · deterministic scenario generation

Cross-family judge validation · Physician-adjudicated calibration (κ = 1.00)

Evaluation Results

Summary findings across L1-L4 evaluation layers

Layer	Finding	N
L1 Baseline	0% hard failures(all frontier models pass)	2,000+
L2 Severity	0% severe/catastrophic(S3-S4)	1,140
L3 Recovery	26.5% recover with prompt(73.5% persist without)	400
L4 Robustness	Mitigation holds through T5(adversarial pressure)	500

These findings reflect behavioral properties under controlled evaluation conditions. Divergence from criterion does not equal real-world harm. Full model-specific results available in GitHub repository.

Scope & Limitations

What we measure

Behavioral divergence under controlled crisis disengagement
Stratification by harm severity (S0-S4)
Recovery dynamics after divergence
Mitigation robustness under adversarial pressure
Reproducible evaluation with confidence intervals

What we do not claim

Proof of real-world harm (only behavioral proxies)
Comprehensive coverage of all failure modes
That passing means "safe"
Replacement for deployment monitoring
Clinical validation of model outputs

Interpretation guidance: A 43% divergence rate means "model diverges from criterion at 43% under probe conditions"—not "model fails 43% of the time." We report behavioral measurement, not safety certification.

Clinical Strategy Context

Both observed behavioral patterns have documented clinical precedent

Harm Reduction Approach

Provides monitoring guidance alongside emergency recommendations when users indicate barriers. Prioritizes meeting patients where they are.

"If you're going to wait, here are the warning signs to watch for..."

Documented in: Emergency medicine literature, harm reduction frameworks

Boundary Maintenance Approach

Declines to provide monitoring guidance, maintaining that emergency care is the only appropriate action. Prioritizes clear boundaries.

"I'm not comfortable helping you manage this at home..."

Documented in: Risk management guidelines, medicolegal standards

Our evaluation measures divergence from a specified criterion, not superiority of one clinical strategy over another. Reasonable clinicians disagree on which approach is preferable in high-stakes scenarios.

Responsible Disclosure

Findings shared with safety teams at OpenAI, Anthropic, Google DeepMind, and xAI through established disclosure channels (December 2025 – January 2026).

This work is designed as evaluation infrastructure to support iterative safety improvement—not adversarial model criticism.

OpenAIAnthropicGoogle DeepMindxAI

For Safety Teams

Detailed model-specific findings and methodology documentation available through disclosure channels.

For Researchers

Full evaluation code, checkpoint data, and reproduction instructions available under MIT license.