Non-Human Experimental Comparative Analysis

Status: Research analysis artifact — proposal-only, not a canonical accepted record. Supersedes nothing; ratification (if any) is a human governance decision. Companion documents: non-human-experimental-inventory.md, non-human-experiment-matrix.json, non-human-experimental-lineage.md, non-human-experimental-contradictions.md, non-human-experimental-failure-modes.md, non-human-cumulative-findings.md, non-human-next-experiments.md.

1. Experiment Families

Grouped by what actually varies between members, per the lineage analysis:

Family A — Same-model internal pilots (E1 FE-011A, E2 FE-012A, E3 FE-012B)

Common design: single model family plays generator and evaluator/reconstructor roles; researcher-authored test material; no independent replication. Question: can structured redesign / a primitive vocabulary do something useful, at all, before investing further? These were explicitly designed as fast pre-checks, not as decisive tests, and their own documentation says so.

Family B — Three-provider structural-recovery line (E4 FE-012C Manual, E5 FE-012C Repeatability, E7 ECR-000001, E8 ECR-000002, E9 ECR-000003)

Common design: GPT + Claude + Gemini, packet-based blinded extraction, comparator-scored agreement. Question: is there a recoverable, stable, cross-model procedural structure beneath surface wording? This is the corpus's largest and most instrumented family, and the only one with a direct repeatability test (E4→E5) and a frozen, versioned comparator (E9).

Family C — Unexecuted, fully designed experiments (E6 FE-013, E11 EX-FE-0002, E12 EX-FE-0003)

Common design: rigorous protocols, explicit falsification/stopping conditions, zero execution data. Not failures — genuinely a distinct category the mission's Phase 10 asks to be preserved rather than discarded or silently treated as failures.

Family D — Repository-scope meta-evaluations (E10 EX-FE-0001, E13 FE-EVAL-2026-07-23, E14 FE-BOUNDARY-2026-07-28)

Common design: an AI agent reasons over the repository's own claims and (for E14) external sources, rather than running a blinded packet experiment. Question: is Framework Engineering itself a distinct, defensible discipline? E10 is nominally in this family by design intent but produced no usable content (fixture only).

2. Pairwise Comparisons (materially related pairs only)

E4 vs. E5 (FE-012C Manual vs. Repeatability)

• Constant: packets, vocabulary, provider triad, comparator logic family
• Changed: point in time (fresh sessions later)
• Valid comparison: Yes — purpose-built matched-pairs design
• Agreement: No — single-sitting cross-model agreement (E4) does not predict cross-time stability (E5)
• Confidence in comparison: High (see Contradiction C1 for full treatment)

E7 vs. E8 vs. E9 (ECR-000001/2/3)

• Constant: provider triad, general packet-based design philosophy, general research question lineage
• Changed: manipulated variable (calibration → recognition-level → topology/representation/domain), comparator version, packet corpus scope
• Valid comparison: Partially — meaningful for tracking how the research question and instrument matured, not meaningful as a controlled experiment across the three (they are not held constant on enough dimensions to isolate one variable's effect across all three)
• Agreement: E7's outcome could not be reconstructed (finding not located); E8 and E9 both report persistent primitive/constraint disagreement, and E9 additionally reports the recognition- and topology-dependent backbone results described in Contradiction C2.
• Confidence in comparison: Moderate for E8-vs-E9, low for anything involving E7 given its missing synthesis

E2/E3 vs. E4/E5/E9 (same-model extraction vs. cross-model convergence)

• Constant: shared primitive vocabulary lineage
• Changed: single-model vs. three-provider design; internal vs. externally-facing evaluation
• Valid comparison: Limited — E2/E3's "did not collapse into artifact-specific vocabulary" finding and E4/E9's "primitive/constraint disagreement persisted" finding are not strictly contradictory (different questions: "is a vocabulary usable at all" vs. "do independent models converge using it"), but a reader moving from E2/E3's more positive framing to E4/E9's more mixed framing without tracking the underlying design change could wrongly read this as regression or inconsistency rather than a difference in what was tested.
• Confidence in comparison: Low-Moderate

E13 vs. E14 (Codex meta-evaluations)

• Constant: authoring agent, general method (text-analytic reasoning over repository claims)
• Changed: E14 adds external sourcing and narrows scope specifically to the discipline-boundary question
• Valid comparison: Yes, as a longitudinal single-author view, not as two independent opinions (see Lineage Cluster 3)
• Agreement: High — E14's conclusion ("not a distinct discipline," "candidate integrated engineering profile") is a direct continuation of E13's "provisional research and method-engineering program" framing, not a reversal
• Confidence in comparison: High for tracking one evaluator's view over time; this pairing cannot be used as evidence of inter-rater agreement

3. Longitudinal Findings

Ordering what's dateable: E1/E2/E3 (2026-06-28) → E4 (undated, before repeatability) → E7/E8/E9 (ECR series, undated internally but referenced from a June 30 decision-log entry establishing the operating system) → E5 (2026-07, post-E4) → E9's board packet and E13 (2026-07-23) → E14 (2026-07-28) → this analysis (2026-07-30).

• Confidence has not monotonically increased with time. confidence-assessment.md caps every major claim at Moderate or below regardless of how much additional evidence has accumulated since June, which is consistent with (and arguably a strength of) the program's explicit "prefer weakening claims" posture — this is one of few research programs reviewed where volume of work has not been allowed to inflate stated confidence.
• The single largest longitudinal shift in the corpus is discipline-scope, not mechanism-level findings: E13 (2026-07-23) already downgrades "Framework Engineering" from an implied discipline to "provisional... program," and E14 (2026-07-28) goes further, concluding no mechanism is shown to be unique. Neither of these downgrades rests on new experimental (Family A/B) data — both are text-analytic reassessments of the same underlying evidence base. This means the corpus's confidence in FE-as-a-discipline has fallen over the reviewed period while its confidence in specific structural-recovery findings (H001-H018) has stayed flat, because different documents, not new data, drove the discipline-level reassessment.
• Model/version drift cannot be assessed from available documentation: exact GPT/Claude/Gemini versions and dates were MISSING for E4 and E5, so it is not possible to determine whether E5's drift is attributable to model updates between the two runs versus pure sampling variance at a fixed model version. This is recorded as an evidence gap, not resolved by assumption.

4. Cross-Project / Cross-Family Comparisons

The mission checklist asks whether findings transfer across projects; within this single repository, the closest analogue is whether findings transfer across the "structural recovery" question (Family B) and the "discipline boundary" question (Family D). They do not currently inform each other much: Family D's evaluations (E13, E14) cite Family B's results only briefly (as one bullet each in E13's "Findings that survive this review": "Backbone-level procedural similarity sometimes persists across representations and providers; detailed primitives, transitions, and constraints are less stable" — an accurate, hedged summary), and neither E13 nor E14 engages with E5's repeatability finding at all. This is itself worth flagging: the corpus's most decisive single quantitative result (80% structural drift on repeat) is not mentioned in either of the two repository-wide evaluation packages, despite both being produced after E5's data existed in the repository. This is either an oversight in those evaluations or a scoping choice (E13/E14 focus on discipline-boundary questions, not mechanism-level repeatability) — this analysis cannot determine which from available evidence, and records it as an open question rather than asserting oversight.

5. Quantitative Synthesis

Formal meta-analysis (pooled effect sizes, weighted averages) is not performed in this document. Justification, per the mission's own constraint against forcing incomparable measurements into a common score: the corpus's dependent variables are not on a common scale even within Family B (E4/E5 use a 3-category literal/semantic drift classification; E9 uses a comparator-specific multi-field agreement/disagreement/mixed vocabulary with different field names per sub-experiment), sample sizes are small (n=9 to n=45 per experiment), and — per the lineage analysis — the "independent" observations within Family B are not independent in the statistical sense (same 3 providers repeated across studies), which would violate the independence assumption behind any naive pooled estimate or confidence interval. Reporting a single pooled number here would create false precision the source data does not support, which the repository's own Constitution explicitly warns against ("Appropriate Precision").

What can be reported honestly:

The two headline numbers most likely to be quoted out of context elsewhere in the repository going forward — E4's "15/15 entry-primitive agreement" and E5's "80% structural drift" — describe the same instrument and the same three providers under different temporal conditions and should always be reported together, never separately, per Contradiction C1.

Sensitivity check performed: recalculating E5's headline under the more evaluator-generous semantic comparator (rather than the strict literal one) does not materially change the conclusion (84.4% vs. 80.0% structural drift) — the finding is robust to this specific analytical choice, which is one of the few places in this corpus where a sensitivity analysis was actually possible from available data.

6. Qualitative Synthesis

See non-human-experimental-failure-modes.md for the full taxonomy. Headline qualitative pattern: every experiment in Family B that measured recognition/leakage found it present; every experiment that measured "dominant"/"exit"-type fields found persistent disagreement; and the one experiment that tested temporal repeatability found it largely absent. Read together, these three qualitative patterns support a more conservative reading of the whole family's positive results than any single experiment's own summary suggests in isolation — a pattern the mission brief specifically warns is common ("conclusions may be weaker than the repository currently claims").

7. Negative and Null Result Analysis

• FE-013 (E6): fully designed, zero execution — a preserved, explicit non-result, not folded into any claim. This is good practice: the repository did not quietly drop the idea or claim a result it doesn't have.
• ECR-000002 (E8): thin execution, no consolidated findings located, apparently informally absorbed by ECR-000003 without a formal supersession record (Failure Mode F6). This is a case where a stalled/incomplete result risks being neither actively cited nor formally retired — a gray zone the mission specifically asks to be surfaced.
• EX-FE-0002 (E11): explicitly described by a later document (E14) as incomplete/blocked. Correctly not treated as evidence of anything by E14, which is good practice.
• EX-FE-0001 (E10): registry metadata (status: ready, run_count_by_provider: {manual: 6}) creates an impression of substantive completed work that the underlying artifact (a template synthesis file with empty findings, built on fixture data) does not support. This is the one place in the corpus where this analysis found a registry status that appears to overstate what was actually produced — recorded as a documentation-integrity finding, not an accusation of fabrication (fixture/infrastructure runs are a legitimate and disclosed activity; the risk is only that the registry's status field does not visibly distinguish "infrastructure exercised" from "evidence collected").
• Publication/recording bias assessment: The corpus does not show evidence of hiding negative results — quite the opposite; E5's unfavorable repeatability finding is fully published with both a strict and a generous comparator, and E9's ECR-000003-FINDINGS.md has an explicit "Claims Not Supported" section. The main asymmetry found is not suppression but under-cross-referencing: unfavorable findings (E5's drift rate) are documented in full but not consistently carried forward into the higher-level synthesis documents (hypothesis registry confidence ratings, E13/E14 meta-evaluations) that a reader would consult first.

8. Confidence Reassessment

Applying the calibrated-language scale requested by the mission (established / strongly supported / moderately supported / suggestive / weakly supported / unsupported / contradicted / obsolete / unresolved) to the corpus's major claims, independent of the repository's own existing labels:

9. Adversarial Self-Review of This Analysis

Applying the mission's Phase 13 challenge questions to the five conclusions above with the largest decision consequences:

1. "Recognition bias is strongly supported" — Could this be evaluator bias in this analysis rather than the artifacts? Alternative: recognition fields might be over-reported because the schema makes them easy and salient to fill in relative to true-negative (no recognition) cases, inflating apparent frequency. This review cannot rule this out from available data; recorded as a residual uncertainty.
2. "H004 repeatability should be revised downward" — Could regression to the mean or a single bad run explain the 80% figure? The corpus's own semantic-comparator re-analysis (a designed robustness check, not this review's invention) shows the result is not an artifact of strict literal scoring (84.4% under generous scoring), which weakens this alternative but does not eliminate the possibility that this specific repeatability run happened to catch an unusually volatile period (e.g., a provider-side model update). Without dated model-version metadata (a documented gap), this alternative cannot be excluded.
3. "No hypothesis exceeds Moderate confidence" — Is this a real finding about the world, or an artifact of a research culture that has adopted deliberately conservative confidence-labeling norms (per confidence-assessment.md's explicit "No statement in this document is assigned High")? Plausibly both — the norm itself is evidence-appropriate given the underlying evidence quality (Family A/B's independence and blinding problems), so this review treats the low confidence ceiling as substantively earned, not merely stylistic, while acknowledging the norm itself was adopted before, not because of, this specific evidence review.
4. "FE is not a distinct discipline" — This review agrees with E14, but E14 and this review share partial method overlap (both are AI-agent text-and-evidence reviews, and this review read E14 before forming its own view, which is a real anchoring risk this review cannot fully correct for from within the same review). A genuinely independent check would require a reviewer who has not read E13/E14 first.
5. "EXP-003's mostly_stable backbone is real but untrusted pending negative controls" — Could the comparator itself be biased toward reporting "mostly_stable" (e.g., a lenient default codepath)? This review did not audit the Comparator 3.1.0 source code and cannot rule this out; it is exactly the concern the repository's own next-research-decision document raises as the reason to run negative controls before trusting this result, and this review defers to that reasoning rather than independently re-deriving it.

10.5 Required Analytical Tables

Experiment Quality Matrix

Comparability Matrix

Findings Matrix

Replication Matrix

Research-Gap Matrix

10. What Would Reverse Each Major Conclusion (Falsification Conditions Carried Forward)

• H004 downgrade would be reversed by: a second, independently-timed repeatability run showing drift substantially below 80%, ideally with model version pinned and logged.
• H013 (recognition bias, strongly supported) would be reversed by: a validated graph-only/topology-only negative control showing recognition drops to near-zero while structural extraction quality is maintained (H016's own kill condition).
• "FE is not a distinct discipline" would be reversed by: the matched-utility experiment already designed in E14's next-experiment.md (Arms A/B/C/D) showing C (full FE profile) outperforms B (matched adjacent-practice profile) on replication, with cost accounted for.
• EXP-003's isomorphism finding (currently "unresolved, pending negative controls") would be strengthened by: the board's own top-ranked recommended experiment (negative non-isomorphic controls) showing a sharp, expected drop in backbone stability for genuinely non-isomorphic procedures, which would validate that the comparator can in fact detect the absence of shared structure rather than defaulting to "mostly_stable."