research-document
Comparator Defect Register
Historical calibration artifact. This document describes Comparator v3.0 before the Comparator v3.1 implementation. Current instrument status is recorded in comparator-v3.1-approval-record.md, comparator-v3.1-regression-report.md, and the generated-v3.1 outputs.
Historical Status: Superseded as current-state guidance by Comparator v3.1 implementation and regression results.
Comparator Defect Register
| ID | Description | Evidence | Severity | Likelihood | Recommended Action | Status | v3.1 Resolution | Regression Evidence | Current Status |
|---|---|---|---|---|---|---|---|---|---|
| CD-001 | Recognition classifier can mislabel broad-family resemblance as recognized. |
Claude P001C says "No confident recognition" and "no specific named artifact is identified", but raw-comparison-data.json records recognized. |
critical | high | Add regression case for negative-recognition phrasing that still includes family resemblance. | open | Recognition classification now maps explicit recognition negation plus family resemblance to partial rather than recognized. |
comparator-v3.1-regression-report.md RG-001 PASS; comparison/generated-v3.1/recognition-persistence-results.csv shows Claude P001A, P001B, and P001C as partial. |
resolved_v3.1 |
| CD-002 | Stored machine-readable output is inconsistent with current human-facing reports. | raw-comparison-data.json shows primary_record_count: 12 and empty data_quality; current summary shows 10/12 with two malformed files. |
critical | high | Freeze report-generation order and add regression to ensure all derived outputs reflect the same run state. | open | Generated v3.1 outputs now share one immutable run state and the machine summary matches the human-facing reports. | comparator-v3.1-regression-report.md RG-005 PASS and RG-013 PASS; comparison/generated-v3.1/run-state.json run ID 179fa743-b92a-4f75-b4a2-492fa9eb7898; same run ID appears in summary, data-quality, recognition, structural, and observation-ledger outputs. |
resolved_v3.1 |
| CD-003 | Structural scope summaries overstate disagreement severity. | Every scope in structural-stability-report.md is disagreement-heavy while the overall structural profile is mostly_stable. |
major | high | Separate backbone/topology stability from field-level lexical disagreement in summary logic. | open | Structural reporting now separates backbone stability from literal, conceptual, and dimensional agreement. | comparator-v3.1-regression-report.md RG-006 PASS and RG-007 PASS; comparison/generated-v3.1/structural-stability-report.md reports separate backbone, literal, conceptual, and dimensional profiles. |
resolved_v3.1 |
| CD-004 | Constraint disagreement labels are dominated by field-placement and schema variance. | All five constraint fields report disagreement despite substantial semantic overlap across responses. |
major | high | Treat constraint fields as calibration-sensitive until cross-field normalization is validated. | open | Constraint reporting now adds conceptual cross-field comparison and field-placement sensitivity, but remains calibration-sensitive. | comparator-v3.1-regression-report.md RG-009 PASS; comparison/generated-v3.1/constraint-stability-report.md reports Dimensional conceptual agreement: partial_agreement and Field-placement sensitivity: high. |
mitigated_v3.1 |
| CD-005 | Representation metrics are scored even when they are mostly prose-style dependent. | Natural summary, canonical summary, ambiguities, and notes all report disagreement. |
major | high | Convert style-heavy representation metrics to informational status or separate review metrics. | open | Representation prose metrics are now informational rather than score-bearing. | comparator-v3.1-regression-report.md RG-010 PASS; comparison/generated-v3.1/representation-stability-report.md marks all prose fields informational_only. |
resolved_v3.1 |
| CD-006 | unknown versus not_recognized boundary is underspecified for values like unrecognized. |
Gemini P001A and P001C are classified as unknown; that is plausible, but the rule text does not explicitly settle it. |
minor | medium | Clarify rule text and add regression coverage for unrecognized, unknown, and empty-string cases. |
open | Recognition boundary is now explicitly frozen: unrecognized maps to unknown, while an empty field maps to not_recognized. |
comparator-v3.1-regression-report.md RG-002 PASS and RG-003 PASS; comparison/generated-v3.1/recognition-persistence-results.csv shows Gemini P001A and P001C as unknown. |
resolved_v3.1 |
| CD-007 | Comparator depends on tolerant parsing for GPT files, but that tolerance is not clearly surfaced as a first-class audit condition. | GPT canonical files use smart quotes and fail strict jq parsing, yet some runs still consume them after sanitization. |
major | medium | Make tolerant-parsing use explicit in data-quality outputs and regression tests. | open | Tolerant parsing is now explicitly surfaced in data-quality reporting and regression coverage. | comparator-v3.1-regression-report.md RG-011 PASS; comparison/generated-v3.1/data-quality-report.md records one tolerant_parse entry and four dataset-level warning rows. |
resolved_v3.1 |
| CD-008 | Recognition leakage detection can surface terms appearing in cautionary prose rather than actual recognition claims. | Claude P001D is not_recognized but still gets leakage findings from other response text. |
observation | medium | Keep leakage reporting separate from recognition classification and label it as contextual rather than decisive. | open | Leakage reporting remains separate from recognition classification and is treated as contextual evidence rather than decisive recognition. | comparison/generated-v3.1/domain-leakage-report.md remains observational; comparison/generated-v3.1/recognition-persistence-report.md keeps Claude P001D at not_recognized. |
mitigated_v3.1 |