research-document
FE-012C Threats To Validity
FE-012C Threats To Validity
- Recognition bias: If models recognized artifacts, blinding was weak.
- Shared training data: Model overlap may inflate agreement through shared exposure rather than independent convergence.
- Prompt and instrument effects: A common packet structure can induce common output structure.
- Researcher-designed primitive vocabulary: The provided vocabulary may constrain or channel the outputs.
- Model-family limitations: Agreement across three model families is still model-based evidence only.
- No human validation: Human analysts were not part of this result set.
- No domain expert validation: Domain experts did not review extraction quality in this run.
- Ambiguity in primitive boundaries:
Boundaries between fields such as
Evaluate,Compare,Prioritize,Verify, andDecideremain contestable. - JSON schema effects: Required fields may force artificial precision or convergence.
- Likely over-agreement due to provided vocabulary: Lack of candidate missing primitives may partly reflect the constraint of the available vocabulary rather than true sufficiency.