research-document

FE-012C Threats To Validity

FE-012C Threats To Validity

  • Recognition bias: If models recognized artifacts, blinding was weak.
  • Shared training data: Model overlap may inflate agreement through shared exposure rather than independent convergence.
  • Prompt and instrument effects: A common packet structure can induce common output structure.
  • Researcher-designed primitive vocabulary: The provided vocabulary may constrain or channel the outputs.
  • Model-family limitations: Agreement across three model families is still model-based evidence only.
  • No human validation: Human analysts were not part of this result set.
  • No domain expert validation: Domain experts did not review extraction quality in this run.
  • Ambiguity in primitive boundaries: Boundaries between fields such as Evaluate, Compare, Prioritize, Verify, and Decide remain contestable.
  • JSON schema effects: Required fields may force artificial precision or convergence.
  • Likely over-agreement due to provided vocabulary: Lack of candidate missing primitives may partly reflect the constraint of the available vocabulary rather than true sufficiency.