Blind Review Protocol

Status: Working draft

Purpose

The blind review protocol reduces naming bias during artifact classification.

The protocol should use FEMS-1 consistently once adopted for a study.

Procedure

1. Complete a KACS record for the artifact.
2. Remove branding, advocacy language, and other identifying cues where practical.
3. Present the anonymized KACS record to reviewers.
4. Ask reviewers to answer independently:
   1. Identity
   2. Capabilities
   3. Composition if applicable
   4. Domain
   5. Confidence
   6. Rationale
5. Ask reviewers to note uncertainty explicitly.
6. Compare reviewer results and document disagreements.

Reviewer Instructions

• Use the characterization record, not prior familiarity, as the primary basis for classification.
• Apply operational definitions before relying on common labels.
• Measure agreement separately for identity, capabilities, framework grammar, primitive composition, and domain.
• Keep classifications provisional.
• Use categorical confidence only.
• Do not use numeric scores.

Expected Outputs

• anonymized KACS records
• identity classifications
• capability assignments
• framework grammar assignments when applicable
• primitive composition assignments when applicable
• domain assignments
• categorical confidence assessments
• disagreement notes
• open boundary questions