INTEL128 · DOCS · EVALUATION

How we evaluate

What we measure, how we split data, and what we will not publish. This is the protocol, written before the results.

No performance numbers appear on this page. The evaluation runs against a held-out, source- and time-disjoint test set that stays sealed until prompts, model versions, and analysis plans are frozen. When results exist they will be published here with their confidence intervals — including any that are flat or negative.

The five tasks

Task	What is asked
Entity extraction	Identify supported CTI entities and their exact mention spans in the report text.
Relation extraction	Predict a typed relation between two entities, with the source evidence for it.
ATT&CK mapping	Map a described behavior to a version-pinned technique identifier — with an explicit no-match or uncertain option.
Evidence support	Decide whether the cited report span actually entails the proposed claim.
Abstention	Behave correctly on ambiguous or insufficiently supported input.

Actor attribution is not a primary prediction target. If it appears at all, it is exploratory, is evaluated separately, and is never presented as definitive attribution.

Metrics

Measured	How
Extraction quality	Strict span-level precision, recall and F1 for entities and relations. Relaxed matching is reported only as a secondary measure, never instead of strict.
Relation correctness	Both endpoints and the predicate must be right.
Normalization	Identifier accuracy and the unresolved-ID rate, against a pinned reference version.
Evidence	Exact quote and offset match, plus a blinded human entailment review.
ATT&CK mapping	Precision, recall and F1 against the pinned release, with an explicit adjudication policy.
Schema validity	The share of model outputs that pass the validator.
Unsupported-claim rate	Claims whose evidence does not support them.
Abstention	Coverage, and the risk among non-abstained predictions — i.e. what happens to the answers it does give.
Efficiency	Latency, tokens per second where meaningful, peak GPU memory, host RAM, adapter size, training time.

A confidence output is treated as a score, not a calibrated probability, unless calibration is explicitly evaluated and reported as such.

Statistical analysis

All primary results are computed from frozen predictions and a versioned scorer.
Confidence intervals use document-level bootstrap resampling, so passages from the same report are not treated as independent.
Paired comparisons use the same examples and the same prompts.
The statistical test, significance level, any multiple-comparison correction, and random seeds are specified before the final test evaluation.
Where compute allows, the principal comparison is repeated across seeds. If not, that limitation is disclosed and document-level uncertainty is reported.
No best-run-only result is ever presented as the estimate.

Splitting

Splits are made at the report, source and campaign level, not by sentence, so variants of the same report cannot leak across train and test.
Near-duplicate reports and mirrors are grouped before partitioning.
Where dates allow, the test set is chronologically later than the training data.
The final test labels stay held out until prompts, model versions and the analysis plan are frozen.
Possible pretraining contamination is documented as a threat to validity rather than assumed absent.

Ablations

Each component is removed or varied one at a time, so a gain can be attributed to something specific:

retrieval on / off
the evidence-span requirement on / off
the explicit epistemic-status field present / absent
the abstention option available / withheld
adapter rank and training-data quantity, where tuning is feasible

The adapter feasibility gate

Before any fine-tuning result is claimed, the stack must demonstrate, on synthetic examples: a forward pass, a finite training loss, a backward pass, an optimizer update, adapter save and reload, and reproducible inference. The exact trainable modules, model revision, quantization, library versions, sequence length and peak VRAM are recorded.

If that gate fails, the adaptation claim is dropped — not softened. The model remains an inference baseline and any result from a different model is reported as that model's result.

Reproducibility

Each run records software versions, OS, GPU and VRAM, model revision, tokenizer and prompt-encoding implementation, prompts, schema, reference-data version, dataset hashes, split manifests, seeds and decoding configuration. Every experiment has a unique identifier and a machine-readable configuration, and the tables regenerate from saved predictions.

Public release is limited to code, configs, manifests, schemas, synthetic fixtures, and artifacts whose licenses permit redistribution.

What you will not see

A single headline number without its denominator, split and confidence interval.
A result from the test set before the protocol is frozen.
A benchmark score presented as proof of operational value — throughput or analyst-time claims require a measured study.
Attribution presented as fact.
Confidence scores described as calibrated probabilities unless calibration was evaluated.

Previous: ← The claim schema

Ask us for the evaluation protocol.

If you are assessing PeriscopeCTI for adoption or review, we will share the full protocol, the split design and the metric definitions.

Request a research preview