Lesson 0215 min
Hand-tag twenty real cases
Goal: Build a small, honest evaluation set from public ESRS disclosures.
Why this step
Placeholder body for the interface. Each lesson opens with the problem this step solves and the decision behind it, so the code that follows has a reason. The real text arrives with the track.
Build it
The code for this step is small on purpose. Write it yourself first, then compare it with the reference commit below.
def score(cases: list[Case], answer: Answerer) -> float:
"""Share of cases where the top tag matches the hand-tagged one."""
hits = 0
for case in cases:
result = answer(case.disclosure)
hits += result.tags[0] == case.expected_tag
return hits / len(cases)Check it
Run the evaluation script. The number should match the one at the end of this lesson, or you'll know exactly which step to look at.
Checkpoint
This lesson ends on this commit. Compare your work with it:
8b04e7deval: add 20 hand-tagged disclosures
$ git diff 8b04e7d