Lesson 0810 min
Refuse when the sources don't say
Goal: Teach the system to say it doesn't know, and score that too.
Why this step
Placeholder body for the interface. Each lesson opens with the problem this step solves and the decision behind it, so the code that follows has a reason. The real text arrives with the track.
Build it
The code for this step is small on purpose. Write it yourself first, then compare it with the reference commit below.
def score(cases: list[Case], answer: Answerer) -> float:
"""Share of cases where the top tag matches the hand-tagged one."""
hits = 0
for case in cases:
result = answer(case.disclosure)
hits += result.tags[0] == case.expected_tag
return hits / len(cases)Check it
Run the evaluation script. The number should match the one at the end of this lesson, or you'll know exactly which step to look at.
Checkpoint
This lesson ends on this commit. Compare your work with it:
b83e5f1answer: refuse below the confidence floor
$ git diff b83e5f1