Lesson 018 min
Why the evaluation set comes first
Goal: Explain why a RAG system without an evaluation set can't be improved on purpose.
Why this step
Placeholder body for the interface. Each lesson opens with the problem this step solves and the decision behind it, so the code that follows has a reason. The real text arrives with the track.
Build it
The code for this step is small on purpose. Write it yourself first, then compare it with the reference commit below.
def score(cases: list[Case], answer: Answerer) -> float:
"""Share of cases where the top tag matches the hand-tagged one."""
hits = 0
for case in cases:
result = answer(case.disclosure)
hits += result.tags[0] == case.expected_tag
return hits / len(cases)Check it
Run the evaluation script. The number should match the one at the end of this lesson, or you'll know exactly which step to look at.
Checkpoint
This lesson ends on this commit. Compare your work with it:
3f9a1c2docs: describe the task and how it's scored
$ git diff 3f9a1c2