Eval-set inspector

What an eval set is

An eval set is the list of test cases you measure a model or a prompt against, usually a file where each row holds a prompt and the answer you expect. You change something, run the set, and compare the score with last time.

That only works if the set itself is sound. A score is a claim about your system, but it is computed from this file, so every fault in the file becomes a fault in the claim, and it happens silently, because a malformed row does not announce itself. It just quietly stops being counted.

What this tool reports

Rows, and the ones that did not parse

How many rows were read, and a line-numbered list of the ones that were not, each with the reason: not valid JSON, valid JSON but not an object, or the wrong number of columns for a CSV. Line numbers count physical lines in the file, blanks included, so they match what your editor shows you.

Field coverage

Every field name that appears anywhere in the set, how many rows actually have a value for it, and the percentage. A field that is present but empty counts as a hole, not as coverage. {"expected": ""} is a missing answer, not an answer.

Duplicate prompts

Reported twice over: rows that are identical, and rows that are identical once case and spacing are ignored. The second pass is the one that finds what a plain de-duplication missed. "What is 2+2?" and "what is 2+2?" are one test case counted twice.

The check runs against the first of prompt, input, question or query that your set actually uses.

Length

For every field: the shortest value, the longest, and the mean, in characters and in words.

What it catches

• A truncated export. You expected 500 rows and the count says 480. Something dropped twenty and told nobody.
• A field you believed was populated. Coverage says expected is at 62%. More than a third of your set is scoring against nothing.
• Duplicates that skew the number. A prompt appearing three times carries triple weight. If the model happens to be good at that one, the whole set looks better than it is.
• Duplicates that survived de-duplication, because they differ only in capitalisation or a trailing space.
• Length outliers. A longest prompt of 14,000 characters is probably a whole document pasted into one row. A shortest expected answer of 1 character is probably a truncation bug.
• A merge that went wrong. Two sets combined, and coverage now shows two spellings of the same field: expected at 60% and expected_answer at 40%.

When to use it

• Before you trust a number that came out of the set, especially one you are about to report to someone else.
• After merging sets from more than one source.
• After any hand-edit, export or format conversion.
• When a score moves and you cannot explain why. Check whether the set changed before you conclude the model did.
• Before handing a set to someone else to use.

What it does not do

It does not run a model, score answers or judge quality. It reads the file and reports what is in it. Whether a test case is a good test case is your judgement, not something a parser can tell you.

It also reports no token counts. An exact tokeniser is too large to ship on a page that promises to contact nothing, and an estimated token count presented as a number is the kind of confident guess this site avoids. Characters and words are exact, and they are what you get.

Formats and privacy

Reads .jsonl (one JSON object per line), .json and .csv. Give it a file, paste rows straight in, or point it at a URL. Pasted text is identified by its content, so it does not need a filename.

A file or pasted rows are read in your browser and analysed there. Nothing is uploaded, stored or sent anywhere. A URL is fetched by your browser directly from the address you give it, and we never see it. Some servers refuse to be read from another site; if that happens, download the file and choose it instead.