Study · 8 October 2026
The same question, three ways
One AI model asked 39 regulatory questions, 36 of them with an answer in the law: from memory, with web search, and with MetLex. What each got right, and how long it took.
Joakim Platbarzdis, Mettle AB
| Model alone | With web search | With MetLex | |
|---|---|---|---|
| Right article and paragraph | 26 of 36 | 36 of 36 | 36 of 36 |
| Link to the official source | 31 of 36 | 32 of 36 | 36 of 36 |
| Time to answer, median | 16.8 s | 48.2 s | 15.0 s |
| Text read per question, median | 240 tokens | 58,385 tokens | 6,619 tokens |
What it found
- 36 of 36 right with MetLex and with web search; 26 of 36 from memory, most misses in the technical standards, the Swedish rules and the Q&As.
- Web search got there by reading: about nine times the text of MetLex, and about three times the time.
- When the law had no answer (3 questions), all three ways said so.
- From memory, the quotations drifted: asked about DORA Article 11(6)(a), the model wrote “policy” where the text says “plans”.
- One example of the reading: to quote one paragraph of FFFS 2014:4, web search read 208,165 tokens in 92 seconds; with MetLex, 8,302 tokens in 13 seconds.
How it was measured
- The three ways
- One model, Claude Opus 5.5 at default settings, answered each question three times: with no tools; with web search and fetch, up to eight of each per question; and with MetLex connected as a tool over MCP (the Model Context Protocol), which returns verbatim passages with citation, link and snapshot date.
- The questions
- 31 working questions of the kind a compliance officer asks (DORA and its technical standards, GDPR, FFFS in Swedish, EBA Q&As) and 8 known traps from an earlier Mettle paper, where an answer from memory goes wrong. 36 have an answer in the law; 3 do not.
- The grading
- Automatic, against the official text as built into MetLex from EUR-Lex, fi.se and the ESA Q&A pages: act, article and paragraph must match, and the link must be official. Each failure was read by hand.
Limits
- Each question was asked once per way, with one model. Another run or another model will give other figures.
- Joakim wrote the questions so that every expected answer is in the MetLex corpus. That favours MetLex; a question outside it would favour web search.
- The working questions and the traps ran separately, and a correction to the GDPR text reached MetLex in between, so in the first run its one answer on that passage quoted the uncorrected text.
- Mettle sells MetLex. The questions, the answers, the grader and the raw results are kept with the run, so that it can be repeated and checked.
This page is a summary. The full paper, with every question and result, is being revised; ask for a copy. Ask for the paper
Get in touch
Two lines are enough. You will have an answer by email within two working days.
hello at mettle.se