I rebuilt my Hebrew PII model in a day and a half with an agent. Four questions did the steering.
Between the evening of November 12 and the afternoon of November 13, GolemPII went from v1 to v2. That’s my Hebrew PII detection model. Over five working sessions, Claude Code wrote the data pipeline, ran about 35 training jobs on Hugging Face, scored every one of them and kept a running handover file that passed 1,300 lines.
The model went from 0.709 to 0.917 F1 on the held-out half of an external Hebrew benchmark. The whole thing cost about $42.
I didn’t write the training code. I did set the problem. v1 and its training set were mine, from 2024. I chose the target (enterprise text that mixes Hebrew and English), the 15 labels, and the rule that the external benchmark is split once and its held-out half opened once. After that I read results, approved spending and made calls. Looking back at the handover, four of my questions changed the direction of the work more than any single experiment did. This post is about those four questions, and about what an agent needs from you when it’s running an ML project.
TL;DR
- I built GolemPII v2 with Claude Code over five sessions: 3,316 LLM-written templates, about 35 GPU jobs, 226 tests, more than 30 logged decisions. Total cost about $42.
- The agent was very good at discipline: reading rules fixed before results, a held-out test opened exactly once, every failure written down.
- It was bad at stepping back. It kept polishing a benchmark score until I asked whether we were overfitting. Once it measured, a whole round of “gains” turned out to be span edges.
- The calls that changed the outcome were mine: four questions and a handful of policy decisions.
How do you run an ML project across agent sessions?
You write everything down. The handover file was the project’s memory. Every session started by reading it and ended by updating it. It had status markers (DONE, IN PROGRESS, NOT STARTED), a “results NOT READ” flag for jobs still running, and the exact commands to read each pending job.
It also had a decision log: more than 30 entries, each with a reason. Some were mine (“15 labels”, “enterprise mixed Hebrew-English text”). Some were the agent’s, taken after I said “do what you think is best”, and those were marked that way. When a decision was reversed, the old one stayed with a note pointing at the new one.
And it had a section called “Where we failed or were wrong”. By the end it had 28 items.
Generation ran as Gemini batches. Training ran on Hugging Face Jobs on A10G GPUs, about five times faster than my Mac and roughly $0.40 for a 24-minute run. Jobs could run in parallel while the agent did something else. Credentials stayed in the macOS keychain and were loaded inside each command, never printed.
I wasn’t watching it work. I’d open a session, read the handover, answer whatever was waiting for me and leave it running. More than once my answer was “do what you think is best”, and the agent logged the call as its own so I could find it later. Once I came back to eight cancelled jobs because the credits had run out.
Question 1: “What is REDACT?”
REDACT is an external PII benchmark. Its 621 Hebrew records are enterprise text: logs, JSON, tickets, chat and emails, 69% of them mixing in English. It was the reason for the whole project. v1 had scored 0.998 on its own synthetic test split. On the half of REDACT we used for decisions it scored 0.731.
The answer that stuck was “an exam written by someone else”. v1 had only ever taken exams I wrote. Its failure was over-tagging: 69% of its phone spans landed on timestamps, hashes and session ids, because every number in its training data had been PII.
That answer is why I set the first rule of the project, and the agent held it more strictly than I would have. REDACT was never trained on. It was split once into a select half for every decision and a final half that would be scored once, for the shipped model, at the very end. Before final was opened, the agent wrote down which seed would ship and by what rule. After it was opened, nothing could be retrained or re-chosen.
Question 2: “Are we overfitting to REDACT’s quirks?”
This is the one I’m glad I asked. The pilot model (same encoder as v1, 916 LLM-written templates) scored 0.916 on the select half. The agent read the errors and found many of them at span edges: REDACT puts the word “street” inside address spans and labels “Visa ending 4821” whole. It changed the fill script to match. Round 1 scored 0.925. Round 2 was built the same way.
From inside a session that looks like steady progress: errors read, fixes made, score up, tests passing. Nothing in the loop asks whether the score measures the right thing.
When I asked, the agent’s first answer was an argument. Its second was a measurement. It added a score that ignores span boundaries: an entity counts as found if any span of its category touches it.
| Model | Character F1 | Entity F1 |
|---|---|---|
| v1 | 0.731 | 0.738 |
| v2 pilot | 0.916 | 0.929 |
| v2 after fill fixes, round 1 | 0.925 | 0.928 |
| v2 after fill fixes, round 2 | 0.925 | 0.927 |
The fill fixes were worth nothing at entity level. They were the model learning how this particular benchmark draws its boxes. The way we explained it to each other: a kid with a highlighter, graded letter by letter, can raise the grade by finding more, or by drawing the edges the way this teacher likes.
Then the agent wrote a rule into the decision log: every result is reported on both scores from now on, and no more changes whose only effect is a span edge. It held to that for the rest of the project, including when it meant switching off a fill change that had looked good.
Question 3: “Where do the ID numbers come from?”
My assumption was that LLMs won’t generate realistic Israeli ID numbers, so the training data must be thin on them. I’d suggested pulling them from v1’s old dataset, which has plenty.
The assumption was wrong, and it still pointed at the real gap. The LLM never writes a number in this pipeline. It writes documents with placeholders, a validator rejects any digit outside one, and a script fills in valid IDs. So the agent ran a probe. After the Hebrew abbreviation for “ID number” the model found 6 of 6 IDs. After NATIONAL_ID= in a log line it found 0 of 6. A valid checksum made no difference.
The gap was the text around the number, the part the LLM writes. And our own prompt had a rule telling it to always name the document in Hebrew words, so it never wrote English keys. A batch of 349 templates with English keys, bare chat replies and English phrases cost $0.60 and took the probe to 6 of 6 after the English keys.
Question 4: “What does it do on real text?”
By then v2 was ahead on REDACT. I wanted to know how it did on Hebrew it had never seen in any form, so I asked whether the plan had any test besides REDACT and whether NEMO could be one. We settled on NEMO and IAHLT, two human-labeled news and encyclopedia corpora with person names marked, kept strictly test-only.
On those, a newer open Hebrew PII model from another team beat every v2 model on names. v2 scored 0.820 person F1 on NEMO, and the gap was precision. v2 tagged sports clubs, companies and ordinary words as people.
That reversed an earlier call. Swapping names into real IAHLT sentences had been planned and then deferred, because REDACT showed person F1 already level. Real text said otherwise, so I brought it back. Twenty thousand real sentences with swapped names took NEMO person F1 to 0.873 and 0.879 on two seeds, and REDACT went up with it. After the final tuning round it ended at 0.887, with precision at 0.853. That put v2 ahead of the other model on recall and still behind it on precision.
Where did the agent get it wrong?
It miscounted. Early on the agent reported bank names in 307 templates and countries in 128, so the model should have learned they aren’t cities. The search had matched the Hebrew words for “National Insurance” and “international”, which contain the name of a major Israeli bank. The real figures were about 5 and 9.
Before anyone caught that, it had built a fill-time step to vary countries in the text. That step produced 54 country mentions in 5,496 examples and one nonsense phrase, and was removed. The real fix was a new template batch with country, bank and company slots, which turned out to be the only change after the pilot that moved entity F1.
The pattern I’d watch for: an agent will build confidently on a number it produced itself. It’ll check a result against a rule. It’s much less likely to check the number the rule was applied to.
What did the agent need from me?

It needed questions that change direction, like the four above, and policy calls that no score can make. Is a customer reference number personal data? We said no; other models say yes, and that one choice explains a lot of the precision difference on REDACT. Which encoder ships? The plan’s own selection rule would have picked a smaller Hebrew encoder. I overrode it, because the 0.012 it gave up was the size of the whole lead. Should the model cards compare against another team’s model head to head? I decided they shouldn’t. Which names to publish under, and when.
And permission. Nothing went public without my explicit go-ahead. Every paid batch was approved with an estimate first. The agent’s estimates were close: $4.14 estimated against $4.20 billed for the big batch.
What it gave me back was discipline I don’t usually have at 1 a.m.: reading rules fixed before the results came in, “a difference smaller than the seed spread is not a result”, two seeds before any claim, and a list of its own failures kept up to date.
What did it cost?
| Item | Cost |
|---|---|
| Gemini 3.8 Flash through Google’s Batch API (3,316 templates, plus checks and dry runs) | $7.65 |
| Generator bake-off (four models compared on OpenRouter) | $1.70 |
| Hugging Face Jobs, A10G time (about 35 runs, plus 8 cancelled when credits ran out) | $23.70 |
| Hugging Face Pro, one month (needed for Jobs) | $9.00 |
| Total | about $42 |
The result: GolemPII-v2-base scores 0.917 character F1 and 0.924 entity F1 on the held-out half of REDACT, against 0.709 and 0.725 for v1. GolemPII-v2-small (125M) scores 0.916 and is the best of the three at telling names from organisations in Hebrew prose. GolemPII-v2-tiny (45M) runs about five times faster.
Can I try it?
Yes. There’s a demo that runs the three models side by side on the same text. Or load it with transformers:
from transformers import pipeline
detector = pipeline(
"token-classification",
model="CordwainerSmith/GolemPII-v2-base",
aggregation_strategy="simple",
stride=128,
)
detector("הלקוח יואב בן דוד (ת.ז. 034567891) התקשר הבוקר. order_id=ORD-2024-88127")
This is what it does to a support email. The names, the ID, the address, the phone, the card and the IBAN are replaced. The order id and the amount are left alone, which is the thing v1 couldn’t do:
הלקוח [FIRST_NAME] [LAST_NAME] (ת.ז. [ID_NUM]) התקשר הבוקר וביקש לעדכן את הכתובת למשלוח: [STREET], [CITY], מיקוד [POSTAL_CODE].
אפשר להשיג אותו ב-[PHONE_NUM] או במייל [EMAIL].
החיוב החודשי ירד מכרטיס [CC_PROVIDER] [CC_NUM], והזיכוי יועבר לחשבון [IBAN].
order_id=ORD-2024-88127, סכום: 349.90 ש"ח, תאריך לידה לאימות: [DATE].
Which one to use:
| Model | Size | Use it for |
|---|---|---|
| GolemPII-v2-base | 277M | Hebrew mixed with English in emails, tickets, logs and JSON. Best of the three on ID, card and bank numbers. |
| GolemPII-v2-small | 125M | Hebrew prose. Best of the three at telling people from organisations. |
| GolemPII-v2-tiny | 45M | When speed or size matters most. About five times faster, less accurate on enterprise text. |
It’s a detector that helps with redaction. It doesn’t guarantee anonymisation, so text that has to be anonymised still needs a human to review it.
The base and small models are MIT and tiny is CC-BY-4.0. If you use one in a product, a paper or a fine-tune, credit it as GolemPII v2 by Liran Baba with a link to the model. Each model card has a BibTeX entry.
Frequently asked questions
Can Claude Code run a machine learning project end to end?
It ran this one: data generation, 226 tests, about 35 GPU training jobs on Hugging Face, scoring and model cards, over five sessions. It needed a running handover file to carry state between sessions, and a human to ask whether the metric still measured the right thing.
How do you keep an agent from overfitting to a benchmark?
Split the benchmark once and open the held-out half exactly once. Fix the reading rule before results arrive. Report a second metric that ignores the benchmark’s conventions. In this project a boundary-free score showed one round of fixes worth 0.009 at character level was worth zero.
How much does it cost to train a Hebrew NER model on Hugging Face Jobs?
A 24-minute run of xlm-roberta-base on an A10G cost about $0.40. The final recipe, 38,000 training examples for 3 epochs, took about 70 minutes, roughly $1.17. All compute for this project, about 35 runs, came to about $23.70, plus $9 a month for HF Pro.
What I’d do the same way
The handover file is the part I’d copy into any long agent project. It helped me as much as the agent: half of what’s in this post came from rereading it, including the mistakes, which were written down in the same flat tone as the wins.
The other part is harder to copy. An agent will run a tight loop for hours, and every step in it will look right. Somebody still has to stop and ask what the loop is optimizing.
Would I run the next model this way? Yeah, sure. What I’d change is the order. The second benchmark and the boundary-free score would be there on day one, before the first round of fixes, not after I got suspicious. And I’d collect real failure cases from my own work before starting. I never did, so numeric PII in enterprise text was only ever tested on REDACT.