Ten Models, Four Weeks
The big AI labs shipped ten new models in just the last four weeks.
Claude Sonnet 5 went first, then Grok 4.5, OpenAI’s whole GPT-5.6 family and Meta’s Muse Spark 1.1 within two days of each other. Then two very large open-weight models, Inkling from Thinking Machines and Kimi K3 from Moonshot. Google added more Geminis on the 21st, and DeepSeek V4 and Claude Opus 5 both arrived on the 24th. Each one came with a headline saying it beat the last one. And if you’ve caught yourself wondering whether your firm picked the wrong one, you’re not the only one.
You probably didn’t. Because whichever model your firm settled on, the firm down the street can subscribe to the same one tomorrow at the same price. That part everybody already has, and it keeps getting better for everybody at the same time. So the difference between your firm and theirs was never going to come from the model.
And the legal benchmarks say the same thing, in numbers.
Two Points Apart
Three legal benchmarks put out fresh numbers this month, and Vals AI publishes all three of them. Start with LegalBench, the academic test of legal reasoning that Stanford’s CodeX helped build. The top five models there sit within two points of each other, Claude Fable 5 at 88.56% and three Geminis and a GPT behind it, the last of them at 86.86%. Under two points across three different companies is essentially a rounding error.
Vals also runs a harder one of its own, Legal Research Bench, where agents do multi-step research with real tools instead of just answering questions. Three weeks ago GPT-5.6 Sol was on top of it. By the 21st Claude Fable 5 had taken over. Claude Opus 5 came out on the 24th and looks to have taken it again. That’s three different leaders on one board inside a month.
Then there’s Harvey’s Legal Agent Benchmark, which asks a model to produce complete legal work product. Artificial Analysis runs this benchmark and puts Kimi K3 on top at 26.7%. Vals ran the same benchmark on the 22nd and got Meta’s Muse Spark 1.1 at 20%, with Kimi K3 down in fourth at 10.83%. Two independent scorers, running the same tasks against the same rubric, disagree about who won.
Put the three boards next to each other and they say the same thing. The top is crowded, the order changes every few weeks, and whichever model is winning today, the firm down the street can be using it by this afternoon.
Five Percent, No New Model
Legora published its own benchmark this week, the Legora BAR, and they built it to test the whole system instead of the model on its own. The AI model is only one part of what makes the AI system good, and the rest comes from the harness around it, the tools it can reach, and the setup it runs inside.
They took a set of models, ran them on their platform in June, and ran that same set again in July. The models were identical both times, same versions, yet the output quality improved by 5%. The only thing that changed was the harness around the model.
And that 5% is what Legora got from improving a harness that was already good. Most law firms are not in that position. They have people typing into a chat window one at a time, starting same conversation from scratch every morning, with none of it written down anywhere.
The gap between that and having a harness at all is a lot bigger than 5%.
The Vendors’ Bet
A partner at a mid-sized firm told me he asked a salesperson at one of the big legal AI vendors how many firms adjust the agentic workflows that come with the platform to their own way of working. The salesperson said it was not as many as you would think, and that the firms who do that work get noticeably better results than the out of the box version of their platform. The company’s bet in the market, he said, is that most law firms won’t bother.
So most firms paying for one of these tools are getting less out of it than it can actually do. If a specialized tool only reaches its ceiling once you’ve taught it your firm’s way, and almost nobody does that, then what your firm is paying the premium for is the vendor’s head start - the jurisdiction knowledge, the legal specific setup, the workflows already sitting in the box.
That head start is real, but it only takes you so far. It runs out where your own way of working begins, and past that point the work is yours no matter which tool you bought.
An Hour That Keeps on Paying Back
I wanted to see how close Claude could get to what the big legal AI platforms produce, for a fraction of what those platforms cost.
So I asked a friend to help me test it. Gayk Ayvazyan is a corporate lawyer with twelve years behind him, now legal counsel at EMPIQ in Amsterdam, where negotiating and drafting MSAs is a standing part of the job. For about a year and a half already, Gayk has also been an enthusiastic user of Wordsmith. In fact, he was one of their first users in the Netherlands. He had never opened Claude.
I sent Gayk a master service agreement that was publicly available online and asked him to review it by hand, the way he would for a client. I gave that single review to Claude and asked it to produce two files. The first is a playbook that sets out how Gayk reviews an MSA. The second is a SKILL.md file that teaches Claude how to read that playbook and apply it to other MSAs Gayk has to review.
Now, you might think - you gave Claude only one example? Yes. We intentionally did not do this properly. Doing it properly would mean handing Claude a stack of Gayk’s past reviews, potentially categorized per industry, so it could pick up how he works across many documents and many clients.
We gave it one, because the actual experiment we were running is how long does it take for a lawyer that is not used to working with Claude, to setup a reusable workflow that would give comparable result to what they get with a much more expensive tool.
Gayk and I ran a different MSA through Claude with the new reusable workflow in place. Then we ran the same MSA through Wordsmith. He looked at both, and the verdict was that what we got out of Wordsmith and what we got out of Claude were very comparable.
Setting it all up took about an hour in total.
Without knowing it, Gayk made the same point as the sales rep from one of the biggest legal AI vendors earlier in this newsletter. You have to teach any of these tools, even the specialised legal AI ones, how you work, because none of them does the job your way out of the box.
Your firm does not need a premium legal AI subscription for this. An hour was enough for a lawyer who had never opened Claude to get to a similar result.
So What Tools do Law Firms Need?
Your firm needs four things, and it probably already has three of them. A document management system, a practice management system, and an agentic platform like Claude Cowork connected to both. The fourth is your own way of working, written down in a way AI can read it and apply it to the work you task it with.
Most firms stop after the first three, and that’s what legal AI vendors are counting on. And the fourth is what made the difference in that hour with Gayk, because his positions on the clauses he negotiates every week ended up in a form Claude could actually use. At most firms that knowledge sits in three people’s heads, where no AI can reach it.
And what you write down stays yours. It sits in your firm’s own files rather than inside someone’s platform, so it doesn’t disappear when a subscription ends or when a vendor changes how their product works.
Legal AI in Action
🎬 Wordsmith vs Claude Cowork: A Lawyer Judges Both
Gayk Ayvazyan, a practicing attorney, reviews MSAs for a living and has been a happy Wordsmith user for a year and a half. He had never opened Claude. So we built him a reusable MSA review inside Claude out of a single example of his own work, ran a fresh agreement through both tools, and let him judge the two side by side. You see the playbook and the SKILL.md get written, the run itself, the changes he asked for, and where he landed at the end.
Next time Gayk and I go over how to get Claude to create a good legal skill for you - specifically for lawyers too busy to learn how to write a good one themselves.
🎙 Give Your Firm’s Templates to Your Clients
Sarah Eldon Cullen is a startup and venture capital lawyer at Mace, a boutique firm in Copenhagen, where she leads the firm’s AI strategy. She also sits on the board of Copenhagen Legal Tech community. On top of that, after an LLM due diligence project she got funded stalled the moment a tech team took over, Sarah taught herself backend engineering.
We got into why she would happily make her templates public, since her value sits in the twenty percent that clients actually pay for, and the large firm that charged clients for token usage only to find its partners running up the biggest bills of anyone. She also puts a question to every firm buying its way into this: if you take away all your tools, what are you building towards?
Coming up!
🎙 Next Tuesday at 2pm CET!
Next week’s guests on Rok’s Legal AI Conversations is Kim Mason, who runs Nailed AI in Winchester, getting non-technical leaders and their teams through the first steps of AI. Before that she was the commercial director who took her own company from a few people quietly playing with ChatGPT to paid licences across the business and a roadmap that leadership actually signed off.
We discuss why your IT provider is probably not your AI partner, even though most firms assume they are; why training your team only works when it comes in small doses over months, with someone on hand for when your people get stuck; and why the firms that get somewhere run two things at once, teaching everyone the basics while one real project gets proper backing from the top.
Each edition of Legal AI Brief brings practical lessons from firms using AI safely.