Harvey is training its own AI model

Harvey Builds Its Own

On 18 June, Harvey’s co-founder Gabe Pereyra put out a post on X, announcing they are training the first in a series of their own legal LLMs. The same day Artificial Lawyer published a confirmation from Harvey’s CEO, Winston Weinberg.

I’ll repeat. Just to avoid the confusion. Harvey is building an AI model they post-train themselves rather than another layer riding on top of Claude or GPT, with the stated aim of letting each firm eventually “own their own intelligence”. Weinberg added they are already running studies with law firms to encode how a firm works on its hardest matters, down to how that work is done for specific, longstanding clients.

Harvey’s bet is that a firm’s own way of working is the durable asset, and that turning it into a model no competitor can license is what makes it defensible. That is the same institutional moat Kirkland put $500M behind, now showing up as a vendor’s product strategy rather than one firm’s balance sheet.

I see two practical reasons behind this:

1. An open source and open weight model can run inside your own walls, with full visibility into its reasoning and tool calls - the auditability a hosted API can’t give you.

2. There is real headroom in training for the work: on its own Legal Agent Benchmark, even the best frontier models complete under 10% of tasks end to end and that’s the gap Harvey is hoping to close with post-training the open source models.

Not For Every Task

Harvey is not trading Claude and GPT for their model. They are working with an outside research partner, they trained a 27B parameter open source model on legal-agent work until it performed at frontier level on their own benchmark, and then built the system around it to behave like a senior associate. The trained model does the bulk of the routine reading and structuring, and hands the genuinely hard steps up to a frontier model or a human.

This means that frontier LLMs are not getting fully replaced, they will be working hand in hand with the open source LLMs.

There are two immediate benefits to this, cost and speed. Running at frontier level costs roughly $50 and twenty minutes per task - affordable once, ruinous across every step of a long matter. So the split follows the work: reading through a data room or pulling the structure out of a contract is routine, and a cheaper model trained for it can carry that load. Where the frontier intelligence still earns its cost is for complex tasks, such as judgment on how a particular term cuts. Harvey’s geniuenly smart approach here is matching the model to the task, escalating where it counts. Harvey calls the result “base camp” - early, and still being proven out.

What Makes Sense for You

Until recently, running an open-source model behind your everyday tools needed an engineering team. That changed in April, when Anthropic added a third-party inference option to Claude Desktop, the installed application. It does not appear out of the box: you switch it on in developer mode, and availability still varies by plan. Once it is on, you can point the app’s inference at an LLM other than Claude, including an open source one you host yourself. The components now come ready made, so you can get a long way without hundreds of engineering hours.

Here is how it works. You setup and run the open source model on your own cloud, such as Amazon Bedrock or Google Vertex (and later this year on Microsoft Azure), and you put a small translation layer, a gateway, in front of it so Claude Desktop and the model can talk. From then on, your model is one of the options you choose inside the app, where in the same place you choose other models. Because the whole chain sits in your own cloud, the firm’s data never leaves it, which is what makes this perfect for client work.

Now there are a couple of gateways out there.

OpenRouter is the fast option, connecting almost any model in minutes. It can be set up for client work: on its enterprise tier it will sign a data processing agreement, you can force it to route only to providers that don’t retain or train on your prompts, and you can keep processing inside the EU. By default it logs no prompt content, only metadata. What it won’t give you is a HIPAA agreement, so health related matters are out of the scope, and the data processing terms sit behind the enterprise tier, not the quick self-serve version you can subsrcibe to online. Either way, you are trusting and configuring a second vendor, plus the providers it routes to on your behalf.

LiteLLM is another option. It’s open-source and you run it inside your own cloud (Bedrock, Vertex, or Azure), so the request travels from Claude Desktop to your model and never leaves the boundary your firm has already cleared. Nothing extra to vet, and residency is yours by default. The trade is that you operate it yourself instead of letting someone host it for you.

Before You Fine Tune Your Model

Fine tuning a model, or post-training it, to borrow the jargon, is the part of the Harvey story that does not transfer down to a boutique, and won’t for a while. The reason is simple: fine tuning runs on high quality, structured datasets, and building those is a lot of work which, for most boutique law firms, isn’t realistic to be done as a separate project. It just takes too long to see the payoff.

But you don’t need that. The leverage Harvey gets from matching the model to the task, you can get another way - by building your portable operational layer: the skills, plugins, instructions, and workflows that hold how your firm works. Once that layer is in place, you can add a classification layer and point it at whatever model fits the task, being it a frontier or an open source LLM.

Because that layer lives in files, it travels with you, not locked to one model, one vendor, or a stack that will look different in three months. This is the heart of the Boutique AI Climb.

Legal AI in Action

🎤 Notes From LegalTechTalk 2026 in London

I spent the last week at LegalTechTalk 2026 in London, in person. The panels reaffirmed the lines we already know - AI augments lawyers rather than replacing them, data is the moat, transformation is a people problem more than a tech one, and the perennial “is the billable hour dead”. All true, and all settled years ago. What lawyers seem to be hungry for now is the HOW: the execution, and the real success and failure stories of embedding AI into legal work and redesigning the work itself around it.

While I was there I recorded a series of interviews with the people highly involved in legal AI.

🎙 The first one is live.

I sat down with Niels de Jong and Elgar Weijtmans of HVG Law for the “geek version” - their read on what Harvey’s announcement actually means: token economics more than a gift to customers, why Harvey tried fine-tuning early and walked away from it, and the Trojan-horse worry of a vendor that could one day compete with the very firms it serves.

A new interview drops every few days - watch on YouTube or follow the full series on my site.

🎬 Teach Claude how Your Law Firm Works

The five places your firm’s knowledge actually lives in Claude, from organization and user instructions through projects, skills, and plugins, the trick that gets Claude to write each one for you instead of you writing it cold, and how memory runs underneath it all so you teach Claude your firm once instead of starting over every chat.

🎙 This Lawyer Encoded 3 Decades and 2TB of his Legal Judgment

Oscar Octavio Hijonosa Guerra has practised law in Mexico for 31 years and for 21 of these runs HAA Legal, a wealth-structuring boutique now listed in the AI Firm Index. A fear that AI would replace lawyers pushed him to rebuild it around 19 agents, without writing a line of code. We talked through how he encoded three decades of his own judgment, why the accumulated criterion matters more than his 2 terabytes of stored files, the agent built to tear every draft apart before it reaches him, and how he stays the final gate on everything that goes out.

Coming up!

🎙 Next Tuesday at 2pm CET!

Next week’s guests on Rok’s Legal AI Conversations is George Hannah, a solicitor apprentice at a London law firm who started straight out of school, at the age of 18. Alongside the day job he writes Best Practice, a legal AI newsletter, and covers the industry across his podcast and LinkedIn.

We discuss the junior lawyer paradox from the inside - how you build judgment when the grunt work is the first thing automated away, his idea that legal AI tools should know your seniority and coach juniors instead of handing over finished work, and why, even with the AI providers now coming for legal, a law firm still isn’t a tech company.

Each edition of Legal AI Brief brings practical lessons from firms using AI safely.