The Risks of DIY AI: Why CLO Documents Need a Pro

9 Jul

We live in a golden age of do-it-yourself. Home Depot made it a movement in the 1980s. YouTube videos perfected it. Today, there's a tutorial for everything: tiling a bathroom, rewiring a panel, laying crown molding. And every weekend, thousands of confident homeowners discover the same hard truth: the catalog photo and the finished result are two very different things. Some projects (plumbing, electrical, fine finish work) look simple until you're in the wall. Then they're not. This is why you don't fix your own car's transmission. You get a pro.

The same lesson is now arriving on Wall Street, dressed in the language of tokens.

The Tokenmaxxing Hangover

Shopify told staff that proving work "cannot be done using AI" was a precondition for new headcount. NVIDIA's Jensen Huang said he'd be "deeply alarmed" if a $500,000 engineer wasn't burning $250,000 of tokens a year. Amazon and Meta ran internal leaderboards ranking employees by consumption. The era had a name: tokenmaxxing.

Then the bills came due. Uber burned through its projected annual AI budget in four months. Meta warned staff of an "exponential increase" in internal costs. Microsoft started canceling coding assistant licenses after finding engineers spending up to $2,000 a month each. The new word, per The New York Times, is tokenminning: token-minimizing. Meta's own CTO had to send a memo: "Token usage alone is not a measure of impact of any kind."

Enterprise generative AI spend tripled to $37 billion in 2025, per Menlo Ventures. MIT researchers found roughly 95% of corporate AI pilots showed no measurable P&L impact. The bills got big. The results didn't follow.

Why CLO Docs Break the DIY Model

If you've asked a general-purpose LLM about a CLO indenture, you've noticed: the answers drift, and the costs don't.

Here's the mechanical problem. A CLO indenture runs 400 pages, sometimes 800 with amendments: anywhere from 250,000 to 650,000 tokens. Loading it once into a frontier model costs a dollar or two. Trivial, until you watch how a desk actually works. An analyst pastes the document, asks a question, and gets an answer. A colleague opens a new session the next day because chat sessions don't share memory and reprocesses the same document to ask the same questions. Twenty questions on one indenture: tens of dollars. One hundred deals, a team of analysts, and you've recreated, at desk scale, how Uber spent an annual budget in four months.

Wider context windows make this worse, not better. Yes, you can now feed an LLM four or five CLO documents at once. But a larger input means a less precise answer: the model has more ground to cover, and accuracy drops for anything buried deep in the document. There's a name for this in the research literature: "lost in the middle." Indentures, unhelpfully, bury their definitions exactly there.

And cost is not even the main problem.

Stanford researchers found that general-purpose LLMs hallucinate on direct, verifiable legal questions between 58% and 88% of the time. Purpose-built legal AI tools fared better, and still produced incorrect information 17% to 34% of the time. In most domains, a confidently wrong answer costs you an awkward meeting. In structured credit, a misread par flush condition feeds a cash flow model, a trade, or a portfolio decision. As one structured credit desk head told us: "We spend more time checking the AI's work than doing the analysis ourselves."

The per-token price is falling fast, roughly 10x a year for equivalent capability. But cheaper tokens produce more usage, which is why spending tripled in the year prices fell. And free tokens don't fix hallucination, don't fix lost-in-the-middle, and don't touch the verification burden, which was always the real cost.

The Sharpe Ratio of AI

In investing, raw return is the wrong metric. The Sharpe ratio asks: how much return are you getting per unit of risk? A strategy that earns 15% with wild volatility may be far less valuable than one earning 10% with consistency you can rely on.

The same logic applies to AI in document-intensive work. Token spend is not ROI. The right measure is accuracy per dollar: output you can actually trust, per unit of cost. At Semeris, we run continuous benchmarks across the LLMs we deploy, matching each task to the most cost-efficient model. The goal isn't maximum token consumption. It's the maximum signal per dollar, with verified accuracy, on which you can build decisions. That's the Sharpe ratio of AI deployment, and most DIY LLM workflows score poorly on it.

In Structured Finance, Depth Beats Breadth. Every Time.

The fix is structural. Read the document once, properly. Extract what matters into hundreds of structured data points per deal. Have expert analysts verify the extraction, so the checking happens once, by people who know what "par flush" means. Then let your team query the data and compare it across deals, instead of re-processing the same document through a probabilistic keyhole a thousand times.

No repeated context, so the token economics collapse. Analyst-verified data, so the trust problem goes with it.

That's what we built Semeris Docs around. Fifty-plus firms in the CLO market have already made the same call.

The tokenmaxxing era asked how much AI you could use. The right question was always what you could rely on.

If you were told to adopt AI, then told to justify the bill, and you don't entirely trust the answers in between, we're happy to talk. Some jobs need a pro.

For more on why shallow AI doesn't work in structured finance, see our post The Troubles of Shallow AI.