Build the Test Before the Claim

From the WALDHORN.AI Build Archive
Build date: May 20, 2026

By May 20, I was already working on both sides of the contract problem. One part of the system could take an existing agreement and analyze what was inside it. Another could take structured deal terms and turn them into a new recording agreement.

May 20 was also my birthday.

I spent part of it looking at the product and deciding I wanted to make a fairly fundamental change in how I was building it. I was less interested in adding another visible feature that day. I wanted to know whether the things already there could survive being pushed in directions I had not conveniently chosen for them.

The generation side was the obvious place to start. It had an uncomfortable property: it was very easy to produce a contract that looked correct. Proving that the system behind it was correct was much harder.

Give the system a sensible set of inputs, generate a contract, read fifteen or twenty pages of formal language, and the result can feel convincing almost immediately. Defined terms line up. Sections have legal-sounding names. Numbers appear where numbers should appear.

That is not much of a test.

So my birthday present to the product, apparently, was trying to break it.

A field can be valid while the deal is not

Basic form validation catches simple failures. An email address can be malformed. A percentage can fall outside its allowed range. A required field can be empty.

A contract has a different class of problem. Two perfectly valid answers can become incompatible when they appear in the same deal.

A rights structure might assign copyright outright while another selection assumes those same rights return automatically at the end of a license term. A financial choice can make another economic option nonsensical. A territory decision can conflict with the distribution scope selected elsewhere.

Each field is valid. The combination is not.

I did not want an AI model deciding these cases every time someone filled out the form. On May 20, I added a deterministic conflict layer to the agreement builder. The first version contained ten conflict rules covering four groups of relationships in the deal.

The system could block incompatible choices, reset a field when another selection made its old value impossible, and explain the conflict before generation. There was no AI call involved.

That was intentional.

If the application already knows that A and B cannot coexist, asking a language model to notice the contradiction again is worse engineering. It introduces probability into something the software can know.

At the time, I did not have a grand philosophy around this. I was trying to stop a contract form from contradicting itself.

The principle stayed.

Then the protection system started failing

The first implementation exposed problems almost immediately. I had explanations attached to blocked options, but some depended on hover behavior.

That sounded reasonable until I used it. A disabled control is exactly where someone needs an explanation. Hiding that explanation behind an interaction the user may never try was a bad trade.

So I changed it. The reason for a blocked option became visible directly beneath it.

Then there was a timer bug. Conflict notices could appear after the system automatically changed a field. Dismissing one notice could interfere with the timers controlling another.

That got fixed.

The conflict logic was also watching far more form state than it actually needed. The form already had a large number of fields, and changes unrelated to conflict detection could still trigger recalculation. I narrowed it to the fields that mattered.

Then I found a smaller bug that bothered me more.

The conflict system ran when the form first loaded. Sometimes it normalized a value immediately so the initial state would be internally valid. Technically, that was correct.

But the interface could then display a notice explaining that a setting had changed. The user had not changed anything. The machine had.

That distinction matters.

Software that explains consequential decisions should not describe its own behavior as though the person caused it. So the initial normalization stayed. The first-run notification did not.

None of these changes made the product more impressive in a demo. They made its behavior more truthful.

Four contracts that were never meant for customers

A few hours later, I added a template picker to the builder. It contained four recording deals. They were explicitly marked in the code as testing aids. They were supposed to come out before production.

The first was a major-label-style exclusive deal. It included a copyright assignment, a $250,000 advance, a seven-year term, label-funded recording costs, and New York arbitration.

The second moved in a very different direction: an indie licensing structure with a $5,000 advance, a 60 percent royalty, a three-year term, and much stronger artist approvals.

The third was a development deal with multiple option periods, a $25,000 advance, and a key-person provision.

The fourth was deliberately artist-favorable. It used a 40 percent royalty, broader creative and sync approval rights, and no cross-collateralization.

I did not want four slight variations of the same agreement. That would have defeated the point. I wanted distance between them.

One should push ownership logic. Another should push licensing. Another should stress options. Another should force the economics and approval structure into a substantially different shape.

Before those scenarios went into the picker, I checked them against the conflict rules themselves. Then I could load the same scenario repeatedly, change one thing, and see what broke.

That was much more useful than generating another beautiful contract from scratch.

A good-looking contract can hide a bad system

There is a psychological problem in generative software that I do not think gets enough attention.

Output quality is persuasive.

If a system returns seventeen pages of polished legal prose, the sheer amount of coherent text creates an impression that substantial work has happened underneath. Sometimes it has. Sometimes the system has simply become very good at looking finished.

That distinction bothered me with contracts in particular.

If I personally filled in the generation form each time, I already knew roughly what the inputs were supposed to mean. I could unconsciously avoid strange combinations. I could read an ambiguous clause and supply missing context in my own head. I knew what the system intended to say.

A user would not have that advantage.

Fixed scenarios removed some of my own influence from the process.

Load the major-label case. Generate it. Change the rights grant. Generate it again. Change the term. Switch an approval. Turn on something that should no longer be available.

Now compare.

Did the related clauses change? Did an old assumption survive? Did the interface block something it should have allowed? Did it allow something that no longer made sense? Did the document actually reflect the deal selected in the form?

Those questions are less glamorous than asking whether the contract looks professional. They are also harder to fake.

I did not want AI to become an excuse

AI makes some defects unusually easy to rationalize. Generated language varies. Two clauses can express the same concept differently. One run can sound better than another.

That variability is real. It can also become a convenient place to hide ordinary software failures.

If the form allows two states the application already knows are incompatible, that is not a model problem. If changing ownership leaves an impossible reversion setting behind, that is not creative variation. If a user is told that they caused a state change they never made, that is not an AI hallucination.

Those are product bugs.

I wanted a line between the things that required interpretation and the things the software could know exactly. The conflict layer belonged on one side of that line. Generation belonged on the other. Testing had to cross both.

I would eventually apply the same distinction to much more than contract drafting. At the time it was only beginning to become visible.

The test should be able to embarrass you

There is a bad way to build a test suite for your own product. You choose examples that resemble the cases you already used while building it. You run them. They pass. The number looks excellent.

Then you start believing the number.

My four test deals were nowhere near a serious benchmark. They were still constructed by me, inside the same mental model I was trying to test.

That is a limitation.

But they introduced an important habit: I wanted repeatable cases that survived the code that produced them.

After a prompt changed, run them again. After a form field changed, run them again. After conflict logic changed, run them again.

A product can improve globally and regress locally. Without fixed cases, that regression is easy to miss.

Human memory is especially bad at this. I remember the contract that looked excellent. I am less likely to remember the obscure setting that stopped working three changes ago.

A fixed test case does not have that bias. It asks the same question again.

There is a cost to doing this early

For a small company, this kind of work can feel almost economically irrational. A customer cannot see a conflict map. Nobody pays more because I fixed the lifecycle of an eight-second notification. A QA template marked "remove before production" is difficult to put on a landing page.

A new visible feature would have been easier to explain. I could have spent May 20 building one.

Instead, I spent a surprising amount of my birthday fixing disabled controls, notification behavior, conflict logic, and four fake recording deals nobody was ever supposed to buy.

None of that made the product more impressive in a screenshot. It made the next claim about it harder to make casually.

That is the trade.

The work that makes a system more trustworthy often produces less visible progress than the work that makes a demo more impressive. I was willing to give up some of the second for the first.

Not indefinitely. A company still has to ship.

But contract software has a particular failure mode that changes the calculation for me.

An obvious crash is embarrassing. A confident contract containing the wrong deal is worse.

The crash tells the user something failed. The document may not.

That is why the useful thing I built on May 20 was not really a template picker. It was a way to stop treating one successful output as evidence that the system worked.