the biggest challenge i've encountered isn't building the factory, it's getting others to trust it enough to let it actually work. when you're solo building, you know the system intimately and can trust the outcomes because you built everything and usually have stress-tested it to a real degree of certainty.
... introduce it to a team and eyebrow raised.
trust is the hardest thing to build still, not just in the software but the people around it.
Really like the "lit" metaphor of humans adding light to dark factories.
Verification is really the bottleneck. As engineers, we were often used to figuring things out as we went. Now the game has changed: we need to figure more of it out upfront, if we can.
Automation must be earned. For me, that's where much of the complexity lies, because there is a discovery problem that needs to be solved before delivery can be automated.
Do you have any insights on "Discovery Factories"? Can discovery itself become a factory, or is ambiguity fundamentally what prevents it from becoming one?
Back pressure reads as a per loop rule: only as much autonomy as you can cheaply verify. But review capacity is one shared pool. Ten loops that each clear the bar can still exceed what a team reads in a week, and the number being tracked is token utilization.
The metaphor of the factory, used at face value, is fundamentally misguided right? A compiler is a factory, while software involves design in tasks of huge variability rather than repeating the same actions over and over. Drawing the car rather than building millions of them.
That said I would be very interested to see a tool that allows prescribing a deterministic workflow to follow. For example, I follow outside-in Test Driven Development as a discipline and currently I hold that loop myself with a fine-grained tool like Cursor. I saw someone try to build that loop with Pi. Still, part of the job is also knowing when to stop and change approaches: you know need a diagram, or a conversation, or production data. These are all tools I have not seen agents pick up.
I built a deterministic verification with no model in the verification path. The task gets a scoped packet before execution: which files are allowed, which are forbidden, what evidence is required, what counts as done. The agent produces output. A deterministic check verifies whether the output satisfies the declared conditions. Same input, same conditions, same verdict, every time. Sub-millisecond. No model evaluating the output. No human reviewing every line. The human reviews what survives the structural gate. The human is the final gate, not the only gate.
99% post-repair on the benchmark suite. 185 out of 185 CI tests passed. No model in the verification path. The verification IS the governance. The governance IS deterministic. The bottleneck widens when the verification does not require a model or a human for every check. The human still owns the verdict. The human does not perform every check to reach it.
the biggest challenge i've encountered isn't building the factory, it's getting others to trust it enough to let it actually work. when you're solo building, you know the system intimately and can trust the outcomes because you built everything and usually have stress-tested it to a real degree of certainty.
... introduce it to a team and eyebrow raised.
trust is the hardest thing to build still, not just in the software but the people around it.
Addy,
Another masterpiece.
Really like the "lit" metaphor of humans adding light to dark factories.
Verification is really the bottleneck. As engineers, we were often used to figuring things out as we went. Now the game has changed: we need to figure more of it out upfront, if we can.
Automation must be earned. For me, that's where much of the complexity lies, because there is a discovery problem that needs to be solved before delivery can be automated.
Do you have any insights on "Discovery Factories"? Can discovery itself become a factory, or is ambiguity fundamentally what prevents it from becoming one?
Diego Pacheco
Back pressure reads as a per loop rule: only as much autonomy as you can cheaply verify. But review capacity is one shared pool. Ten loops that each clear the bar can still exceed what a team reads in a week, and the number being tracked is token utilization.
The metaphor of the factory, used at face value, is fundamentally misguided right? A compiler is a factory, while software involves design in tasks of huge variability rather than repeating the same actions over and over. Drawing the car rather than building millions of them.
That said I would be very interested to see a tool that allows prescribing a deterministic workflow to follow. For example, I follow outside-in Test Driven Development as a discipline and currently I hold that loop myself with a fine-grained tool like Cursor. I saw someone try to build that loop with Pi. Still, part of the job is also knowing when to stop and change approaches: you know need a diagram, or a conversation, or production data. These are all tools I have not seen agents pick up.
I built a deterministic verification with no model in the verification path. The task gets a scoped packet before execution: which files are allowed, which are forbidden, what evidence is required, what counts as done. The agent produces output. A deterministic check verifies whether the output satisfies the declared conditions. Same input, same conditions, same verdict, every time. Sub-millisecond. No model evaluating the output. No human reviewing every line. The human reviews what survives the structural gate. The human is the final gate, not the only gate.
99% post-repair on the benchmark suite. 185 out of 185 CI tests passed. No model in the verification path. The verification IS the governance. The governance IS deterministic. The bottleneck widens when the verification does not require a model or a human for every check. The human still owns the verdict. The human does not perform every check to reach it.
https://metacortexdynamics.substack.com/p/stop-letting-ai-coding-agents-decide
https://metacortexdynamics.substack.com/p/your-ci-runner-is-governance-infrastructure
The core thing is human review is limited.
Unread code is still unread code, even when every test is green.