If we expect no mistakes from AI, we will miss the value it can bring
People push back on fully automated systems because they want a human in control. If we expect no mistakes from them, we will miss the real value they can bring. I want control over the intent, an override where it is safe, and the same standard we hold people to.
Fully automated systems that appear to remove a person’s free will from the loop get a lot of pushback. I’m thinking of mechanical things like self-driving cars. The main argument, as far as I can tell, is that we still want a human in control. A soul.
Creative work like films and books gets the same pushback. That is a different question, and I’m leaving it for another post.
How much control do we want
Would we like to be fully in control of our own bodies? Breathing, pumping blood, growing hair, bowel movements? That sounds like a terrible idea to me. I quite enjoy only having to choose where I want to go, or what I want to wear. Those are simple tasks for an entity that cannot focus on too many things at a time.
The body doesn’t give me the same control over everything. For some things, like locomotion, I take full control of the direction. For others, like breathing, I can take temporary control: I can hold my breath, and then it goes back to doing it for me. For the critical ones, like pumping my blood, I don’t get a say, and I’m fine with that. It feels a bit like kernel space and user space. That is the split I want from an autonomous system.
Apply that to software. If there’s a system that does all the coding for me, then I can focus on where I want the project to go and how I’d like it to look. If I have to worry less about which database it uses, or about the latest zero-day exploit, because the system is smart enough to make those choices for me, why wouldn’t I want that?
I still hope there are experts in databases and security driving those projects, just like I want to keep driving the lifecycle management of a system like Kairos.
I’m not sure AI is at that stage yet, but it feels like we’re close.
The mistakes
I don’t understand the point of those who only focus on the mistakes AI makes. Imagine if any of us had to show the same level of accountability for our mistakes, bad behavior and what not. We’d consider that inhumane. If we expect no mistakes from autonomous systems, we’re going to miss the real value they can bring.
If we started calling everything people get wrong “human slop”, we would never stop finding slop. I see it every day in work done by people. Crap, crap, crap, everywhere. And we don’t jump to “put them in jail” because someone didn’t clean properly or the train came late.
The same standard
I’m not arguing that AI should be held to no standard. I’m arguing for the same one.
We already accept human errors at a rate that depends on the stakes. A late train is annoying and acceptable. A wrong drug dose is not. So an AI system should be measured against the human baseline for that task, scaled to the stakes. I haven’t looked up what those baselines actually are for any particular task.
Someone also has to own the mistake when it happens. With a person, it’s that person. With an AI system, I think it lands on the person or team that deployed it. I wrote about one way to set that up for infrastructure in Don’t give AI agents root. Make them propose the next system state.: the agent proposes a change, and people own the gates it has to pass through.
So instead of complaining about the mistakes, I’d rather we seriously experiment with where the pairing of human and machine improves things, where it doesn’t, and how to make it work better.