A working demo is not a working system
Most operations teams we meet have already run an AI pilot. It usually worked. A notebook, two years of historical data, a model that flagged the failure three days early on data it had never seen. Everyone in the room agreed it was impressive. Eighteen months later, nothing is running.
That is the common case, not the exception — and the model is almost never the reason.
What actually kills them
Nobody owns the output. A prediction arrives somewhere, and no named person is accountable for acting on it. An alert with no owner is a notification, and notifications get muted.
It was never tested on live data. Historical extracts are complete, clean, and in order. Live feeds are none of those. A model that assumes a sensor reports every thirty seconds behaves very differently when the link drops for an hour and then backfills.
The threshold was tuned for a demo. In a pilot, catching the event is what impresses. In production, the false-positive rate decides whether operators leave the system switched on. Get that wrong once and you rarely get a second run at it.
There is no route from alert to action. The model says a pump is degrading. Then what? If the answer isn’t a work order, a shift instruction, or an entry in a system someone already uses, the insight dies on the dashboard.
What closes the gap
Run it in shadow mode for a full operational cycle. Live inputs, live schedule, no action taken. You find the data gaps, the seasonal edge cases, and the true alert rate before anyone’s decisions depend on it.
Measure what operations measures. Model metrics belong in the engineering review. The conversation with the business is about alerts per week, how many were useful, and what an early catch was worth.
Give it an owner and an off switch. Someone accountable for the output, and a documented way to disable it that does not require us. Systems people cannot turn off are systems people learn to work around.
Ship the unglamorous parts. Monitoring on the pipeline as well as the model, retraining that someone can actually run, and a plain-English note on what the system does when it is unsure.
The honest test
Before a pilot becomes a commitment, we ask one question: if this ran unattended for six months, what would break, and who would notice? If nobody in the room can answer, the work isn’t deployment — it’s the operational design that should have come first.