The integration is the project
Every operational software proposal has a line item called “integration”. It is almost always the item estimated last and understood least. In heavy industry it is usually the whole job.
The data needed to make a better decision already exists. It sits in a process historian, a SCADA layer, a maintenance system procured in 2004, and a shared drive of spreadsheets that three people understand. None of it was built to be read by anything other than the screen it was written for.
Why it’s harder than it looks
Three things bite, and they bite in this order:
- Access is a negotiation, not an API. The system holding the data is often the system that must not go down. Read access is a conversation with the people accountable for uptime — and it should be.
- The schema is a story, not a spec. Field names lie. Units change halfway through the history. A status code means one thing before a 2019 upgrade and something else after it. That knowledge lives in operators’ heads, not in documentation.
- Volume is uneven. Telemetry arrives in bursts, backfills arrive late, and the link drops when the weather is bad — which is precisely when the data matters most.
What holds up in practice
Read into a copy, never work in the source. We land raw data in our own store first and do nothing clever upstream. The operational system keeps its guarantees; we keep the freedom to reprocess.
Make the pipeline replayable. Keep the raw payloads. When you discover — and you will — that a unit conversion was wrong for three months, you re-run rather than re-collect.
Treat late and duplicate data as normal. Not as an error path bolted on afterwards. Idempotent writes and explicit event timestamps cost little on day one and save the project on day ninety.
Write the meaning down. Every field we map gets a one-line definition agreed with someone who operates the asset. That document outlives the project, and it is often the most valuable thing we hand over.
Where to start
Pick one decision, trace the data it needs back to its source, and integrate that single path end to end. One thin, reliable pipeline teaches you more about the estate than six months of discovery — and it leaves something running.