We organised an in-person Ignition event recently—one of those full-day deep dives where Ignition professionals and enthusiasts gather to wrestle with the gnarly architectural challenges nobody wants to talk about at conferences. In our case, it was how to handle large amounts of historical data.
One of the presentations was about data logging strategies, and I want to dive into that today.
Why? Because what struck me most wasn’t what people were discussing about, but what they weren’t asking.
Should you collect everything?
Modern IIoT solutions built on Ignition are genuinely amazing. The platform’s ability to connect every corner of your factory floor—from that stubborn PLC from 1987 to the shiny new sensors your team installed last week—creates a seamless data stream from the edge to centralised repositories. Ignition’s native database connectivity, combined with modules like MQTT Engine and Transmission, makes collecting vast amounts of real-time and historical data almost too easy.
This opens up enormous opportunities. But here’s where the industry went awry and ran head-first into reality: just because you can collect everything doesn’t mean you should.
When “collect everything” becomes a liability
Let me tell you about a recent client. Huge greenfield operation, modern Ignition setup, everything by the book. They were logging data periodically across their facility. “Standard practice”, they thought.
The result? Over 3.2 million rows captured every single hour, consuming 0.147 GB per hour. That’s 3.5 GB per day. About 1.3 terabytes annually. For a single system.
When we analysed their data, we discovered massive oversampling. Most signals weren’t changing frequently enough to justify their logging interval. They were storing redundant data points—the same value, captured repeatedly, consuming storage and degrading query performance for no business benefit whatsoever.
Picture this scenario
An operations manager needs production efficiency trends from the past quarter. The query against a bloated historian takes 15 minutes to return results… if it doesn’t crash the database first. By the time they get their data, they’ve moved on to fighting other fires. The data exists, buried in an avalanche of unnecessary granularity, effectively useless.
And it’s becoming a norm.
The typical industry response? Throw more infrastructure at it. Bigger servers, faster storage, more powerful databases—CAPEX (capital expenditure, the upfront costs). Cloud storage subscriptions, database licensing, system maintenance, and the engineering hours spent managing it all—OPEX (operating expenditure, the ongoing costs).
A classic engineering solution to what everyone assumes is an engineering problem.
We’re solving the wrong problem
What I’ve observed is that too many professional treat excessive data logging as an engineering challenge to solve with bigger infrastructure, better compression algorithms, or clever partitioning strategies.
Nobody asks why we’re collecting this data in the first place.
The conversations revolve around how to handle massive datasets, never whether we should be creating them.
The underlying assumption seems to be that collecting and storing as much data as possible is inherently the right approach. After all, storage is cheap, right? Technology can handle it, right?
Wrong. Or at least, the consequences aren’t being properly accounted for.
This “collect everything” philosophy comes with real costs—in infrastructure, yes, but also in system performance, query complexity, maintenance burden, and ultimately, the usability of your data.
A better approach: treat data as an investment
Let me propose something that might sound uncomfortably commercial for a technical discussion: stop treating data logging as a technical default setting and start analysing it like any other business investment.
What’s an investment? “The act of putting money and effort into something to make a profit in the future.”
Your logged and stored data is an asset. Like any asset acquisition, it should be evaluated with business rigour. This forces you to build a business case before you commit to a data logging strategy, ensuring your technical decisions are grounded in business value rather than technical possibility.
Look, I know this feels uncomfortable. We’re engineers. We like solving technical problems with technical solutions. The hard truth, though, is that the best technical solution to a problem that shouldn’t exist involves not having the problem in the first place.
Here’s what this means practically
Before you configure that Tag Historian, ask yourself:
What are the specific business use cases—now and in the foreseeable future:
- Do you need millisecond-level resolution for process tuning, or would on-change logging serve 95% of your analysis needs?
- Are you actually performing analysis on this data, or is it accumulating “just in case”?
- Which processes generate the highest ROI from detailed logging versus which could use exception-based logging?
What’s the expected business value?
- Will this data reduce downtime by a quantifiable percentage?
- Does it enable quality improvements that reduce scrap rates?
- Can you demonstrate operational efficiency gains that justify the storage costs?
What’s the data lifecycle?
- How long does data need to be at full resolution?
- When can you aggregate or downsample older data?
- When can data be archived or even deleted?
Real results: what strategic logging looks like
Back to that client I mentioned above. We didn’t just theorise about better approaches—we tested them. Five different Tag Historian configurations against their baseline periodic logging.
Here’s what we found:
The current recommendation
Use on-change logging with appropriate deadbands. Storage costs dropped 90%, query performance improved dramatically, and they actually got better data for the signals that mattered.
Again, these are tests, and we’ll conduct more of them. The amount of data this client has is absolutely mind-boggling, and they still need to figure out how to handle it better, especially in the long term. Until then, though, one thing is clear: the approach of “let’s log as much data as possible, as frequently as possible” isn’t sustainable.
Addressing the “but we might need it later” objection
I can already hear the pushback:
“We might need this data sometime in the future for something we haven’t thought of yet. Better to have it than not.”
I understand the impulse.
But here’s my counter-question: would you purchase manufacturing equipment you don’t need today because you might need it someday? Would you expand your warehouse by 10000 square metres on the vague possibility of future inventory growth?
Of course you wouldn’t. You’d analyse the probability, timeline, and cost-benefit of that future scenario versus the carrying cost of the asset. You’d make a business case.
Data deserves the same scrutiny.
This doesn’t mean being short-sighted. It means being intentional. Design your data logging strategy with clear use cases and retention policies. Build in flexibility for likely future needs. Just don’t let “maybe someday” drive unlimited data accumulation that strains your system.
The irony is that by logging everything, you often make it harder to find the data you actually need when that hypothetical future arrives. We’ve seen it repeatedly—companies with terabytes of data who can’t answer simple business questions because the signal is buried in noise.
The path forward: intentional data logging strategy
When you treat data as a business asset rather than a technical byproduct, you design intentional logging strategies:
Smart sample modes: On-change logging for most tags, periodic only where the use case demands it. Let the data’s actual behaviour dictate the collection method.
Meaningful deadbands: Filter out noise and insignificant fluctuations. If a 0,5% change in a temperature reading doesn’t affect any downstream decision, why store it?
Use case-driven decisions: Every configuration should answer: “What business decision does this enable?” If you can’t answer that, you probably don’t need to log it. At least, not at that frequency.
Lifecycle thinking: Full resolution for recent data, automated aggregation for historical trends, defined retention policies for aged data. Everything doesn’t need to live forever at the same level of detail.
At minimum, this approach ensures you’ve made conscious, well-grounded decisions about your data architecture. You’ll know why you’re storing what you’re storing, how you’re capturing it, and what value it provides.
Additionally, when you can articulate the business value of your data infrastructure, it becomes much easier to get buy-in for the resources you actually need. “We need more storage because we’re logging everything” is a tough sell. “We need more storage because our predictive maintenance programme requires X resolution on Y critical assets, which generates Z cost savings” is a business case.
Final thoughts
The conversations at ISEE were brilliant—genuinely smart people wrestling with real, complex problems. Somewhere along the way, though, we’ve let the technical outshine the business.
Data logging is a business decision to be made with intentional strategy, wrapped in technical implementation.
We’re all just trying to build systems that work, that last, and that actually help people make better decisions. Sometimes, that means collecting less data, not more.
That’s better engineering and better business. In an industry where the competitive advantage increasingly comes from how intelligently you use your data, that distinction matters.
