Data Integrity Begins Before Data Reaches the Database
The database is not the beginning of the record
A database can enforce access control, preserve an audit history, apply retention rules and detect unauthorised alteration. These are necessary controls. They do not establish that the value written to the database was measured correctly, associated with the right asset, transmitted completely, interpreted using the correct configuration or timestamped against a reliable clock.
In operational and regulated systems, the stored record is usually the output of a chain: a physical phenomenon is sensed; firmware converts a signal into a digital value; communications carry that value to a gateway; software normalises, enriches or aggregates it; and a storage service commits it as a record. Integrity can be lost at every transition.
This distinction matters because a technically intact record may still be indefensible. A temperature value with an unbroken database audit trail is of limited use if the sensor was out of calibration, its identity was incorrectly bound to the monitored location, or a gateway silently retransmitted stale values after a network interruption. Data integrity must therefore be designed as an end-to-end property of the measurement and information flow.
Start with the measurement, not the message
For sensor-derived data, raw digital output is not automatically a measurement. A defensible measurement depends on the sensor's characteristics, calibration status, installation, operating range, environmental influences, signal conditioning and processing method. The uncertainty associated with the result may be material even where the communications path is reliable.
A system should retain sufficient context to establish what generated a value and under which conditions. This commonly includes device serial number, sensor type, firmware version, calibration identifier and due date, engineering units, configuration revision, sampling interval and any applied conversion or compensation coefficients. Where a measurement is used to support a quality, safety or compliance decision, the relationship between those items must be controlled rather than inferred later from spreadsheets or informal knowledge.
Identity is particularly important. A unique device identifier does not by itself prove that a device was installed at a particular point, connected to the intended probe, or configured for the intended service. Installation and commissioning records provide part of that evidence. Controlled replacement procedures are equally important: replacing a failed sensor without updating its digital identity can produce a continuous-looking trend assembled from two different instruments with different calibration histories.
Firmware and edge processing are integrity boundaries
Firmware is often treated as an implementation detail. In practice, it determines how a system samples, filters, scales, buffers and labels the physical signal. A change to averaging behaviour, overflow handling, timekeeping, engineering-unit conversion or alarm thresholds can change the meaning of every subsequent record.
Edge devices also need defined behaviour under degraded conditions. If connectivity is unavailable, do they buffer records locally? If so, how is buffer capacity monitored, what happens on exhaustion, and can the system distinguish delayed records from newly acquired records? If a device reboots, does it retain configuration and clock state? If it uses local sequence numbers, can a receiving service detect missing, duplicated or reordered messages?
These are not merely availability questions. A gateway that accepts data after a reboot with an invalid clock can create apparently precise timestamps that misrepresent the order of events. A device that discards failed samples without recording the failure may create a smooth trend that conceals intermittent faults. Engineering for integrity means preserving quality indicators, acquisition time, receipt time, sequence information and explicit exception states where they are relevant to intended use.
Transport must preserve context as well as payload
Network protection is necessary, but encryption alone does not prove message completeness, origin or correct interpretation. The receiving system should be able to establish which device or service sent a message, whether it arrived intact, whether it has already been processed and which schema or protocol version applies.
Appropriate mechanisms depend on the architecture. They may include mutually authenticated transport, signed messages, payload checksums, monotonic sequence numbers, replay protection and durable acknowledgements. Each mechanism addresses a different failure mode. A checksum can identify accidental corruption but does not authenticate the sender. A secure transport session protects data in transit but may not prevent a compromised authorised endpoint from sending incorrect values.
Store-and-forward architectures require particular care. Gateways may batch measurements, transform formats or retransmit records after disruption. The central platform should preserve the original acquisition timestamp and source identity rather than replacing them with gateway receipt time. It should also make late arrival, duplicate detection and rejected messages observable. Silent correction of these conditions can make later reconstruction unnecessarily difficult.
Transformation requires controlled semantics
Many systems modify data before storage: units are converted, readings are rounded, multiple samples are aggregated and external reference data is joined. These transformations can be valid, but they must be specified, tested and traceable.
A common failure is to retain only the calculated result. If a daily average is used for reporting, the system should normally retain or be able to retrieve the constituent readings, the aggregation method, the applicable time boundary and the handling of missing or invalid samples. Without that information, an investigator cannot determine whether a changed result reflects a real process change, a configuration change or a processing defect.
Controlled change is central here. Configuration, transformation logic, reference tables and time-zone rules can all alter outputs without changing source measurements. In a controlled environment, changes should be assessed against intended use, authorised, tested and deployed in a manner that permits the affected data population to be identified. The exact validation evidence required will depend on the system and applicable regulatory framework, but the engineering objective is consistent: demonstrate that the system continues to produce reliable records for its intended purpose.
Integrity at rest still matters
Database controls remain an essential final layer. Role-based access, least-privilege administration, immutable or appropriately controlled audit records, backup and restore testing, retention controls and monitoring for anomalous change all contribute to trustworthy data at rest. They should be designed alongside, not instead of, controls at the edge.
The useful question is not simply whether a database record can be altered. It is whether an organisation can reconstruct the path from stored value to source measurement, identify the hardware, software and configuration involved, explain exceptions and demonstrate that changes were controlled. That requires evidence across the full system boundary.
Data integrity is therefore not a feature added at the persistence layer. It is a property built through measurement design, device management, secure communications, explicit processing rules, controlled change and recoverable operational evidence. By the time data reaches the database, the most important integrity decisions have often already been made.