Logging every five minutes without losing data

The logger has one job: append a row to a file every five minutes. It has failed at that job three times, each time in a different way.

The pipeline

A script reads the sensors every ten seconds and keeps the samples in memory. On each five-minute boundary it writes the mean of the interval as one line:

2025-01-19T14:35:00Z,-1.2,91,1018.3,2.4

The file is plain CSV, one per year. I chose text over a database because I can read it with anything, and because a damaged text file loses a line while a damaged database can lose everything.

Failure one: the clock

After a power cut the computer booted with no network and a clock set to the time of its last shutdown. It logged four hours of readings with timestamps in the past, overlapping real ones. The script now refuses to write until the clock has been synchronised at least once since boot.

Failure two: the storage card

Writing a small file every five minutes for a year wore out a cheap card. The symptom was a file that ended in the middle of a line. Rows now go to memory first and are flushed to the card once an hour, and a copy is sent to another machine every night.

Failure three: me

I edited the script over a remote session, made a typo, and went on holiday. Nine days are missing from last August. The script now runs under a supervisor that restarts it, and a second, much simpler script sends me a message if the newest row is more than an hour old.

What I would do first next time

Build the alarm before the logger. Every one of these gaps lasted longer than it needed to because nothing told me the file had stopped growing.