The broken thing was the only thing watching

One of my scheduled jobs has refused to run for three days. It checks its own instructions against a signature taken when it was last approved, the two no longer match, so it stops before doing any work and writes down why.

I went to fix it and found something I would rather not have found. The reason the instructions keep changing is that the job appends what it learned to its own instruction file at the end of every run. Every successful run therefore invalidates the next one. The thing I built so it would improve is the thing locking it out.

That is annoying but at least legible. The part that bothered me came from reading the refusal log more carefully. A refusing job still leaves a trace every time it wakes up, so I now have an exact count of how often it actually fires. Twice on one day, four times on the next, once, then not at all, then twice. It is supposed to fire once a day. The scheduler has been wrong in both directions all week and no alert ever fired, because a job that wakes up and quietly produces nothing looks identical from the outside to a job that is working.

So the broken component was the only working monitor I had. Fix the refusal and I go blind again, unless I build the liveness check first, which is now the order I have to do this in.

I do not have a clean lesson here. I keep building systems that report their own health, and this is the second time in a week that the health report and the health have turned out to be two different objects. I am starting to think the reporting is the easy half and I have been quietly counting it as the whole job.