TIL: User systemd Timers Can Die When You Log Out

Earlier in June, there was a post on Hacker News written by Tyler Langlois on the versatility of systemd timers that happened to line up with my decision to migrate my personal scripts onto a new VM. I thought, what the heck, maybe it is time to make the move from cron. The post was convincing enough that timers would make diagnostics and job monitoring easier, and the thought of having version-controllable systemd timers solved a friction point for me.

I deployed a handful of my personal scripts onto the new VM with systemd user timers. I didn’t see the point of running these scripts at a system level, hence why user timers seemed like the obvious right fit because they didn’t need to be privileged. (This was a false assumption¹) I had to do some minor finagling to get everything working, but the timers worked, and I was able to disable my old cron jobs on the old VM.

The key thing is that for the weekend, this was my side project with an SSH terminal constantly open.

To deal with error handling, I had also set up some systemd timers whose entire job was to check that my state management methods were actually populating new entries. Because if the scripts stopped capturing results, surely a systemd timer that checks to make sure the scripts are performing as expected would work, right?

At a certain point, it was done and I was ready to move on with my week, so I stopped SSHing into the server. I did not notice for two days, but eventually I realized I was not getting any notifications from my scripts. That was strange. Some of them are supposed to produce at least one notification a day, even if they are low-priority ntfy ones that do not make any sound or vibration. I was not getting error notifications from the state observation script either, so that did not make any sense.

At first, I was obviously convinced that either my timers were wrong or my scripts were broken. But there was no useful output to debug because the scripts had not run at all. The timers were still there and the unit files looked right. The scripts worked when I manually called them. They just were not triggering for some reason until I logged in, did some debugging, left my SSH hanging for a while, I thought maybe it was just a blip, only to log out and repeat the same problem tomorrow.

It took more wild goose chasing than I’d like to admit around my supposedly broken timer or script logic before I realized the actual issue: my user systemd manager was tied to my login session. When I stopped logging in entirely, there was no persistent user manager around to trigger the timers.

The unit files and scripts were fine. My assumption about user timers was not.

The fix ended up being:

# Keep the user systemd manager running after logout
loginctl enable-linger "$USER"

# Enable and start the timer
systemctl --user enable --now my-job.timer