this post was submitted on 16 Apr 2025
692 points (99.4% liked)

Programmer Humor

22921 readers
114 users here now

Welcome to Programmer Humor!

This is a place where you can post jokes, memes, humor, etc. related to programming!

For sharing awful code theres also Programming Horror.

Rules

founded 2 years ago
MODERATORS
 

So today I discovered that there's a cron job that holds non-reproducible state that died, and now our system is fucked.

The cron job doesn't live inside any source control. This morning it entered a terminal state, and because it overwrites its state there's no way to revert it.

I'm currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.

top 47 comments
sorted by: hot top controversial new old
[–] wise_pancake@lemmy.ca 164 points 2 weeks ago (2 children)

What's extra frustrating is the previous guy did create a git repo of these types of hacks, but this one doesn't live in it for no discernible reason.

[–] Colloidal@programming.dev 139 points 2 weeks ago (2 children)
[–] wise_pancake@lemmy.ca 114 points 2 weeks ago (3 children)

He does charge a consulting fee to "fix" these issues

Almost all of them are dumb shit like this, where something is built in super hacky and dumbass ways.

[–] blarth@thelemmy.club 102 points 2 weeks ago

It’s his kill switch and he forgot to check in.

[–] peregrin5@lemm.ee 58 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

Smart man. This is how we fight being replaced by AI.

[–] Agent641@lemmy.world 19 points 2 weeks ago (1 children)

Judgement day postponed indefinitely due to "Object reference not set to an instance of an object"

[–] kiagam@lemmy.world 2 points 2 weeks ago

I don't know why but this is the first time I read this phrase and it actually makes sense.

I knew exactly what it meant before, but it didn't make sense until now

[–] Stanley_Pain@lemmy.dbzer0.com 14 points 2 weeks ago (1 children)

Super hacky and dumb? Sign me up 😂

[–] swab148@lemm.ee 7 points 2 weeks ago (2 children)

Me running all my services in tmux

[–] Sherry@programming.dev 4 points 2 weeks ago (2 children)

that might be a stupid question, but why would you running all services in tmux be a bad idea? a co-worker of mine is doing exactly that right now, which is why I'm asking.

[–] swab148@lemm.ee 4 points 2 weeks ago (2 children)

It's not horrible, like it'll do the job just fine, it's just probably a better idea to use systemd and like, containers and whatnot, but I couldn't be arsed to fiddle with all that for Jellyfin, caddy reverse proxy, and two modded Minecraft servers, so shell scripts and tmux won the day. It takes a little extra time to restart everything after an update, and maybe I'll get the motivation to do things "correctly™" one day, but today is not that day.

[–] mosiacmango@lemm.ee 6 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

Use the tmux resurrect plugin. It will restore your tmux session to its previous state after a restart, including programs if you like.

You can put off doing things "correctly™" even longer.

[–] whats_all_this_then@programming.dev 6 points 2 weeks ago (1 children)

The "here's how you keep doing this poorly but more efficiently" energy on display here is a refreshing change of pace from the usual "here's how you do this correctly" crap peddled by normies (including me). You have my respect.

[–] mosiacmango@lemm.ee 4 points 1 week ago

If you're going to do something wrong, do it wrong right.

[–] Sherry@programming.dev 2 points 2 weeks ago

thank you very much for the detailed response :)

[–] qaz@lemmy.world 2 points 1 week ago* (last edited 1 week ago)
  1. They're all gone when you restart
  2. It doesn't properly deal with logging
  3. You can't set up dependencies between services but that doesn't matter due to point 1

I recommend using systemd services and/or docker compose instead. systemd services are files that describe which program / script to run and when (like after networking is active or after a certain other service is loaded).

[–] qaz@lemmy.world 1 points 2 weeks ago
[–] koper@feddit.nl 20 points 2 weeks ago (1 children)

But the whole point of the doomsday machine is lost... if you keep it a secret! Why didn't you tell the world, eh?

[–] MyBrainHurts@lemmy.ca 10 points 2 weeks ago

It was going to be announced at his retirement party on Monday... You know the dev likes surprises.

[–] some_guy@lemmy.sdf.org 3 points 2 weeks ago

And a kbase with no entry for it.

[–] symbioticremnant@lemmy.world 87 points 2 weeks ago

So do you work for Spotify or Zoom?

[–] arotrios@lemmy.world 73 points 2 weeks ago* (last edited 2 weeks ago) (2 children)

This is almost exactly what happened to me on Monday, resulting in a fifteen hour day.

My particular jenga piece was an Access query that none of my predecessors had deigned to document or even tell me about... but was critical to run monthly or you had obsolete data embedded deep within multi-million dollar reports.

Thank god I don't work on salary anymore, or I'd be really upset.

[–] floo@retrolemmy.com 56 points 2 weeks ago

I stopped reading at “Access” and just wept a silent tear for you.

[–] mesamunefire@lemmy.world 33 points 2 weeks ago

Oh god Access.

You have my condolences.

[–] MonkderVierte@lemmy.ml 33 points 2 weeks ago* (last edited 2 weeks ago)

Only tangentially related, but "What a elegant house of cards" is an insult i'm going to use someday.

[–] Primer81@lemmy.zip 23 points 2 weeks ago (1 children)

Idempotent code/repositories are great - I love making everything as reproducible as possible. Particularly in make where every 'all' type command should have a corresponding 'clean' command. Many times I'll see code bases where they skip defining the 'clean' command... or worse, have no 'all' command to begin with and rely on the developer knowing all the build and environment setup commands...

[–] wise_pancake@lemmy.ca 21 points 2 weeks ago (1 children)

Yeah, I don’t consider most code complete unless it’s safe and reproducible. I love make, currently using npm but you can set up scripts with it. Automating the build process was the very first thing I did.

This project is a piece of work. There’s effectively no documentation, and every now and then I find something new like this. The stuff I’ve fixed up so far has been much much more reliable and performant.

Part of me just wants to rewrite the whole thing, but I need to ship features so we can sell the product and pay my salary.

At least I’m not a cog in a huge corporation getting my soul crushed every day. I actually love fixing weird stuff.

[–] SchwertImStein@lemmy.dbzer0.com 5 points 2 weeks ago (1 children)

since you are currently using npm, check out pnpm

also "just" seems to be a more modern replacement for make

[–] wise_pancake@lemmy.ca 4 points 2 weeks ago

I'll check out both, thanks!

[–] SubArcticTundra@lemmy.ml 21 points 2 weeks ago

AHAHAHAHAHAHA you couldn't make this up

[–] mesamunefire@lemmy.world 17 points 2 weeks ago

We have a couple of those at work. Black boxes that are used.

I'm rebuilding one after it failed on one morning for SQL odbc reasons. And its just a binary that shuffles data around.

[–] Godort@lemm.ee 14 points 2 weeks ago (1 children)

Time to restore a whole machine backup to a VM with no network connectivity, and manually pull the command?

[–] wise_pancake@lemmy.ca 22 points 2 weeks ago (1 children)

I was able to do that

Turns out there was a second bug which triggered this one, and a bug I found in this script that I thought was responsible was happening silently for months.

Now three bugs are squashed

[–] slacktoid@lemmy.ml 14 points 2 weeks ago

Why does this sound like maintaining my nextcloud instance from time to time?

[–] someguy3@lemmy.world 11 points 2 weeks ago (4 children)
[–] wise_pancake@lemmy.ca 36 points 2 weeks ago

Cron is a scheduler to run a program at a set frequency

[–] JasonDJ@lemmy.zip 15 points 2 weeks ago

The executive branch of the US government.

[–] NewDark@lemmings.world 14 points 2 weeks ago

Scheduled job, but implies that it uses a cron format.

[–] ulterno@programming.dev 0 points 2 weeks ago

An older way of automating stuff.

It's not there by default nowadays, because systemd tends to fulfil their requirements.

[–] some_guy@lemmy.sdf.org 6 points 2 weeks ago

Somebody’s having a fun day! /s

[–] JackbyDev@programming.dev 4 points 2 weeks ago (1 children)

Had a similar thing once. Some how, some way, the DBA copied and pasted something wrong. Oracle DB had some odd extra syntax for left and right joins that other DBs didn't (or at least that I'd never seen). My best guess is that he auto formatted out of habit and maybe it took those symbols out.

It took a long time to find that. Because the only evidence something was wrong was that ONE of our customers wasn't being billed for ONE product. Everyone else was fine. Basically they were using it in a very atypical way. The left joins made sure to include them in the billing even because they didn't have whatever was on the right of that join. Everyone else did.

[–] wise_pancake@lemmy.ca 2 points 2 weeks ago* (last edited 2 weeks ago)

SQL auto format is still mostly terrible

The only half decent format is to start from the Mozilla style and then make it more sane.

I’ve been playing with sqlglot lately and want to start using it for diffs.

[–] Korhaka@sopuli.xyz 3 points 2 weeks ago

For us it's a task that no one is even aware of and the first issue is the customer saying their data export doesn't work. You had a data export?

[–] floo@retrolemmy.com 3 points 2 weeks ago

Just update your Clang library!

[–] grrgyle@slrpnk.net 2 points 1 week ago (1 children)

I have also mixed up crontab -l with crontab -r. 😔

Let this be a lesson to start versioning your crontabs.

[–] tiredofsametab@fedia.io 1 points 1 week ago

We never had our crons in source control, but I always saved it somewhere (usually on my machine and the target machine) so we had some history just in case of typing r instead of l for some reason. You can also create an alias called backupCrontab or something that runs the command for you and puts the output somewhere safe.