How my VPS spent 135 days mining crypto for a stranger

Two years ago I wrote an article about how GitLab broke my VPS. Two years? I had to check the date twice, because in my head that was maybe last spring. Apparently not - the blog has been sitting here quietly gathering dust while I wasn't looking, which, as you'll see, turns out to be a bit of a theme.

Anyway. I ended that article by saying that maybe I should switch to Gitea, or Forgejo, or something else, but that I'm quite lazy and already committed to GitLab, so for now it can stay.

I stayed lazy for two more years. Then one Tuesday morning it all came due at once, and by the evening I had migrated my Git server, evicted a cryptocurrency miner that had been living on my machine since February, discovered my server had never had a firewall in its entire life, and locked myself out of it. Not necessarily in that order of embarrassment.

It started, as these things do, with something completely trivial. I have a little hobby project - a PWA game - and I wanted a .gitignore for it. That's it. That was the whole plan for the morning.

I pushed the project to my self-hosted GitLab and it said "Deploy in progress. Please try again in a few minutes."

Fine. A few minutes passed. Still deploying. Then the entire GitLab GUI became "Deploy in progress" - every page, every URL. I couldn't cancel anything, because cancelling things requires a GUI, and the GUI was the thing that was gone.

My first theory was Auto DevOps. But once the GitLab came online I disabled Auto DevOps, saved, and got "Deploy in progress" again without pushing anything at all.

So I pulled the logs, and the logs told a different story. Puma - the Rails server that is the GitLab web interface - was restarting every three to four minutes. And every boot took 116 seconds. So the cycle was: spend two minutes booting, be alive for one minute, get killed, repeat. That "Deploy in progress" page was never a deploy. It was Apache's ErrorDocument 503 fallback, dutifully showing me a maintenance page every time the backend died. Which was constantly.

Why was it dying? df and free answered that in about one second. Disk at 96 percent, 1.2 GB free out of 30. RAM completely exhausted: 3.8 GB total, 105 MB free, and swap 100 percent full. There was simply nothing left. GitLab wasn't broken, it was suffocating.

And here I finally admitted the thing I'd written two years earlier and then ignored: this VPS does not meet GitLab's minimum requirements, and no amount of tuning Puma workers was going to change that. So I did what past me had been too lazy to do. Gitea. Docker container, SQLite, about 150 MB of RAM at idle instead of multiple gigabytes.

The migration had one nice twist. To get my repositories out of GitLab I first had to make GitLab healthy enough to hand them over, which it refused to do - every clone over HTTPS died with "remote: Nil JSON web token" and a 403. That's not a credentials problem, that's GitLab's Workhorse and Gitaly failing to talk to each other because the instance was too sick to sign its own tokens. My Git server was too broken to give me my own source code.

So I went around it. GitLab stores repositories on disk under a @hashed directory, named by hash rather than by project, which is great for GitLab and useless for humans. But each one keeps its original path in its git config, so you can walk the directory, ask every repository what it thinks its name is, and build the mapping yourself. Then it's just a bare clone and a push into Gitea. All four repositories came over, source, branches and tags, without GitLab's cooperation.

I repointed the Apache vhost from port 8080 to Gitea on 3000, kept the same address, and then purged GitLab entirely. Five gigabytes came back. Disk went from 96 percent to 80.

It runs like butter now. Honestly, I should have done it in 2024.

But the memory was still wrong.

That's what bothered me. GitLab was gone - the single biggest consumer on the box, uninstalled, purged, its data directories deleted - and I still had 3.5 GB of 3.8 GB used and swap completely full. So I sorted the process list by memory, and there it was, right at the top:

   lmstudio 3365393 93.8 59.4 /dev/shm/2775695008

Roughly translated: some user was running a process at 94 percent CPU, using 59 percent of all the RAM in the machine, and the program it was running was a file called 2775695008 in /dev/shm.

I did once install LM Studio on this server. It was an experiment, I played with it for an afternoon, and I never thought about it again. So my honest first reaction was "I still have LM Studio running on this machine?"

I did not. LM Studio was sitting on the disk doing nothing. What was running was a cryptocurrency miner.

Everything about that line is wrong, and it's wrong in a very deliberate way. /dev/shm is shared memory - a filesystem that lives in RAM. Nothing legitimate installs itself there. It's chosen precisely because there is nothing on disk to scan and no file to find later. The random numeric filename is the same idea. And 94 percent CPU on an otherwise idle server that hosts a few small websites has exactly one plausible explanation.

It wasn't alone, either. There were three parts to it, and this is the part I found genuinely interesting once I got over being annoyed:

• The miner itself, a stripped, statically linked binary in /dev/shm, connected out to 170.168.103.27 on port 29205 and quietly earning money for somebody who isn't me.
• A watchdog, hiding in /home/lmstudio/.config/systemd/ under the filename "journalctl" - a 15 MB fake copy of a real system utility, whose only job was to notice the miner dying and immediately start a new one. It had been running for 135 days straight.
• An @reboot entry in the lmstudio user's crontab, so that if the machine ever restarted, the whole thing would come back on its own.

That last one is almost funny. It had never once fired, because the server hadn't rebooted in 1155 days. Three years of uptime. I used to be a bit proud of that number.

The ordering matters when you clean this up. If you just kill the miner, the watchdog notices within a second and starts another one, and you sit there killing miners forever wondering why you're losing. You have to take the watchdog and the miner down together, with a signal that can't be caught, then remove the payloads, then the cron entry.

Since the entire lmstudio account was compromised and I don't use it, I deleted the whole thing - user, home directory, LM Studio and all. free -h afterwards showed 2.4 GB available where there had been 105 MB. The mysterious memory pressure that I had spent the morning blaming on GitLab had been a stranger's mining rig for about four and a half months.

Which brings me to the front door, and this is the genuinely embarrassing bit, because it was entirely my fault.

The lmstudio account had never logged in over SSH - not once, the login records were empty. So nobody guessed a password. What happened is that LM Studio 0.3.4 has a network server, I had it listening on 0.0.0.0 rather than localhost, and I left it that way when I lost interest in the experiment. An old inference server, open to the entire internet, running as its own user. Somebody found it around February 21st, got code execution as that user, dropped a miner, and settled in. They never got root, they never got a shell, they never touched anything else. They just quietly took most of my RAM and left everything else alone, which is why I never noticed.

Then, while checking what else was exposed, I ran iptables -L and got the actual headline of the day:

The server did not have a firewall. At all. Not a misconfigured one, not a permissive one - the INPUT policy was ACCEPT and there were zero rules. Every single port bound to 0.0.0.0 on that machine was directly reachable from the open internet, and had been for years.

That included MariaDB on 3306. It included an FTP server on port 21, plaintext, which I did not even remembered was installed. It included two Node development servers, a Next.js app, a snippet manager, and a handful of other things I'd started at some point and forgotten. Fourteen open ports, of which I actually meant for six to be public.

So the real fix wasn't reconfiguring fourteen applications one at a time. It was installing one default-deny firewall and opening only what I actually serve: SSH, HTTP, HTTPS, Gitea's git-over-SSH port, and two ports for a backend that genuinely has to be public because its own frontend JavaScript calls it directly from the browser. Everything else went dark. I probed it from outside afterwards to be sure, and the database, the dev servers and the FTP had all vanished from the internet.

One thing worth knowing if you ever do this: ufw does not filter ports that Docker publishes. Docker writes its own iptables rules further up the chain and cheerfully bypasses your firewall. If you want a container off the internet you have to fix the container's port binding, not the firewall. That's how I found out my snippet manager had also been publicly exposed the entire time.

While I was in there I looked at SSH, and found PermitRootLogin yes, PasswordAuthentication yes, and 17,282 failed login attempts in the log, with a live swarm of bots hammering away as I read it. The only thing standing between the internet and a root shell on my server was how good my password is. I switched to key-only authentication, which I should have done years ago.

Small trap for anyone copying this: on this box, /etc/ssh/sshd_config has no Include directive, so the drop-in file I wrote into sshd_config.d/ was read by absolutely nobody. The config validated, sshd said everything was fine, and the effective settings hadn't changed at all. Always check with sshd -T instead of trusting that the file you wrote is being read.

Then the reboot. 1155 days of uptime means the kernel running in memory is not the kernel installed on disk, and a lot of security-patched binaries were still running their old versions. But you don't reboot a three-year-old production box casually - the risk isn't the kernel, it's the services that somebody (me) started by hand in 2023 and never configured to start automatically. So I checked every one first. Good thing, too: it turned out my snippet manager had no restart policy and had silently died hours earlier when I restarted Docker, and I hadn't noticed. It would have stayed dead through the reboot too.

Everything came back on the new kernel. Sites serving, containers up, all six Node apps resurrected.

And then, because the day wasn't complete, I added fail2ban to stop the bots from filling my logs. But then I started an automated script that's using SSH, went AFK for an hour, came back, and found that SSH was timing out.

The websites were all fine. Port 443 fine, port 80 fine, Gitea fine. Only port 22, only for me. I had set fail2ban to aggressive mode, which counts every pre-authentication disconnect as a failure - and an hour of short SSH sessions from my own IP looks, from a certain angle, exactly like an attack. So fail2ban did precisely what I told it to do and banned the administrator.

This is where the VNC console in my provider's admin panel earned its keep, luckily I had deliberately tested it earlier that same day, before disabling password logins, specifically so that I'd have a way back in if I locked myself out. I did not expect to need it four hours later. Unban my own IP, add it to the whitelist, drop back to normal mode, done. Fail2ban is now banning about ten actual bots and leaving me alone.

So: a .gitignore.

What I keep coming back to isn't the miner. The miner was almost polite - it never escalated, never touched root, never broke anything, and if it had used 20 percent of my RAM instead of 60 I would probably still not know it was there. What bothers me is that it got in through an experiment I ran for one afternoon and then abandoned. Not a vulnerability in something I depend on. Just a thing I switched on, found mildly interesting, and forgot to switch off.

That's the actual hobbyist failure mode, I think. It's not that we write insecure code. It's that we accumulate. Every side project, every "let me just try this," every service started on 0.0.0.0 because that was the fastest way to see if it worked, all of it stays running for years on a box nobody audits. Three years of uptime isn't a badge of honour. It's three years of not looking.

The box is fine now. Clean, patched, firewalled, key-only, and running a Git server that uses a twentieth of the memory of the one it replaced. There's still an old service in there trying to start a World of Warcraft server whose binary I deleted at some point, failing and retrying forever, which tells you I haven't fully learned my lesson.

But at least I finally switched to Gitea. Only took two years and a stranger's mining rig to get me moving.