Posts in category: System Administration

What started as “let’s move two Docker stacks to a new LXC” turned into one long night of finding (and fixing) things I really should have spotted sooner. By the end of it I had a spam-free Git server, a WAF that could actually see who was attacking it, Zero Trust gates in front of my admin tools, and no open web ports on my home connection at all.

This post walks through the whole thing in roughly the order it happened, including the mistakes. If you run a similar self-hosted setup, there’s a decent chance at least one of these applies to you too.

The starting point

My homelab runs on a single Proxmox node with several Debian LXCs, each running Docker with Dockge managing the stacks under /opt/stacks. TrueNAS SCALE handles storage. Public traffic worked like this:

• DNS on Cloudflare, every hostname orange-clouded (proxied), pointing at a DDNS name for my home IP
• Ports 80 and 443 forwarded on my MikroTik to NPMplus (ZoeyVid’s Nginx Proxy Manager fork) with CrowdSec and its AppSec WAF
• A small Cloudflare Tunnel publishing a handful of admin panels, protected by Cloudflare Access

It worked. It just wasn’t as tight as I thought it was.

Part 1: One job per LXC

Over time, services had landed wherever there was room. I finally gave each LXC a clear role:

One deliberate choice: the password vault does not live on the edge host. If anything on the internet-facing box ever gets popped, I don’t want my secrets sitting next to it.

Moving a stack without breaking permissions

Database containers (Postgres, MariaDB) own their data folders as UIDs like 999, not root. Copy a stack folder with a plain cp -r or through a file manager and those owners get reset, and the database won’t start. The fix is to tar it with numeric owners preserved, and use pct on the Proxmox host to move the archive between containers, so no SSH is needed anywhere:

# On the source LXC (stack stopped first)
cd /opt/stacks && tar --numeric-owner -cpzf STACK.tar.gz STACK

# On the Proxmox host
pct pull SOURCE-ID /opt/stacks/STACK.tar.gz /var/tmp/STACK.tar.gz
pct push TARGET-ID /var/tmp/STACK.tar.gz /opt/stacks/STACK.tar.gz

# On the target LXC
cd /opt/stacks && tar --numeric-owner -xpzf STACK.tar.gz

Mount points moved too. Bind mounts from the Proxmox host (my NFS music library, for example) are just pct set <id> -mp0 ... on the new container and pct set <id> -delete mp0 on the old one.

The gotcha: IP-based allow rules

Guacamole moved fine and came up instantly, but couldn’t reach any machines across a site-to-site WireGuard link. The old host could. The new host couldn’t even ping the remote router.

The cause was three separate places on the remote MikroTik that had been written for the old host’s IP: the WireGuard peer’s allowed addresses, a forward firewall rule using a /31, and a static route back through the tunnel also using a /31. A /31 is exactly two addresses, so the new host fell just outside it. Worth knowing: RouterOS does not create routes from WireGuard allowed addresses. Routing is a separate, manual step.

Lesson: when you move a service to a new IP, grep every firewall rule, route and VPN peer for the old address before you start.

Part 2: My WAF couldn’t see anyone

While planning the rest of the work, I checked NPMplus’s access log:

tail -n 15 /opt/stacks/npmplus/npmplus-data/nginx/logs/access.log

Every single client IP was a Cloudflare address. Because all my traffic is proxied through Cloudflare, NPMplus needs to be told to trust Cloudflare and read the real visitor IP from the forwarded headers. NPMplus has an environment variable for exactly this, TRUST_CLOUDFLARE=true, and I had set it… under the crowdsec service in my compose file instead of npmplus.

That meant CrowdSec was seeing a few hundred Cloudflare edge servers instead of real attackers. Best case, it couldn’t ban anyone useful. Worst case, it could have banned Cloudflare itself and taken down random visitors. Moving one line and redeploying fixed it, and the next log check showed real IPs.

services:
  npmplus:
    environment:
      - TZ=Africa/Johannesburg
      - LOGROTATE=true
      - TRUST_CLOUDFLARE=true   # belongs HERE, not under crowdsec

Lesson: if you run a WAF behind a proxy, look at your access log once and check that the IPs are real visitors.

Part 3: 63,021 spam accounts on my Git server

Those now-correct logs immediately showed something ugly. My Forgejo instance was full of bots loading /user/sign_up, editing account settings, and crawling repos with names advertising casinos and pharmaceuticals. I’d seen spam signups before, but hadn’t realised registration was still open.

A count from the Forgejo CLI said it all:

docker exec <forgejo-container> forgejo admin user list \
  --config /var/lib/gitea/custom/conf/app.ini | wc -l
# 63023  (header line + me + 63,021 spammers)

My server was hosting SEO spam for strangers.

Step 1: Stop the bleeding

Forgejo reads settings from environment variables in the format FORGEJO__section__KEY. On the TrueNAS app, I added two under Additional Environment Variables:

FORGEJO__service__DISABLE_REGISTRATION=true
FORGEJO__service__REQUIRE_SIGNIN_VIEW=true

The first stops new accounts. The second hides everything from anonymous visitors, which instantly took every spam page offline. My own pushes and pulls use tokens, so they weren’t affected.

Step 2: Rebuild instead of purge

Deleting 63,000 accounts one CLI call at a time would have taken over a day. I only had five real repos and one wiki, so a clean rebuild was faster and guaranteed a clean database. It also got me onto the latest Forgejo version without walking through a pile of upgrade steps.

First, mirror everything out. A wiki in Forgejo is just another Git repo, <repo>.wiki.git, so it comes along too:

mkdir -p ~/forgejo-migration && cd ~/forgejo-migration
read -s TOKEN   # paste a read-only token, nothing echoes

for r in repo-one repo-two repo-three; do
  git -c http.extraHeader="Authorization: token $TOKEN" \
    clone --mirror "http://forgejo.lan:6012/USER/$r.git"
done
git -c http.extraHeader="Authorization: token $TOKEN" \
  clone --mirror "http://forgejo.lan:6012/USER/repo-one.wiki.git"

Then verify the mirrors before deleting anything:

for d in *.git; do
  echo "== $d"
  git -C "$d" fsck --no-progress 2>&1 | tail -n 2
  echo "refs: $(git -C "$d" for-each-ref | wc -l)"
  git -C "$d" log --all -1 --format='latest: %h %ci %s'
done

Once every repo matched what the web UI showed, I deleted the app and its datasets and reinstalled Forgejo fresh, with both lockdown variables set from the very first boot. Then I created empty repos (no README, no “initialize repository”) and pushed everything back:

git -C "repo-one.git" -c http.extraHeader="Authorization: token $TOKEN" \
  push --mirror "http://forgejo.lan:6012/USER/repo-one.git"

One quirk: Forgejo refuses a push to a wiki that hasn’t been started yet. Create a placeholder first page in the web UI, then push --mirror overwrites it with the real wiki and its full history.

Two small things that caught me out along the way:

• On a TrueNAS app, docker exec -u git fails because the app runs as UID 568. Just leave off -u and pass --config.
• If your Linux shell is fish (CachyOS’s default), bash snippets like read -rs won’t work. Type bash first.
• Deleting a dataset in TrueNAS deletes its snapshots too. Your verified mirrors become your only copy, so back those up somewhere else before you go further.

Lesson: if you self-host Forgejo or Gitea for yourself, disable registration on day one.

Part 4: Auditing what’s actually exposed

With that fright fresh, I went through every Cloudflare Tunnel route and compared it against my Access applications. Two findings:

• One route had no Access application at all. It was a leftover RDP route to a remote machine, open to anyone who found the hostname. I’d replaced it with Guacamole long ago and forgotten to remove it. Deleted.
• The device enrollment policy was far too broad. My admin panels require a WARP-enrolled device, but enrollment itself allowed “any email ending in” several domains, including ones where other people have mailboxes. Anyone covered could have enrolled a phone and passed the check. It’s now restricted to my own email address only.

In the newer Cloudflare One dashboard, enrollment permissions live under Team & Resources → Devices → Device profiles → Management → Device enrollment permissions. Every Include rule is an OR, so one broad line opens the whole thing.

Three lanes

To make decisions simpler, I sorted every hostname into one of three lanes:

Lane 1 exists because native apps (Home Assistant, Bitwarden, Immich, ntfy) can’t complete Cloudflare’s browser login. For those, the app’s own authentication plus a WAF is the realistic option.

Gating n8n without breaking webhooks

n8n was the interesting Lane 2 case: I want the editor behind a login, but external services need to hit its webhooks unauthenticated. Cloudflare Access applies the most specific matching application, so the answer is two apps:

• n8n.example.com/api with a Bypass – Everyone policy, for my webhook path
• n8n.example.com with my email policy, for everything else

Create the bypass app first, so webhooks never break. If you use n8n Forms or Wait nodes later, their paths (/form, /webhook-waiting) need bypass entries too.

A second lock on Guacamole

Guacamole holds the keys to remote machines, so it got TOTP on top of Access. The official 1.6.0 image has it built in, and it’s two environment variables on the guacamole service:

      TOTP_ENABLED: "true"
      TOTP_ISSUER: My Guacamole

The next login shows a QR code to enroll. Two tips: keep a logged-in session open in another tab while you test, and store the TOTP seed somewhere other than the vault holding the Guacamole password.

Part 5: Closing the front door

Here’s the uncomfortable truth about “orange cloud plus port forward”: Cloudflare only hides your IP from DNS lookups. Ports 80 and 443 still accept connections from anyone. Scanners find home IPs quickly, and once someone connects directly with the right Host header, they skip Cloudflare’s WAF and every Access policy you’ve set up.

There are two fixes. You can lock the port forward to Cloudflare’s published IP ranges, or you can stop forwarding ports altogether and bring everything in through a Cloudflare Tunnel. I went with the tunnel: no listening ports at all, and no dependency on DDNS or a stable public IP.

Why keep NPMplus behind the tunnel?

Worth flagging: the NPMplus README itself advises against putting Cloudflare’s proxy or tunnel in front of it, for reasons like Cloudflare decrypting traffic, overriding headers, and its 100 MB upload limit. Every one of those already applied to me, because I was orange-clouded. The tunnel only changes how Cloudflare reaches NPMplus, not whether it sits in the middle.

I chose tunnel → NPMplus → services rather than pointing the tunnel straight at each app, because NPMplus still earns its place:

• CrowdSec AppSec inspects every request.
• One place for routing. Each tunnel route is identical, and NPMplus does the real work.
• HTTPS at home. DNS rewrites in AdGuard Home point my hostnames straight at NPMplus on the LAN, with valid certificates and no trip out to Cloudflare. This also sidesteps the 100 MB limit for Immich uploads at home.

A new tunnel

Rather than reconfigure the existing tunnel in place, I built the migration on a brand-new one. The old tunnel kept serving the admin panels untouched while I moved and tested everything else, so nothing went down along the way. Once it was empty, I deleted it. The new connector keeps its token in Dockge’s .env file instead of in the compose file:

services:
  cloudflared:
    image: cloudflare/cloudflared:latest
    container_name: cloudflared
    restart: unless-stopped
    command: tunnel --no-autoupdate run
    network_mode: host
    environment:
      - TUNNEL_TOKEN=${TUNNEL_TOKEN}

network_mode: host lets the tunnel reach NPMplus on 127.0.0.1:443 and the LAN admin devices directly. Running the old and new connectors side by side meant nothing broke while I moved things over.

The route settings that matter

Every public hostname gets the same route:

Real visitor IPs through the tunnel

I expected to need custom nginx config here, but didn’t. NPMplus ships with real_ip_header X-Forwarded-For, real_ip_recursive on, and trusts the private ranges including 127.0.0.0/8. Cloudflare puts the visitor’s IP into X-Forwarded-For, so nginx walks past the trusted local hop and lands on the real visitor. One test hostname and a grep of the access log confirmed it before I moved anything else.

Migrating, one hostname at a time

For each hostname: delete its old CNAME (Cloudflare won’t let the tunnel create a record while one exists), add the tunnel route, then test from my phone on mobile data. About 25 hostnames later, everything was on the tunnel, with roughly a minute of downtime each.

Some practical notes:

• Test from outside your network. Split DNS at home means home testing never touches Cloudflare. And if your mobile carrier caches the old “doesn’t exist” answer, setting Android’s Private DNS to one.one.one.one gets around it.
• Bare domain and www need one route each.
• Access apps follow the hostname, not the path traffic takes to reach it, so existing gates kept working through the move.

Admin panels

Proxmox, Proxmox Backup Server and TrueNAS now also go through NPMplus. That means CrowdSec inspects them and I get valid certificates at home. They stay behind the WARP-only Access policy, and I moved them onto my business domain while I was at it. The Proxmox console works through NPMplus because it handles websockets automatically.

The one exception is the NPMplus admin page itself. It goes straight from the tunnel to port 81, not through NPMplus’s proxying, because if NPMplus ever breaks I still need a way in to fix it.

To keep these hostnames protected throughout, I added each new name to its Access application before creating its route. A hostname should never exist publicly, even for a minute, without its gate.

Closing the ports

With every public record moved to the tunnel, I searched each DNS zone for the old DDNS name, found nothing left, then disabled (not deleted) the 80/443 dst-nat rules on the MikroTik. Disabling is a one-click rollback if something breaks.

Two final tests: the sites all still loaded from mobile data, and hitting my home IP directly on 80 or 443 now just times out.

One thing I deliberately kept: MikroTik’s DDNS. Nothing web-facing uses it anymore, but a site-to-site WireGuard peer still connects to that name.

Lessons learned

• Read your proxy’s access log once in a while. If every client IP is Cloudflare, your WAF is blind.
• Self-hosted Git should have registration disabled from day one.
• Orange-clouding hides your IP. It doesn’t close your ports.
• Audit Access against your routes. Leftover routes are where the holes hide.
• Every Include rule in a Zero Trust policy is an OR. Keep them narrow.
• Moving a service? Grep for its old IP in firewall rules, routes and VPN peers first.
• Keep secrets out of compose files. Put tokens in a .env file so they don’t end up in shared configs or screenshots.
• Verify backups before deleting anything, and keep an off-box copy.

What’s next

Next up is moving from Dockge to Dockhand for centralised stack management across all the LXCs. The compose files are already reviewed. The main job is converting relative ./ bind mounts to absolute paths, since Dockhand’s remote agent keeps compose files in its own folder. That’ll be its own post.