Skip to content

Commit debdfbc

Browse files
thewismitclaude
thewismit
and
claude
committed
v2.0.0: expanded remediation (17 mappings, 6 playbooks), 23 alert rules, fleet-patch pipeline
- Remediation bridge: 4 → 17 alert-to-playbook mappings with label_vars, extra_vars, defer strategy - New playbooks: host-recovery.yml (Proxmox VM/LXC restart), remount-nfs.yml (NFS recovery) - Alert rules: parameterized Jinja2 template covering 7 categories (host, container, TLS, backup, infra, network, services) - Fleet patching: 5-stage pipeline (audit, snapshot, patch, smoke, rollback) with Proxmox integration - Updated vars, README, and CHANGELOG Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 26f10e9 commit debdfbc

11 files changed

Lines changed: 1072 additions & 17 deletions

‎CHANGELOG.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,34 @@ Format: `## vX.Y.Z — YYYY-MM-DD`
66

77
---
88

9+
## v2.0.0 — 2026-03-12
10+
11+
Major expansion: alert rules, fleet patching, and full remediation coverage.
12+
13+
### Added
14+
15+
**Alert rules**
16+
- `playbooks/deploy-alert-rules.yml` — deploy 23 Prometheus alert rules
17+
- `templates/alert-rules.yml.j2` — parameterized alert rules across 7 categories
18+
- Alert threshold variables in `group_vars/all/vars.yml`
19+
20+
**Fleet patching**
21+
- `playbooks/fleet-patch.yml` — 5-stage safe patching pipeline (audit, snapshot, patch, smoke, rollback)
22+
- `playbooks/vars/fleet-patch-vars.yml.example` — host VMID mapping and smoke test template
23+
24+
**Remediation playbooks**
25+
- `remediation/host-recovery.yml` — Proxmox VM/LXC auto-recovery for `HostDown` alerts
26+
- `remediation/remount-nfs.yml` — NFS mount recovery for `NFSMountUnavailable` alerts
27+
28+
### Changed
29+
30+
- Remediation bridge map expanded: 4 → 17 alert-to-playbook mappings
31+
- Added `label_vars` and `extra_vars` support to remediation mappings
32+
- Added `strategy: defer_to_timer` for alerts that shouldn't auto-retry (e.g. `ResticBackupFailed`)
33+
- README updated with alert rules, fleet patching, and expanded remediation docs
34+
35+
---
36+
937
## v1.0.0 — 2026-02-28
1038

1139
Initial public release.

‎README.md‎

Lines changed: 59 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -36,15 +36,19 @@ Prometheus ──alerts──▶ Alertmanager ──webhook──▶ Remediation
3636
| **Loki** | Log aggregation backend (Docker Compose, NFS storage) | `deploy-loki.yml` |
3737
| **Promtail** | Log shipper on every host → Loki | `deploy-promtail-fleet.yml` |
3838
| **Remediation bridge** | Alertmanager → Ansible auto-remediation | `deploy-remediation-bridge.yml` |
39+
| **Alert rules** | 23 Prometheus alert rules (parameterized) | `deploy-alert-rules.yml` |
40+
| **Fleet patching** | 5-stage safe patching pipeline (audit→snapshot→patch→smoke→rollback) | `fleet-patch.yml` |
3941

40-
**4 targeted remediation playbooks** — called automatically by the bridge or manually:
42+
**6 targeted remediation playbooks** — called automatically by the bridge or manually:
4143

4244
| Playbook | When it runs |
4345
|---------|-------------|
44-
| `remediation/restart-container.yml` | `ContainerDown` alert |
45-
| `remediation/service-restart.yml` | `SystemdServiceFailed` alert |
46-
| `remediation/cleanup-disk.yml` | `DiskSpaceHigh` alert |
47-
| `remediation/caddy-reload.yml` | `TLSCertExpiringSoon` alert |
46+
| `remediation/restart-container.yml` | `ContainerDown`, `ContainerOOM` alerts |
47+
| `remediation/service-restart.yml` | `SystemdServiceFailed`, `HighMemoryUsage`, `DNSResolutionFailed` alerts |
48+
| `remediation/cleanup-disk.yml` | `DiskSpaceWarning`, `DiskSpaceCritical` alerts |
49+
| `remediation/caddy-reload.yml` | `TLSCertExpiringSoon/Critical/Expired`, `TLSHandshakeFailed` alerts |
50+
| `remediation/host-recovery.yml` | `HostDown` alert — Proxmox VM/LXC auto-recovery |
51+
| `remediation/remount-nfs.yml` | `NFSMountUnavailable` alert — NFS mount recovery |
4852

4953
---
5054

@@ -182,6 +186,7 @@ Secrets go in `group_vars/all/vault.yml` (see `vault.yml.example` for all requir
182186
| `roles/common` | Passwordless sudo for ansible_user | [README](roles/common/README.md) |
183187
| `roles/restic` | Restic install, repo init, backup + prune | [README](roles/restic/README.md) |
184188
| `roles/restic_backup` | Systemd timer/service units for scheduled backups | [README](roles/restic_backup/README.md) |
189+
| `roles/server_hardening` | SSH hardening, UFW firewall, sysctl tuning | [README](roles/server_hardening/README.md) |
185190

186191
---
187192

@@ -215,6 +220,55 @@ See [`playbooks/REMEDIATION-BRIDGE.md`](playbooks/REMEDIATION-BRIDGE.md) for:
215220

216221
---
217222

223+
## Alert rules
224+
225+
23 Prometheus alert rules across 7 categories, all parameterized via `group_vars`:
226+
227+
| Category | Alerts |
228+
|----------|--------|
229+
| **Host health** | `HostDown`, `DiskSpaceWarning`, `DiskSpaceCritical`, `HighMemoryUsage`, `EnabledServiceInactive` |
230+
| **Container** | `ContainerDown`, `ContainerOOM`, `ContainerRestartLoop` |
231+
| **TLS** | `TLSCertExpiringSoon`, `TLSCertExpiringCritical`, `TLSCertExpired`, `TLSProbeFailure`, `TLSHandshakeFailed` |
232+
| **Backup** | `ResticBackupStale`, `ResticBackupFailed` |
233+
| **Infrastructure** | `NFSMountUnavailable`, `HighSwapUsage`, `HighLoadAverage`, `HighDiskIOWait` |
234+
| **Network** | `HTTPEndpointDown`, `DNSResolutionFailed` |
235+
| **Services** | `SystemdServiceFailed`, `SystemdTimerFailed` |
236+
237+
Deploy:
238+
```bash
239+
ansible-playbook playbooks/deploy-alert-rules.yml --check --limit docker-host
240+
ansible-playbook playbooks/deploy-alert-rules.yml --limit docker-host
241+
```
242+
243+
---
244+
245+
## Fleet patching
246+
247+
5-stage safe patching pipeline. Each stage is tag-gated — nothing runs unless you explicitly ask for it.
248+
249+
| Stage | Tag | What it does |
250+
|-------|-----|-------------|
251+
| **Audit** | `audit` | Dry-run `apt upgrade`, map packages to running services |
252+
| **Snapshot** | `snapshot` | Proxmox VM/LXC snapshot before patching |
253+
| **Patch** | `patch` | `apt safe` upgrade + automatic service restarts |
254+
| **Smoke** | `smoke` | Post-patch service health validation |
255+
| **Rollback** | `rollback` | Revert to pre-patch Proxmox snapshot |
256+
257+
```bash
258+
# Audit all hosts
259+
ansible-playbook playbooks/fleet-patch.yml --tags audit
260+
261+
# Full patch cycle on one host
262+
ansible-playbook playbooks/fleet-patch.yml --tags snapshot,patch,smoke --limit wiki-host
263+
264+
# Rollback if something broke
265+
ansible-playbook playbooks/fleet-patch.yml --tags rollback -e target_host=wiki-host -e rollback_date=20260312
266+
```
267+
268+
Requires `playbooks/vars/fleet-patch-vars.yml` — see `fleet-patch-vars.yml.example`.
269+
270+
---
271+
218272
## Security Notes
219273

220274
- All secrets encrypted via `ansible-vault` — no plaintext passwords anywhere

‎group_vars/all/vars.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,3 +77,31 @@ restic_exclude_paths:
7777
# ── Remediation bridge ────────────────────────────────────────────────────────
7878
ntfy_topic: "homelab-alerts"
7979
bridge_dir: "/opt/remediation-bridge"
80+
81+
# ── Alert thresholds (override per host in host_vars if needed) ─────────────
82+
# Used by templates/alert-rules.yml.j2
83+
disk_warning_threshold_pct: 20
84+
disk_critical_threshold_pct: 10
85+
memory_threshold_pct: 90
86+
swap_threshold_ratio: 0.8
87+
load_avg_threshold: 2.0
88+
disk_io_threshold: 0.9
89+
tls_warning_days: 14
90+
tls_critical_days: 7
91+
restic_stale_threshold_seconds: 129600 # 36 hours
92+
container_down_threshold_seconds: 300 # 5 minutes
93+
94+
# Prometheus job names (match your prometheus.yml scrape config)
95+
node_exporter_job: "node_exporter_fleet"
96+
http_probe_job: "http_uptime"
97+
dns_probe_job: "dns_probe"
98+
https_probe_job: "blackbox-https"
99+
100+
# NFS mount pattern for NFSMountUnavailable alert
101+
nfs_mount_pattern: "/mnt/synology.*"
102+
103+
# Systemd services to monitor for EnabledServiceInactive alert
104+
monitored_services_regex: "promtail.service|restic-backup.timer"
105+
106+
# ── Fleet patching ──────────────────────────────────────────────────────────
107+
# Used by playbooks/fleet-patch.yml (vars file: playbooks/vars/fleet-patch-vars.yml)

‎playbooks/REMEDIATION-BRIDGE.md‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ route:
5757
# Optionally limit to specific alert groups:
5858
routes:
5959
- match_re:
60-
alertname: "ContainerDown|DiskSpaceHigh|SystemdServiceFailed|TLSCertExpiringSoon"
60+
alertname: "DiskSpaceWarning|DiskSpaceCritical|HighMemoryUsage|TLSCertExpiringSoon|TLSCertExpiringCritical|TLSCertExpired|ContainerDown|SystemdServiceFailed|ContainerOOM|NFSMountUnavailable|ResticBackupFailed|ResticBackupStale|DNSResolutionFailed|SystemdTimerFailed|HostDown|TLSHandshakeFailed|EnabledServiceInactive"
6161
receiver: remediation-bridge
6262
```
6363
@@ -81,10 +81,34 @@ mappings:
8181

8282
| Alert | Playbook | Cooldown |
8383
|-------|---------|---------|
84+
| `DiskSpaceWarning` | `remediation/cleanup-disk.yml` | 60 min |
85+
| `DiskSpaceCritical` | `remediation/cleanup-disk.yml` | 30 min |
86+
| `HighMemoryUsage` | `remediation/service-restart.yml` | 30 min |
87+
| `TLSCertExpiringSoon` | `remediation/caddy-reload.yml` | 360 min |
88+
| `TLSCertExpiringCritical` | `remediation/caddy-reload.yml` | 120 min |
89+
| `TLSCertExpired` | `remediation/caddy-reload.yml` | 60 min |
8490
| `ContainerDown` | `remediation/restart-container.yml` | 10 min |
85-
| `DiskSpaceHigh` | `remediation/cleanup-disk.yml` | 60 min |
8691
| `SystemdServiceFailed` | `remediation/service-restart.yml` | 15 min |
87-
| `TLSCertExpiringSoon` | `remediation/caddy-reload.yml` | 360 min |
92+
| `ContainerOOM` | `remediation/restart-container.yml` | 15 min |
93+
| `NFSMountUnavailable` | `remediation/remount-nfs.yml` | 30 min |
94+
| `ResticBackupFailed` | *(defer to timer)* | 60 min |
95+
| `ResticBackupStale` | `remediation/service-restart.yml` | 120 min |
96+
| `DNSResolutionFailed` | `remediation/service-restart.yml` | 30 min |
97+
| `SystemdTimerFailed` | `remediation/service-restart.yml` | 30 min |
98+
| `HostDown` | `remediation/host-recovery.yml` | 30 min |
99+
| `TLSHandshakeFailed` | `remediation/caddy-reload.yml` | 120 min |
100+
| `EnabledServiceInactive` | `remediation/service-restart.yml` | 30 min |
101+
102+
## Remediation playbooks
103+
104+
| Playbook | Purpose |
105+
|----------|---------|
106+
| `remediation/restart-container.yml` | Restart a Docker container by name |
107+
| `remediation/cleanup-disk.yml` | Free disk space (journals, tmp, apt cache) |
108+
| `remediation/service-restart.yml` | Restart a systemd service or timer |
109+
| `remediation/caddy-reload.yml` | Reload Caddy to pick up renewed TLS certs |
110+
| `remediation/host-recovery.yml` | Start a VM/LXC via Proxmox API when host is unreachable |
111+
| `remediation/remount-nfs.yml` | Remount NFS shares from fstab |
88112

89113
## Cooldown behavior
90114

‎playbooks/deploy-alert-rules.yml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
---
2+
# Deploy Prometheus alert rules to the monitoring host.
3+
# Copies alert rule files from template, reloads Prometheus.
4+
# Run: ansible-playbook playbooks/deploy-alert-rules.yml --check --limit docker-host
5+
# Run: ansible-playbook playbooks/deploy-alert-rules.yml --limit docker-host
6+
7+
- name: Alert Rules | Deploy Prometheus alert rules
8+
hosts: docker-host
9+
become: true
10+
vars:
11+
prometheus_config_dir: "{{ synology_docker_mount }}/prometheus_config"
12+
prometheus_rules_dir: "{{ prometheus_config_dir }}/rules"
13+
prometheus_url: "http://localhost:9090"
14+
15+
tasks:
16+
17+
- name: PRE-FLIGHT | Confirm Prometheus is running
18+
tags: [deploy]
19+
ansible.builtin.uri:
20+
url: "{{ prometheus_url }}/-/healthy"
21+
status_code: 200
22+
timeout: 10
23+
register: prom_check
24+
failed_when: prom_check.status != 200
25+
when: not ansible_check_mode
26+
27+
- name: INSTALL | Create rules directory
28+
tags: [deploy]
29+
ansible.builtin.file:
30+
path: "{{ prometheus_rules_dir }}"
31+
state: directory
32+
mode: "0755"
33+
34+
- name: CONFIG | Deploy alert rules
35+
tags: [deploy, configure]
36+
ansible.builtin.template:
37+
src: "{{ playbook_dir }}/../templates/alert-rules.yml.j2"
38+
dest: "{{ prometheus_rules_dir }}/{{ item }}"
39+
mode: "0644"
40+
loop:
41+
- backup_alerts.yml
42+
- container_alerts.yml
43+
- host_alerts.yml
44+
- infrastructure_alerts.yml
45+
- network_alerts.yml
46+
- service_alerts.yml
47+
- tls_alerts.yml
48+
notify: Reload Prometheus
49+
50+
- name: VALIDATE | Check Prometheus config
51+
tags: [deploy, validate]
52+
ansible.builtin.uri:
53+
url: "{{ prometheus_url }}/-/healthy"
54+
status_code: 200
55+
timeout: 10
56+
register: prom_post
57+
when: not ansible_check_mode
58+
59+
- name: VALIDATE | Report
60+
tags: [deploy, validate]
61+
ansible.builtin.debug:
62+
msg: "Alert rules deployed to {{ prometheus_rules_dir }}. Prometheus {{ 'healthy' if prom_post.status | default(0) == 200 else 'needs check' }}."
63+
when: not ansible_check_mode
64+
65+
handlers:
66+
- name: Reload Prometheus
67+
ansible.builtin.uri:
68+
url: "{{ prometheus_url }}/-/reload"
69+
method: POST
70+
status_code: [200, 204]
71+
timeout: 10
72+
when: not ansible_check_mode

‎playbooks/deploy-remediation-bridge.yml‎

Lines changed: 86 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -264,28 +264,105 @@
264264
# playbook: path relative to ansible_dir
265265
# cooldown_minutes: min time between re-runs of the same alert+host combo
266266
# extra_vars: additional vars passed to the playbook (optional)
267+
# label_vars: map playbook var names to Prometheus label names (optional)
268+
# strategy: defer_to_timer — skip remediation, let systemd timer retry (optional)
267269
268270
mappings:
269271
270-
# Container exited / not running
271-
ContainerDown:
272-
playbook: remediation/restart-container.yml
273-
cooldown_minutes: 10
272+
# Disk < 20% free (warning)
273+
DiskSpaceWarning:
274+
playbook: remediation/cleanup-disk.yml
275+
cooldown_minutes: 60
274276
275-
# Disk > 85%
276-
DiskSpaceHigh:
277+
# Disk < 10% free (critical)
278+
DiskSpaceCritical:
277279
playbook: remediation/cleanup-disk.yml
280+
cooldown_minutes: 30
281+
282+
# Memory > 90%
283+
HighMemoryUsage:
284+
playbook: remediation/service-restart.yml
285+
cooldown_minutes: 30
286+
287+
# TLS cert expiring within 14 days
288+
TLSCertExpiringSoon:
289+
playbook: remediation/caddy-reload.yml
290+
cooldown_minutes: 360
291+
292+
# TLS cert expiring within 7 days (critical)
293+
TLSCertExpiringCritical:
294+
playbook: remediation/caddy-reload.yml
295+
cooldown_minutes: 120
296+
297+
# TLS cert has expired
298+
TLSCertExpired:
299+
playbook: remediation/caddy-reload.yml
278300
cooldown_minutes: 60
279301
302+
# Container exited / crashed
303+
ContainerDown:
304+
playbook: remediation/restart-container.yml
305+
cooldown_minutes: 10
306+
280307
# Systemd service failed
281308
SystemdServiceFailed:
282309
playbook: remediation/service-restart.yml
283310
cooldown_minutes: 15
311+
label_vars:
312+
service_name: name
284313
285-
# Caddy cert expiry < 7 days (should auto-renew, but just in case)
286-
TLSCertExpiringSoon:
314+
# Container OOM killed
315+
ContainerOOM:
316+
playbook: remediation/restart-container.yml
317+
cooldown_minutes: 15
318+
319+
# NFS mount dropped
320+
NFSMountUnavailable:
321+
playbook: remediation/remount-nfs.yml
322+
cooldown_minutes: 30
323+
324+
# Restic backup failed — defer to timer retry
325+
ResticBackupFailed:
326+
strategy: defer_to_timer
327+
cooldown_minutes: 60
328+
329+
# Restic backup timer stale
330+
ResticBackupStale:
331+
playbook: remediation/service-restart.yml
332+
cooldown_minutes: 120
333+
extra_vars:
334+
service_name: restic-backup.timer
335+
336+
# DNS resolution failed
337+
DNSResolutionFailed:
338+
playbook: remediation/service-restart.yml
339+
cooldown_minutes: 30
340+
extra_vars:
341+
service_name: named.service
342+
343+
# Systemd timer failed
344+
SystemdTimerFailed:
345+
playbook: remediation/service-restart.yml
346+
cooldown_minutes: 30
347+
label_vars:
348+
service_name: name
349+
350+
# Host unreachable — attempt VM/LXC start via Proxmox
351+
HostDown:
352+
playbook: remediation/host-recovery.yml
353+
cooldown_minutes: 30
354+
355+
# TLS handshake failing
356+
TLSHandshakeFailed:
287357
playbook: remediation/caddy-reload.yml
288-
cooldown_minutes: 360
358+
cooldown_minutes: 120
359+
360+
# Expected service enabled but not running
361+
EnabledServiceInactive:
362+
playbook: remediation/service-restart.yml
363+
cooldown_minutes: 30
364+
label_vars:
365+
service_name: name
289366
notify: Restart remediation-bridge
290367

291368
# -------------------------------------------------------------------------

0 commit comments

Comments
 (0)