rpert
e96a8b03fc
- 10 hosts (mo1, ams, ams2, ro1, ca1, ca2, ca3, fr1, sony, termux) - discover-logs.sh: portable inventory (Linux/FreeBSD/Termux) - scan-anomalies.sh: ERROR/WARN/CRITICAL counts + journalctl + kubectl - run-all.sh: parallel SSH fan-out - build-summary.py: aggregates into reports/SUMMARY.md - 5 HIGH-severity findings identified on ro1 (apache scanner traffic, mount_monitor warnings)
4.9 KiB
4.9 KiB
Cross-Server Log Inspection — Summary
Generated: 2026-04-10T21:49:07+00:00
Coverage
| Host | Inventory entries | Status | Top log dirs |
|---|---|---|---|
| ams | 31 | ok | /var/log/borg-backup.log (5.9M), /var/log/auth.log (612.1K), /var/log/utx.log.1 (468.9K) |
| ams2 | 73 | ok | /var/log/auth.log (648.3K), /var/log/messages (647.8K), /var/log/daemon.log (646.9K) |
| ca1 | 92 | ok | /var/log/syslog (8.6M), /var/log/kern.log.1 (7.3M), /var/log/ufw.log.1 (7.3M) |
| ca2 | 48 | ok | /var/log/ufw.log.1 (3.7M), /var/log/auth.log.1 (3.3M), /var/log/auth.log (1.6M) |
| ca3 | 9 | ok | /var/log/dpkg.log (134.3K), /var/log/apt (85.1K), /var/log/syslog (28.1K) |
| fr1 | 106 | ok | /var/log/syslog (37.7M), /var/log/kern.log.1 (7.3M), /var/log/ufw.log.1 (7.2M) |
| mo1 | 50 | ok | /var/log/syslog (64.4M), /var/log/rclone-media.log (17.5M), /var/log/syslog.3.gz (6.9M) |
| ro1 | 59 | ok | /var/log/webmail-ssl-access.log (23.8M), /var/log/borg-backup.log (12.7M), /var/log/httpd (9.6M) |
| sony | 128 | ok | /var/log/borg (4.8M), /var/log/apt (261.3K), /var/log/installer (237.7K) |
| termux | 29 | ok | /data/data/com.termux (45.9M) |
Top 25 largest log files (cluster-wide)
| Host | Path | Size | Mtime | Service |
|---|---|---|---|---|
| mo1 | /var/log/syslog |
64.4M | 2026-04-10 21:46:09 | syslog |
| fr1 | /var/log/syslog |
37.7M | 2026-04-10 21:46:28 | syslog |
| termux | /data/data/com.termux/files/usr/var/log/borg/borg.log |
36.2M | 2024-06-30 23:05:56 | borg |
| ro1 | /var/log/webmail-ssl-access.log |
23.8M | webmail-ssl-access.log | |
| mo1 | /var/log/rclone-media.log |
17.5M | 2026-04-10 21:45:33 | rclone-media.log |
| ro1 | /var/log/borg-backup.log |
12.7M | borg-backup.log | |
| ro1 | /var/log/httpd/i47i.tk-access.log |
9.3M | httpd | |
| ca1 | /var/log/syslog |
8.6M | 2026-04-10 21:46:08 | syslog |
| ca1 | /var/log/kern.log.1 |
7.3M | 2026-04-04 23:59:51 | kern.log.1 |
| fr1 | /var/log/kern.log.1 |
7.3M | 2026-04-05 00:00:01 | kern.log.1 |
| ca1 | /var/log/ufw.log.1 |
7.3M | 2026-04-04 23:59:51 | ufw.log.1 |
| fr1 | /var/log/ufw.log.1 |
7.2M | 2026-04-05 00:00:01 | ufw.log.1 |
| mo1 | /var/log/syslog.3.gz |
6.9M | 2026-03-22 00:00:04 | syslog.3.gz |
| ro1 | /var/log/redis/redis.log |
6.2M | redis | |
| fr1 | /var/log/kern.log |
6.0M | 2026-04-10 21:46:25 | kern.log |
| fr1 | /var/log/ufw.log |
6.0M | 2026-04-10 21:46:25 | ufw.log |
| ca1 | /var/log/kern.log |
6.0M | 2026-04-10 21:45:55 | kern.log |
| ca1 | /var/log/ufw.log |
6.0M | 2026-04-10 21:45:55 | ufw.log |
| ams | /var/log/borg-backup.log |
5.9M | borg-backup.log | |
| ro1 | /var/log/httpd-error.log |
5.5M | httpd-error.log | |
| fr1 | /var/log/postfix.log |
5.2M | 2026-02-02 23:37:02 | postfix.log |
| mo1 | /var/log/auth.log.1 |
4.3M | 2026-04-05 00:00:03 | auth.log.1 |
| ro1 | /var/log/freedns-ssl-access.log |
4.2M | freedns-ssl-access.log | |
| fr1 | /var/log/syslog.2.gz |
3.8M | 2026-03-29 00:00:00 | syslog.2.gz |
| ca2 | /var/log/ufw.log.1 |
3.7M | 2026-03-18 04:08:03 | ufw.log.1 |
Anomalies — files with errors or excessive warnings
| Host | Severity | Errors | Warns | Size | Path |
|---|---|---|---|---|---|
| ro1 | HIGH | 72 | 0 | 1.3M | /var/log/freedns-ssl-error.log |
| ro1 | HIGH | 62 | 0 | 27.5K | /var/log/webmail-ssl-error.log |
| ro1 | HIGH | 51 | 0 | 391.4K | /var/log/httpd/i47i.tk-error.log |
| ro1 | HIGH | 7 | 3614 | 1.0M | /var/log/mount_monitor.log.old |
| ro1 | HIGH | 0 | 1808 | 514.3K | /var/log/mount_monitor.log |
| ams | MED | 21 | 0 | 5.9M | /var/log/borg-backup.log |
| ro1 | MED | 0 | 886 | 500.3K | /var/log/messages |
| ro1 | LOW | 6 | 0 | 3.4M | /var/log/rclone_1fichier.log |
| ro1 | LOW | 5 | 0 | 12.7M | /var/log/borg-backup.log |
| ro1 | LOW | 3 | 0 | 2.4K | /var/log/manual-upgrades/upgrade-2026-04-05_0400.log |
| ams | LOW | 1 | 0 | 53.9K | /var/log/debug.log.0.bz2 |
| ams2 | LOW | 1 | 0 | 259.3K | /var/log/borg/cron.log |
systemd journal error volume (24h)
| Host | journalctl -p err lines |
|---|---|
| ams | 0 |
| ams2 | 0 |
| ca1 | 1 |
| ca2 | 1 |
| ca3 | 2 |
| fr1 | 1 |
| mo1 | 37 |
| ro1 | 0 |
| sony | 100 |
| termux | 0 |
Recommendations
- Investigate 5 HIGH-severity log file(s) immediately — see table above. These have either ≥50 error lines or ≥1000 warning lines in the last 7 days.
- Sparse inventories on ca3, termux — these likely require sudo to enumerate /var/log fully. Re-run discovery as root for a complete picture (the runner can be extended to use
sudo -non Linux hosts as it already does on FreeBSD). - Re-run
./scripts/run-all.shon a schedule (cron / systemd timer) and commit the diff to track regressions over time. - Consider centralising logs (Loki / Vector → VictoriaLogs on mo1) so this scan becomes a single query rather than 10 SSH fan-outs.