rpert/log_analysis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial cross-server log inventory + anomaly scan

Browse Source 
- 10 hosts (mo1, ams, ams2, ro1, ca1, ca2, ca3, fr1, sony, termux)
- discover-logs.sh: portable inventory (Linux/FreeBSD/Termux)
- scan-anomalies.sh: ERROR/WARN/CRITICAL counts + journalctl + kubectl
- run-all.sh: parallel SSH fan-out
- build-summary.py: aggregates into reports/SUMMARY.md
- 5 HIGH-severity findings identified on ro1 (apache scanner traffic, mount_monitor warnings)
This commit is contained in:
rpert
2026-04-10 21:49:17 +00:00
parent cabf4c587f
commit e96a8b03fc
26 changed files with 1636 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
reports/SUMMARY.md Normal file
View File

@@ -0,0 +1,87 @@
# Cross-Server Log Inspection — Summary
_Generated: 2026-04-10T21:49:07+00:00_
## Coverage
| Host | Inventory entries | Status | Top log dirs |
|------|-------------------:|--------|--------------|
| ams | 31 | ok | /var/log/borg-backup.log (5.9M), /var/log/auth.log (612.1K), /var/log/utx.log.1 (468.9K) |
| ams2 | 73 | ok | /var/log/auth.log (648.3K), /var/log/messages (647.8K), /var/log/daemon.log (646.9K) |
| ca1 | 92 | ok | /var/log/syslog (8.6M), /var/log/kern.log.1 (7.3M), /var/log/ufw.log.1 (7.3M) |
| ca2 | 48 | ok | /var/log/ufw.log.1 (3.7M), /var/log/auth.log.1 (3.3M), /var/log/auth.log (1.6M) |
| ca3 | 9 | ok | /var/log/dpkg.log (134.3K), /var/log/apt (85.1K), /var/log/syslog (28.1K) |
| fr1 | 106 | ok | /var/log/syslog (37.7M), /var/log/kern.log.1 (7.3M), /var/log/ufw.log.1 (7.2M) |
| mo1 | 50 | ok | /var/log/syslog (64.4M), /var/log/rclone-media.log (17.5M), /var/log/syslog.3.gz (6.9M) |
| ro1 | 59 | ok | /var/log/webmail-ssl-access.log (23.8M), /var/log/borg-backup.log (12.7M), /var/log/httpd (9.6M) |
| sony | 128 | ok | /var/log/borg (4.8M), /var/log/apt (261.3K), /var/log/installer (237.7K) |
| termux | 29 | ok | /data/data/com.termux (45.9M) |
## Top 25 largest log files (cluster-wide)
| Host | Path | Size | Mtime | Service |
|------|------|-----:|-------|---------|
| mo1 | `/var/log/syslog` | 64.4M | 2026-04-10 21:46:09 | syslog |
| fr1 | `/var/log/syslog` | 37.7M | 2026-04-10 21:46:28 | syslog |
| termux | `/data/data/com.termux/files/usr/var/log/borg/borg.log` | 36.2M | 2024-06-30 23:05:56 | borg |
| ro1 | `/var/log/webmail-ssl-access.log` | 23.8M | | webmail-ssl-access.log |
| mo1 | `/var/log/rclone-media.log` | 17.5M | 2026-04-10 21:45:33 | rclone-media.log |
| ro1 | `/var/log/borg-backup.log` | 12.7M | | borg-backup.log |
| ro1 | `/var/log/httpd/i47i.tk-access.log` | 9.3M | | httpd |
| ca1 | `/var/log/syslog` | 8.6M | 2026-04-10 21:46:08 | syslog |
| ca1 | `/var/log/kern.log.1` | 7.3M | 2026-04-04 23:59:51 | kern.log.1 |
| fr1 | `/var/log/kern.log.1` | 7.3M | 2026-04-05 00:00:01 | kern.log.1 |
| ca1 | `/var/log/ufw.log.1` | 7.3M | 2026-04-04 23:59:51 | ufw.log.1 |
| fr1 | `/var/log/ufw.log.1` | 7.2M | 2026-04-05 00:00:01 | ufw.log.1 |
| mo1 | `/var/log/syslog.3.gz` | 6.9M | 2026-03-22 00:00:04 | syslog.3.gz |
| ro1 | `/var/log/redis/redis.log` | 6.2M | | redis |
| fr1 | `/var/log/kern.log` | 6.0M | 2026-04-10 21:46:25 | kern.log |
| fr1 | `/var/log/ufw.log` | 6.0M | 2026-04-10 21:46:25 | ufw.log |
| ca1 | `/var/log/kern.log` | 6.0M | 2026-04-10 21:45:55 | kern.log |
| ca1 | `/var/log/ufw.log` | 6.0M | 2026-04-10 21:45:55 | ufw.log |
| ams | `/var/log/borg-backup.log` | 5.9M | | borg-backup.log |
| ro1 | `/var/log/httpd-error.log` | 5.5M | | httpd-error.log |
| fr1 | `/var/log/postfix.log` | 5.2M | 2026-02-02 23:37:02 | postfix.log |
| mo1 | `/var/log/auth.log.1` | 4.3M | 2026-04-05 00:00:03 | auth.log.1 |
| ro1 | `/var/log/freedns-ssl-access.log` | 4.2M | | freedns-ssl-access.log |
| fr1 | `/var/log/syslog.2.gz` | 3.8M | 2026-03-29 00:00:00 | syslog.2.gz |
| ca2 | `/var/log/ufw.log.1` | 3.7M | 2026-03-18 04:08:03 | ufw.log.1 |
## Anomalies — files with errors or excessive warnings
| Host | Severity | Errors | Warns | Size | Path |
|------|----------|-------:|------:|-----:|------|
| ro1 | **HIGH** | 72 | 0 | 1.3M | `/var/log/freedns-ssl-error.log` |
| ro1 | **HIGH** | 62 | 0 | 27.5K | `/var/log/webmail-ssl-error.log` |
| ro1 | **HIGH** | 51 | 0 | 391.4K | `/var/log/httpd/i47i.tk-error.log` |
| ro1 | **HIGH** | 7 | 3614 | 1.0M | `/var/log/mount_monitor.log.old` |
| ro1 | **HIGH** | 0 | 1808 | 514.3K | `/var/log/mount_monitor.log` |
| ams | **MED** | 21 | 0 | 5.9M | `/var/log/borg-backup.log` |
| ro1 | **MED** | 0 | 886 | 500.3K | `/var/log/messages` |
| ro1 | **LOW** | 6 | 0 | 3.4M | `/var/log/rclone_1fichier.log` |
| ro1 | **LOW** | 5 | 0 | 12.7M | `/var/log/borg-backup.log` |
| ro1 | **LOW** | 3 | 0 | 2.4K | `/var/log/manual-upgrades/upgrade-2026-04-05_0400.log` |
| ams | **LOW** | 1 | 0 | 53.9K | `/var/log/debug.log.0.bz2` |
| ams2 | **LOW** | 1 | 0 | 259.3K | `/var/log/borg/cron.log` |
## systemd journal error volume (24h)
| Host | journalctl -p err lines |
|------|------------------------:|
| ams | 0 |
| ams2 | 0 |
| ca1 | 1 |
| ca2 | 1 |
| ca3 | 2 |
| fr1 | 1 |
| mo1 | 37 |
| ro1 | 0 |
| sony | 100 |
| termux | 0 |
## Recommendations
- **Investigate 5 HIGH-severity log file(s) immediately** — see table above. These have either ≥50 error lines or ≥1000 warning lines in the last 7 days.
- **Sparse inventories on ca3, termux** — these likely require sudo to enumerate /var/log fully. Re-run discovery as root for a complete picture (the runner can be extended to use `sudo -n` on Linux hosts as it already does on FreeBSD).
- Re-run `./scripts/run-all.sh` on a schedule (cron / systemd timer) and commit the diff to track regressions over time.
- Consider centralising logs (Loki / Vector → VictoriaLogs on mo1) so this scan becomes a single query rather than 10 SSH fan-outs.