Self Hosted (part 4) : Logs
Intro
This post will show how to make journald logs persist on the hard drive and how to rotate SELinux audit logs.
Persist journald logs
By default, journald will not persist logs on the hard drive. The logs are saved in the volatile memory (/run/log/journald) and are lost on system reboot which makes debugging harder when system reboots due to errors.
There are two ways to make journald logs persistent:
- Create a directory /var/log/journal. Once the directory is created, journald will automatically start writing logs to the directory.
mkdir -p /var/log/journal
- Edit /etc/systemd/journald.conf file and set
Storage
topersistent
. Do it manally with vim or nanoor use[Journal] Storage=persistent
sed
commandFinally, restart the journald service withsed -i 's/#Storage=auto/Storage=persistent/' /etc/systemd/journald.conf
NOTE: Make sure the key and value are not in quotes otherwise it will not work and errors will be shown when journald starts.systemctl restart systemd-journald.service
MORE: https://access.redhat.com/solutions/696893
Rotate journald logs
By defaut, journald will rotate logs and use a maximum of 10% of partition and leave at least 15% free space for other uses, both capped at 4GB.
I will leave my settings at defaults, but if needed these values can be changed with SystemMaxUse
and SystemKeepFree
in /etc/systemd/journal.conf
MORE: https://www.freedesktop.org/software/systemd/man/journald.conf.html
Rotate SELinux audit logs
By default, the SELinux audit logs will not be rotated and sooner or later there will not be enough space for more logs. Depending on the low disk space action set in the config file, the system might stop booting up. The error messages in journald logs will be something like:
Audit daemon is low on disk space for logging
The audit daemon is now halting the system
To rotate audit logs based on size, set max_log_file_action
to rotate
in /etc/audit/auditd.confg
max_log_file_action = rotate
To implement time based instead of size based log rotation: https://access.redhat.com/solutions/661603
Thank you for reading. Check out the other parts in the series below.