Systemd is nowadays the init system adopted by almost all Linux distributions, from Red Hat Enterprise Linux to Debian and Ubuntu. One of the things that made Systemd the target of a lot of critics is that it tries to be a lot more than a simple init system and tries to re-invent some Linux subsystems.
The traditional logging system used on Linux, for example was rsyslog, a modern version of the traditional syslog. Systemd introduced its own logging system: it is implemented by a daemon, journald, which stores logs in binary format into a “journal”, which can be queried by the journalctl utility.
In this tutorial we will learn some parameters we can use to modify the journald daemon behavior, and some examples of how to query the journal and format the output resulting from said queries.
In this tutorial you will learn:
- How to change default journald settings
- How journald can coexist with syslog
- How to query the journal and some ways to format the queries output
Software requirements and conventions used
Category | Requirements, Conventions or Software Version Used |
---|---|
System | A Linux distribution using systemd (almost all do) |
Software | No specific software is needed |
Other | Root privileges to (eventually) change default configurations |
Conventions | # – linux-commands to be executed with root privileges either directly as a root user or by use of sudo command$ – linux-commands to be executed as a regular non-privileged user |
Journald configuration file
The behavior of the journald daemon can be modified by changing settings in its configuration file: /etc/systemd/journald.conf
. The direct modification of this file is not recommended; instead, we should create some separate configuration files containing the parameters we intend to change, save them with the .conf
extension, and place them inside the /etc/systemd/journald.conf.d
directory.
The files placed inside the /etc/systemd/journald.conf.d
directory have a bigger precedence than /etc/systemd/journald.conf
: they are sorted by their name in lexicographic order and parsed in that order, all after the main file. In case the same option setting exists in more than one file, the last one to be parsed will be effective.
The /etc/systemd/jourlnald.conf
file, by default, contains a commented list of options inside the [Journal]
stanza: they represent the default values used at compilation time (the content below is from a Fedora system):
[Journal] #Storage=auto #Compress=yes #Seal=yes #SplitMode=uid #SyncIntervalSec=5m #RateLimitIntervalSec=30s #RateLimitBurst=10000 #SystemMaxUse= #SystemKeepFree= #SystemMaxFileSize= #SystemMaxFiles=100 #RuntimeMaxUse= #RuntimeKeepFree= #RuntimeMaxFileSize= #RuntimeMaxFiles=100 #MaxRetentionSec= #MaxFileSec=1month #ForwardToSyslog=no #ForwardToKMsg=no #ForwardToConsole=no #ForwardToWall=yes #TTYPath=/dev/console #MaxLevelStore=debug #MaxLevelSyslog=debug #MaxLevelKMsg=notice #MaxLevelConsole=info #MaxLevelWall=emerg #LineMax=48K #ReadKMsg=yes #Audit=yes
Let’s see what is the meaning of some of those options, and how they can change the behavior of the journald daemon.
The “Storage” option
The first option we encounter in the file is Storage. This option controls where the journal data is stored. The default value used at compilation time here is auto
, but it’s possible to choose among:
- volatile
- persistent
- auto
- none
If we use volatile
as the value of this option, the journal data will be stored only in memory under /run/log/journal
(/run
is a tmpfs: its content is stored in memory), so it will not survive a system reboot.
If persistent
is used instead, the journal data will be stored on disk, under /var/log/journal
, which is created if doesn’t exist. If for some reason the disk is not writable, however, /run/log/journal
is used as a fallback.
The auto
value for the Storage
option, which here is used as default, works basically like persistent
in the sense that when it is used the journal data is stored under /var/log/journal
. The difference is that if the path doesn’t exist, it is not created, and logs will be stored only in memory.
Finally, if the none
value is used, all storage is turned off: while forwarding to other logging systems such as syslog will still work, all received data will be dropped.
The “Compress” option
The “compress” option controls whether if data exceeding the threshold of 512
bytes is compressed before being stored to the disk. This option accepts two types of values: a boolean as in the case above (yes
), or a number which sets the compression threshold itself. If the latter is provided, the compression is activated implicitly. The threshold value is, by default, express in bytes, but the K
, M
or G
suffixes can be used instead.
The “ForwardToSysLog” option
As already mentioned, in the pre-Systemd era, the logs where managed by the syslog
logging system (rsyslog
actually). This logging system is able to forward logs to many destinations, like text files, terminals, or even other machines on the network. Systemd implemented its own logging system, which is the object of this tutorial: journald.
The two systems can coexist (this is sometimes necessary since journald misses some features like centralized logging, or just because we, as administrators may like logs to be stored in text files instead of in binary format, so they can be manipulated with standard Unix tools).
This ForwardToSysLog
option takes a boolean value: if set to yes
, messages will be forwarded to the /run/systemd/journal/syslog
socket, where can be read by syslog
. This behavior can also be set at boot via the systemd.journald.forward_to_syslog
option.
Similar options can be used to forward messages to kmsg
(kernel log buffer), to console or to “wall” (sent as log messages to logged in users). Only the latter is set to yes
by default.
Querying the journal
The tool we can use to examine the system logs and query the systemd journal is journalctl
. If the command is called without further parameters, all the content of the journal is displayed. Fortunately, several strategies can be implemented to filter the logs. Let’s see some of them.
Filtering messages by units
One of the most useful options we can pass to journalctl
is -u
, which is the short version of --unit
. With this option we can filter the content of the journal so that only messages from the specific systemd-unit passed as the option argument is returned. For example, to display only messages coming from the NetworkManager.service
unit, we can run:
$ journalctl -u NetworkManager -- Logs begin at Wed 2020-07-01 21:47:23 CEST, end at Sat 2020-07-25 15:26:59 CEST. -- Jul 01 21:48:07 eru systemd[1]: Starting Network Manager... Jul 01 21:48:07 eru NetworkManager[1579]: <info> [1593632887.7408] NetworkManager (version 1.22.10-1.fc32) is starting... (for the first time) Jul 01 21:48:07 eru NetworkManager[1579]: <info> [1593632887.7413] Read config: /etc/NetworkManager/NetworkManager.conf Jul 01 21:48:07 eru systemd[1]: Started Network Manager.
Furthermore, a specific option is dedicated to filtering only kernel messages: -k
, which is the short form of --dmesg
.
Filtering logs by date
If we want to filter messages stored in the journal by date we can use two dedicated options: -S
(short for --since
) and -U
(short for --until
). Both options accept a date in the format YYYY-MM-DD hh:mm:ss
. The “time” part of the date can be omitted, and in that case 00:00:00
is assumed. Suppose we want to filter the logs starting from the current date; we would run the following command:
$ journalctl --since 2020-07-25
To further restrict logs with a time from 16:04:21
to 16:04:26
:
$ journalctl --since "2020-07-25 16:04:21" --until "2020-07-25 16:04:26"
A series of aliases also exist: they can be used instead of plain dates:
String | Meaning |
---|---|
“yesterday” | 00:00:00 of the day before the current one |
“today” | the current day |
“tomorrow” | the day after the current one |
“now” | the current time |
Displaying only the latest logs
If we launch the journalctl
command with the -f
(--follow
) option, we can visualize only the latest received logs, and still observe as new logs are appended to it (it is basically like calling tail
with the -f
option). On the other hand, if we just want to visualize the end of the journal we can use the -e
option (--pager-end
).
Formatting the journalctl output
The output we receive when using journalctl
can be easily formatted using a dedicated option: -o
, or its long version, --output
. When using this option we can specify among a series of “styles”. Among the (many) others:
- short
- verbose
- json-pretty
The short
format is the default: one line per entry is displayed in an output similar to that of traditional syslog:
Jul 01 21:48:07 eru systemd[1]: Starting Network Manager...
The verbose
format, instead, makes all the fields of the entry to be displayed:
Wed 2020-07-01 21:48:07.603130 CEST [s=d61cdf3710e84233bda460d931ebc3bb;i=6be;b=1c06b8c553624a5f94e1d3ef384fb50d;m=2e82666;t=5a966922b0155;x=6668aad5e895da03] PRIORITY=6 _BOOT_ID=1c06b8c553624a5f94e1d3ef384fb50d _MACHINE_ID=afe15f1a401041f4988478695a02b2bf _HOSTNAME=eru SYSLOG_FACILITY=3 SYSLOG_IDENTIFIER=systemd _UID=0 _GID=0 _TRANSPORT=journal _CAP_EFFECTIVE=3fffffffff CODE_FILE=src/core/job.c CODE_LINE=574 CODE_FUNC=job_log_begin_status_message JOB_TYPE=start MESSAGE_ID=7d4958e842da4a758f6c1cdc7b36dcc5 _PID=1 _COMM=systemd _EXE=/usr/lib/systemd/systemd _SYSTEMD_CGROUP=/init.scope _SYSTEMD_UNIT=init.scope _SYSTEMD_SLICE=-.slice _SELINUX_CONTEXT=system_u:system_r:init_t:s0 _CMDLINE=/usr/lib/systemd/systemd --switched-root --system --deserialize 34 MESSAGE=Starting Network Manager... JOB_ID=243 UNIT=NetworkManager.service INVOCATION_ID=6416439e51ff4543a76bded5984c6cf3 _SOURCE_REALTIME_TIMESTAMP=1593632887603130
The json-pretty
format displays the entries as JSON objects in a human-readable way. In this format the entries are separated by a newline:
{ "__REALTIME_TIMESTAMP" : "1593632887603541", "PRIORITY" : "6", "_SYSTEMD_UNIT" : "init.scope", "_SYSTEMD_CGROUP" : "/init.scope", "_UID" : "0", "_COMM" : "systemd", "_SYSTEMD_SLICE" : "-.slice", "_CAP_EFFECTIVE" : "3fffffffff", "_BOOT_ID" : "1c06b8c553624a5f94e1d3ef384fb50d", "_SELINUX_CONTEXT" : "system_u:system_r:init_t:s0", "__CURSOR" : "s=d61cdf3710e84233bda460d931ebc3bb;i=6be;b=1c06b8c553624a5f94e1d3ef384fb50d;m=2e82666;t=5a966922b0155;x=6668aad5e895da03", "_HOSTNAME" : "eru", "_PID" : "1", "MESSAGE_ID" : "7d4958e842da4a758f6c1cdc7b36dcc5", "CODE_FUNC" : "job_log_begin_status_message", "MESSAGE" : "Starting Network Manager...", "_EXE" : "/usr/lib/systemd/systemd", "__MONOTONIC_TIMESTAMP" : "48768614", "_TRANSPORT" : "journal", "SYSLOG_FACILITY" : "3", "UNIT" : "NetworkManager.service", "JOB_ID" : "243", "JOB_TYPE" : "start", "_GID" : "0", "CODE_FILE" : "src/core/job.c", "_MACHINE_ID" : "afe15f1a401041f4988478695a02b2bf", "_CMDLINE" : "/usr/lib/systemd/systemd --switched-root --system --deserialize 34", "SYSLOG_IDENTIFIER" : "systemd", "CODE_LINE" : "574", "INVOCATION_ID" : "6416439e51ff4543a76bded5984c6cf3", "_SOURCE_REALTIME_TIMESTAMP" : "1593632887603130" }
Conclusions
In this tutorial we approached journald the systemd daemon which implements the logging journal. This logging system is meant to be used instead of syslog which was the traditional system used on Linux. On many distributions, for a reason or another the two systems still coexists.
We saw what is the journald configuration file and what is the meaning of some important options that can be used to modify its behavior, and we learned how we can query the systemd journal with the journalctl utility. If you want to know more about journald and journalctl. I suggest you to read the respective manuals (man journald.conf
and man journalctl
are the commands you are searching for).