Introduction to the Systemd journal

Systemd is nowadays the init system adopted by almost all Linux distributions, from Red Hat Enterprise Linux to Debian and Ubuntu. One of the things that made Systemd the target of a lot of critics is that it tries to be a lot more than a simple init system and tries to re-invent some Linux subsystems.

The traditional logging system used on Linux, for example was rsyslog, a modern version of the traditional syslog. Systemd introduced its own logging system: it is implemented by a daemon, journald, which stores logs in binary format into a “journal”, which can be queried by the journalctl utility.

In this tutorial we will learn some parameters we can use to modify the journald daemon behavior, and some examples of how to query the journal and format the output resulting from said queries.

In this tutorial you will learn:

  • How to change default journald settings
  • How journald can coexist with syslog
  • How to query the journal and some ways to format the queries output

systemd-logo

Software requirements and conventions used

Software Requirements and Linux Command Line Conventions
Category Requirements, Conventions or Software Version Used
System A Linux distribution using systemd (almost all do)
Software No specific software is needed
Other Root privileges to (eventually) change default configurations
Conventions # – linux-commands to be executed with root privileges either directly as a root user or by use of sudo command
$ – linux-commands to be executed as a regular non-privileged user

Journald configuration file

The behavior of the journald daemon can be modified by changing settings in its configuration file: /etc/systemd/journald.conf. The direct modification of this file is not recommended; instead, we should create some separate configuration files containing the parameters we intend to change, save them with the .conf extension, and place them inside the /etc/systemd/journald.conf.d directory.

The files placed inside the /etc/systemd/journald.conf.d directory have a bigger precedence than /etc/systemd/journald.conf: they are sorted by their name in lexicographic order and parsed in that order, all after the main file. In case the same option setting exists in more than one file, the last one to be parsed will be effective.

The /etc/systemd/jourlnald.conf file, by default, contains a commented list of options inside the [Journal] stanza: they represent the default values used at compilation time (the content below is from a Fedora system):

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
#ReadKMsg=yes
#Audit=yes


Let’s see what is the meaning of some of those options, and how they can change the behavior of the journald daemon.

The “Storage” option

The first option we encounter in the file is Storage. This option controls where the journal data is stored. The default value used at compilation time here is auto, but it’s possible to choose among:

  • volatile
  • persistent
  • auto
  • none

If we use volatile as the value of this option, the journal data will be stored only in memory under /run/log/journal (/run is a tmpfs: its content is stored in memory), so it will not survive a system reboot.

If persistent is used instead, the journal data will be stored on disk, under /var/log/journal, which is created if doesn’t exist. If for some reason the disk is not writable, however, /run/log/journal is used as a fallback.

The auto value for the Storage option, which here is used as default, works basically like persistent in the sense that when it is used the journal data is stored under /var/log/journal. The difference is that if the path doesn’t exist, it is not created, and logs will be stored only in memory.

Finally, if the none value is used, all storage is turned off: while forwarding to other logging systems such as syslog will still work, all received data will be dropped.

The “Compress” option

The “compress” option controls whether if data exceeding the threshold of 512 bytes is compressed before being stored to the disk. This option accepts two types of values: a boolean as in the case above (yes), or a number which sets the compression threshold itself. If the latter is provided, the compression is activated implicitly. The threshold value is, by default, express in bytes, but the K, M or G suffixes can be used instead.

The “ForwardToSysLog” option

As already mentioned, in the pre-Systemd era, the logs where managed by the syslog logging system (rsyslog actually). This logging system is able to forward logs to many destinations, like text files, terminals, or even other machines on the network. Systemd implemented its own logging system, which is the object of this tutorial: journald.

The two systems can coexist (this is sometimes necessary since journald misses some features like centralized logging, or just because we, as administrators may like logs to be stored in text files instead of in binary format, so they can be manipulated with standard Unix tools).

This ForwardToSysLog option takes a boolean value: if set to yes, messages will be forwarded to the /run/systemd/journal/syslog socket, where can be read by syslog. This behavior can also be set at boot via the systemd.journald.forward_to_syslog option.

Similar options can be used to forward messages to kmsg (kernel log buffer), to console or to “wall” (sent as log messages to logged in users). Only the latter is set to yes by default.

Querying the journal

The tool we can use to examine the system logs and query the systemd journal is journalctl. If the command is called without further parameters, all the content of the journal is displayed. Fortunately, several strategies can be implemented to filter the logs. Let’s see some of them.

Filtering messages by units

One of the most useful options we can pass to journalctl is -u, which is the short version of --unit. With this option we can filter the content of the journal so that only messages from the specific systemd-unit passed as the option argument is returned. For example, to display only messages coming from the NetworkManager.service unit, we can run:

$ journalctl -u NetworkManager
-- Logs begin at Wed 2020-07-01 21:47:23 CEST, end at Sat 2020-07-25 15:26:59 CEST. --
Jul 01 21:48:07 eru systemd[1]: Starting Network Manager...
Jul 01 21:48:07 eru NetworkManager[1579]: <info>  [1593632887.7408] NetworkManager (version 1.22.10-1.fc32) is starting... (for the first time)
Jul 01 21:48:07 eru NetworkManager[1579]: <info>  [1593632887.7413] Read config: /etc/NetworkManager/NetworkManager.conf
Jul 01 21:48:07 eru systemd[1]: Started Network Manager.

Furthermore, a specific option is dedicated to filtering only kernel messages: -k, which is the short form of --dmesg.

Filtering logs by date

If we want to filter messages stored in the journal by date we can use two dedicated options: -S (short for --since) and -U (short for --until). Both options accept a date in the format YYYY-MM-DD hh:mm:ss. The “time” part of the date can be omitted, and in that case 00:00:00 is assumed. Suppose we want to filter the logs starting from the current date; we would run the following command:

$ journalctl --since 2020-07-25


To further restrict logs with a time from 16:04:21 to 16:04:26:

$ journalctl --since "2020-07-25 16:04:21" --until "2020-07-25 16:04:26"

A series of aliases also exist: they can be used instead of plain dates:

String Meaning
“yesterday” 00:00:00 of the day before the current one
“today” the current day
“tomorrow” the day after the current one
“now” the current time

Displaying only the latest logs

If we launch the journalctl command with the -f (--follow) option, we can visualize only the latest received logs, and still observe as new logs are appended to it (it is basically like calling tail with the -f option). On the other hand, if we just want to visualize the end of the journal we can use the -e option (--pager-end).

Formatting the journalctl output

The output we receive when using journalctl can be easily formatted using a dedicated option: -o, or its long version, --output. When using this option we can specify among a series of “styles”. Among the (many) others:

  • short
  • verbose
  • json-pretty

The short format is the default: one line per entry is displayed in an output similar to that of traditional syslog:

Jul 01 21:48:07 eru systemd[1]: Starting Network Manager...

The verbose format, instead, makes all the fields of the entry to be displayed:

Wed 2020-07-01 21:48:07.603130 CEST [s=d61cdf3710e84233bda460d931ebc3bb;i=6be;b=1c06b8c553624a5f94e1d3ef384fb50d;m=2e82666;t=5a966922b0155;x=6668aad5e895da03]
    PRIORITY=6
    _BOOT_ID=1c06b8c553624a5f94e1d3ef384fb50d
    _MACHINE_ID=afe15f1a401041f4988478695a02b2bf
    _HOSTNAME=eru
    SYSLOG_FACILITY=3
    SYSLOG_IDENTIFIER=systemd
    _UID=0
    _GID=0
    _TRANSPORT=journal
    _CAP_EFFECTIVE=3fffffffff
    CODE_FILE=src/core/job.c
    CODE_LINE=574
    CODE_FUNC=job_log_begin_status_message
    JOB_TYPE=start
    MESSAGE_ID=7d4958e842da4a758f6c1cdc7b36dcc5
    _PID=1
    _COMM=systemd
    _EXE=/usr/lib/systemd/systemd
    _SYSTEMD_CGROUP=/init.scope
    _SYSTEMD_UNIT=init.scope
    _SYSTEMD_SLICE=-.slice
    _SELINUX_CONTEXT=system_u:system_r:init_t:s0
    _CMDLINE=/usr/lib/systemd/systemd --switched-root --system --deserialize 34
    MESSAGE=Starting Network Manager...
    JOB_ID=243
    UNIT=NetworkManager.service
    INVOCATION_ID=6416439e51ff4543a76bded5984c6cf3
    _SOURCE_REALTIME_TIMESTAMP=1593632887603130


The json-pretty format displays the entries as JSON objects in a human-readable way. In this format the entries are separated by a newline:

{
        "__REALTIME_TIMESTAMP" : "1593632887603541",
        "PRIORITY" : "6",
        "_SYSTEMD_UNIT" : "init.scope",
        "_SYSTEMD_CGROUP" : "/init.scope",
        "_UID" : "0",
        "_COMM" : "systemd",
        "_SYSTEMD_SLICE" : "-.slice",
        "_CAP_EFFECTIVE" : "3fffffffff",
        "_BOOT_ID" : "1c06b8c553624a5f94e1d3ef384fb50d",
        "_SELINUX_CONTEXT" : "system_u:system_r:init_t:s0",
        "__CURSOR" : "s=d61cdf3710e84233bda460d931ebc3bb;i=6be;b=1c06b8c553624a5f94e1d3ef384fb50d;m=2e82666;t=5a966922b0155;x=6668aad5e895da03",
        "_HOSTNAME" : "eru",
        "_PID" : "1",
        "MESSAGE_ID" : "7d4958e842da4a758f6c1cdc7b36dcc5",
        "CODE_FUNC" : "job_log_begin_status_message",
        "MESSAGE" : "Starting Network Manager...",
        "_EXE" : "/usr/lib/systemd/systemd",
        "__MONOTONIC_TIMESTAMP" : "48768614",
        "_TRANSPORT" : "journal",
        "SYSLOG_FACILITY" : "3",
        "UNIT" : "NetworkManager.service",
        "JOB_ID" : "243",
        "JOB_TYPE" : "start",
        "_GID" : "0",
        "CODE_FILE" : "src/core/job.c",
        "_MACHINE_ID" : "afe15f1a401041f4988478695a02b2bf",
        "_CMDLINE" : "/usr/lib/systemd/systemd --switched-root --system --deserialize 34",
        "SYSLOG_IDENTIFIER" : "systemd",
        "CODE_LINE" : "574",
        "INVOCATION_ID" : "6416439e51ff4543a76bded5984c6cf3",
        "_SOURCE_REALTIME_TIMESTAMP" : "1593632887603130"
}

Conclusions

In this tutorial we approached journald the systemd daemon which implements the logging journal. This logging system is meant to be used instead of syslog which was the traditional system used on Linux. On many distributions, for a reason or another the two systems still coexists.

We saw what is the journald configuration file and what is the meaning of some important options that can be used to modify its behavior, and we learned how we can query the systemd journal with the journalctl utility. If you want to know more about journald and journalctl. I suggest you to read the respective manuals (man journald.conf and man journalctl are the commands you are searching for).



Comments and Discussions
Linux Forum