Advanced Logging and Auditing on Linux

One of the most important aspects of Linux system administration is to set up logging of critical and important events, and regularly audit your server for weaknesses or errors. Since Linux servers are typically powered on 24/7, administrators must rely on logging and auditing tools to help them keep track of what is happening on the system throughout the day. These useful tools can collect and summarize mass amounts of data, notify the sysadmin when a critical event occurs, and identify trends and potential issues.

In this tutorial, we will go over some techniques for advanced logging and auditing on Linux systems. Linux comes with a lot of logging tools built in by default, but there are also a lot of third party system monitors which can aid us with additional features, such as the ability to send IT administrators an alert when the server’s resources are being overwhelmed.

In this tutorial you will learn:

How to use journald for log parsing
Basic SELinux and AppArmor commands and configuration
What is ELK, Logwatch, and logrotate

Software Requirements and Linux Command Line Conventions
Category	Requirements, Conventions or Software Version Used
System	Any Linux distro
Software	journald, SELinux, AppArmor, ELK, Logwatch, logrotate
Other	Privileged access to your Linux system as root or via the `sudo` command.
Conventions	# – requires given linux commands to be executed with root privileges either directly as a root user or by use of `sudo` command $ – requires given linux commands to be executed as a regular non-privileged user

Techniques for Advanced Logging and Auditing in Linux

We will go over various tools and techniques to log and audit your Linux system below. Some of these tools come installed by default, some must be installed manually, and some are made for specific Linux distributions. The right tool for you will depend on which Linux distro you are using, and the depth of logging and auditing you would like to do. One or more of the tools below will surely fit your criteria.

journald / rsyslog / syslog

journald is now ubiquitous across Linux systems because of its integration with systemd. Of course, not all Linux distros use systemd, but the vast majority do. rsyslog and syslog are also viable logging tools, but we have grouped them together in this section with journald because we feel that journald has largely replaced rsyslog and syslog in many situations.

NOTE
journald is not available on systems with a different init system, such as SysV init. In these situations, installing rsyslog or syslog-ng would be recommended for a general purpose logging tool that collects everything in a centralized location.

Some of journald’s advantages include:

Indexed log files: looking up specific events is much faster and easier than in the plain text files of syslog
Access control on log files: users can only view pertinent log files, and root can see all of them
Log filtering: easy to find events based on time or level (critical vs warning, etc)

The journalctl command can be used to view all of the logs collected by systemd. This includes logs related to the system’s kernel, initrd, various services and applications, as well as systemd itself. The journalctl command makes querying all of these logs pretty painless, since systemd gathers and stores all these various logs in a central location for administrators to view.

You will also find that other security tools like SELinux and AppArmor work hand in hand with journald, as it will store logs that are generated by these other tools. Mastering journald and some of the journalctl Linux command syntax will also help you in becoming more proficient with these other security tools and being able to quickly identify relevant log entries.

Journald can and will log millions of entries in very little time, so it is not generally recommended to run the journalctl command by itself. Instead, you can use the --since and --until options can be used to help you isolate relevant logs that were logged during a certain timeframe. For example, to see all of the logs since yesterday:
```
$ journalctl --since yesterday
```
You can also use the options in conjunction with each other.
```
$ journalctl --since yesterday --until "2 hours ago"
```
To see entries that have been logged for a particular system service, use the -u flag. For example, to see all entries logged by Apache:
```
$ journalctl -u apache2.service
```
To see only kernel related messages, use the -k option.
```
$ journalctl -k
```
Since there are so many log entries, it can be helpful to only see those of a certain priority. The highest priority is level 0, and the lowest is level 7. The log levels are as follows:
```
0: emergency
1: alert
2: critical
3: error
4: warning
5: notice
6: info
7: debug
```
Use the -p option to see logs of a certain level, plus any above it in priority. In this example, we will go with level 3, which is any entry marked as an error, critical, alert, or emergency.
```
$ journalctl -p 3
```

More info: Introduction to the Systemd journal

SELinux

SELinux, which stands for Security Enhanced Linux, is an extra layer of security control built for Linux systems. The original version of SELinux was developed by the NSA. Other key contributors include Red Hat, which has enabled it by default in their own RHEL and its derivative Linux distributions.

SELinux (Security Enhanced Linux) is an implementation of a Mandatory Access Control permission system (MAC) in the Linux kernel. This type of access control differs from Discretionary Access Control systems (DAC) like ACLs and standard Unix ugo/rwx permissions, in how the access to a resource is provided. In the case of MAC is not the owner of a resource the one who decides who and how can access it: this access is based on the relationships between domains and labels, dictated by a policy and enforced at the kernel level. It’s important to say that SELinux enforced rules and standard system permissions are not mutually exclusive, and the former are implemented after the latter.

Check out our other tutorial for information on SELinux Concepts and Management. This will show you how to assign SELinux policies to processes and files.

To check the current status of SELinux:

$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

We can use the ausearch command to search for recent SELinux log entries. For example, to search for Access Vector Cache (AVC) messages (the kind that SELinux issues when an action is denied by it), we can execute:
```
$ sudo ausearch -m AVC -ts today
```
Note: The -ts option is “timestamp.” Another valid argument would be “recent” to see logs from the last 10 minutes. To see more relevant information, take the time stamp revelead by ausearch and search the journald logs as shown in the previous section.
We can also use sealert to see more information about a specific SELinux logged message. Just replace log_id with the ID number of the log you wish to inspect:
```
$ sudo sealert -l [log_ID]
```

AppArmor

AppArmor is another type of Mandatory Access Control permission system (MAC) commonly implemented in distributions like Ubuntu Linux. AppArmor works by assigning “profiles” to processes, networking ports, files, and users. AppArmor allows users to audit and log what entities are attempting access to confined resources.

Check the current status of AppArmor:
```
$ sudo apparmor_status
```
Possible modes are enforced, complain, and unconfined.
Put a profile into complain mode (issues warnings but takes no action):
```
$ sudo aa-complain /path/to/binary
```
Put a profile into enforce mode (generates logs but also enforces access control policies):
```
$ sudo aa-enforce /path/to/binary
```
All log messages generated by AppArmor can be seen in the /var/log/dmesg log file.
```
$ cat /var/log/dmesg | grep apparmor
```
As discussed earlier, it is more common and efficient to use journald (the journalctl command) to view relevant AppArmor log entries.

Elasticsearch, Logstash, and Kibana (ELK)

ELK is the answer to managing large amounts of log data on Linux servers. The ELK stack combines Elasticsearch, Logstash, and Kibana, which are open source tools that work in tandem to provide you with the ability to manage log data from a convenient graphical web interface. These three tools are developed by Elastic and specifically designed to work together.

ELK offers users security analysis, performance metrics, logs of critical and important events, and other useful data all from a web interface. You can install a local Apache or NGINX server to present the data, and then access it from localhost. Setting it up is a multi-step process and will vary by distribution, but here is how to install ELK on Ubuntu.

Logwatch

Logwatch works by allowing administrators to set up log parsing for particular files, and configuring alerts for certain events. These alerts can be sent to the administrator via email, so you do not need worry about constantly sifting through log files or aggregate data to manually check for any issues.

Logwatch is a simple implementation and ideal for medium sized servers and infrastructures that do not need a robust solution like ELK, but still want a way to notify staff on important events.

Logrotate

Logrotate is not a solution in and of itself, but works with other logging services as a means to rotate and compress log files. It is built into the Linux kernel already, and system services or third party applications have the option to utilize it. Without the logrotate function, log files would grow unwieldy in size and not be very useful when an administrator actually needs to peak inside of one.

You may find yourself working with logrotate quite regularly, especially if you plan to pasrse the log files for a variety of different services. It is definitely recommended that you familiarize yourself with the logrotate Linux command.

Closing Thoughts

In this tutorial, we learned about advanced logging and auditing techniques on a Linux system. We saw how journald is the most common way to parse indexed log data, and the journalctl command can be used to look for relevant entries. SELinux and AppArmor are the two main access control programs, implemented on a variety of Linux distros. Finally, ELK and Logwatch enable administrators to stay on top of large amount of log data in order to track critical events and make sense of information.