There are times when it’s useful to inspect what a running application is doing under the hood, and what system calls it is performing during its execution. To accomplish such a task on Linux, we can use the strace utility. In this article we will see how to install it and we will learn its basic usage.
In this tutorial you will learn:
- How to install strace
- How to use strace to trace system calls made by a process
- How to filter specifics system calls
- How to attach to an already running process
- How to generate a system call summary
Software requirements and conventions used
| Category | Requirements, Conventions or Software Version Used |
|---|---|
| System | Distribution-independent |
| Software | Strace |
| Other | Familiarity with the command line interface and Linux processes management |
| Conventions | # – linux-commands to be executed with root privileges either directly as a root user or by use of sudo command$ – linux-commands to be executed as a regular non-privileged user |
Installation
Although not installed by default, the strace utility is available in the official repositories of all the major Linux distributions; this means that we can install it very easily by using our favorite package manager.
If we are running on Fedora (or any other distribution in the Red Hat family), for example, we must use dnf:
$ sudo dnf install strace
If we are more comfortable using Debian, or Debian-based distributions such as Ubuntu or Linux Mint, we can use apt to achieve the same result:
$ sudo apt install strace
If Arch Linux is our distribution of choice, we can use pacman to install the application, which is available in the extra repository:
$ sudo pacman -S strace
With the software installed, we can proceed forward, and see some examples of its usage.
Introducing strace
As we already said, strace is a tool used to keep track of the system calls made by a running process and the signals received by it. System calls are the fundamental interface between an application and the Linux kernel; when we use strace, the name of the calls made by a process, along with their arguments and return values are displayed on stderr (standard error file descriptor).
Let’s see a basic usage of strace, in order to familiarize with its output. In its most basic usage, we call strace followed by the program we want to execute and whom the behavior we want to analyze. For the sake of this example we will just copy a file using the cp command:
$ strace cp ~/.bashrc bashrc
The output of the command is quite long, and of course here we cannot analyze it in detail; let’s just see the first line. Each line in the strace output contains:
- The system call name
- The arguments passed to the system call in parentheses
- The system call return value
The first system call we can see in the output is execve. This call is used to execute a program with a specified array of arguments. The first argument accepted by execv is the path of the file we want to execute; the second is an array of strings which represents the arguments that will be passed to the program (the first argument, by convention, being the name of the program itself).
In our case, as expected, the binary that is called is /usr/bin/cp, and the array of arguments passed to the call are: the name of the program (cp), the source and the destination paths:
execve("/usr/bin/cp", ["cp", "/home/egdoc/.bashrc", "bashrc"], 0x7fff53d4e4c0 /* 46 vars */) = 0
The /* 46 vars */ notation means that 46 variables where inherited from the calling process (in the execv function the environment is taken from the external environ variable). Finally, we have the return value, which in this case is 0 (actually the exec family of function returns a value only if an error occurs).
Filtering only specific system calls
When using strace sometimes we may want keep track of only specifics system calls made by a process. In those situations we can use the -e option followed by an expression which indicates what system calls should be traced. Suppose we run the same command we used in the previous example, but we only want the read system calls to be displayed in the output, we would run:
$ strace -e read cp ~/.bashrc bashrc
As expected, only read calls are reported:
read system call takes three arguments: the first is a file descriptor associated with the file that should be read; the second is the buffer into which the file should be read, and third is the number of bytes that should be read. On success, the function returns the number of bytes read from the file, as we can observe in the output of the above.Attaching strace on a running process
Until now we invoked strace passing to it the command to be executed and to keep trace of; what if we want to trace an existing and already running process? In that case, we must invoke strace with the -p (or --attach) option, and pass the PID (Process Id) of the process we want to attach it to.
To find the PID of a program, among the other solutions, we can use the pidof utility. For the sake of this example we will attach strace to a running instance of gnome-terminal-server:
$ pidof gnome-terminal-server 121316
The pidof command returned 121316, which is the PID of gnome-terminal-server. Knowing this, we can attach strace to the process:
$ strace -p 121316
The command above initially will return something like:
Ctrl+C on the keyboard; we will be notified of the “detachment”, but the traced process will continue to run:strace: Process 121316 detached
Tracing signals
Thanks to strace we can also observe when a process receives a signal, and how it reacts to it. Let me demonstrate it. First, we launch a long running process as top, which is a process monitor:
$ top
We than attach strace to it, after obtaining its PID, which in this case is 44825:
$ strace -p 44825
At this point strace starts tracking the system calls made by top, but also the signals received by it. To prove it we send a SIGTERM to PID 44825:
$ kill 44825
As expected, the event is reported in the strace output:
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=44888, si_uid=1000} ---
In the above output si_signo is the number of signal being delivered (SIGTERM = 15), si_code contains a code which identifies the cause of the signal (SI_USER = 0): in this case the signal was generated by a user process. The si_pid and si_uid fields report, respectively, the PID of the sending process and its UID.
Save the output of strace to a file
If we use the -o option (short for --ouput) when launching strace, we can redirect its output to a file, passing a path as argument, for example:
$ strace -p 121316 -o strace_output strace: Process 121316 attached
The strace_output file will be created and the output of strace will be written inside it. To watch the update in the file we can use the tail: normally this command reads the last 10 lines of a file and exits, but if we call it with the -f option (short for --follow) we can observe as new content is appended:
$ tail -f strace_output
Print a summary of the system calls
The strace utility comes with a very useful feature: the ability to generate a summary of all the system calls made by a specified process. If we want to generate such a report, all we have to do is to invoke the program with the -c or --summary-only option. Let’s take as an example the cp command we used before:
$ strace -c cp ~/.bashrc bashrc
The command above will generate this report:
% time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 25.71 0.000298 7 38 13 openat 19.24 0.000223 4 51 mmap 11.48 0.000133 4 28 close 9.92 0.000115 57 2 1 newfstatat 7.94 0.000092 10 9 mprotect 6.99 0.000081 3 25 fstat 2.85 0.000033 3 11 read 2.76 0.000032 16 2 munmap 2.50 0.000029 14 2 statfs 1.90 0.000022 22 1 write 1.55 0.000018 2 8 pread64 1.38 0.000016 8 2 1 access 1.04 0.000012 4 3 brk 0.78 0.000009 4 2 rt_sigaction 0.60 0.000007 7 1 futex 0.52 0.000006 3 2 1 arch_prctl 0.43 0.000005 5 1 rt_sigprocmask 0.43 0.000005 5 1 set_tid_address 0.43 0.000005 5 1 fadvise64 0.43 0.000005 5 1 set_robust_list 0.43 0.000005 5 1 prlimit64 0.26 0.000003 3 1 1 stat 0.26 0.000003 3 1 1 lseek 0.17 0.000002 2 1 geteuid 0.00 0.000000 0 1 execve ------ ----------- ----------- --------- --------- ---------------- 100.00 0.001159 5 196 18 total
As you can see, since we generated a summary, the normal output of strace is not displayed. If we want to generate the summary but still obtain the regular output of the program, we must use the -C option instead, which is the short form of --summary.
Conclusions
In this tutorial we learned how to install and use strace, a nice utility useful for debugging purposes and more generally to keep track of the system calls performed by a process. We saw how the output of strace is organized, how to launch a program and keep track of the system calls it makes, how to attach strace to an already running process and how signals received by a process are notified; finally, we saw how to generate a summary of all the calls made by a process. Here we barely scratched the surface of what we can do with strace: if you want to know more about it, the advice is, as always, to read the manual!
