There are times when it’s useful to inspect what a running application is doing under the hood, and what system calls it is performing during its execution. To accomplish such a task on Linux, we can use the strace utility. In this article we will see how to install it and we will learn its basic usage.

In this tutorial you will learn:

  • How to install strace
  • How to use strace to trace system calls made by a process
  • How to filter specifics system calls
  • How to attach to an already running process
  • How to generate a system call summary
How to trace system calls made by a process with strace on Linux
How to trace system calls made by a process with strace on Linux

Software requirements and conventions used

Software Requirements and Linux Command Line Conventions
Category Requirements, Conventions or Software Version Used
System Distribution-independent
Software Strace
Other Familiarity with the command line interface and Linux processes management
Conventions linux-commands to be executed with root privileges either directly as a root user or by use of sudo command
linux-commands to be executed as a regular non-privileged user

Installation

Although not installed by default, the strace utility is available in the official repositories of all the major Linux distributions; this means that we can install it very easily by using our favorite package manager.

If we are running on Fedora (or any other distribution in the Red Hat family), for example, we must use dnf:

$ sudo dnf install strace

SUBSCRIBE TO NEWSLETTER
Subscribe to Linux Career NEWSLETTER and receive latest Linux news, jobs, career advice and tutorials.


If we are more comfortable using Debian, or Debian-based distributions such as Ubuntu or Linux Mint, we can use apt to achieve the same result:

$ sudo apt install strace

If Arch Linux is our distribution of choice, we can use pacman to install the application, which is available in the extra repository:

$ sudo pacman -S strace

With the software installed, we can proceed forward, and see some examples of its usage.

Introducing strace

As we already said, strace is a tool used to keep track of the system calls made by a running process and the signals received by it. System calls are the fundamental interface between an application and the Linux kernel; when we use strace, the name of the calls made by a process, along with their arguments and return values are displayed on stderr (standard error file descriptor).

Let’s see a basic usage of strace, in order to familiarize with its output. In its most basic usage, we call strace followed by the program we want to execute and whom the behavior we want to analyze. For the sake of this example we will just copy a file using the cp command:

$ strace cp ~/.bashrc bashrc

The output of the command is quite long, and of course here we cannot analyze it in detail; let’s just see the first line. Each line in the strace output contains:

  • The system call name
  • The arguments passed to the system call in parentheses
  • The system call return value

The first system call we can see in the output is execve. This call is used to execute a program with a specified array of arguments. The first argument accepted by execv is the path of the file we want to execute; the second is an array of strings which represents the arguments that will be passed to the program (the first argument, by convention, being the name of the program itself).

In our case, as expected, the binary that is called is /usr/bin/cp, and the array of arguments passed to the call are: the name of the program (cp), the source and the destination paths:

execve("/usr/bin/cp", ["cp", "/home/egdoc/.bashrc", "bashrc"], 0x7fff53d4e4c0 /* 46 vars */) = 0

The /* 46 vars */ notation means that 46 variables where inherited from the calling process (in the execv function the environment is taken from the external environ variable). Finally, we have the return value, which in this case is 0 (actually the exec family of function returns a value only if an error occurs).

Filtering only specific system calls

When using strace sometimes we may want keep track of only specifics system calls made by a process. In those situations we can use the -e option followed by an expression which indicates what system calls should be traced. Suppose we run the same command we used in the previous example, but we only want the read system calls to be displayed in the output, we would run:

$ strace -e read cp ~/.bashrc bashrc

As expected, only read calls are reported:

strace -e read output
The output of the “strace -e read cp ~/.bashrc bashrc” command

By the way, the read system call takes three arguments: the first is a file descriptor associated with the file that should be read; the second is the buffer into which the file should be read, and third is the number of bytes that should be read. On success, the function returns the number of bytes read from the file, as we can observe in the output of the above.

Attaching strace on a running process

Until now we invoked strace passing to it the command to be executed and to keep trace of; what if we want to trace an existing and already running process? In that case, we must invoke strace with the -p (or --attach) option, and pass the PID (Process Id) of the process we want to attach it to.

To find the PID of a program, among the other solutions, we can use the pidof utility. For the sake of this example we will attach strace to a running instance of gnome-terminal-server:

$ pidof gnome-terminal-server
121316


The pidof command returned 121316, which is the PID of gnome-terminal-server. Knowing this, we can attach strace to the process:

$ strace -p 121316

The command above initially will return something like:

strace -p output
The output of the “strace -p 121316” command

The above (truncated) output will be updated “on the fly” as system calls are performed. To “detach” strace we can simply press Ctrl+C on the keyboard; we will be notified of the “detachment”, but the traced process will continue to run:

strace: Process 121316 detached

Tracing signals

Thanks to strace we can also observe when a process receives a signal, and how it reacts to it. Let me demonstrate it. First, we launch a long running process as top, which is a process monitor:

$ top

We than attach strace to it, after obtaining its PID, which in this case is 44825:

$ strace -p 44825

At this point strace starts tracking the system calls made by top, but also the signals received by it. To prove it we send a SIGTERM to PID 44825:

$ kill 44825

As expected, the event is reported in the strace output:

--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=44888, si_uid=1000} ---

In the above output si_signo is the number of signal being delivered (SIGTERM = 15), si_code contains a code which identifies the cause of the signal (SI_USER = 0): in this case the signal was generated by a user process. The si_pid and si_uid fields report, respectively, the PID of the sending process and its UID.

Save the output of strace to a file

If we use the -o option (short for --ouput) when launching strace, we can redirect its output to a file, passing a path as argument, for example:

$ strace -p 121316 -o strace_output
strace: Process 121316 attached

The strace_output file will be created and the output of strace will be written inside it. To watch the update in the file we can use the tail: normally this command reads the last 10 lines of a file and exits, but if we call it with the -f option (short for --follow) we can observe as new content is appended:

$ tail -f strace_output


Print a summary of the system calls

The strace utility comes with a very useful feature: the ability to generate a summary of all the system calls made by a specified process. If we want to generate such a report, all we have to do is to invoke the program with the -c or --summary-only option. Let’s take as an example the cp command we used before:

$ strace -c cp ~/.bashrc bashrc

The command above will generate this report:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 25.71    0.000298           7        38        13 openat
 19.24    0.000223           4        51           mmap
 11.48    0.000133           4        28           close
  9.92    0.000115          57         2         1 newfstatat
  7.94    0.000092          10         9           mprotect
  6.99    0.000081           3        25           fstat
  2.85    0.000033           3        11           read
  2.76    0.000032          16         2           munmap
  2.50    0.000029          14         2           statfs
  1.90    0.000022          22         1           write
  1.55    0.000018           2         8           pread64
  1.38    0.000016           8         2         1 access
  1.04    0.000012           4         3           brk
  0.78    0.000009           4         2           rt_sigaction
  0.60    0.000007           7         1           futex
  0.52    0.000006           3         2         1 arch_prctl
  0.43    0.000005           5         1           rt_sigprocmask
  0.43    0.000005           5         1           set_tid_address
  0.43    0.000005           5         1           fadvise64
  0.43    0.000005           5         1           set_robust_list
  0.43    0.000005           5         1           prlimit64
  0.26    0.000003           3         1         1 stat
  0.26    0.000003           3         1         1 lseek
  0.17    0.000002           2         1           geteuid
  0.00    0.000000           0         1           execve
------ ----------- ----------- --------- --------- ----------------
100.00    0.001159           5       196        18 total

As you can see, since we generated a summary, the normal output of strace is not displayed. If we want to generate the summary but still obtain the regular output of the program, we must use the -C option instead, which is the short form of --summary.

Conclusions

In this tutorial we learned how to install and use strace, a nice utility useful for debugging purposes and more generally to keep track of the system calls performed by a process. We saw how the output of strace is organized, how to launch a program and keep track of the system calls it makes, how to attach strace to an already running process and how signals received by a process are notified; finally, we saw how to generate a summary of all the calls made by a process. Here we barely scratched the surface of what we can do with strace: if you want to know more about it, the advice is, as always, to read the manual!

FIND LATEST LINUX JOBS on LinuxCareers.com
Submit your RESUME, create a JOB ALERT or subscribe to RSS feed.
LINUX CAREER NEWSLETTER
Subscribe to NEWSLETTER and receive latest news, jobs, career advice and tutorials.
DO YOU NEED ADDITIONAL HELP?
Get extra help by visiting our LINUX FORUM or simply use comments below.

You may also be interested in:



Comments and Discussions