Even the most basic installation of any Linux distribution comes with a set of really useful utilities: “xargs” is undoubtedly one of those. By using xargs we can build and execute command lines using items from standard input as arguments of a command. This is especially useful when dealing with programs which don’t read standard input directly.
In this article we learn how to install the xargs utility on the most used Linux distributions, and how to use it to build and execute command lines.
In this tutorial you will learn:
- How to install xargs on the most used Linux distributions
- How to use xargs to build and execute command lines
- How to handle whitespaces in file names
- How to make xargs ask for confirmation before executing a command
- How to run multiple commands on each item
| Category | Requirements, Conventions or Software Version Used |
|---|---|
| System | Distribution agnostic |
| Software | findutils, xargs |
| Other | Privileged access to your Linux system as root or via the sudo command in order to perform system-wide installation of required packages |
| Conventions | # – requires given linux-commands to be executed with root privileges either directly as a root user or by use of sudo command$ – requires given linux-commands to be executed as a regular non-privileged user |
Installation
The xargs utility is usually distributed in the findutils package, which includes another very useful command line tool: “find”. The package is part of even the most minimal installation of practically every Linux distribution, therefore we shouldn’t need to install it explicitly. If for some odd reason, however, it is missing from our system, we can pull it back using our favorite distribution package manager.
On Debian and Debian-based distributions, for example, we can use
apt:
$ sudo apt install findutils
On Fedora and other distributions of the Red Hat family, we can use dnf, instead:
$ sudo dnf install findutils
The package is also available in the Archilinux “Core” repository, therefore can be installed using pacman:
$ sudo pacman -S findutils
Let’s see how we can use xargs to build and execute command lines.
Using xargs to build command lines
What the xargs utility does, exactly? Xargs uses items passed on the standard input, typically by using a pipe redirection operator, as additional arguments of a command we can specify together with some optional initial arguments. xargs syntax is pretty easy to understand:
xargs [options] [command [initial-arguments]]
Let’s see some real-world examples of xargs usage.
Example 1: Finding files with a specific extension containing a string
One typical xargs use case is when we have to search for a string in files with a specific extension. Recently, for example, I wanted to modify a WordPress theme, so I had to find in what PHP template file, the code snippet I wanted to modify was located. Here is how I used xargs to accomplish the task:
$ find theme -type f -name "*.php" | xargs grep -i "text-to-search"
Let’s see what this command does. First we used
find to search recursively inside the theme directory and we instructed the utility to look for for regular files (-type f) whose name ends in “.php” (-name "*.php"), then, we piped the standard output of the command to xargs, so that the files which matched the above creteria were passed as arguments to the grep command.
At this point you may think: “the find utility has itself the -exec option, which executes a given command using the files which matches the search as arguments, why should I use xargs?”. The answer is: “because xargs is faster”. Let’s take a look at this example. Suppose we want to find all “.txt” files in the ~/.config directory containing the string “wizard”. We use the time to get time statistic on the command execution. First we use -exec:
$ time find ~/.config -type f -name "*.txt" -exec grep -i "wizard" {} \;
Here are the results:
real 0m0.031s user 0m0.010s sys 0m0.020s
Now, let’s try with xargs:
$ time find ~/.config -type f -name "*.txt" | xargs grep -i "wizard"
This time we obtain the following statistics:
real 0m0.015s user 0m0.006s sys 0m0.010s
Although time cannot be considered a proper benchmark tool, especially for short running commands, it gives us a general idea of what command is faster. The gap between the two commands would grow even bigger when searching in a larger set of files.
Example 2: creating a tarball archive containing potentially non-existing files
Here is another case in which xargs could be useful. Suppose we want to create a tarball containing certain files, which, however, may not exist. If we try to include a non existing file in a tarball, tar exits with a failure status:
$ tar -cpf archive.tar ~/nonexistingfile ~/.bashrc ~/.bash_profile tar: Removing leading `/' from member names tar: /home/doc/nonexistingfile: Cannot stat: No such file or directory tar: Removing leading `/' from hard link targets tar: Exiting with failure status due to previous errors
A possible solution is to tell tar to ignore read errors, using the --ignore-failed-read option; ignoring errors, however is seldom a good idea. A better solutions is to use “find” to look for the files we want to put in the archive:
$ find ~ -maxdepth 1 -type f \( -name .bashrc -o -name .bash_profile -o -name nonexistingfile \) | xargs tar -cpf archive.tar
In the example above I assumed the files to be all directly inside the home directory, therefore I used the
-maxdepth 1 option, which actually disables recursive search. I also used the -type f filter, then, I included multiple -name filters (one for each file), separated by the -o operator, which works as a logical OR.
The “find” utility puts and implicit logical AND operator (-a) between given tests, which has an higher precedence than OR. That’s why we grouped the “name” tests inside parenthesis (escaped to avoid the shell interpreting them as special characters). If we hadn’t, the expression would have been equivalent to the following:
$ find ~ -maxdepth 1 -type f -a -name .bashrc -o -name .bash_profile -o -name nonexistingfile
In plain English this would have meant: “find a regular file named “.bashrc”, or any type of file named “.bash_profile”, or any type of file named “nonexistingfile”, in the home directory, without descending into subdirectories.
Using the strategy, if a file doesn’t exist it is not included in the list of files to be archived.
Handling whitespaces in file names
There is one thing we didn’t consider in previous examples: files whose name contains white spaces. xargs, by default, uses a whitespace character as items separator, therefore it will get confused if a file contains a whitespace as part of its name. Here is an example. Suppose there is a directory containing three files, one of those including a whitespace as part of its name:
test ├── file1.txt ├── file2.txt └── file 3.txt
Now, suppose we want to search for a string in one of those files, like we did in the previous examples:
$ find test -type f | xargs grep -i "content"
As expected, the command will fail:
grep: test/file: No such file or directory grep: 3.txt: No such file or directory
See what happened? Since “file 3.txt” has a whitespace in its name, and xargs uses a whitespace as items delimiter, it considered “file” and “3.txt” as two separate items! We can solve this problem by changing the character xargs uses as delimiter.
When we combine find and xargs, we can simple use the former -print0 option: this makes so that each found item is followed by a null character (default is a newline). On the xargs side, instead, we can pass the --null (or -0) option, which instructs the utility to use a null character as items separator, and to consider quotes, backslashes and end of file strings, literally:
$ find test -type f -print0 | xargs --null grep -i "content"
As an alternative, we can use the xargs -d option, which let us specify which character we want to use as a delimiter.
Making xargs ask confirmation before executing a command
In certain cases we may want to double check the command which would be executed by xargs. In those cases we can use the -p or --interactive option. Let’s apply it to the previous example:
$ find test -type f -print0 | xargs --interactive --null grep -i "content"
Here is the result:
grep -i content test/file1.txt test/file2.txt 'test/file 3.txt'?...
The command will be executed only after an affirmative answer (by using
y or Y). If we only want the command to be printed on the standard output before it is actually executed, instead, we can use the -t or --verbose option:
$ find test -type f -print0 | xargs --verbose --null grep -i "content" grep -i content test/file1.txt test/file2.txt 'test/file 3.txt' test/file 3.txt:content
Running multiple commands on each item
One interesting option we can use with xargs is -I, which let us create a sort of “template”. It takes a “placeholder” as argument: each occurrence of the placeholder is replaced at runtime by items read from the standard input. As a trivial example, suppose we want to notify the user we are searching for a string in a file, before actually using the grep command. We would run:
$ find test -type f | xargs -I % /bin/sh -c 'echo "searching in %"; grep -i "content" "%"'
When the -I option is used, the “newline” character (\n) is used as separator, therefore, as you can see, we don’t need to use --null, and we can simply quote the % placeholder.
Conclusions
In this tutorial we learned how to use the xargs utility to build and execute command lines from standard input. The xargs utility ships in the “findutils” package, together with another very useful program: “find”, and is often used in conjunction with it, as we saw in the examples.