Objective
Learn to use redirections, pipes and tee in the Bash shell
Operating System and Software Versions
- Operating System: – Linux distribution agnostic
Requirements
- Access to a Bash shell
Conventions
- # – requires given linux commands to be executed with root privileges either directly as a root user or by use of
sudocommand - $ – requires given linux commands to be executed as a regular non-privileged user
Introduction
Redirection is the ability to redirect the input and output of various commands to and from files or devices. We are going to see how redirecting works in Bash: the default shell in the majority of Linux distributions.
File descriptors
Every time you execute a program, three file descriptors are created by default:
- 0 –
stdin(standard input) - 1 –
stdout(standard output) - 2 –
stderr(standard error)
By default the stdout and stderr descriptors are “attached” to the screen, meaning that the program output and its errors are not saved to any file, but just displayed, while the standard input is attached to the keyboard. Redirection operators let us manipulate those associations.
Redirecting standard output
As said above, by default, the standard output of a program is sent to the screen, but in some circumstances, as for example in the context of a script, we may want to discard it, or perhaps send it to a file. How do we accomplish this? The key here is the > operator:
ls -l > output.txt
In this little example, we redirected the output of the ls command to the file output.txt (notice that the file doesn’t need to exist, it is created automatically). Nothing appeared on screen, but if we check the content of the file, we are going to see something quite familiar:
$ cat output.txt total 36 drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Desktop drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Documents drwxr-xr-x. 2 egdoc egdoc 4096 Jun 23 02:40 Downloads drwxrwxr-x. 13 egdoc egdoc 4096 Jun 23 08:13 git drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Music -rw-rw-r--. 1 egdoc egdoc 0 Jun 23 09:38 output.txt drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:39 Pictures drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Public drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Templates drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Videos
What we see is the output of the ls command. If we now try the redirection again, the current content of the file will be replaced by the new output. How can we preserve the previous content, and just append new lines to it? In this case we use the >> operator:
ls -l >> output.txt
This way, if the file doesn’t exist, or it has no content, the redirection will have the same effect as if we used the > operator, otherwise the new content will be appended to the existing one, as you can see by observing the file again:
total 36 drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Desktop drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Documents drwxr-xr-x. 2 egdoc egdoc 4096 Jun 23 02:40 Downloads drwxrwxr-x. 13 egdoc egdoc 4096 Jun 23 08:13 git drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Music -rw-rw-r--. 1 egdoc egdoc 0 Jun 23 09:38 output.txt drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:39 Pictures drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Public drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Templates drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Videos total 40 drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Desktop drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Documents drwxr-xr-x. 2 egdoc egdoc 4096 Jun 23 02:40 Downloads drwxrwxr-x. 13 egdoc egdoc 4096 Jun 23 08:13 git drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Music -rw-rw-r--. 1 egdoc egdoc 541 Jun 23 09:38 output.txt drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:39 Pictures drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Public drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Templates drwxr-xr-x. 2 egdoc egdoc 4096 Jun 22 19:36 Videos
We may also need to redirect the output of multiple commands at once: we can obtain the wanted result using curly braces to group them:
$ { echo "linuxconfig"; ls -l; } > output.txt
The output.txt file, will now contain both the string ‘linuxconfig’ and the result of the ls -l command.
Another common operation is to discard the output of a command completely, this time redirecting it to a special device: /dev/null. In unix-like operating systems /dev/null (also known as bit bucket), is a device which discards all data written to it:
ls -l > /dev/null
Redirect both standard output and standard error
In the examples above we just redirected the standard output. If some kind of error occurs, we will still be able to see the error message on screen:
$ ls -l nonexistingfile.txt > /dev/null ls: cannot access 'nonexistingfile.txt': No such file or directory
This happens because, as said above, stdout and stderr descriptors are completely separated by each other. What can we do, then, to redirect them both? There are two syntaxes we can use to accomplish this task: the first one, which works even in old versions of the shell, is the following:
ls -l > output.txt 2>&1
What have we done? First of all we redirected the stdout of the command to the output.txt file, just like we did before, then we redirected the stderr to the stdout. Please notice how we referenced file descriptors by their respective numbers. For a reasonably modern Bash version, we can use this other, more streamlined syntax:
ls -l &> output.txt
Redirect standard output to standard error
Imagine you are writing a script, and you want to handle a case when a specific instruction fails, by showing the user an error message. How would you accomplish this? First thing that comes to mind is to just echo the wanted message and then probably exit the script with the appropriate error code. This would be perfectly fine, but ask yourself, on what descriptor this message will be ‘sent’? It is the stdout of the echo command, but at the same time, if we see things from the script perspective, as an error message, it should use the stderr descriptor. What we want to do here, is to redirect stdout to stderr. We use the following syntax to accomplish the task:
echo "An error occurred, bye!" >&2
Surely it is not the most useful of the error messages, but it is enough for our example.
Redirecting standard input
As we said before, by default, the standard input is associated to the keyboard, but by using the < operator, we can make some programs to accept input from other sources. Let’s see a quick example using the tr command (as you probably know tr is used to delete or translate characters). It normally works this way:
tr 'goot tay!' t d
You give tr a string, first specifying the character you want to change, and then the one it should use to substitute it. In this case we pass the string ‘goot tay!’ directly, by using the keyboard: it will be translated to ‘good day!’. What we will do to demonstrate stdin redirection, is to write the string to a file and then redirect the content of the file to the stdin of the tr command.
First we write ‘goot tay!’ to the output.txt file
$ echo 'goot tay!' > output.txt
Then we send its content to the stdin of tr:
$ tr < output.txt t d good day!
As you can see everything went as expected, and a nice message has been printed on the screen.
Pipelines
Using the pipe operator | we can chain multiple commands together, so that the stdout of the command at the left of the operator is passed to the stdin of the command at the right of it. We can quickly demonstrate this, by using the tr command again:
$ echo 'goot day!'| tr t d good day!
What happened? The standard output of the echo command (consisting in the string ‘goot tay!’) is piped to the standard input of the tr command, which translates the string. Finally, we see tr standard output on screen. But of course the pipe can continue. Imagine we want just the word ‘good’ to be displayed:
$ echo 'goot tay!' | tr t d | cut -f 1 -d ' '
What we have done here is to add the cut command to the pipe, passing the stdout of tr to its stdin. The cut command uses the space as delimiter (-d switch) and selects only the first field, returning the string ‘good’.
Using tee
The tee command reads standard input and redirects it both to standard output and to a file at the same time, making possible to create a ‘T’ in our pipe. Let’s reuse the example above, this time sending the intermediate result (‘good day!’) also to the output.txt file:
$ echo 'goot tay!' | tr t d | tee ouput.txt | cut -f 1 -d ' '
The output on screen will be the same as before (‘good’), but if we read the output.txt file, we can see that the ‘good day!’ string has been written to it. This is because ‘good day!’ was the standard output flowing in the pipe when we inserted our tee.
Tee is also useful is some specific circumstances. For example, if you try to ‘echo’ something to a file which needs root privileges to be written, you will notice that things will not go as expected:
$ sudo echo "linuxconfig.org" > protected.txt -bash: protected.txt: Permission denied
What happened? You probably expected the command to be successful, because you prefixed it with sudo, but it failed anyway. This is because you just ran the echo command with privileges, but that didn’t gave you write permissions on the file. Let’s try this way instead:
$ echo "linuxconfig.org" | sudo tee protected.txt > /dev/null
Here we run echo as a normal user, but the redirection itself is performed with root privileges, so this time the command succeeds. We also added an extra redirection to /dev/null, because we didn’t need the output to be displayed on the screen.
Note that using this technique, the output will not be appended to the destination file: the latter will be overwritten, and its previous content will be lost. To append to the file, we must add the -a switch to tee (short for –append).
Be careful, just a little distraction here can cause horrible things!
