The uniq command in Linux, short for “unique,” is used to isolate unique strings in text documents or standard input. It’s a pretty basic command with a simple purpose, but comes with a few options under the hood that prove themselves handy under a lot of scenarios.
uniq can do things like suppress duplicate lines or count the number of times each word occurs in a piece of text. Linux administrators will find this useful when parsing large log files. For example, rather than seeing the same “error” line repeatedly, a log file can be passed to uniq, which will just tell you how many times the error occurs, giving you a much cleaner, human readable output.
In this guide, you’ll learn how to use the uniq command in Linux through examples. Follow along below to see learn about the various options that you can use with this command.
In this tutorial you will learn:
- How to use the uniq command on Linux
| Category | Requirements, Conventions or Software Version Used |
|---|---|
| System | Any Linux distro |
| Software | uniq |
| Other | Privileged access to your Linux system as root or via the sudo command. |
| Conventions | # – requires given linux commands to be executed with root privileges either directly as a root user or by use of sudo command$ – requires given linux commands to be executed as a regular non-privileged user |
Frequently Used Options
The uniq command allows us to get text files and input full of repeat lines under control. In order to show you how the uniq command works, we’ve created a text document full of Linux distro names, with repeated lines along the way. Check out the examples below to see how we use the uniq command to clean up this list of names.
uniq command in Linux Basic Examples
- First, let’s see what our text looks like with the cat command.
$ cat distros.txt AlmaLinux Arch Linux Arch Linux CentOS Debian Fedora Fedora Fedora Gentoo Kali Linux Kali Linux Manjaro openSUSE openSUSE openSUSE openSUSE Red Hat Ubuntu Ubuntu Ubuntu Ubuntu Ubuntu
Now let’s use the uniq command to view the same document.
$ uniq distros.txt AlmaLinux Arch Linux CentOS Debian Fedora Gentoo Kali Linux Manjaro openSUSE Red Hat Ubuntu
As you can see, the default behavior of uniq command in Linux is to discard duplicate lines. Note that this only works for lines that are adjacent. So, if our “Ubuntu” text wasn’t repeated one after another, they wouldn’t have been omitted from the output.
- It’s common to use the sort command in conjunction with uniq, since uniq can only do its job when lines are adjacent. The sort command can put lines in alphabetical or numerical order, before sending the text to uniq.
$ cat distros.txt | sort | uniq
- Use the
-coption with uniq if you want to count the number of times each line is repeated.$ uniq -c distros.txt 1 AlmaLinux 2 Arch Linux 1 CentOS 1 Debian 3 Fedora 1 Gentoo 2 Kali Linux 1 Manjaro 4 openSUSE 1 Red Hat 5 Ubuntu - Use the
-doption to only list the lines that have repeats.$ uniq -d distros.txt Arch Linux Fedora Kali Linux openSUSE Ubuntu
- By default, uniq is case sensitive, meaning it will not consider “Ubuntu” and “ubuntu” to be repeat lines. This behavior can be changed with the
-ioption.$ uniq -i distros.txt
You can always use the man command to read more about the uniq command and its official documentation. Click the previous link to see how to open the manual pages for any command on a Linux system.
Advanced Usage
The uniq command in Linux is pretty simple, but it comes packed with a lot of options. Many of these options fly under the radar, and even some seasoned system administrators may not know of them. However, they can definitely come in handy in various situations. In this section of the guide, we’ll show you a few of the lesser known options of the uniq command that we think are useful.
uniq command in Linux Advanced Examples
- Use the
-uoption to only output lines that don’t have any repeats.$ uniq -u distros.txt AlmaLinux CentOS Debian Gentoo Manjaro Red Hat
- Use the
-soption to instruct uniq to skip comparing the first X number of characters in each line. For example, this command will only compare the characters that come after the fourth character in each line:$ uniq -s 4 distros.txt
- To compare only the first X number of characters, you can use the
-woption with the uniq command. For example, this command will only compare the first three characters to determine if there are any repeat lines.$ uniq -w 3 distros.txt
Closing Thoughts
In this guide, we learned all about the uniq command on Linux. The uniq command proves very useful when parsing large portions of text that have a tendency to repeat themselves. Since repeat data is rarely useful, uniq can make the data easier to digest, while optionally preserving the number of repeat lines with the -c option. If you have a tendency to look through lots of log files, knowing the ins and outs of the uniq command will be a life saver.