How can I remove duplicate lines from my text file?
The easiest way to remove duplicate files from a text file as by use sort and uniq commands. Consider a following text file:
$ cat file.txt 
How can I remove duplicate lines from my text file?
use sort and uniq commands
use sort and uniq commands
How can I remove duplicate lines from my text file?

To remove all duplicate lines we first need to pipe the content of the file to sort. Once the content is sorted we use uniq command to remove duplicates:
$ sort file.txt | uniq 
How can I remove duplicate lines from my text file?
use sort and uniq commands

Free Linux eBooks

Do you have the right skills?

Our IT Skills Watch page reflects an up to date IT skills demand leaning towards the Linux and Unix environment. We have considered a number of skills and operating systems.

See the result...

Linux Online Training

Learn to run Linux servers and prepare for LPI certification with Linux Academy. 104 available video lessons with PDF course notes with your own server!

Go to top