scripting Archives

Various examples for removing duplicate lines from a text file on Linux

Removing duplicate lines from a text file using Linux command line

17 May 2021 by Luke Reynolds

Removing duplicate lines from a text file can be done from the Linux command line. Such a task may be more common and necessary than you think. The most common scenario where this can be helpful is with log files. Oftentimes log files will repeat the same information over and over, which makes the file nearly impossible to sift through, sometimes rendering the logs useless.

In this guide, we’ll show various command line examples that you can use to delete duplicate lines from a text file. Try out some of the commands on your own system, and use whichever one is most convenient for your scenario.

In this tutorial you will learn:

How to remove duplicate lines from file when sorting
How to count the number of duplicate lines in a file
How to remove duplicate lines without sorting the file

Various Bash examples for redirecting standard output and standard error to the same file

Bash – Redirect both standard output and standard error to same file

7 May 2021 by Luke Reynolds

The Bash shell is the most popular shell on Linux systems, and to use the shell efficiently, you need a little knowledge about Bash shell redirections. This is also an essential step in learning Bash scripting.

In this guide, we’ll show how to redirect standard output and standard error to the same file on the Bash shell command line. This will include several examples so you can pick the right method in any scenario.

In this tutorial you will learn:

How to redirect standard output and standard error to same file
How to redirect standard output and standard error to file and terminal
How to redirect standard output and standard error to /dev/null

Renaming multiple files at once on Linux

How to rename multiple files on Linux

30 April 2021 by Korbin Brown

Renaming files on Linux systems is usually handled by the mv (move) command. The syntax is just mv old.txt new.txt. Simple enough, but what if we have multiple files that need to be renamed at once, even hundreds of them? The default mv utility can’t handle renaming more than one file unless we do a bit of scripting. There are also other utilities we can install to solve the problem, like rename and mmv.

In this guide, we’ll show you how to use the mv command as well as the rename and mmv tools to rename multiple files on your Linux distro. We’ll go over several examples so you can understand the syntax and how to use each method.

In this tutorial you will learn:

How to rename multiple files at once with mv command
How to install rename on major Linux distros
How to install mmv on major Linux distros
How to use mmv, through command examples
How to use rename, through command examples

Appending text to a file in Bash on Linux

How to append to file on bash shell command line

27 April 2021 by Korbin Brown

In this guide, we’ll show how to append text or command output to a file on the Bash shell command line. This will include several examples so you can pick the right method in any scenario.

In this tutorial you will learn:

How to append text or command output to a file
How to append and view command output at the same time
How to append multiple lines of text to a file

Appending text to a file in Bash on Linux

How to set and list environment variables on Linux

21 April 2021 by Korbin Brown

Environment variables are part of the Linux system shell that contain changing values. They help facilitate scripts and system programs, so that code can accommodate a variety of scenarios. Unlike regular shell variables, environment variables can be accessed system-wide, by any user or process.

Let’s look at a very simple example to see how environment variables work and why they exist. There are many system programs and user made scripts that need to access a current user’s home directory. This can be done reliably through the HOME environment variable. Therefore, a script that contains the following line can be used by any user on the system and it will generate the same result.

$ echo $SHELL > $HOME/current-shell.log

$ cat $HOME/current-shell.log
/bin/bash

In this guide, we’ll show how to list all the environment variables on a Linux system, as well as set new ones. Setting new environment variables can either be done temporarily, or permanently if you need them to survive a reboot. We’ll show instructions for both methods below.

In this tutorial you will learn:

How to list environment variables on Linux
How to set a temporary environment variable on Linux
How to set a permanent environment variable on Linux

How to Discover, From Inside a Bash Script, the Path the Script Is In

23 March 2021 by Roel Van de Paar

When you develop complex Bash scripts and start putting various scripts into a folder, where one script interacts with another by, for example, starting it, it quickly becomes necessary to ensure we know the path the script was started from, so we can start the other scripts with a fully qualified pathname. This is important because the first script may have been started from outside the script’s directory. We could have also done so by using a relative path, so even – somehow – reading the command that started the current script will not work.

In this tutorial, you will learn:

What the pwd command is, and what it does
How to discover from inside a Bash script what path that same script is in

Special Bash Variables with examples

16 February 2021 by Roel Van de Paar

Bash is a great coding language, which allows you to do complex things like Big Data Manipulation, or simply create sever or desktop management scripts.

The entry level skill required to use the Bash language is quite low, and one-liner scripts (an often used jargon, which indicates multiple commands executed at the command line, forming a mini-script), as well as regular scripts, can grow in complexity (and how well written they are) as the Bash developer learns more.

Learning to use special variables in Bash is one part of this learning curve. Whereas originally the special variables may look cryptic: $$, $?, $*, \$0, $1, etc., once you understand them and use them in your own scripts, things will soon become clearer and easier to remember.

In this tutorial you will learn:

How to use special variables in Bash
How to correctly quote variables, even special ones
Examples using special variables from the command line and scripts

Introduction to named pipes on Bash shell

1 January 2021 by Egidio Docile

On Linux and Unix-based operating systems, pipes are very useful since they are a simple way to achieve IPC (inter-process communication). When we connect two processes in a pipeline, the output of the first one is used as the input of the second one. To build a so called “anonymous” pipe, all we have to do is to use the | operator. Anonymous, or unnamed pipes last just as long as the processes they connect. There is, however, another type of pipe we can use: a FIFO, or named pipe. In this article we will see how named pipes work and in what they are different from the standard pipes.

In this tutorial you will learn:

What is a named pipe
How to create a named pipe
How to recognize a named pipe
How named pipes work
How to delete a named pipe

How to Use Bash Subshells Inside if Statements

15 December 2020 by Roel Van de Paar

If you have ever used Bash subshells ($(...)), you know how flexible subshells can be. It only takes a few characters to start a subshell to process anything required, inline to another statement. The number of possible use cases is virtually unlimited.

We can also use Bash subshells inside if statements, inline with the statement. Doing so gives the user and developer much additional flexibility when it comes to writing Bash if statements.

If you are not familiar yet (or would like to learn more about) Bash if statements, please see our Bash If Statements: If Elif Else Then Fi article.

In this tutorial you will learn:

How to incorporate Bash subshells inside if statements
Advanced methods to incorporate Bash subshells inline with other commands
Examples demonstrating the use of Bash subshells in if statements

Process List Management and Automatic Process Termination

15 December 2020 by Roel Van de Paar

As sever optimal use/maximization continues to grow, it becomes more and more important to manage processes well. One aspect of this is automatic process termination. When a process has gone rogue, and is consuming too much resources, it can be terminated automatically.

This is especially suited to servers which have a lot of temporary or disposable processes. It is also well suited for testing servers which are running many test trials and where such test trials prove to be unstable or cause the software under testing to behave erratically (for example by using too much memory)

In this tutorial you will learn:

How to manage processes in an automated fashion
Which resources you may want to monitor, and why
Example code showing how automatic process termination can work for memory hogging issues

Time Your Bash Scripts and Procedures From Inside the Code

10 December 2020 by Roel Van de Paar

In general, one may use the time Bash utility (see man time for more information) to run a program, and obtain runtime duration and system resource usage summaries. But how can one time particular sections of code, directly from within the Bash source code?

Using some easy variable assignments and calculations, it is possible to achieve accurate timing metrics for Bash script executions.

In this tutorial you will learn:

How to time Bash scripts using variable assignments and calculations
How to use overlapping timers to time specific sections of your scripts
Examples which exemplify how specific sections of code can be timed

How to Correctly Grep for Text in Bash Scripts

1 December 2020 by Roel Van de Paar

grep is a versatile Linux utility, which can take a few years to master well. Even seasoned Linux engineers may make the mistake of assuming a given input text file will have a certain format. grep can also be used, directly in combination with if based searches to scan for the presence of a string within a given text file. Discover how to correctly grep for text independent of character sets, how to use the -q option to text for string presence, and more!

In this tutorial you will learn:

How to do correct character set-independent text searches with grep
How to use advanced grep statements from within scripts or terminal oneliner commands
How to test for string presence using the -q option to grep
Examples highlighting grep usage for these use cases