A shell is a crucial part of an Unix-based operating system and is the main interface we can use to interact with the system itself. Bash is without doubt the most used shell on the majority of Linux distributions: it was born as thefree software replacement for the Bourne shell (bash is the acronym for Bourne-again shell) inside the GNU project. In this tutorial we will learn how some of the most useful bash expansions works.
In case you are not familiar with Bash yet, or you simply need to refresh your memory, then you are recommended to visit our Bash Scripting Tutorial for Beginners, before you dive into the Bash Shell expansions concept below.
In this tutorial you will learn:
- How to use various bash parameter expansions
Software Requirements and Conventions Used
| Category | Requirements, Conventions or Software Version Used |
|---|---|
| System | Distribution-independent |
| Software | A Bash shell |
| Other | Basic knowledge of Bash |
| Conventions | # – requires given linux commands to be executed with root privileges either directly as a root user or by use of sudo command$ – requires given linux commands to be executed as a regular non-privileged user |
The simplest possible expansion
The simplest possible parameter expansion syntax is the following:
${parameter}
When we use this syntax, parameter is substituted by its value. Let’s see an example:
$ site="linuxconfig.org"
$ echo "${site}"
linuxconfig.org
We created the site variable and assigned the "linuxconfig.org" string to it. We then used the echo command to display the result of the variable expansion. Being this a basic expansion, it would have worked even without the use of curly braces around the variable name:
$ echo "$site" linuxconfig.org
Why did we use the curly braces then? Curly braces, when performing parameter expansions, are used to delimit the variable name:
$ echo "You are reading this article on $site_!" You are reading this article on
What happened? Since the variable name was not delimited, the _ character was considered as part of it. The shell tried to expand thenon-existent $site_ variable, therefore nothing was returned. Wrapping the variable with curly braces solves this problem:
$ echo "You are reading this article on
${site}_!"
You are reading this article on linuxconfig_!
While the use of curly braces is not always needed with basic parameter expansion, it is mandatory to perform all the other expansions we will see in this article.
Before proceeding further, let me give you one tip. In the example above the shell tried to expand a non-existing variable, producing a blank result. This can be very dangerous, especially when working with path names, therefore, when writing scripts, it’s always recommended to use the nounset option which causes the shell to exit with error whenever a non existing variable is referenced:
$ set -o nounset $ echo "You are reading this article on $site_!" bash: site_: unbound variable
Working with indirection
The use of the ${!parameter} syntax, adds a level of indirection to our parameter expansion. What does it mean? The parameter which the shell will try to expand is not parameter ; instead it will try to use the the value of parameter as the name of the variable to be expanded. Let’s explain this with an example. We all know the HOME variable expands in the path of the user home directory in the system, right?
$ echo "${HOME}"
/home/egdoc
Very well, if now we assign the string “HOME”, to another variable, and use this type of expansion, we obtain:
$ variable_to_inspect="HOME"
$ echo "${!variable_to_inspect}"
/home/egdoc
As you can see in the example above, instead of obtaining “HOME” as a result, as it would have happened if we performed a simple expansion, the shell used the value of variable_to_inspect as the name of the variable to expand, that’s why we talk about a level of indirection.
Case modification expansion
This parameter expansion syntax let us change the case of the alphabetic characters inside the string resulting from the expansion of the parameter. Say we have a variable called name; to capitalize the text returned by the expansion of the variable we would use the ${parameter^} syntax:
$ name="egidio"
$ echo "${name^}"
Egidio
What if we want to uppercase the entire string, instead of capitalize it? Easy! we use the ${parameter^^} syntax:
$ echo "${name^^}"
EGIDIO
Similarly, to lowercase the first character of a string, we use the ${parameter,} expansion syntax:
$ name="EGIDIO"
$ echo "${name,}"
eGIDIO
To lowercase the entire string, instead, we use the ${parameter,,} syntax:
$ name="EGIDIO"
$ echo "${name,,}"
egidio
In all cases a pattern to match a single character can also be provided. When the pattern is provided the operation is applied only to the parts of the original string that matches it:
$ name="EGIDIO"
$ echo "${name,,[DIO]}"
EGidio
In the example above we enclose the characters in square brackets: this causes anyone of them to be matched as a pattern.
When using the expansions we explained in this paragraph and the parameter is an array subscripted by @ or *, the operation is applied to all the elements contained in it:
$ my_array=(one two three)
$ echo "${my_array[@]^^}"
ONE TWO THREE
When the index of a specific element in the array is referenced, instead, the operation is applied only to it:
$ my_array=(one two three)
$ echo "${my_array[2]^^}"
THREE
Substring removal
The next syntax we will examine allows us to remove a pattern from the beginning or from the end of string resulting from the expansion of a parameter.
Remove matching pattern from the beginning of the string
The next syntax we will examine, ${parameter#pattern}, allows us to remove a pattern from the beginning of the
string resulting from the parameter expansion:
$ name="Egidio"
$ echo "${name#Egi}"
dio
A similar result can be obtained by using the "${parameter##pattern}" syntax, but with one important difference: contrary to the one we used in the example above, which removes the shortest matching pattern from the beginning of the string, it removes the longest one. The difference is clearly visible when using the * character in the pattern:
$ name="Egidio Docile"
$ echo "${name#*i}"
dio Docile
In the example above we used * as part of the pattern that should be removed from the string resulting by the expansion of the name variable. This wildcard matches any character, so the pattern itself translates in “‘i’ character and everything before it”. As we already said, when we use the ${parameter#pattern} syntax, the shortest matching pattern is removed, in this case it is “Egi”. Let’s see what happens when we use the "${parameter##pattern}" syntax instead:
$ name="Egidio Docile"
$ echo "${name##*i}"
le
This time the longest matching pattern is removed (“Egidio Doci”): the longest possible match includes the third ‘i’ and everything before it. The result of the expansion is just “le”.
Remove matching pattern from the end of the string
The syntax we saw above remove the shortest or longest matching pattern from the beginning of the string. If we want the pattern to be removed from the end of the string, instead, we must use the ${parameter%pattern} or ${parameter%%pattern} expansions, to remove, respectively, the shortest and longest match from the end of the string:
$ name="Egidio Docile"
$ echo "${name%i*}"
Egidio Doc
In this example the pattern we provided roughly translates in “‘i’ character and everything after it starting from the end of the string”. The shortest match is “ile”, so what is returned is “Egidio Doc”. If we try the same example but we use the syntax which removes the longest match we obtain:
$ name="Egidio Docile"
$ echo "${name%%i*}"
Eg
In this case the once the longest match is removed, what is returned is “Eg”.
In all the expansions we saw above, if parameter is an array and it is subscripted with * or @, the removal of the matching pattern is applied to all its elements:
$ my_array=(one two three)
$ echo "${my_array[@]#*o}"
ne three
Search and replace pattern
We used the previous syntax to remove a matching pattern from the beginning or from the end of the string resulting from the expansion of a parameter. What if we want to replace pattern with something else? We can use the ${parameter/pattern/string} or ${parameter//pattern/string} syntax. The former replaces only the first occurrence of the pattern, the latter all the occurrences:
$ phrase="yellow is the sun and yellow is the
lemon"
$ echo "${phrase/yellow/red}"
red is the sun and yellow is the lemon
The parameter (phrase) is expanded, and the longest match of the pattern (yellow) is matched against it. The match is then replaced by the provided string (red). As you can observe only the first occurrence is replaced, so the lemon remains yellow! If we want to change all the occurrences of the pattern, we must prefix it with the / character:
$ phrase="yellow is the sun and yellow is the
lemon"
$ echo "${phrase//yellow/red}"
red is the sun and red is the lemon
This time all the occurrences of “yellow” has been replaced by “red”. As you can see the pattern is matched wherever it is found in the string resulting from the expansion of parameter. If we want to specify that it must be matched only at the beginning or at the end of the string, we must prefix it respectively with the # or % character.
Just like in the previous cases, if parameter is an array subscripted by either * or @, the substitution happens in each one of its elements:
$ my_array=(one two three)
$ echo "${my_array[@]/o/u}"
une twu three
Substring expansion
The ${parameter:offset} and ${parameter:offset:length} expansions let us expand only a part of the parameter, returning a substring starting at the specified offset and length characters long. If the length is not specified the expansion proceeds until the end of the original string. This type of expansion is called substring expansion:
$ name="Egidio Docile"
$ echo "${name:3}"
dio Docile
In the example above we provided just the offset, without specifying the length, therefore the result of the expansion was the substring obtained by starting at the character specified by the offset (3).
If we specify a length, the substring will start at offset and will be length characters long:
$ echo "${name:3:3}"
dio
If the offset is negative, it is calculated from the end of the string. In this case an additional space must be added after : otherwise the shell will consider it as another type of expansion identified by :- which is used to provide a default value if the parameter to be expanded doesn’t exist (we talked about it in the article about managing the expansion of empty or unset bash variables):
$ echo "${name: -6}"
Docile
If the provided length is negative, instead of being interpreted as the total number of characters the resulting string should be long, it is considered as an offset to be calculated from the end of the string. The result of the expansion will therefore be a substring starting at offset and ending at length characters from the end of the original string:
$ echo "${name:7:-3}"
Doc
When using this expansion and parameter is an indexed array subscribed by * or @, the offset is relative to the indexes of the array elements. For example:
$ my_array=(one two three)
$ echo "${my_array[@]:0:2}"
one two
$ echo "${my_array[@]: -2}"
two three
A negative length , instead, generates an expansion error:
$ echo "${my_array[@]:0:-2}"
bash: -2: substring expression < 0
“Length” expansion
When using the ${#parameter} expansion, the result of the expansion is not the value of the parameter, by its length:
$ name="Egidio"
$ echo "${#name}"
6
When parameter is an array, and it is subscripted with * or @, the number of the elements contained in it is returned:
$ my_array=(one two three)
echo "${#my_array[@]}"
3
When a specific element of the array is referenced, its length is returned instead:
$ echo "${#my_array[2]}"
5
Putting all together
In this article we saw many expansions syntax. We saw how to lowercase or uppercase the first letter of the string resulting from the expansion of a variable, how to use a level of indirection, how to perform substring removal and substring expansion, how to replace a pattern with a provided string and how to make a parameter be expanded in the length of its value, instead of its value itself.
This is not an exhaustive list of all the possible expansions we can perform with bash: consult the GNU documentation if you want to know more. In the article we also mentioned bash arrays: to know more about them you can read our dedicated bash arrays article.
