The wget command is used to retrieve content from servers via HTTP, HTTPS, and FTP. It simplifies many downloading tasks that you’d normally have to do yourself by perusing a website and manually clicking links to download. Wget is able to perform the same function from the command line and has a lot of added abilities that can save you time, such as downloading directories recursively.
In this article, we’ll show you what wget is capable of and provide you with example commands that you can use in your own Linux terminal.
In this tutorial you will learn:
- How to download a file from a website with wget
- How to download a directory
- How to mirror a website
- How to download and untar a file automatically
- How to authenticate with wget
- How to use quiet mode with wget
| Category | Requirements, Conventions or Software Version Used |
|---|---|
| System | Linux (any distribution) |
| Software | wget |
| Other | Privileged access to your Linux system as root or via the sudo command. |
| Conventions |
# – requires given linux commands to be executed with root privileges either directly as a root user or by use of sudo command$ – requires given linux commands to be executed as a regular non-privileged user |
Download a file from a website with wget
Wget makes file downloads very painless and easy. It’s probably the best command line tool on Linux suited for the job, though other tools can also perform the task, like cURL.
Let’s take a look at a few examples of how we could use wget to download a Linux distribution, which are offered on developer websites as ISO files.
The most basic command you can execute with wget is just supplying the URL of the file you wish to download.
$ wget http://example.com/linux.iso
Wget will download the specified file to whatever location you are running the command from. It will show the progress of the download, current speed, and estimated time of completion. It also spits out some other information about its process of connecting to the server and requesting the file. That output can be helpful when diagnosing a connection issue.
Specify directory and file name
You can also specify a directory for the file to download to, as well as choose a name for the download. Use the -O (output) option and enter the directory and file name after the URL.
$ wget http://example.com/linux.iso -O /path/to/dir/myfile.iso
Resuming downloads
A nifty feature of wget is that it can resume downloads. If your file download was interupted, either unintentionally or because you needed to stop it with Ctrl+C, you can pick up right where you left off by using the -c option. Just make sure you are in the correct directory or you tell wget where to find the partial file with -O.
wget -c http://example.com/linux.iso
Download multiple files
If you want to download more than one file, create a text document that contains a list of download links, with each URL on a separate line. Then, run the wget command with the -i option and specify the path to your text document.
$ wget -i download-links.txt
Limit download speed
Another handy option of wget is to limit its download speed. This is useful if you don’t want a large download to steal all your network bandwidth, which might give latency to other users on your network. Use the --limit-rate flag and specify k for kilobytes, m for megabytes, or g for gigabytes. For example, this would download a file at a maximum rate of 500 KB per second:
$ wget --limit-rate 500k http://example.com/linux.iso
Download a directory
Wget can download an entire directory, recursively, from either an FTP or web (HTTP/HTTPS) server. For FTP, just use the -r (recursive) option in your command and specify the directory you want to get.
$ wget -r ftp://example.com/path/to/dir
If you are trying to download the directory of a website, the command is pretty much the same but in most cases you will also want to append the --no-parent (or just -np) option so wget doesn’t try to follow any links back to the index of the site.
$ wget -r -np http://example.com/directory
How to mirror a website
Wget has the ability to follow all the links on a website, downloading everything it comes across as it goes. This makes wget an extremely powerful tool because not only can it download a directory or multiple files, it can actually mirror an entire website.
Websites are made up of HTML files, and usually you’ll also find some .jpg or .png image files, .css (style sheets), .js (JavaScript), and a variety of others. Wget can find all these files automatically and download them into the same directory structure as the website, which would essentially give you an offline version of that site.
Include the -m (mirror) flag in your wget command and the URL of the site you want to mirror.
$ wget -m http://example.com
In most cases, you’ll also want to include the -p option in your command, which tells wget to download all the files that would be required to display the offline website correctly, such as style sheets. The -k option can also make the site display better, as it will rename the directories and references as necessary for offline viewing. Whether or not you’ll need these commands just depends on the site you’re mirroring.
$ wget -m -p -k http://example.com
Download and untar a file automatically
You can save some time when downloading a tar archive by piping your wget command to tar so it downloads and decompresses all in one command. To do so, use the -O - option, which tells wget to download the file to standard output. Then just pipe directly to your tar command.
For example, to download latest version of WordPress and open the tar archive in a single command:
$ wget https://wordpress.org/latest.tar.gz -O - | tar -xz
How to authenticate with wget
If the HTTP or FTP server you are trying to download from requires authentication, there are a couple of options you have for supplying a username and password with wget. These example commands will work with both FTP and HTTP.
The first option is to supply the username and password in the wget command itself, which is not the safest method since your password is visible to anyone looking at your screen or viewing your user’s command history:
$ wget --user=USERNAME --password=SECRET http://example.com/SecureFile.txt
You just need to replace USERNAME and SECRET with the appropriate information.
The second option is to let wget prompt you for a password, which keeps it hidden from those that can see your screen or look through your command history:
$ wget --user=USERNAME --ask-password http://example.com/SecureFile.txt
Use quiet mode with wget
To suppress all the output that wget displays, you can use the -q (quiet) option. This is especially useful when saving to standard output (-O -) as that can spam your terminal with a ton of text. When using -q, you’ll know your download completed when your terminal returns to a normal prompt, since wget can’t give you any indication itself.
$ wget -q http://example.com
A somewhat similar option is to background the wget command with -b. This will allow you to close your terminal or continue using it for something else while the wget command continues its job in the background.
$ wget -b http://example.com/linux.iso
Wget will log the usual output in a text file (wget-log by default) and tell you the process ID. If you want to cancel the download, you can always use the kill command followed by the PID.
Conclusion
Wget is simply the best command line utility you can use to download files on Linux. It has so many options, many of which are built to save you time – such as the feature to download recursively. In this article, we covered some of the basic uses of the wget command. Believe it or not, this is only scraping the surface of what it can do.
