Curl command basics tutorial with examples

Curl is a free and open source software we can use to exchange data with servers using one of the many supported protocols, such as HTTP, HTTPS, FTP, FTP, SFTP. Since Curl is a command line tool, we can use it in our scripts, to automatize repetitive tasks, for example. There are many use-cases Curl can cover. In this tutorial, however, we see some of the most common ones.

In this tutorial we see how to use Curl to perform basic GET and POST requests and how to set the header parameters of a request.

In this tutorial you will learn:

  • How to install Curl in some of the most used Linux distributions
  • How to perform GET and POST requests with Curl
  • How to send JSON and XML data in a POST request with Curl
  • How to set a request header parameters with Curl
Introduction to Curl
Introduction to Curl
 
Software requirements and conventions used
Category Requirements, Conventions or Software Version Used
System Distribution independent
Software curl
Other Administrative privileges to install software
Conventions # – requires given linux-commands to be executed with root privileges either directly as a root user or by use of sudo command $ – requires given linux-commands to be executed as a regular non-privileged user

Installation

Curl is free and open source software which is developed on github. It is included in the repositories of all the most common Linux distributions, and chances are you will find it is already installed on your system. Should you need to install it explicitly, however, you can follow this instructions. To install Curl on Fedora:

$ sudo dnf install curl

To install the package on Ubuntu, Debian and other Debian-based distributions, instead, we can launch the following command:

$ sudo apt-get update && sudo apt-get install curl

Curl is also available in the “Core” Archlinux repository. From there, we can install it by running:

$ sudo pacman -Sy curl

Performing GET requests

The GET http method is used to retrieve resources from a server. When we request a web page from our browser or we download a file, we are making use of this method. Performing such requests with Curl is a very easy operation. In the following example we request the https://www.google.com page:

$ curl https://www.google.com



When invoked this way, curl prints the body of the response sent by the server on the standard output. In this case the content we receive is the HTML code of the page. Since the content is sent on the standard output, we can use a pipe to feed it as input to some other command and further process and manipulate it. Here is a trivial example. Suppose we want to count how many times the word “google” appears in the source code of the page returned by the request in the previous example:

$ curl -s https://www.google.com | grep -i -o google | wc -w
85

In the example above we invoked Curl and used the `-s` option: this is the short for `–silent` and modify Curl behavior so that it doesn’t display the progress counter for the request. As we said, the body of the response is sent to the standard output file descriptor, so we use the | redirection operator (a pipe) to pass it as standard input to grep which we invoke with the -i option, to make it case-insensitive, and with -o to make it display only the matching part of a line (in this case it will display only the world “google”, instead of the whole line in which the word is found). Finally, we send the output produced by grep to the wc -w command, which returns the number of words.

Downloading a file

Curl can also be used to download files. Since, as we said, the body of a response is displayed on the standard output, we need to specify that we want it to be saved as a file on our filesystem, otherwise we would see gibberish in our terminal. In such cases we can make use of the -o option (short for --output), and provide, as argument, the name to use for the local file. Suppose we want to download the latest version of Firefox from the Mozilla official site. To save the tarball locally as firefox.tar.bz2, we would run the following command:

$ curl \
  --output firefox.tar.gz \
  https://download-installer.cdn.mozilla.net/pub/firefox/releases/106.0.1/linux-x86_64/en-US/firefox-106.0.1.tar.bz2

One important thing to notice is that we quoted the URL in order to avoid the shell to interpret possible special characters. As soon as we launch the command, the download progress will be displayed on screen:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   137  100   137    0     0    176      0 --:--:-- --:--:-- --:--:--   176
 87 75.1M   87 65.6M    0     0  1486k      0  0:00:51  0:00:45  0:00:06 1454k^C

As an alternative to the --output option, we can use --remote-name or -O. This will cause the file to be saved with its remote name, firefox-106.0.1.tar.bz2 in this case:

$ curl \
  --remote-name \
  'https://download-installer.cdn.mozilla.net/pub/firefox/releases/106.0.1/linux-x86_64/en-US/firefox-106.0.1.tar.bz2'

Performing POST requests

The POST http method is used to create new resources on a server. A POST requests usually include data, which can be send in different ways. Let’s see some examples.

Using the application/x-www-form-urlencoded content type

The simplest way we can include data in a POST request with Curl is to use the --data option (-d) or --data-urlencode. The data we send must be urlencoded (percent-encoded). If we use the former, we need to encode it beforehand; if we use the latter (available in recent versions of Curl), instead, the data will automatically encoded for us:

curl \
  --data-urlencode "name=John Smith" \
  --data-urlencode "age=30" \
  https://httpbin.org/post



When we perform a request like the one you see above, x-www-form-urlencoded is set automatically in the request header as value of the Content-Type parameter. In the above example, we sent the request to the https://httpbin.org site, which can be used to test requests. The server returns a response which reflects the data sent in the request:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "age": "30", 
    "name": "John Smith"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "22", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.82.0", 
    "X-Amzn-Trace-Id": "Root=1-6356ba6c-325eb51e69a7a95021ee6249"
  }, 
  "json": null, 
  "origin": "xxx.xxx.x.xx", 
  "url": "https://httpbin.org/post"
}

Sending JSON or XML data

In some occasions we may need to send a JSON string as data. In such cases, we need to do two things:

  • Set the Content-Type header paramater to application/json explicitly
  • Pass the JSON string as argument of the --data option.

Here is an example of such request:

$ curl \
  --header "Content-Type: application/json" \
  --data '{"name": "John Smith", "age": 30 }' \
  https://httpbin.org/post

The same principle should be applied in case we want to send XML data, for example:

$ curl \
  --header "Content-Type: text/xml" \
  --data '<person><name>John Smith</name><age>30</age></person>' \
  https://httpbin.org/post

Using the multipart/form-data content type

To make a POST request and include data using the multipart/form-data content type we can run Curl with the --form (-F) option. This is what we want to use, for example, when we need to upload a file. Here is the same request we made in the previous example, but performed with this content type:

$ curl \
  --form "name=John Smith" \
  --form "age=30" \
   https://httpbin.org/post

And here is the corresponding response sent by the server:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "age": "30", 
    "name": "John Smith"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "243", 
    "Content-Type": "multipart/form-data; boundary=------------------------e231a6560f5118bd", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.82.0", 
    "X-Amzn-Trace-Id": "Root=1-6356c1d1-05e70c5919d0971a3bd299e6"
  }, 
  "json": null, 
  "origin": "xxx.xx.x.xx", 
  "url": "https://httpbin.org/post"
}

Suppose, as part of the request, we want to upload a picture which currently exists in our working directory, associated to the “profile” field. Here is the command we would run:

curl \
  --form "name=John Smith" \
  --form "age=30" \
  --form "profile=@pic.jpg" \
  https://httpbin.org/post



As you can notice we used the @ symbol followed by the path of the image, which can be absolute or relative. The server returns this response when we perform the request. Notice the file we uploaded is in the “files” section of the response, associated to the corresponding form field  (I used the […] notation to omit the very long value, for convenience):

{
  "args": {}, 
  "data": "", 
  "files": {
    "profile-picture": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAADwAAAAhwCAYAAACQvstyAAABhWlDQ1BJQ0MgcHJvZmlsZQAAKJF9kT1Iw0AcxV9btVoqIu0g4pChOlkQFXHUKhShQqgVWnUwufRDaNKQpLg4Cq4FBz8Wqw4uzro6uAqC[...]"
  }, 
  "form": {
    "age": "30", 
    "name": "John Smith"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "153871", 
    "Content-Type": "multipart/form-data; boundary=------------------------94b6dea0b7f448d5", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.82.0", 
    "X-Amzn-Trace-Id": "Root=1-6356c368-2f39e19a67106e4141410178"
  }, 
  "json": null, 
  "origin": "xxx.xx.x.xx", 
  "url": "https://httpbin.org/post"
}

Sending the content of a file

Curl provides a special syntax we can use to get the content of a file, instead of uploading the file. Suppose, for example we have our name stored in the name.txt file in our current working directory. To pass its content as value of the “name” form field, we would use the < symbol, in the following manner:

curl --form "name=<name.txt" https://httpbin.org/post

Notice that all the content of the file will be used the value of the field, newline characters included.

Setting the request Headers

Header parameters are used to provide additional information into a request: some examples are Keep-Alive which is used to control how much a connection should stay open or User-Agent, which is used to identify the type of application and operating system, or Accept, used to specify what MIME types are accepted by the client. To set header parameters with Curl we can use the -H option (short for --header):

$ curl \
  --header "Accept: text/xml" \
  --header "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0" \
   https://httpbin.org/get

By the way, to set the User agent of a request, Curl provides a shortcut via the --user-agent option (-A):

$ curl --user-agent "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0" https://httpbin.org/get

Conclusions

In this article we see the basic usage of Curl. We learned how to perform requests using the most common http methods: GET and POST. We saw how to download a file, how to send different types of data in a POST requests, and how we can set the header parameters of a request. Curl is an immensely useful tool: here we just introduced it. Take a look at the manual, you will be surprised at its many features!