Back to blog

How to Download Files With cURL?

When working with the cURL command to interact with web services or APIs, saving the response from a remote server directly to a file is useful for various scenarios, such as logging responses, processing data offline, or simply preserving the output for review at a later time. cURL provides a straightforward way to download files, enhancing the tool's utility for developers and system administrators. In this blog post, we’ll demonstrate how to download files with cURL, including key commands for saving files, handling redirects, and managing authentication.

Martin Ganchev

Aug 26, 2024

6 min read

cURL request

Saving a file with the original name

To save a remote file to your local system with the same filename as in the URL, use the -O option:

curl -O http://example.com/api/resource

In this case, the downloaded file will keep its original name.

Saving a file with a specified name

To download a response to a file with a specific name, use the -o (or --output) argument, followed by the local file name to which you’d like to save the contents.

curl -o example.txt http://example.com/api/resource

In this example, the file will be saved under the "example.txt" name, and you'll avoid overwriting the existing file with the same name.

How to append responses to a file?

If you want to append the response to an existing file rather than overwrite it, you can use shell redirection in Unix-like systems or the equivalent in other operating systems. Here's how to append to a file in a Unix-like system:

curl http://example.com/api/resource >> response.txt

Note that when using shell redirection to append or save output (> for overwrite; >> for append), you're capturing the standard output (stdout) of the cURL command. This means that any headers or verbose information sent to standard error (stderr) won't be captured in the downloaded file unless you explicitly redirect stderr to stdout using 2>&1. Here’s how this would look in a complete cURL request, including both the redirection of output to a file and capturing verbose information:

curl -v https://example.com -o output.txt 2>&1
  • curl is the command to use cURL.
  • -v enables verbose mode, which sends detailed request and response information to stderr.
  • -o output.txt directs stdout (typically the response body) to a file named output.txt.
  • 2>&1 redirects stderr (where the verbose information goes) to stdout so that both the response data and the verbose details are captured in output.txt.

Handling redirects in cURL

Redirects are a standard mechanism on the web, instructing clients to fetch a resource from a different URL than the one initially requested. When saving a remote file using cURL, handling redirects ensures that your application can follow the server’s response just as a web browser would.

How to automatically follow redirects?

By default, cURL doesn’t follow redirect responses (like 301 Moved Permanently or 302 Found). To enable automatic following of redirects, use the -L or --location flag:

curl -L http://example.com/api/resource

With -L, if the server responds with a redirect, cURL will make a new request to the URL specified in the response's Location header.

Dealing with cookies across redirects

You might need to manage cookies across different domains or paths when handling redirects, especially with authentication flows. Use the -b option to send cookies and -c to store cookies provided in the response:

curl -L -b cookies.txt -c cookies.txt -X POST http://example.com/api/login

In this cURL command, -b cookies.txt tells cURL to use the existing cookies from the cookies.txt file for the request, while -c cookies.txt updates cookies.txt with any new cookies received in the response.

Bypassing authentication

To bypass HTTP or FTP authentication when downloading files, use the -u or --user flag:

curl -u username:password -O http://example.com/api/protected/resource

This method is straightforward but not the most secure, as the credentials are sent in plaintext. It's strongly recommended to use HTTPS when employing basic authentication to prevent eavesdropping.

Downloading files via proxy

For a step-by-step guide, check our help documentation on setting up and implementing our proxies to download files, but it can be as simple as generating a proxy list on our dashboard and using it like this:

curl -U "username:password" -x "us.smartproxy.com:10000" "https://ip.smartproxy.com/json"

Here, the -U option introduces proxy authentication, which should be followed with your proxy user’s credentials. The -x option specifies the proxy server: in this case, it’s a location from the US, but you can customize this and other parameters on our dashboard.

It's also possible to simply whitelist your IP and use the -x option, followed by the address of the proxy server, port, and target website:

curl -x "us.smartproxy.com:10000" "https://ip.smartproxy.com/json"

Saving headers and body

To save both the response headers and body to the same file, you can use the -i (or --include) option, which includes the HTTP response headers in the output:

curl -i http://example.com/api/resource -o response_with_headers.txt

Alternatively, if you want to save headers and body to separate files, you can use the -D option for headers:

curl http://example.com/api/resource -o body.txt -D headers.txt

Downloading multiple files

To download multiple files from different URLs, you can simply specify URLs and filenames in one cURL command line:

curl -O https://example1.com/image1.jpg -O https://example2.com/image2.jpg

To download multiple files for URLs that share a common pattern, use curly braces {}:

curl -O https://example.com/images/{image1.jpg,image2.jpg,image3.jpg}

In this case, all files share the same "https://example.com/images/" path.

Handling binary files

When downloading binary files, such as an image or an archive, ensure that you write the output in binary mode to prevent data corruption. cURL handles this automatically when using -o or -O, but it's an important consideration when redirecting output through the shell:

curl http://example.com/api/download -o image.jpg

Saving file in verbose mode

If you're using verbose mode (-v) to debug a request and want to save the output (including verbose information) to a file, you can redirect both stdout and stderr to the file:

curl -v http://example.com/api/resource -o response_verbose.txt 2>&1

What is the verbose mode in cURL?

Verbose mode in cURL is an invaluable feature providing a detailed view of the request-response cycle. This mode outputs additional details to stderr about the HTTP conversation between the client (cURL) and the server. It includes information such as request headers, response headers, and the negotiation process of the protocols. Understanding and effectively using verbose mode can significantly aid in debugging, optimizing, and understanding web interactions.

How to enable verbose mode?

To enable verbose mode in cURL, you use the -v or --verbose flag. Here's a basic example:

curl -v http://example.com/api/resource

The -v flag instructs cURL to print detailed information about the request and response, including the full set of HTTP headers.

How can the verbose mode help to debug a request?

Verbose mode provides insights into several aspects of the HTTP request and response process:

  • Connection details. Shows the IP address and port number of the server to which cURL connects.
  • TLS/SSL handshake. If connecting over HTTPS, verbose mode displays the TLS handshake process, including the chosen protocol version and cipher suite.
  • Request headers. Displays all headers sent to the server, which can help ensure that your request is formatted correctly.
  • Response headers. Lists all headers returned by the server, useful for debugging caching issues, cookies, and more.
  • Redirects. Shows the process of following redirects if the -L flag is used, including the response codes and the Location headers that guide the redirects.

How to combine verbose mode with output redirection?

When using verbose mode, the details are printed to stderr, while the actual content of the response is sent to stdout. This separation allows you to redirect them separately, for instance, saving the response to a file while still viewing the verbose output on the screen:

curl -v http://example.com/api/resource -o response.txt

To capture both the verbose data and the response in files, you can redirect stderr to a separate file:

curl -v -http://example.com/api/resource -o response.txt 2> debug.txt

Mind the security

While the verbose mode is highly useful for debugging, be cautious with the output, especially in shared environments or when saving to logs, as it can contain sensitive information like authentication tokens or cookies.

Rate limiting

You can use the --limit-rate [speed] flag in cURL to restrict the maximum download speed, which is particularly helpful on slow internet connections. You can specify the rate limit value with a letter suffix: K for kilobytes, M for megabytes, or G for gigabytes.

curl https://example.com/ --limit-rate 300K

In this example, the maximum download speed is 300 kilobytes.

Final thoughts

If you're working with web services and APIs, using cURL for file downloads can enhance your efficiency when saving responses, handling redirects, or managing authentication. Using various options to download multiple files, save files via proxy, automatically follow redirects, and other tasks, cURL becomes an essential tool for optimizing your workflow.

About the author

Martin Ganchev

VP Enterprise Partnerships

Martin, aka the driving force behind our business expansion, is extremely passionate about exploring fresh opportunities, fostering lasting relationships in the proxy market, and, of course, sharing his insights with you.

LinkedIn

All information on Smartproxy Blog is provided on an as is basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Smartproxy Blog or any third-party websites that may belinked therein.

Frequently asked questions

What is the difference between wget and curl for downloading files?

wget is a command-line utility designed to download files from the web and automatically handle file downloads, recursion, and mirroring. Meanwhile, curl is more versatile, offering not only file downloads but also a wide range of options for interacting with web services and APIs, such as handling different protocols, managing authentication, and following redirects.

What is a silent mode download?

Web scraping SERPs with cURL and terminal tutorial

Alternative Google SERP Scraping Techniques - Terminal and cURL [VIDEO]

Google has become a gateway to easily-accessible information. And one of the best ways to make use of Google’s limitless knowledge is data scraping. We’ve just released a detailed blog post about scraping Google SERPs with Python, where we cover lots of useful info, including the technical part. So before you dive into this tutorial – check it out. But what if Python is not exactly your forte? This blog post will show you how to scrape SERPs using a simpler method. One that doesn't require much tech knowledge or downloading loads of applications and software. So, what do you know about web scraping with cURL and Terminal?

Mariam Nakani

Dec 23, 2021

10 min read

© 2018-2024 smartproxy.com, All Rights Reserved