cURL - what is it and how to use it with a proxy

A short history of cURL

cURL was developed in 1997 by Daniel Stenberg as HttpGet. At first, it was a tool to get URLs, but soon developers went wild and expanded its upload capabilities, added additional protocols.

Now, cURL is a client-side URL transfer library. On top of supporting all major protocols, it also works on most platforms, including Android, MS Windows, iOS, macOS, NetBSD, or DOS. In addition, it supports over 40 programming languages and is compatible with IPv6.

cURL has become a global standard throughout the years, installed in over 5 billion devices worldwide.

What can cURL do?

cURL can do data transfer to and from servers through syntax commands. Since it doesn’t require any user interaction, cURL is often the first choice for automation.

Getting multiple URLs with cURL

cURL lets you get alphanumeric sequences by using brackets. Here’s an example:

Pro tip: if you’re using cURL from a command-line prompt, it’s best to put the full URL into double quotes. Otherwise, the shell might interfere with the syntax, especially if you use special symbols (e.g. ‘&’ or ‘?’) in the URL.

If you do not specify a protocol at the start of a syntax, cURL will use HTPP by default. Otherwise, it will use other protocols based on the prefix.

What is cURL used for

cURL works with multiple programming languages, so developers use it to create various apps. But your average Joe probably knows this command-line tool for calling APIs.

As for proxies, using cURL command is a great way for checking whether a proxy connection is set up correctly.

cURL installation

All new macOS and Linux distributions come with cURL already installed, so you don't need to put any additional effort to start using cURL. All gadgets with Windows 10 have cURL installed too.

If cURL is not provided, you can install it by running the install command on your Linux distributions. You just need to open Terminal on Ubuntu and run this command:

If you're running one of the older versions of Windows, download cURL from the official download page. You're welcome!

How to start using cURL

On a Mac, launch Terminal and run a command ‘curl –man’ to pull up the “man pages”.

In Windows open PowerShell and run ‘curl –man’. This will pull up the “man pages”.

The usual cURL syntax looks like this:

For example, you can use a cURL command to show the output of a URL with:

cURL can also use a proxy:

To learn more, check out the documentation in the cURL website.

Using cURL with HTTP/HTTPS proxy

As we have mentioned, most Windows and macOS devices come with pre-installed cURL libraries (libcurl). The unfortunate part is that many users are unaware of how to use these libraries properly and cURL with a proxy.

In reality, curling with a proxy is very simple. You simply need to add a variable in the cURL syntax so that your command line would include the -x or --proxy (both mean the same thing and can be used interchangeably). Just note that the command lines are case-sensitive!

For instance, this command will use the HTTP protocol with the proxy.

While this sets HTTPS as the protocol for the proxy:

You can set proxy protocols through the prefix by using any of the following prefixes:

This syntax sets the proxy protocol to the one you specify for the URL:

Since cURL is syntax-dependent, it offers no user interaction. That means you have to define a proxy in the cURL syntax. To do this, use the command below and enter proxy information, for example:

We use ipinfo.io in this example because it returns the proxy information and is very helpful when you want to be sure that you are using a proxy correctly.

Remember, you can also use the --proxy command instead of -x.

To check a random, rotating proxy on https://ipinfo.io with a cURL, use the following syntax:

This syntax will send a request to ipinfo.io with the HTTPS protocol and the gate.smartproxy.com:7000 endpoint (gate). This will work only if you have whitelisted your IP address in the Dashboard.

To check the same proxy with proxy user credentials (login:pass), you must use the -U parameter. Note that the parameter uses the capital U:

Why check proxies with cURL?

We use cURL because it is the easiest way to check your proxy endpoint setup before starting to scrape data. 

Smartproxy allows you to send unlimited concurrent requests from different IP addresses (including sticky IP sessions). This lets you scrape the web quicker and pulls a lot more data from public APIs.

Read more about proxy settings in our Help documents.

Using domain:port format in cURL

Here are some helpful tips on how to use cURL command with proxies.

Example for a rotating session, random location option

The gate.smartproxy.com:7000 will rotate your IP to an IP address from a random location with every new request.

cURL (with user:pass authorization) will look like:

cURL (with whitelist authorization) will look like:

Example for a rotating session, specific location option

You can use a single city/country specific proxy address (endpoint) to get a random proxy in that city/country with every query. For example, use us.smartproxy.com:10000 for IP addresses in the US.

cURL (with user:pass authorization) will look like:

cURL (with whitelist authorization) will look like:

Example for a sticky session, random location option

Your new IP address will not change with each new request but will stay the same for a longer period (from 1 and up to 30min). When you need to establish sticky sessions with IP addresses from random locations, use a single backconnect entry node with the sticky session parameter.

cURL (with user:pass authorization) will look like:

cURL (with whitelist authorization) will look like:

Example for a sticky session, specific location option

You need to choose sticky ports to keep an IP from a specific location for longer period. The most popular countries have up to 10,000 ports for sticky IPs. For example, it.smartproxy.com: 25762 will give the same Italian IP address for up to 10 minutes. Please note that this endpoint (it.smartproxy.com) has 10,000 sticky ports, from 20001 to 29999. If you need, you can download them all by clicking ‘Download this list’.

Also, if you need longer session time (for up to 30 min.), write the command as follows:

cURL (with user:pass authorization) will look like:

cURL (with whitelist authorization) will look like:

Note that with whitelisted IPs, sessions will always last for up to 10 minutes.

Using a backconnect entry node in cURL

The backconnect entry node can be used in various apps. You will need to pass every parameter for your session alongside the username in the username field. Here are some examples to guide you:

Example for random rotating proxy from a random country/city

Example for random proxy from a specific country and random city

In this case, we use rotating Italian proxy:

Example for a specific country with a sticky proxy (using session ID)

We use sticky Portuguese proxy with session ID. Randomstring123 denotes a session ID defined by the string of your choice:

Example for a specific country and city with sticky proxy (using session ID)

In this case, we use sticky US, New York city proxy with session ID. Randomstring123 denotes a session ID defined by string of your choice:

The case about environment variables

An environment variable is a dynamic-named value that can influence the behavior of the running processes on a computer. They are a part of the particular environment in which the process runs and can be used in two different ways:

Store information about your system

You can use environment variables to look up your username on CMD:

In addition, you can access hidden AppData folder with a command:

It'll show you in which system AppData is stored in your computer.

Path tracking

PATH is one of the main environment variables that locates all operating systems' executable programs. Find each executing process or user session with this command:

If you want to use cURL or Python, you need to run them through command-line that looks similar to this:

The best part is that if you use this environment variable in the cURL command, Windows will always recognize it and will run: