Proxy Integration with Scrapy Proxy Middleware

Getting residential proxies

Log in to your Smartproxy dashboard, find residential proxies by choosing Residential under the Residential Proxies column on the left panel, and select a plan that suits your needs. Then, follow these steps:

1. Open the Proxy setup tab.
2. Navigate to the Endpoint generator below.
3. Configure the parameters. Set your authentication method, location, session type, and protocol. 
4. Select the number of proxy endpoints you want to generate (default – 10). 
5. Copy the endpoints by clicking the Copy button.

Configuration

1. Open the settings.py file in your project folder.

2. Edit the file by adding the following properties at the bottom:

DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'yourprojectname.smartproxy_auth.ProxyMiddleware': 100,
}

SMARTPROXY_USER = 'username'## Smartproxy username
SMARTPROXY_PASSWORD = 'password'## Smartproxy password
SMARTPROXY_ENDPOINT = 'gate.smartproxy.com'## Endpoint you'd like to use
SMARTPROXY_PORT = '7000'## Port of the endpoint you're using

3. Change the yourprojectname in the above to the name of your project folder.

4. Enter the Smartproxy credentials and endpoint information you’ve received from the dashboard.

5. Your web crawlers will now be going through Smartproxy proxies.