smartproxy
  • Smartproxy >
  • Integrations >
  • Proxy Integration with Scrapy Proxy Middleware

Proxy Integration with Scrapy Proxy Middleware

Learn how to set up residential proxies with the Scrapy proxy middleware.

55M+ IP pool
Advanced targeting
195+ global locations
24/7 tech support
Easy to integrate
99.99% uptime
Get proxies
See pricing
Proxy Integration with Scrapy Proxy Middleware

What is the Scrapy proxy middleware?

Scrapy is an open-source web crawling framework for Python that extracts data from websites, processes it, and stores it. The proxy middleware enables it to bypass IP-based rate limiting and access geographically restricted content, improving the efficiency and reach of web scraping and crawling tasks.

Asynchronous scraping

Asynchronous scraping

Ensure lightning-fast data collection by allowing multiple requests to be processed simultaneously.

Built-in selectors

Built-in selectors

Streamline data extraction by quickly and accurately gathering the information you need from web pages.

A range of add-ons

Middleware support

Integrate proxies in just minutes thanks to simple middleware support.

Why residential proxies?

Why residential proxies?

A residential proxy serves as a mediator, allowing users to get an IP address from an authentic desktop or mobile device connected to a local network. Due to its origin, residential proxies are a perfect match for overcoming geo-restrictions, bypassing CAPTCHAs, managing multiple accounts, and conducting web testing with the CapSolver platform.

Smartproxy offers top-notch residential proxies with an extensive IP pool of over 55M IPs across 195+ locations. With an unparalleled responsive rate, clocking in at under 0.6 seconds, a success rate of 99.47%, and an affordable entry point with Pay As You Go, Smartproxy is a great deal for hustlers and fast-growing companies.

Set up Smartproxy proxies with Scrapy

To install the Scrapy proxy middleware, you’ll need to set up a Scrapy project first. Follow the official installation and documentation to create your project. Then, follow the instructions below to set up the middleware:

smartproxy

Getting residential proxies

Log in to your Smartproxy dashboard, find residential proxies by choosing Residential under the Residential Proxies column on the left panel, and select a plan that suits your needs. Then, follow these steps:

  1. Open the Proxy setup tab.
  2. Navigate to the Endpoint generator below.
  3. Configure the parameters. Set your authentication method, location, session type, and protocol
  4. Select the number of proxy endpoints you want to generate (default – 10). 
  5. Copy the endpoints by clicking the Copy button.
Installation

Installation

Once you’ve obtained the endpoint information, you can install the middleware:

  1. Open your terminal tool and navigate to the project folder using cd yourprojectname.
  2. Download the proxy middleware using the following command: curl https://raw.githubusercontent.com/Smartproxy/Scrapy-Middleware/master/smartproxy_auth.py > smartproxy_auth.py or get it from the GitHub repository.
Configuration

Configuration

1. Open the settings.py file in your project folder.

2. Edit the file by adding the following properties at the bottom:

DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'yourprojectname.smartproxy_auth.ProxyMiddleware': 100,
}

SMARTPROXY_USER = 'username'## Smartproxy username
SMARTPROXY_PASSWORD = 'password'## Smartproxy password
SMARTPROXY_ENDPOINT = 'gate.smartproxy.com'## Endpoint you'd like to use
SMARTPROXY_PORT = '7000'## Port of the endpoint you're using

3. Change the yourprojectname in the above to the name of your project folder.

4. Enter the Smartproxy credentials and endpoint information you’ve received from the dashboard.

5. Your web crawlers will now be going through Smartproxy proxies.

Configurations & Integrations

Easy Smartproxy proxy setup with popular applications and free third-party tools. Check out these guides to get started right away.

Configurations
Integrations
smartproxy

Chrome

Learn more

Reliable proxies at affordable prices

Pick a plan that suits you. Get ahead with unblockable and stable proxies.

Regular
Enterprise
PAY AS YOU GO

No commitment

$7/GB

Total:

$7

2 GB

Save 14%

$7

$6/GB

Total: $14

$12 +VAT billed monthly

8 GB

Save 21%

$7

$5.5/GB

Total: $56

$44 +VAT billed monthly

MOST POPULAR

25 GB

Save 26%

$7

$5.2/GB

Total: $175

$130 +VAT billed monthly

50 GB

Save 30%

$7

$4.9/GB

Total: $350

$245 +VAT billed monthly

With each plan you access

smartproxy

55M+ ethically-sourced IPs

smartproxy

HTTP(S) & SOCKS5 support

smartproxy

99.68% success rate

smartproxy

24/7 tech support

smartproxy

Unlimited concurrent sessions

smartproxy

Country, city & ZIP code targeting

smartproxy

<0.5s average response time

smartproxy

Rotating and sticky sessions

100 GB

Save 36%

$7

$4.5/GB

Total: $700

$450 +VAT billed monthly

SSL Secure Payment

Your information is protected by 256-bit SSL

What people are saying about us

We're thrilled to have the support of our 50K+ clients and the industry's best

Clients
Awards
Industry experts

Recognized by:

Cybernews
Hackernoon
Techjury
Techradar
Yahoo news

Explore our other proxy line products

  • What are proxies?

    A proxy is an intermediary between your device and the internet, forwarding requests between your device and the internet while masking your IP address.

  • Residential Proxies
    Residential Proxies from $2.2/GB

    Real, physical device IPs that provide a genuine online identity and enhance your anonymity online. Learn more

    Quick start guide
  • ISP Proxies
    ISP Proxies from $2/IP

    IPs assigned by Internet Service Providers (ISPs), offering efficient and location-specific online access with minimal latency. Learn more

  • Mobile Proxies
    Mobile Proxies from $6/GB

    Mobile device based IPs offering anonymity and real user behavior for mobile-related activities on the internet. Learn more

  • Datacenter Proxies
    Datacenter Proxies from $0.09/IP

    Remote computers with unique IPs for tasks requiring scalability, fast response times, and reliable connections. Learn more

  • Site Unblocker
    Site Unblocker from $6/GB

    A powerful application for all proxying activities offering dynamic rendering, browser fingerprinting, and much more. Learn more

Smartproxy Blog

Build knowledge on residential proxies, or pick up some dope ideas for your next project - our blog is just the perfect place.

Most recent

Web scraping Excel hero
data collection

Excel is an incredibly powerful data management and analysis tool. But did you know that it can also...

smartproxy

Zilvinas Tamulis

May 27, 2024

7 min. read

Most popular

smartproxy
PythonData Collection

How to Scrape Google Maps: A Step-By-Step Tutorial 2024

Dominykas Niaura

Mar 29, 2024

10 min. read

Frequently asked questions

What is Scrapy?

Scrapy is an open-source web crawling framework for Python designed to extract, process, and store data from websites.

What is Scrapy used for?

Scrapy is used to create web crawlers (spiders) that define how to navigate and scrape web pages, making it a powerful tool for web scraping and data mining tasks.

Be aware that any tools mentioned in this article belong to a third party, not Smartproxy. Therefore, Smartproxy will not be responsible for any of the services offered by the third-party. Make sure to thoroughly review the third-party’s policies and practices, or do your due diligence, before using or accessing any of their services.

The Fastest Residential Proxies

Dive into a 55M+ ethically-sourced residential IP pool from 195+ locations worldwide.

14-day money-back option

Get in touch

Follow us

Company

© 2018-2024 smartproxy.com, All Rights Reserved