Get Started With Web Scraping API

Collect raw HTML data in bulk without worrying about anti-scraping restrictions, CAPTCHAs, or proxy management. Learn how to use this scraping solution with our quick start guide.

Get scraping API

14-day money-back option

Any country

100% success rate

Results in HTML

Headless scraping

Synchronous or asynchronous requests

Real-time integration

Proxy-like integration

Free 7-day trial

What is Web Scraping API?

Our Web Scraping API provides a versatile data collection solution for websites not covered by our other scraping services. It leverages a vast pool of proxies alongside an advanced web scraper to retrieve raw HTML data.

Most popular Web Scraping API use cases

Choose your target and gather data with confidence using our state-of-the-art scraper.

Public data collection

Effortlessly gather real-time public data for analysis or easy integration into your projects.

Travel fare aggregation

Collect relevant pricing data to get the best deals for hotels, flights, and car rentals.

Fraud protection

Detect malicious websites and prevent fraud from global locations by scanning scraped HTML data.

Start 7-day free trial

How to set up Web Scraping API

Get started with your Web Scraping API configuration by following our guide.

Choose a subscription

After creating your account, find the Web section under the Scraping side menu.

Next, click the Pricing tab and select a subscription plan that suits your needs. Alternatively, start a free trial to get 1K requests for 7 days.

Authenticate with username:password

In the API Authentication tab, click on either the username or password to copy it. Use the buttons on the right to reveal your password or generate a new one.

Create a request command

Go to the Scrapers tab, where you can create a new project or use a ready-made scraper from the Popular Scrapers section.

When creating a new project, select a target, then enter the URL or query you wish to scrape. If applicable, choose the relevant parameters. You can find more parameter options in our help documentation.

On the right, you can copy the code in cURL, Node.js, or Python for use in your own environment.

Alternatively, click Save and Scrape to send your request directly. Once the task is complete, the response will be displayed on the screen.

Copy or download the result in HTML

After sending a request using the Scrapers capabilities on the dashboard, you can copy or download the response in JSON or CSV.

Track scraping usage

To track your scraping API usage, navigate to the Statistics tab. There, you’ll find traffic usage details over a selected timeframe.

Easy to integrate scraper

Our Web Scraping API works with all popular programming languages, ensuring a smooth connection to other tools in your business suite.

import requests
  
url = "https://scraper-api.smartproxy.com/v2/scrape"
  
payload = {
      "target": "universal",
      "url": "https://ip.smartproxy.com/",
      "headless": "html"
}
  
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "authorization": "Basic [YOUR_BASE64_ENCODED_CREDENTIALS]"
}
  
response = requests.post(url, json=payload, headers=headers)
  
print(response.text)

import requests
  
url = "https://scraper-api.smartproxy.com/v2/scrape"
  
payload = {
      "target": "universal",
      "url": "https://ip.smartproxy.com/",
      "headless": "html"
}
  
headers = {
    "accept": "application/json",
    "content-type": "application/json",
    "authorization": "Basic [YOUR_BASE64_ENCODED_CREDENTIALS]"
}
  
response = requests.post(url, json=payload, headers=headers)
  
print(response.text)

<?php
$url = "https://scraper-api.smartproxy.com/v2/scrape";

$payload = json_encode([
    "target" => "universal",
    "url" => "https://ip.smartproxy.com/",
    "headless" => "html"
]);

$headers = [
    "Accept: application/json",
    "Content-Type: application/json",
    "Authorization: Basic [YOUR_BASE64_ENCODED_CREDENTIALS]"
];

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $payload);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$response = curl_exec($ch);
if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

echo $response;
?>

const response = await fetch("https://scraper-api.smartproxy.com/v2/scrape", {
    method: "POST",
    body: {
      "target": "universal",
      "url": "https://ip.smartproxy.com/",
      "headless": "html"
    },
    headers: {
      "Content-Type": "application/json",
      "Authorization": "Basic [YOUR_BASE64_ENCODED_CREDENTIALS]"
    },
  }
).catch(error => console.log(error));
  
console.log(response)

curl --request 'POST' \
        --url 'https://scraper-api.smartproxy.com/v2/scrape' \
        --header 'Accept: application/json' \
        --header 'Authorization: Basic [YOUR_BASE64_ENCODED_CREDENTIALS]' \
        --header 'Content-Type: application/json' \
        --data '
    {
      "target": "universal",
      "url": "https://ip.smartproxy.com/",
      "headless": "html"
    }
'