How to set up a proxy in Scraper API

Scraper API is a sophisticated tool designed for website scraping with affordable proxies, offering vital solutions for efficient business operations. It enables rapid data extraction from targeted websites and formats it for easy analysis. Applications include sourcing SEO-optimized product descriptions from competitors' sites, monitoring product prices in online stores, collecting statistics on user activities like likes, reposts, and views, and tracking the latest news on information portals — high-volume jobs where you can buy IPv6 proxy addresses from a large, low-cost pool. However, websites often block IP addresses associated with web scraping activities. To ensure uninterrupted functionality, configuring high-trust affordable ISP proxies — whose provider-assigned residential IPs are less likely to be flagged — is essential for Scraper API. To ensure uninterrupted functionality, configuring a proxy for Scraper API is essential.

Step-by-step proxy setup in Scraper API

To set up a proxy for Scraper API scraping, follow these steps:

1. Create an account on ScraperAPI and navigate to the “Dashboard” category on the main page. Locate the “Sample Proxy Code” field.

   

2. The “Sample Proxy Code” field contains code similar to this:

   curl -x"http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001" -k "http://httpbin.org/ip"
   

3. Copy this code and paste it into your script. Make the following modifications to the code:

   Replace “scraperapi” with your proxy username.

   Substitute "APIKEY" with your password.

   Change “proxy-server.scraperapi.com” to your new IP address.

   After the IP address, add the port number, separated by a colon.

   After “-k” and within the quotes, specify the URL of the page you wish to scrape.

The modified code should look like this:

curl -x"http://LOGIN:PASSWORD@IP-ADRESS:8001" -k "http://httpbin.org/ip"

This command can be adapted for different programming languages, and the code may vary accordingly. Example for Python:

import requests

proxies = {
    "http": "http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"
}

r = requests.get('http://httpbin.org/ip', proxies=proxies, verify=False)
print(r.text)

# Scrapy users can likewise simply pass their API key in headers.
# NB: Scrapy skips SSL verification by default.
# ...other scrapy setup code

start_urls = ['http://httpbin.org/ip']

meta = {
    "proxy": "http://scraperapi:APIKEY@proxy-server.scraperapi.com:8001"
}

def parse(self, response):
    # ...your parsing logic here
    yield scrapy.Request(url, callback=self.parse, headers=headers, meta=meta)

Ruby:

require 'httparty'

HTTParty::Basement.default_options.update(verify: false)

response = HTTParty.get('http://httpbin.org/ip', {
  http_proxyaddr: "proxy-server.scraperapi.com",
  http_proxyport: "8001",
  http_proxyuser: "scraperapi",
  http_proxypass: "APIKEY"
})

results = response.body
puts results

NodeJS:

const axios = require('axios');

axios.get('http://httpbin.org/ip', {
    method: 'GET',
    proxy: {
        host: 'proxy-server.scraperapi.com',
        port: 8001,
        auth: {
            username: 'scraperapi',
            password: 'APIKEY'
        },
        protocol: 'http'
    }
})
.then(response => {
    console.log(response.data);
})
.catch(error => {
    console.log(error);
});

With the proxy setup complete, you can send requests from various IPs, reducing the likelihood of IP blocking. This also enables requests to resources that are geo-blocked — for which you can buy IPv4 proxy addresses with a static IP in the target region.