How to Implement Proxy Rotation in Puppeteer and Playwright

Proxy rotation is a crucial technique used in web scraping and automation to avoid IP blocking and rate-limiting. When using tools like Puppeteer and Playwright, proxy rotation ensures that each request is routed through a different IP address, mimicking the behavior of a real user. This article will delve into how to implement proxy rotation with both Puppeteer and Playwright, including the necessary configuration and coding examples.

Why Proxy Rotation is Necessary

Proxy rotation serves several purposes:

Avoiding IP blocks: Web scraping often triggers anti-bot systems that block repeated requests from the same IP.
Bypassing rate-limiting: Many websites impose restrictions based on the number of requests coming from the same IP address.
Ensuring anonymity: Proxies help mask the origin of requests, which is useful for privacy and security.
Enhancing efficiency: Rotating proxies enables faster data extraction by preventing delays due to blocking.

Proxy Rotation in Puppeteer

Puppeteer is a Node.js library that provides a high-level API for controlling Chromium browsers. To implement proxy rotation in Puppeteer, you need to set up proxies and configure Puppeteer to switch between them during requests.

Setting Up Proxies

Before rotating proxies, ensure you have a list of working proxies. These proxies can be either HTTP or SOCKS proxies. For proxy rotation, you’ll need a proxy pool, which can be a simple array of proxy addresses.

javascript
const puppeteer = require(‘puppeteer’);

const proxies = [
‘http://proxy1.example.com:8080’,
‘http://proxy2.example.com:8080’,
‘http://proxy3.example.com:8080’,
];

async function launchBrowserWithProxy(proxy) {
const browser = await puppeteer.launch({
headless: true,
args: [–proxy-server=${proxy}]
});
return browser;
}

async function scrapeWithProxies() {
for (let i = 0; i < proxies.length; i++) {
const browser = await launchBrowserWithProxy(proxies[i]);
const page = await browser.newPage();
await page.goto('https://example.com');
// Perform scraping tasks here...
await browser.close();
}
}

scrapeWithProxies();

Rotating Proxies for Every Request

To further enhance rotation, you can change the proxy on each request. This can be achieved by setting the proxy for each page individually.
javascript
async function scrapeWithRotatingProxies() {
for (let i = 0; i < proxies.length; i++) {
const browser = await puppeteer.launch({
headless: true
});
const page = await browser.newPage();
await page.setRequestInterception(true);

page.on('request', (request) => {
request.continue({
url: request.url(),
headers: {
…request.headers(),
‘Proxy’: proxies[i] // Use the proxy for this request
}
});
});

await page.goto(‘https://example.com’);
// Perform scraping tasks here…
await browser.close();
}
}

Proxy Rotation in Playwright

Playwright is another Node.js library similar to Puppeteer, but it supports multiple browsers including Chromium, Firefox, and WebKit. Proxy rotation in Playwright can be implemented in a similar manner.

Setting Up Proxies in Playwright

javascript
const { chromium } = require(‘playwright’);

const proxies = [
‘http://proxy1.example.com:8080’,
‘http://proxy2.example.com:8080’,
‘http://proxy3.example.com:8080’,
];

async function launchBrowserWithProxy(proxy) {
const browser = await chromium.launch({
headless: true,
proxy: { server: proxy }
});
return browser;
}

scrapeWithProxies();

Rotating Proxies on Each Request

In Playwright, you can rotate proxies for each individual request by setting up a proxy for each page or request. This allows you to avoid having multiple requests originate from the same IP address.
javascript
const { chromium } = require(‘playwright’);

async function scrapeWithRotatingProxies() {
for (let i = 0; i < proxies.length; i++) {
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

await page.route('**/*', (route, request) => {
route.continue({
url: request.url(),
headers: {
…request.headers(),
‘Proxy’: proxies[i] // Use the proxy for this request
}
});
});

await page.goto(‘https://example.com’);
// Perform scraping tasks here…
await browser.close();
}
}

Managing Proxy Failures

When implementing proxy rotation, there is always a risk of proxies failing due to various reasons, such as proxy blacklistings or network issues. To handle this, it’s important to build a mechanism for checking proxy health before using them.

Proxy Health Check

One way to ensure proxies are working is by performing a health check before rotating to a new proxy.
javascript
const axios = require(‘axios’);

async function checkProxyHealth(proxy) {
try {
const response = await axios.get(‘https://example.com’, {
proxy: {
host: proxy.split(‘:’)[0],
port: proxy.split(‘:’)[1]
}
});
return response.status === 200;
} catch (error) {
return false;
}
}

async function scrapeWithReliableProxies() {
for (let i = 0; i < proxies.length; i++) {
if (await checkProxyHealth(proxies[i])) {
const browser = await launchBrowserWithProxy(proxies[i]);
const page = await browser.newPage();
await page.goto('https://example.com');
// Perform scraping tasks here...
await browser.close();
} else {
console.log(Proxy ${proxies[i]} is down.);
}
}
}

Conclusion

html

Proxy rotation is a powerful technique to enhance the robustness of your web scraping or automation tasks. Both Puppeteer and Playwright provide flexible ways to integrate proxy rotation, either for individual requests or for entire browser sessions. With proxy health checks and the ability to switch proxies dynamically, you can ensure that your automation remains smooth and efficient while bypassing restrictions effectively.

We earn commissions using affiliate links.