import scrapy
import urllib
class Demo(scrapy.Spider):
name = 'demo'
def parse(self, response):
print(response.body)
def start_requests(self):
username = '{PROXY_USERNAME}'
password = '{API_KEY}'
url = 'https://cloudflare.com/cdn-cgi/trace' # Insert your target URL here
host = 'isp.joinmassive.com'
port = 4080
proxy = f'http://{username}:{password}@{host}:{port}'
yield scrapy.Request(url, callback=self.parse, meta={'proxy': proxy})
Sample Integrations
Scrapy (for Python)
After installing Scrapy (pip install scrapy) and creating a project (scrapy startproject [project name]), you can connect to the Massive Network by saving code like that below to a file in the new [project name]/[project name]/spiders subdirectory then running scrapy crawl demo from the top-level [project name] directory (Scrapy doesn’t seem to support HTTPS proxy connections, but you can use the network’s HTTP port):