Description:

osscrapyhttpproxymiddleware 0.0.2

os-scrapy-httpproxy-middleware




This project provide middleware to enhance Scrapy built-in HttpProxyMiddleware. You can set proxy without sheme as the following format:
[user:password@]proxy:port

Install
pip install os-scrapy-httpproxy-middleware

You can run example spider directly in the project root path
scrapy crawl example

Usage
Settings


enable middleware, it is better disable Scrapy built-in HttpProxyMiddleware
DOWNLOADER_MIDDLEWARES = {
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": None,
"os_scrapy_httpproxy_middleware.middlewares.HttpProxyMiddleware": 750,
}



Unit Tests
sh scripts/test.sh

License
MIT licensed.

License

For personal and professional use. You cannot resell or redistribute these repositories in their original state.

Share

• Share on Facebook
• Share on Twitter