os-scrapy-ipfilter-resolver 0.0.4

Description:

osscrapyipfilterresolver 0.0.4

os-scrapy-ipfilter-resolver




This project provide a DNS Resolver for ip blacklist/whitelist and config DNS expire time.
Install
pip install os-scrapy-ipfilter-resolver

You can run example spider directly in the project root path.
scrapy crawl example

Usage
Settings


enable DNS Resolver, in the project settings.py file:
DNS_RESOLVER = "os_scrapy_ipfilter_resolver.Resolver"



config IP blacklist, it will raise IPFilteredException when the request ip in the blacklist
IP_DISALLOWED = ["192.168.0.0/16", "10.143.0.1"]



config IP whitelist, priority greater than blacklist
IP_ALLOWED = ["192.168.0.1"]



config DNS cache size and expire time(seconds)
DNSCACHE_ENABLED = True
DNSCACHE_SIZE = 10000
DNSCACHE_EXPIRE = 24 * 60 * 60



config DNS lookup timeout(seconds)
DNS_TIMEOUT = 60



Unit Tests
tox

License
MIT licensed.