Description:

osscrapyaioresolver 0.0.3

os-scrapy-aio-resolver




This project provide a async DNS resolver.
Install
pip install os-scrapy-aio-resolver

You can run example spider directly in the project root path
scrapy crawl example

Usage
Settings


enable DNS resolver in the project settings file:
DNS_RESOLVER = "os_scrapy_aio_resolver.AsyncResolver"



use asyncio reactor
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

or you can use os-scrapy command line to start crawling with -r asyncio
os-scrapy crawl example -r asyncio



enable/disable DNS cache, set cache size
DNSCACHE_ENABLED = True
DNSCACHE_SIZE = 10000



Unit Tests
sh scripts/test.sh

License
MIT licensed.

License

For personal and professional use. You cannot resell or redistribute these repositories in their original state.

Share

• Share on Facebook
• Share on Twitter