Description:
osscrapyuvicorn 0.0.1
os-scrapy-uvicorn
This project provide a extension to start a ASGI http server(Uvicorn) along with Scrapy in the same process.
You can use the ASGI framework(recommend FastAPI) to create app to communicate with Scrapy.
Require: Python 3.6+, Scrapy 2.0+
Install
pip install os-scrapy-uvicorn
You can run example spider directly in the project root path
scrapy crawl example
Settings
use asyncio reactor
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
or you can use os-scrapy(installed with this project) to start crawling with -r command line option
os-scrapy crawl -r asyncio example
enable extension
EXTENSIONS = {
"os_scrapy_uvicorn.Uvicron": 1,
}
app path, you can use ASGI app frameworks to create your app, FastAPI is recommended
UVICORN_APP = "app_module:app"
uvicorn server settings, supported settings
UVICORN_CONFIG = {"host": "0.0.0.0", "port": 5000}
when the server started, Scrapy crawler instance is attached to the app, it is the entrypoint to commnicate with Scrapy
Unit Tests
sh scripts/test.sh
License
MIT licensed.
License
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
Customer Reviews
There are no reviews.
