os-scrapy-uvicorn 0.0.1

Description:

osscrapyuvicorn 0.0.1

os-scrapy-uvicorn




This project provide a extension to start a ASGI http server(Uvicorn) along with Scrapy in the same process.
You can use the ASGI framework(recommend FastAPI) to create app to communicate with Scrapy.
Require: Python 3.6+, Scrapy 2.0+
Install
pip install os-scrapy-uvicorn

You can run example spider directly in the project root path
scrapy crawl example

Settings


use asyncio reactor
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

or you can use os-scrapy(installed with this project) to start crawling with -r command line option
os-scrapy crawl -r asyncio example



enable extension
EXTENSIONS = {
"os_scrapy_uvicorn.Uvicron": 1,
}



app path, you can use ASGI app frameworks to create your app, FastAPI is recommended
UVICORN_APP = "app_module:app"



uvicorn server settings, supported settings
UVICORN_CONFIG = {"host": "0.0.0.0", "port": 5000}



when the server started, Scrapy crawler instance is attached to the app, it is the entrypoint to commnicate with Scrapy


Unit Tests
sh scripts/test.sh

License
MIT licensed.