Description:
gallerycrawler 0.1.0
https://github.com/idlesign/gallerycrawler
Description
Generic crawling for galleries
Crawler starts from gallery listing URL;
It visits every details page mentioned on current listing page;
It gathers information from each details page;
It moves to the next listing URL.
Etc.
from galerycrawler.toolbox import Crawler, dump
# Define crawler.
class MyCrawler(Crawler):
selector_listing_next: str = '.page-next a'
selector_listing_thumbnails: str = '.thumbnail img'
selector_details: str = '.page-details a'
selector_details_title: str = '.page-title'
selector_details_img: str = '.image img'
selector_details_author: str = '.image-author'
# Run dumping.
dump(
crawler=MyCrawler,
url='https://mysite.some/gallery/',
fpath='dumped.html',
probe=True, # Use this to quick test your crawler
)
License
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
Customer Reviews
There are no reviews.
