GitLocker: The Coding Marketplace

Description:

gdprCrawlerTest19 0.0.2

GDPRxiv Crawler (README)

An efficient tool to crawl GDPR legal documents!

About The Project
With the introduction of the Europeans Union's General Data Protection Regulation (GDPR), there has been an explosion in the number of legal
documents pertaining to case reviews, analyses, legal decisions, etc... that mark the enforcement of the GDPR.
Additionally, these documents are spread across over 30 Data Protection (DPA) and Supervisory Authorities. As a result, it is
cumbersome for researchers/legal teams to access and download a large quantity of GDPR documents at once.
To address this, we have created GDPRxiv Crawler, a command-line tool that allows users to efficiently filter and
download GDPR documents. Users may select their desired DPA and document_type, and GDPRxiv Crawler will scrape the web
and download all up-to-date documents.
Of course, it is impossible to entirely keep up with DPA website redesigns and newly added document categories.
However, we hope that this tool will eliminate the bulk of the workload and allow users to focus on more important tasks.
Built With

BeautifulSoup4
Selenium
Add more later...

Getting Started
Prerequisites
Python 3.9 is required. This python version includes the pip installer and the venv module, which is needed to create a
virtual environment.
It is strongly recommended that users utilize a virtual environment when installing this package.
See below to create and activate one.
In a directory:

venv:
virtualenv <virtual env name>

Activate the virtual environment:
source <virtual env name>/bin/activate

Installation
At any moment, use command 'pip3 list' to view all installed packages.

Download requirements.txt
and place it in the directory that contains the virtual environment.
Install package requirements
pip3 install -r requirements.txt

Install the GDPRxiv Crawler package
pip3 install -i https://test.pypi.org/simple/ gdprCrawlerTest15

Usage
Downloaded documents will be organized into a set of folders based on DPA and document type.
A file called visitedDocs.txt is always created upon an initial run within a new directory. This file records each downloaded document's
unique hash, which allows the tool to avoid overwriting existing documents (if desired) in future runs.

Scrape desired documents:
gdprCrawler scrape --country <country name> --document_type <document type> --path <directory to store documents>

The same directory can be used for multiple countries: the scraper automatically organizes documents based on country and document type.

Optionally, the --overwrite argument can be included if users would like to overwrite existing documents:
gdprCrawler scrape --country <country name> --document_type <document type> --path <directory to store documents> --overwrite <True/False>

Overwrite is False by default.

Country and document type arguments should be written exactly as they appear below:
SUPPORTED COUNTRIES: DOCUMENTS TYPES:

Austria Decisions
Belgium Annual Reports, Decisions, Opinions
Bulgaria Annual Reports, Opinions
Croatia Decisions
Cyprus Annual Reports, Decisions
Czech Republic Annual Reports, Completed Inspections, Court Rulings, Decisions, Opinions, Press Releases
Denmark Annual Reports, Decisions, Permissions
EDPB (Agency) Annual Reports, Decisions, Guidelines, Letters, Opinions, Recommendations
Estonia Annual Reports, Instructions, Prescriptions
Finland Advice, Decisions, Guides, Notices
France FUTURE UPDATE
Germany N/A
Greece Annual Reports, Decisions, Guidelines, Opinions, Recommendations
Hungary Annual Reports, Decisions, Notices, Recommendations, Resolutions
Ireland Decisions, Judgements, News
Italy Annual Reports, Hearings, Injunctions, Interviews, Newsletters, Publications
Latvia Annual Reports, Decisions, Guidances, Opinions, Violations
Lithuania Decisions, Guidelines, Inspection Reports
Luxembourg Annual Reports, Opinions
Malta Guidelines, News Articles
Netherlands Decisions, Opinions, Public Disclosures, Reports
Poland Decisions, Tutorials
Portugal Decisions, Guidelines, Reports
Romania Decisions, Reports
Slovakia Fines, Opinions, Reports
Slovenia Blogs, Guidelines, Infographics, Opinions, Reports
Spain Blogs, Decisions, Guides, Infographics, Reports
Sweden Decisions, Guidances, Judgements, Publications
United Kingdom Decisions, Judgements, Notices

Contributing
All suggestions and contributions you make are greatly appreciated.
License
Distributed under the MIT License. See LICENSE.txt for more information.
Contact

Project Link: https://github.com/GDPRxiv/crawler
Acknowledgments
Thank you to everyone who has supported the project in any way. We greatly appreciate your time and effort!

Choose an Open Source License
Img Shields

(back to top)