sloth_link_crawler

Description:

sloth link crawler

ARE YOU A SLOTH? - SLOTH LINK CRAWLER #
This package is WIP!
It will crawl an url and list all links. We try to respect the robots.txt file. See example on how to implement into your codebase.
Features #

Crawl URL
List all links (internal OR all)
Delay crawl requests
Custom user agent

Getting started #
With dart:
$ dart pub add sloth_link_crawler
copied to clipboard
With Flutter:
$ flutter pub add sloth_link_crawler
copied to clipboard
Usage #
Import the package
import 'package:sloth_link_crawler/sloth_link_crawler.dart';
copied to clipboard
Access the extensions:
final crawler = SlothLinkCrawler(
// Domain to crawl
baseUrl: 'https://example.com',
// if true only urls of the domain will get listed
onlyInternalDomainLinks: true,
// if true some debugging information will be displayed in the console
debugMode: true,
// set the duration between crawl calls
duration: const Duration(seconds: 2),
// set a custom user agent
userAgent: 'SlothLinkCrawler');

// List of all crawled urls
List<String> result = await crawler.crawl();
copied to clipboard