GitLocker: The Coding Marketplace

Description:

puppeteer

Puppeteer in Dart #
A Dart library to automate the Chrome browser over the DevTools Protocol.
This is a port of the Puppeteer Node.JS library in the Dart language.

What can I do?
Most things that you can do manually in the browser can be done using Puppeteer! Here are a few examples to get you started:

Generate screenshots and PDFs of pages.
Crawl a SPA (Single-Page Application) and generate pre-rendered content (i.e. "SSR" (Server-Side Rendering)).
Automate form submission, UI testing, keyboard input, etc.
Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.

Flutter
See limitations with Flutter section.
API #

See the full API in a single-page document: doc/api.md
See the Dart Doc for this package: API reference
The Dart version of Puppeteer is very similar to the original Javascript code. Every sample available for Puppeteer Node.JS could be converted in Dart very easily.

Examples #

Launch chrome
Generate a PDF from an HTML page
Take a screenshot of a page
Take a screenshot of an element in a page
Create a static version of a Single Page Application
Capture a screencast of the page
Execute JavaScript code

Launch Chrome #
Download the last revision of chrome and launch it.
import 'package:puppeteer/puppeteer.dart';

void main() async {
// Download the Chrome binaries, launch it and connect to the "DevTools"
var browser = await puppeteer.launch();

// Open a new tab
var myPage = await browser.newPage();

// Go to a page and wait to be fully loaded
await myPage.goto('https://dart.dev', wait: Until.networkIdle);

// Do something... See other examples
await myPage.screenshot();
await myPage.pdf();
await myPage.evaluate<String>('() => document.title');

// Gracefully close the browser's process
await browser.close();
}
copied to clipboard
Generate a PDF from a page #
import 'dart:io';
import 'package:puppeteer/puppeteer.dart';

void main() async {
// Start the browser and go to a web page
var browser = await puppeteer.launch();
var page = await browser.newPage();
await page.goto('https://dart.dev', wait: Until.networkAlmostIdle);

// For this example, we force the "screen" media-type because sometime
// CSS rules with "@media print" can change the look of the page.
await page.emulateMediaType(MediaType.screen);

// Capture the PDF and save it to a file.
await page.pdf(
format: PaperFormat.a4,
printBackground: true,
pageRanges: '1',
output: File('example/_dart.pdf').openWrite());
await browser.close();
}
copied to clipboard
Take a screenshot of a complete HTML page #
import 'dart:io';
import 'package:puppeteer/puppeteer.dart';

void main() async {
// Start the browser and go to a web page
var browser = await puppeteer.launch();
var page = await browser.newPage();

// Setup the dimensions and user-agent of a particular phone
await page.emulate(puppeteer.devices.pixel2XL);

await page.goto('https://dart.dev', wait: Until.networkIdle);

// Take a screenshot of the page
var screenshot = await page.screenshot();

// Save it to a file
await File('example/_github.png').writeAsBytes(screenshot);

await browser.close();
}
copied to clipboard
Take a screenshot of a specific node in the page #
import 'dart:io';
import 'package:puppeteer/puppeteer.dart';

void main() async {
// Start the browser and go to a web page
var browser = await puppeteer.launch();
var page = await browser.newPage();
await page.goto('https://pub.dev/documentation/puppeteer/latest/',
wait: Until.networkIdle);

// Select an element on the page
var form = await page.$('input[id="search-box"]');

// Take a screenshot of the element
var screenshot = await form.screenshot();

// Save it to a file
await File('example/_element.png').writeAsBytes(screenshot);

await browser.close();
}
copied to clipboard
Interact with the page and scrap content #
import 'package:puppeteer/puppeteer.dart';

void main() async {
var browser = await puppeteer.launch();
var page = await browser.newPage();

await page.goto('https://developers.google.com/web/',
wait: Until.networkIdle);

// Type into search box.
await page.type('.devsite-search-field', 'Headless Chrome');

// Wait for suggest overlay to appear and click "show all results".
var allResultsSelector = '.devsite-suggest-all-results';
await page.waitForSelector(allResultsSelector);
await page.click(allResultsSelector);

// Wait for the results page to load and display the results.
const resultsSelector = '.gsc-results .gsc-thumbnail-inside a.gs-title';
await page.waitForSelector(resultsSelector);

// Extract the results from the page.
var links = await page.evaluate<List<dynamic>>(r'''resultsSelector => {
const anchors = Array.from(document.querySelectorAll(resultsSelector));
return anchors.map(anchor => {
const title = anchor.textContent.split('|')[0].trim();
return `${title} - ${anchor.href}`;
});
}''', args: [resultsSelector]);
print(links.join('\n'));

await browser.close();
}
copied to clipboard
Create a static version of a Single Page Application #
import 'package:puppeteer/puppeteer.dart';

void main() async {
var browser = await puppeteer.launch();
var page = await browser.newPage();
await page.goto('https://w3c.github.io/');

// Either use the helper to get the content
var pageContent = await page.content;
print(pageContent);

// Or get the content directly by executing some Javascript
var pageContent2 =
await page.evaluate<String>('document.documentElement.outerHTML');
print(pageContent2);

await browser.close();
}
copied to clipboard
Capture a screencast of the page #
The screencast feature is not part of the Puppeteer API.
This example uses the low-level protocol API to send the commands to the browser.
import 'dart:convert';
import 'dart:io';
import 'package:image/image.dart' as image;
import 'package:puppeteer/puppeteer.dart';
import 'package:shelf/shelf_io.dart' as io;
import 'package:shelf_static/shelf_static.dart';

void main() async {
// Start a local web server and open the page
var server =
await io.serve(createStaticHandler('example/html'), 'localhost', 0);
var browser = await puppeteer.launch();
var page = await browser.newPage();
await page.goto('http://localhost:${server.port}/rubiks_cube/index.html');

// On each frame, decode the image and use the "image" package to create an
// animated gif.
image.Image? animation;
page.devTools.page.onScreencastFrame.listen((event) {
var frame = image.decodePng(base64.decode(event.data));
if (frame != null) {
if (animation == null) {
animation = frame;
} else {
animation!.addFrame(frame);
}
}
});

// For this example, we change the CSS animations speed.
await page.devTools.animation.setPlaybackRate(240);

// Start the screencast
await page.devTools.page.startScreencast(maxWidth: 150, maxHeight: 150);

// Wait a few seconds and stop the screencast.
await Future.delayed(Duration(seconds: 3));
await page.devTools.page.stopScreencast();

// Encode all the frames in an animated Gif file.
File('example/_rubkis_cube.gif')
.writeAsBytesSync(image.GifEncoder().encode(animation!));

// Alternatively, we can save all the frames on disk and use ffmpeg to convert
// it to a video file. (for example: ffmpeg -i frames/%3d.png -r 10 output.mp4)

await browser.close();
await server.close(force: true);
}
copied to clipboard
Launch with visible window #
By default, puppeteer launch Chrome in "headless" mode, the browser is invisible.
For better development & debugging, you can force "headful" mode with parameter headless: false
import 'package:puppeteer/puppeteer.dart';

void main() async {
var browser = await puppeteer.launch(headless: false);
// Do something...
await browser.newPage();
await browser.close();
}
copied to clipboard
Low-level DevTools protocol #
This package contains a fully typed API of the Chrome DevTools protocol.
The code is generated from the JSON Schema provided by Chrome.
With this API you have access to the entire capabilities of Chrome DevTools.
The code is in lib/protocol
import 'package:puppeteer/puppeteer.dart';

void main() async {
var browser = await puppeteer.launch();
// Create a chrome's tab
var page = await browser.newPage();

// You can access the entire Chrome DevTools protocol.
// This is useful to access information not exposed by the Puppeteer API
// Be aware that this is a low-level, complex API.
// Documentation of the protocol: https://chromedevtools.github.io/devtools-protocol/

// Examples:

// Start a screencast
await page.devTools.page.startScreencast();

// Change the animation speed for the document
await page.devTools.animation.setPlaybackRate(10);

// Access the memory information for the page
await page.devTools.memory.getDOMCounters();

// Go to https://chromedevtools.github.io/devtools-protocol/ to read more about
// the protocol and use the code in `lib/protocol`.

await browser.close();
}
copied to clipboard
Execute JavaScript code #
The Puppeteer API exposes several functions to run some Javascript code in the browser.
Like in this example from the official Node.JS library:
test(async () => {
const result = await page.evaluate(x => {
return Promise.resolve(8 * x);
}, 7);
});
copied to clipboard
In the Dart port, you have to pass the JavaScript code as a string.
The example above will be written as:
main() async {
var result = await page.evaluate('''x => {
return Promise.resolve(8 * x);
}''', args: [7]);
}
copied to clipboard
The javascript code can be:

A function declaration (in the classical form with the function keyword
or with the shorthand format (() => ))
An expression. In which case you cannot pass any arguments to the evaluate method.

import 'package:puppeteer/puppeteer.dart';

void main() async {
var browser = await puppeteer.launch();
var page = await browser.newPage();

// function declaration syntax
await page.evaluate('function(x) { return x > 0; }', args: [7]);

// shorthand syntax
await page.evaluate('(x) => x > 0', args: [7]);

// Multi line shorthand syntax
await page.evaluate('''(x) => {
return x > 0;
}''', args: [7]);

// shorthand syntax with async
await page.evaluate('''async (x) => {
return await x;
}''', args: [7]);

// An expression.
await page.evaluate('document.body');

await browser.close();
}
copied to clipboard
If you are using IntellJ (or Webstorm), you can enable the syntax highlighter and the code-analyzer
for the Javascript snippet with a comment like // language=js before the string.
main() {
page.evaluate(
//language=js
'''function _(x) {
return x > 0;
}''', args: [7]);
}
copied to clipboard
Limitations with Flutter #
This library does 2 things:

Download the Chrome binaries and launch a Chrome process.
Connect to this process with Websocket and send json commands to control the browser.

Due to limitations on mobile platforms (iOS and Android), it is not possible to launch an external Chrome process on iOS and Android.
So, step 1) does not work on mobile.
You can still use puppeteer-dart on Flutter either with:

Flutter on Desktop (macOS, Windows, Linux)
Flutter on Mobile BUT with the actual Chrome instance running on a server and accessed from the mobile app using puppeteer.connect

Related work #

chrome-remote-interface
puppeteer
webkit_inspection_protocol
dart webdriver