jusTextlid 2.2.9
@jordiae Changes
Use Fasttext language identifier instead of stopwords.
Install:
pip install justext-lid
Usage (no stopwords required):
from justext_lid import justext
first_paragraph = justext(HTML, langs={'en', 'es'}, langid_th=0.3)[0]
first_paragraph_lang = first_paragraph.lang
EleutherAI Changes
Modified the classification function in order to remove elements that have quotes (in order to improve results on forums)
jusText
Program jusText is a tool for removing boilerplate content, such as navigation
links, headers, and footers from HTML pages. It is
designed to preserve
mainly text containing full sentences and it is therefore well suited for
creating linguistic resources such as Web corpora. You can
try it online.
This is a fork of original (currently unmaintained) code of jusText hosted
on Google Code.
Adaptations of the algorithm to other languages:
C++
Go
Java
Some libraries using jusText:
chirp
lazynlp
off-topic-memento-toolkit
pears
readability calculator
sky
Some currently (Jan 2020) maintained alternatives:
dragnet
html2text
inscriptis
newspaper
python-readability
trafilatura
Installation
Make sure you have Python 2.7+/3.4+ and pip
(Windows,
Linux) installed.
Run simply:
$ [sudo] pip install justext
Dependencies
lxml (version depends on your Python version)
Usage
$ python -m justext -s Czech -o text.txt http://www.zdrojak.cz/clanky/automaticke-zabezpeceni/
$ python -m justext -s English -o plain_text.txt english_page.html
$ python -m justext --help # for more info
Python API
import requests
import justext
response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
if not paragraph.is_boilerplate:
print paragraph.text
Testing
Run tests via
$ py.test-2.7 && py.test-3.4 && py.test-3.5 && py.test-3.6 && py.test-3.7 && py.test-3.8
Acknowledgements
This software has been developed at the Natural Language Processing Centre of
Masaryk University in Brno with a financial support from PRESEMT and
Lexical Computing Ltd. It also relates to PhD research of Jan Pomikálek.
Changelog for jusText
2.2.0 (2016-03-06)
INCOMPATIBLE CHANGE: Stop words are case insensitive.
INCOMPATIBLE CHANGE: Dropped support for Python 3.2
BUG FIX: Preserve new lines from original text in paragraphs.
2.1.1 (2014-05-27)
BUG FIX: Function decode_html now respects parameter errors when falling to default_encoding #9.
2.1.0 (2014-01-25)
FEATURE: Added XPath selector to the paragrahs. XPath selector is also available in detailed output as xpath attribute of <p> tag #5.
2.0.0 (2013-08-26)
FEATURE: Added pluggable DOM preprocessor.
FEATURE: Added support for Python 3.2+.
INCOMPATIBLE CHANGE: Paragraphs are instances of
justext.paragraph.Paragraph.
INCOMPATIBLE CHANGE: Script ‘justext’ removed in favour of
command python -m justext.
FEATURE: It’s possible to enter an URI as input document in CLI.
FEATURE: It is possible to pass unicode string directly.
1.2.0 (2011-08-08)
FEATURE: Character counts used instead of word counts where possible in
order to make the algorithm work well in the language independent
mode (without a stoplist) for languages where counting words is
not easy (Japanese, Chinese, Thai, etc).
BUG FIX: More robust parsing of meta tags containing the information about
used charset.
BUG FIX: Corrected decoding of HTML entities € to Ÿ
1.1.0 (2011-03-09)
First public release.
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
There are no reviews.