Learn how to process XML sitemaps to extract all texts present on a website. Go straight from a news or blog homepage to a fully processed document collection.
This introductory level tutorial shows how the web #crawling and #scraping functions are run in Python and on the shell.
--
Table of Contents:
0:00 Introduction: How can we discover content within a website?
0:44 The Python library
1:00 What are sitemaps and what are they good for?
1:57 Discovering and processing sitemaps with Python
2:40 List comprehensions to filter the links
4:00 Scraping and content extraction
5:30 Usage on the command-line
--
Trafilatura is a freely available scraping tool which seamlessly downloads, parses, and scrapes web page data while preserving parts of the text formatting and page structure. It features parallel online and offline processing, robust and efficient extraction, link discovery in feeds and sitemaps, as well as language detection.
The output can be converted to different formats: TXT, CSV, JSON, XML and XML-TEI.
▶️ More information
#webscraping​​ #pythonTutorial #pythonBeginners