In this chapter, you'll be introduced to Hyper Text Markup Language (HTML), a declarative language used to structure modern websites. Using the rvest library, you'll learn how to query simple HTML elements and scrape your first table.

Introductie tot HTML

HTML inlezen

Let op syntaxfouten!

Navigeren door HTML

Selecteer alle kinderen van een lijst

Zet hyperlinks om naar een data frame

Scrape je eerste tabel

Table

De juiste volgorde van tabelelementen

Zet een tabel om in een data frame met html_table()

Introduction to HTML and Web Scraping

Cascading Style Sheets (CSS) describe how HTML elements are displayed on a web page, including colors, fonts, and general layout. In this chapter, you'll learn why CSS selectors and combinators are a crucial ingredient for web scraping.

Introduction to CSS

Select multiple HTML types

Order CSS selectors by the number of results

CSS classes and IDs

Identify the correct selector types

Leverage the uniqueness of IDs

Select the last child with a pseudo-class

CSS combinators

Select direct descendants with the child combinator

How many elements get returned?

Simply the best!

Not every sibling is the same

Navigation and Selection with CSS

The CSS selectors you got to know in the last chapter are powerful but have their limitations. For example, if you want to select nodes based on the properties of their descendants. XPath to the rescue! Using this query language, you can navigate and scrape even the most hideous HTML.

Introduction to XPATH

Find the correct CSS equivalent

Select by class and ID with XPATH

Use predicates to select nodes based on their children

XPATH functions and advanced predicates

Find a more elegant XPATH alternative

Get to know the position() function

Extract nodes based on the number of their children

The XPATH text() function

The shortcomings of html_table() with badly structured tables

Select directly from a parent element with XPATH's text()

Combine extracted data into a data frame

Scrape an element based on its text

Advanced Selection with XPATH

Now that you know how to extract content from web pages, it's time to look behind the curtains. In this final chapter, you’ll learn why HTTP requests are the foundation of every scraping action and how they can be customized to comply with best practices in web scraping.

The nature of HTTP requests

Which of these statements about HTTP is false?

Do it the httr way

Houston, we got a 404!

Telling who you are with custom user agents

Check out your user agent

Add a custom user agent

How to be gentle and slow down your requests

Custom arguments for throttled functions

Apply throttling to a multi-page crawler

Recap: Web Scraping in R

Scraping Best Practices

Ben je weleens een website tegengekomen met veel gegevens, zoals statistieken, productreviews of prijzen, maar dan in een formaat dat niet klaar is voor data-analyse? Vaak publiceren overheden en andere dataproducenten hun gegevens in netjes opgemaakte tabellen. Maar niet al deze sites hebben een downloadknop — geen paniek. In deze cursus leer je hoe je met R efficiënt gegevens van elke website verzamelt en downloadt. Je leert hoe je het scrapen en parsen van Wikipedia automatiseert met de pakketten rvest en httr. Met praktische oefeningen verdiep je ook je kennis van HTML en CSS, de bouwstenen van webpagina’s, terwijl je je dataverzamelingsworkflows minder foutgevoelig en efficiënter maakt.

Intermediate R

Introduction to the Tidyverse

Leer hoe je R gebruikt om data van websites te verzamelen en Wikipedia te scrapen met rvest en httr.

Webscraping in R

Leer hoe je met R makkelijk gegevens van elke website kunt verzamelen en downloaden.

Let op syntaxfouten!

Webscraping in R

Praktische interactieve oefening