Turning a webpage into data using BeautifulSoup: getting the hyperlinks
In this exercise, you'll figure out how to extract the URLs of the hyperlinks from the BDFL's webpage. In the process, you'll become close friends with the soup method find_all()
.
This exercise is part of the course
Intermediate Importing Data in Python
Exercise instructions
- Use the method
find_all()
to find all hyperlinks insoup
, remembering that hyperlinks are defined by the HTML tag<a>
but passed tofind_all()
without angle brackets; store the result in the variablea_tags
. - The variable
a_tags
is a results set: your job now is to enumerate over it, using afor
loop and to print the actual URLs of the hyperlinks; to do this, for every elementlink
ina_tags
, you want toprint()
link.get('href')
.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Import packages
import requests
from bs4 import BeautifulSoup
# Specify url
url = 'https://www.python.org/~guido/'
# Package the request, send the request and catch the response: r
r = requests.get(url)
# Extracts the response as html: html_doc
html_doc = r.text
# create a BeautifulSoup object from the HTML: soup
soup = BeautifulSoup(html_doc)
# Print the title of Guido's webpage
print(soup.title)
# Find all 'a' tags (which define hyperlinks): a_tags
# Print the URLs to the shell
for ____ in ____:
____