1. Using the Census API
The DataFrame used in the first lesson was downloaded from the Census API server. Now we'll learn how to construct an API request and load the response into a pandas DataFrame.
2. Structure of a Census API Request
This is a basic Census API request.
3. Structure of a Census API Request
The part up to the question mark is referred to as the
"base URL", and specifies
the host,
year,
and dataset.
4. Structure of a Census API Request
The part after the question mark is referred to as the query string.
This is where we specify parameters such as the variables being requested,
and the desired geography. Although the URL can be constructed using simple string manipulation...
5. The requests Library
...we will use the requests library, imported here, to construct the URL.
We define variables for the HOST, year, and dataset,
then build the base URL by joining these variables with a slash. dataset = "dec/sf1" stands for "Summary File 1", and refers to the full count data from the decennial census.
The requests.get method accepts query parameters as a dictionary. The Census API documentation refers to these parameters as "predicates", so we name the dictionary "predicates".
The variables to request are assigned to a list, get_vars. These include the name of the geographic unit, the land area in square meters, and the full population count. Don't worry about the obscure name "P001001". We'll learn where to find these variable names later.
A dictionary key "get" is created by joining the variable names into a comma-separated string.
Set the "for" key to the geographic level. We use "state", followed by the asterisk wild card, to request all states.
Finally, execute the request and store the return value in the response object "r".
6. Examine the Response
Now inspect the text attribute of the response object.
This is a string with the form of a list of lists. Each sublist is a "row" of data, with the first row being the table header. Notice that all values, including numerics, are double-quoted. When we load this data into pandas, we will have to fix these data types.
7. Response Errors
In case of a poorly formed request, the API server will return an error message.
One common error is specifying the variable name incorrectly.
8. Create User-Friendly Column Names
The json() method of the response object returns a list of lists.
The first sublist contains the column names. Every API call returns geographic identifiers. This one returned a column named "state".
Throughout this course, we will be renaming columns to make them easier to work with.
Begin by creating a list containing the new column names, all lower case.
9. Load into Pandas DataFrame
Now construct the DataFrame using pd.DataFrame. The new column names get passed to the columns parameter. The json list of lists gets passed to the data parameter. Use slicing to skip row 0, which contains the header.
Two columns have integer data currently stored as text. Fix the data types by passing the correct type, int, to the astype method.
Use head to view the result.
The "state" identifier, like a ZIP code, has leading zeroes and should be left as text.
10. Find 3 Most Densely Settled States
Now that the data is loaded, let's see an example analysis.
Create a new column "pop_per_km2" by dividing the population by the land area. The result is in square meters. Multiply by 1000 squared to get the density in square kilometers.
Then use nlargest to return the 3 top records. The first parameter is the number of rows to return, the second is the column to sort by.
11. Let's practice!
You've seen some key elements of the Census API. Now it's your turn to try some examples.