1. Learn
    2. /
  2. Courses
    3. /
  3. Python Data Science Toolbox (Part 2)

Exercise

Writing an iterator to load data in chunks (4)

In the previous exercises, you've only processed the data from the first DataFrame chunk. This time, you will aggregate the results over all the DataFrame chunks in the dataset. This basically means you will be processing the entire dataset now. This is neat because you're going to be able to process the entire large dataset by just working on smaller pieces of it!

You're going to use the data from 'ind_pop_data.csv', available in your current directory. The packages pandas and matplotlib.pyplot have been imported as pd and plt respectively for your use.

Instructions

100 XP
  • Initialize an empty DataFrame data using pd.DataFrame().
  • In the for loop, iterate over urb_pop_reader to be able to process all the DataFrame chunks in the dataset.
  • Concatenate data and df_pop_ceb by passing a list of the DataFrames to pd.concat().