1. Learn
    2. /
  2. Courses
    3. /
  3. Tidy Data in Python Mini-Course
  • 1

    Tidy Data in Python

    It is often said that data scientists spend only 20% of their time analyzing their data, and 80% of time cleaning it. Indeed, maintaining a tidy, easy-to-use dataset is crucial in our age of big data. In the paper Tidy Data, veteran statistician Hadley Wickham gives definitions of tidy and messy data so that all data scientists can keep their work organized. In this mini-course, you'll learn to transform messy datasets into tidy datasets using the pandas package in python. Let's get started!

Initializing

Exercise

Tidy Data and Messy Data

What exactly marks the difference between tidy data and messy data? It is not only how organized and intuitive the datasets look to our human eyes, but also how easily and efficiently they can be processed by computers. In his seminal paper Tidy Data, Hadley Wickham proposed three standards for tidy data:

  1. Each variable forms a column
  2. Each observation forms a row
  3. Each type of observation forms a unit.

In this course, we'll focus on the first two rules and show you how we can use the Python package pandas to deal with datasets violating them. To get started, execute messy in the IPython shell. This dataset, which appears in Wickham's paper, shows the number of people who choose either of two treatments in a hospital. Observe its structure in comparison with Wickham's rules. This dataset is messy because it violates rule #2: it combines Treatment A and Treatment B, two distinct observations, in a single row.

Now let's look at two more datasets. Execute df1 and df2 in your IPython shell to check out two other preloaded datasets, both featured in DataCamp's Cleaning Data in R course. The former shows the type and number of pets owned by three co-workers, and the latter shows the average BMI in three countries over several years. Which one of these datasets is messy, and why?

Instructions

50 XP

Possible answers