1. Sampling and point estimates
Hi! Welcome to the course! I’m James, and I’ll be your host as we delve into the world of sampling data with Python. To start, let’s look at what sampling is and why it might be useful.
2. Estimating the population of France
Let's consider the problem of counting how many people live in France.
The standard approach is to take a census. This means contacting every household and asking how many people live there.
3. There are lots of people in France
Since there are millions of people in France, this is a really expensive process. Even with modern data collection technology, most countries will only conduct a census every five or ten years due to the cost.
4. Sampling households
In 1786, Pierre-Simon Laplace realized you could estimate the population with less effort. Rather than asking every household who lived there, he asked a small number of households and used statistics to estimate the number of people in the whole population.
This technique of working with a subset of the whole population is called sampling.
5. Population vs. sample
Two definitions are important for this course. The population is the complete set of data that we are interested in.
The previous example involved the literal population of France, but in statistics, it doesn't have to refer to people.
One thing to bear in mind is that there is usually no equivalent of the census, so typically, we won't know what the whole population is like - more on this in a moment.
The sample is the subset of data that we are working with.
6. Coffee rating dataset
Here's a dataset of professional ratings of coffees.
Each row corresponds to one coffee, and there are thirteen hundred and thirty-eight rows in the dataset. The coffee is given a score from zero to one hundred, which is stored in the total_cup_points column. Other columns contain contextual information like the variety and country of origin and scores between zero and ten for attributes of the coffee such as aroma and body. These scores are averaged across all the reviewers for that particular coffee.
It doesn't contain every coffee in the world, so we don't know exactly what the population of coffees is. However, there are enough here that we can think of it as our population of interest.
7. Points vs. flavor: population
Let's consider the relationship between cup points and flavor by selecting those two columns. This dataset contains all thirteen hundred and thirty-eight rows from the original dataset.
8. Points vs. flavor: 10 row sample
The pandas dot-sample method returns a random subset of rows. Setting n to ten means ten random rows are returned. By default, rows from the original dataset can't appear in the sample dataset multiple times, so we are guaranteed to have ten unique rows in our sample.
9. Python sampling for Series
The dot-sample method also works on pandas Series.
Here, using square-bracket subsetting retrieves the total_cup_points column as a Series, and the n argument specifies how many random values to return.
10. Population parameters & point estimates
A population parameter is a calculation made on the population dataset. We aren't limited to counting values either; here, we calculate the mean of the cup points using NumPy.
By contrast, a point estimate, or sample statistic, is a calculation based on the sample dataset. Here, the mean of the total cup points is calculated on the sample.
Notice that the means are very similar but not identical.
11. Point estimates with pandas
Working with pandas can be easier than working with NumPy. These mean calculations can be performed using the dot-mean pandas method.
12. Let's practice!
Let's start sampling!