In this chapter, you'll learn how to overcome some of the most common dirty data problems. You'll convert data types, apply range constraints to remove future data points, and remove duplicated data points to avoid double-counting.

Data type constraints

Common data types

Converting data types

Trimming strings

Range constraints

Ride duration constraints

Back to the future

Uniqueness constraints

Full duplicates

Removing partial duplicates

Aggregating partial duplicates

Common Data Problems

Categorical and text data can often be some of the messiest parts of a dataset due to their unstructured nature. In this chapter, you’ll learn how to fix whitespace and capitalization inconsistencies in category labels, collapse multiple categories into one, and reformat strings for consistency.

Checking membership

Members only

Not a member

Categorical data problems

Identifying inconsistency

Correcting inconsistency

Collapsing categories

Cleaning text data

Detecting inconsistent text data

Replacing and removing

Invalid phone numbers

Categorical and Text Data

In this chapter, you’ll dive into more advanced data cleaning problems, such as ensuring that weights are all written in kilograms instead of pounds. You’ll also gain invaluable skills that will help you verify that values have been added correctly and that missing values don’t negatively impact your analyses.

Uniformity

Date uniformity

Currency uniformity

Cross field validation

Validating totals

Validating age

Completeness

Options

Missing completely at random

Missing at random

Missing not at random

Types of missingness

Visualizing missing data

Treating missing data

Advanced Data Problems

Record linkage is a powerful technique used to merge multiple datasets together, used when values have typos or different spellings. In this chapter, you'll learn how to link records by calculating the similarity between strings—you’ll then use your new skills to join two restaurant review datasets into one clean master dataset.

Comparing strings

Calculating distance

Small distance, small difference

Fixing typos with string distance

Generating and comparing pairs

Link or join?

Pair blocking

Comparing pairs

Scoring and linking

Score then select or select then score?

Putting it together

Congratulations!

Record Linkage

Zagat

Fodor's

Bike Sharing

SFO Satisfaction Survey

Customer Accounts

<h2>Overcome Common Data Problems Like Removing Duplicates in R </h2>
It's commonly said that data scientists spend 80% of their time cleaning and manipulating data and only 20% of their time analyzing it. The time spent cleaning is vital since analyzing dirty data can lead you to draw inaccurate conclusions.
<br><br>
In this course, you’ll learn a variety of techniques to help you clean dirty data using R. You’ll start by converting data types, applying range constraints, and dealing with full and partial duplicates to avoid double-counting.
<br><br>
<h2>Delve into Advanced Data Challenges </h2>
Once you’ve practiced working on common data issues, you’ll move on to more advanced challenges such as ensuring consistency in measurements and dealing with missing data. After every new concept, you’ll have the chance to complete a hands-on exercise to cement your knowledge and build your experience.
<br><br>
<h2>Learn to Use Record Linkage During Data Cleaning </h2>
Record Linkage is used to merge datasets together when the values have issues such as typos or different spellings. You’ll explore this useful technique in the final chapter and practice the application by using it to join two restaurant review datasets together into a single dataset.

Joining Data with dplyr

Learn to clean data as quickly and accurately as possible to help your business move from raw data to awesome insights.

Cleaning Data in R

Learn to clean data as quickly and accurately as possible to help you move from raw data to awesome insights.

Associate Data Scientist in R

Importing & Cleaning Data  in R

Questionnaires were randomly lost in the mail.

Test scores are missing for students from a certain school district, but not the others.

Younger people were less likely to get their blood pressure checked at the doctor, so their medical records were missing blood pressure data.

A scale has a limit of 30 lbs, so shipments above 30 lbs don't have a recorded weight.

Blood samples tested during the night shift were more likely to be contaminated, creating missing test results, but time of testing wasn't recorded.

Types of missingness

Cleaning Data in R

Praktische interactieve oefening