In this chapter, you'll learn how to overcome some of the most common dirty data problems. You'll convert data types, apply range constraints to remove future data points, and remove duplicated data points to avoid double-counting.

Data type constraints

Common data types

Converting data types

Trimming strings

Range constraints

Ride duration constraints

Back to the future

Uniqueness constraints

Full duplicates

Removing partial duplicates

Aggregating partial duplicates

Common Data Problems

Categorical and text data can often be some of the messiest parts of a dataset due to their unstructured nature. In this chapter, you’ll learn how to fix whitespace and capitalization inconsistencies in category labels, collapse multiple categories into one, and reformat strings for consistency.

Checking membership

Options

Membership constraint

Other constraint

Members only

Not a member

Categorical data problems

Identifying inconsistency

Correcting inconsistency

Collapsing categories

Cleaning text data

Detecting inconsistent text data

Replacing and removing

Invalid phone numbers

Categorical and Text Data

In this chapter, you’ll dive into more advanced data cleaning problems, such as ensuring that weights are all written in kilograms instead of pounds. You’ll also gain invaluable skills that will help you verify that values have been added correctly and that missing values don’t negatively impact your analyses.

Uniformity

Date uniformity

Currency uniformity

Cross field validation

Validating totals

Validating age

Completeness

Types of missingness

Visualizing missing data

Treating missing data

Advanced Data Problems

Record linkage is a powerful technique used to merge multiple datasets together, used when values have typos or different spellings. In this chapter, you'll learn how to link records by calculating the similarity between strings—you’ll then use your new skills to join two restaurant review datasets into one clean master dataset.

Comparing strings

Calculating distance

Small distance, small difference

Fixing typos with string distance

Generating and comparing pairs

Link or join?

Pair blocking

Comparing pairs

Scoring and linking

Score then select or select then score?

Putting it together

Congratulations!

Record Linkage

Zagat

Fodor's

Bike Sharing

SFO Satisfaction Survey

Customer Accounts

<h2>Overcome Common Data Problems Like Removing Duplicates in R </h2>
It's commonly said that data scientists spend 80% of their time cleaning and manipulating data and only 20% of their time analyzing it. The time spent cleaning is vital since analyzing dirty data can lead you to draw inaccurate conclusions.
<br><br>
In this course, you’ll learn a variety of techniques to help you clean dirty data using R. You’ll start by converting data types, applying range constraints, and dealing with full and partial duplicates to avoid double-counting.
<br><br>
<h2>Delve into Advanced Data Challenges </h2>
Once you’ve practiced working on common data issues, you’ll move on to more advanced challenges such as ensuring consistency in measurements and dealing with missing data. After every new concept, you’ll have the chance to complete a hands-on exercise to cement your knowledge and build your experience.
<br><br>
<h2>Learn to Use Record Linkage During Data Cleaning </h2>
Record Linkage is used to merge datasets together when the values have issues such as typos or different spellings. You’ll explore this useful technique in the final chapter and practice the application by using it to join two restaurant review datasets together into a single dataset.

Joining Data with dplyr

Learn to clean data as quickly and accurately as possible to help your business move from raw data to awesome insights.

Cleaning Data in R

Learn to clean data as quickly and accurately as possible to help you move from raw data to awesome insights.

Associate Data Scientist in R

Importing & Cleaning Data  in R

Content Developer @ DataCamp

Example: animal classification

Checking categories

Case inconsistency

Whitespace inconsistency

Too many categories

Let's practice!

https://s3.amazonaws.com/assets.datacamp.com/production/course_24098/subtitles/course_24098_59f29309fb02d5fcd0ea0dfd91acfb03.vtt

Categorical and Text Data - Categorical data problems

Categorical data problems

Create Your Free Account