Data cleaning and enhancement

TechCorp is migrating to a new HR system. The dataset needs cleaning: remove low-salary outliers that indicate data entry errors, drop columns the new system won't use, and add a calculated bonus field. Data cleaning typically takes 80% of analysis time—these skills are essential.

The Table, Selection, and DoubleColumn classes have been imported for you.

This exercise is part of the course

Importing Data in Java

Exercise instructions

Remove employees with salaries below $40,000.
Remove the "JobTitle" column.
Add the PerformanceBonus column (5% of salary).

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

public class DataExploration {
	public static void main(String[] args) {

        Table employees = Table.read().csv("employees.csv");

        // Remove employees with salaries below $40,000
        Selection lowSalaries = employees.intColumn("Salary").isLessThan(____);
        Table cleanedEmployees = employees.____(lowSalaries);

        // Remove the JobTitle column
        Table streamlined = cleanedEmployees.____("JobTitle");

        DoubleColumn performanceBonus = streamlined.intColumn("Salary").asDoubleColumn()
            .map(salary -> salary * 0.05);
        performanceBonus.setName("PerformanceBonus");

        // Add the PerformanceBonus column
        Table enhancedEmployees = streamlined.____(performanceBonus);

        System.out.println("Total employees after cleaning: " + enhancedEmployees.rowCount());
        System.out.println("\nFirst 5 rows of enhanced dataset:");
        System.out.println(enhancedEmployees.first(5));
	}
}

Edit and Run Code

This exercise is part of the course

Importing Data in Java

IntermediateSkill Level

4.9+

Start Course for Free

Kickstart your journey into data handling with Java! In this first chapter, you'll learn how to read, validate, and manage data using Java I/O Streams and then dive into the powerful Tablesaw library to easily create, explore, and manipulate data tables, the perfect foundation for modern data analysis in Java.

Exercise 1: Understanding Data Import Fundamentals Exercise 2: Creating basic file readers Exercise 3: Implementing data validation Exercise 4: Java I/O Streams overview Exercise 5: Reading files with BufferedReader Exercise 6: Writing files with BufferedWriter Exercise 7: Stream chaining Exercise 8: Introduction to Tablesaw Exercise 9: Creating a table Exercise 10: Exploring table metadata Exercise 11: Modifying table structure

In this second chapter, we will move on to covering CSV processing in Tablesaw. Initially, you will learn to read and write CSV files with varying formats, and then we move on to modifying tables, adding and removing columns, filtering tables and manipulating the data within tables for data cleaning and processing.

Exercise 1: CSV processing with Tablesaw Exercise 2: Reading CSV files Exercise 3: Handling CSV parsing options Exercise 4: Writing formatted CSV files Exercise 5: Data structures in Tablesaw Exercise 6: Working with column types Exercise 7: Table filtering Exercise 8: Table queries Exercise 9: Data manipulation and analysis Exercise 10: Initial data exploration Exercise 11: Department analysis Exercise 12: Advanced Data Manipulation Techniques Exercise 13: Data cleaning and enhancement

Current Exercise

Exercise 14: Advanced filtering and analysis

Complete your Java data journey by mastering transformation patterns that allow you to modify columns, process rows, and summarize datasets into meaningful insights. You’ll also work with JSON data, from simple files to complex, nested structures - preparing you to handle modern APIs and configuration files.

Exercise 1: Data Transformation Patterns Exercise 2: Employee salary analysis Exercise 3: Total compensation calculator Exercise 4: Department summary statistics Exercise 5: JSON Data Basics with Tablesaw Exercise 6: Loading JSON files Exercise 7: JSON data analysis Exercise 8: JSON validation Exercise 9: Complex JSON Processing Exercise 10: Importing nested data Exercise 11: Merging nested data Exercise 12: Handling missing JSON values Exercise 13: Congratulations!