Save yourself time when performing complex repetitive tasks. You’ll begin this course by refreshing your knowledge of common command-line programs and arguments before quickly moving into the world of Bash scripting. You’ll create simple command-line pipelines and then transform these into Bash scripts. You’ll then boost your skills and learn about standard streams and feeding arguments to your Bash scripts.

Introduction and refresher

Extracting scores with shell

Searching a book with shell

Your first Bash script

A simple Bash script

Shell pipelines to Bash scripts

Extract and edit using Bash scripts

Standard streams & arguments

Using arguments in Bash scripts

Using arguments with HR data

From Command-Line to Bash Script

Master the bread and butter of Bash scripts—variables! In this chapter, you’ll learn how to create basic string and numeric variables, and perform calculations on these variables. You’ll also learn about the magic of a shell-within-a-shell (shell-ception), opening up huge opportunities for advanced scripting.

Basic variables in Bash

Using variables in Bash

Shell within a shell

Numeric variables in Bash

Converting Fahrenheit to Celsius

Extracting data from files

Arrays in Bash

Creating an array

Creating associative arrays

Climate calculations in Bash

Variables in Bash Scripting

Life isn't always linear and neither should your Bash scripts. In this chapter, you’ll take your Bash scripts to the next level by learning control statements. You’ll learn the differences between FOR, WHILE, IF, and CASE statements and how to use them in your Bash scripts. Armed with these new tools, you’ll be ready to create more advanced Bash scripts with conditional logic. 

IF statements

Sorting model results

Moving relevant files

FOR loops & WHILE statements

A simple FOR loop

Correcting a WHILE statement

Cleaning up a directory

CASE statements

Days of the week with CASE

Moving model results with CASE

Finishing a CASE statement

Control Statements in Bash Scripting

Give your Bash scripts a powerful new addition by building useful functions to process data and variables. In this last chapter, you will learn the structure of functions in Bash, how to use functions to help automate frequent tasks, and program your scripts to run on schedule without needing to even lift a finger.

Basic functions in Bash

# Create function
____ upload_to_cloud () ____
  # Loop through files with glob expansion
  for file in output_dir/____results____
  do
    # Echo that they are being uploaded
    ____ "Uploading $file to cloud"
  done
____

# Call the function
upload_to_cloud


# Create function
function upload_to_cloud () {
  # Loop through files with glob expansion
  for file in output_dir/*results*
  do
    # Echo that they are being uploaded
    echo "Uploading $file to cloud"
  done
}

# Call the function
upload_to_cloud


Uploading model results to the cloud

# Create function
_____ _____ {

  # Parse the results of date
  ____=$(date | cut -d " " -f1)

  # Echo the result
  echo ____
}

# Call the function
____


# Create function
what_day_is_it () {

  # Parse the results of date
  current_day=$(date | cut -d " " -f1)

  # Echo the result
  echo $current_day
}

# Call the function
what_day_is_it


Get the current day

Arguments, return values, and scope

# Create a function 
____ return_percentage () ____

  # Calculate the percentage using bc
  percent=$(echo "scale=2; 100 * ____ / ____" | ____)

  # Return the calculated percentage
  echo $____
}

# Call the function with 456 and 632 and echo the result
return_test=$(____ ____ 632)
echo "456 out of 632 as a percent is $____%"


# Create a function 
function return_percentage () {

  # Calculate the percentage using bc
  percent=$(echo "scale=2; 100 * $1 / $2" | bc)

  # Return the calculated percentage
  echo $percent
}

# Call the function with 456 and 632 and echo the result
return_test=$(return_percentage 456 632)
echo "456 out of 632 as a percent is $return_test%"


A percentage calculator

# Create a function
____ get_number_wins () ____

  # Filter aggregate results by argument
  ____=$(cat soccer_scores.csv | cut -d "," -f2 | egrep -v 'Winner'| sort | uniq -c | egrep "____")

____

# Call the function with specified argument
____ "____"

# Print out the global variable
echo "The aggregated stats are: ____"


# Create a function
function get_number_wins () {

  # Filter aggregate results by argument
  win_stats=$(cat soccer_scores.csv | cut -d "," -f2 | egrep -v 'Winner'| sort | uniq -c | egrep "$1")

}

# Call the function with specified argument
get_number_wins "Etar"

# Print out the global variable
echo "The aggregated stats are: $win_stats"


Sports analytics function

# Create a function with a local base variable
function sum_array () {
  ____ sum=0
  # Loop through, adding to base variable
  for number in "$____"
  do
    sum=$(echo "$sum + $____" | bc)
  done
  # Echo back the result
  echo $____
  }
# Call function with array
test_array=(14 12 23.5 16 19.34)
total=$(sum_array "${____[@]}")
echo "The total sum of the test array is $____"


# Create a function with a local base variable
function sum_array () {
  local sum=0
  # Loop through, adding to base variable
  for number in "$@"
  do
    sum=$(echo "$sum + $number" | bc)
  done
  # Echo back the result
  echo $sum
  }
# Call function with array
test_array=(14 12 23.5 16 19.34)
total=$(sum_array "${test_array[@]}")
echo "The total sum of the test array is $total"


Summing an array

Scheduling your scripts with Cron

Analyzing a crontab schedule

Creating cronjobs

Scheduling cronjobs with crontab

Thanks and wrap up

Functions and Automation

New Hires

Soccer Scores

Hire Data

Model Results

Rob's Files

Model Output

Inherited Folder

Bash is a concise, superfast, and robust scripting language for data and file manipulation. It’s a vital skill for building analytics pipelines in the cloud, favored by Linux users to work with data stored across multiple files.

In this course, we’ll guide you through the basics of Bash scripting. We begin with an introduction to Bash script structures, including inputting arguments and outputting results. You’ll then work through data structures, such as variables and arrays, and control statements, including loops and conditionals. You’ll then put what you’ve learned into practice, by creating your own Bash functions and scheduling automated scripts to run like clockwork with cron.

Introduction to Shell

Learn how you can use Bash scripting to build automated scripts, create analytics pipelines in the cloud, and work with data stored across multiple files.

Introduction to Bash Scripting

Bash scripting allows you to build analytics pipelines in the cloud and work with data stored across multiple files.

Analyzing a crontab schedule

Introduction to Bash Scripting

Interaktive Übung