Spark is a framework for working with Big Data. In this chapter you'll cover some background about Spark and Machine Learning. You'll then find out how to connect to Spark using Python and load CSV data.

Machine Learning & Spark

Kenmerken van Spark

Onderdelen in een Spark-cluster

Verbinden met Spark

Locatie van de Spark-master

Een SparkSession maken

Gegevens laden

Vluchtgegevens laden

SMS-spamgegevens laden

Introduction

Now that you are familiar with getting data into Spark, you'll move onto building two types of classification model: Decision Trees and Logistic Regression. You'll also find out about a few approaches to data preparation.

Data Preparation

Removing columns and rows

Column manipulation

Categorical columns

Assembling columns

Decision Tree

Train/test split

Build a Decision Tree

Evaluate the Decision Tree

Logistic Regression

Build a Logistic Regression model

Evaluate the Logistic Regression model

Turning Text into Tables

Punctuation, numbers and tokens

Stopwords and hashing

Training a spam classifier

Classification

Next you'll learn to create Linear Regression models. You'll also find out how to augment your data by engineering new predictors as well as a robust approach to selecting only the most relevant predictors.

One-Hot Encoding

Encoding flight origin

Encoding shirt sizes

Regression

Flight duration model: Just distance

Interpreting the coefficients

Flight duration model: Adding origin airport

Interpreting coefficients

Bucketing & Engineering

Bucketing departure time

Flight duration model: Adding departure time

Regularization

Flight duration model: More features!

Flight duration model: Regularization!

Finally you'll learn how to make your models more efficient. You'll find out how to use pipelines to make your code clearer and easier to maintain. Then you'll use cross-validation to better test your models and select good model parameters. Finally you'll dabble in two types of ensemble model.

Pipeline

Flight duration model: Pipeline stages

Flight duration model: Pipeline model

SMS spam pipeline

Cross-Validation

Cross validating simple flight duration model

Cross validating flight duration model pipeline

Grid Search

Optimizing flights linear regression

Dissecting the best flight duration model

SMS spam optimised

How many models for grid search?

Ensemble

Delayed flights with Gradient-Boosted Trees

Delayed flights with a Random Forest

Evaluating Random Forest

Closing thoughts

Ensembles & Pipelines

Flights

Spark is een krachtig, algemeen hulpmiddel voor het werken met Big Data. Spark regelt transparant de verdeling van rekentaken over een cluster. Dat maakt bewerkingen snel én zorgt ervoor dat jij je kunt richten op de analyse in plaats van op technische details. In deze cursus leer je hoe je data in Spark krijgt en ga je vervolgens in op drie fundamentele Machine Learning-algoritmen in Spark: lineaire regressie, logistische regressie/classifiers en het bouwen van pipelines. Onderweg analyseer je een grote gegevensset met vertraagde vluchten en spam-sms'jes. Met deze basis ben je klaar om de kracht van Spark te benutten en toe te passen in je eigen Machine Learning-projecten!

Supervised Learning with scikit-learn

Introduction to PySpark

Leer voorspellingen maken met Apache Spark via beslisbomen, logistische en lineaire regressie.

Machine Learning met PySpark

Big Data met PySpark

Wetenschapper op het gebied van machine learning in Python

Onderdelen in een Spark-cluster

Machine Learning met PySpark

Praktische interactieve oefening