This chapter introduces the exciting world of Big Data, as well as the various concepts and different frameworks for processing Big Data. You will understand why Apache Spark is considered the best framework for BigData.

What is Big Data?

The 3 V's of Big Data

PySpark: Spark with Python

Understanding SparkContext

Interactive Use of PySpark

Loading data in PySpark shell

Review of functional programming in Python

Use of lambda() with map()

Use of lambda() with filter()

Introduction to Big Data analysis with Spark

The main abstraction Spark provides is a resilient distributed dataset (RDD), which is the fundamental and backbone data type of this engine. This chapter introduces RDDs and shows how RDDs can be created and executed using RDD Transformations and Actions.

Abstracting Data with RDDs

RDDs from Parallelized collections

RDDs from External Datasets

Partitions in your data

Basic RDD Transformations and Actions

Map and Collect

Filter and Count

Pair RDDs in PySpark

ReduceBykey and Collect

SortByKey and Collect

Advanced RDD Actions

CountingBykeys

Create a base RDD and transform it

Remove stop words and reduce the dataset

Print word frequencies

Programming in PySpark RDD’s

In this chapter, you'll learn about Spark SQL which is a Spark module for structured data processing. It provides a programming abstraction called DataFrames and can also act as a distributed SQL query engine. This chapter shows how Spark SQL allows you to use DataFrames in Python.

Abstracting Data with DataFrames

RDD to DataFrame

Loading CSV into DataFrame

Operating on DataFrames in PySpark

Inspecting data in PySpark DataFrame

PySpark DataFrame subsetting and cleaning

Filtering your DataFrame

Interacting with DataFrames using PySpark SQL

Running SQL Queries Programmatically

SQL queries for filtering Table

Data Visualization in PySpark using DataFrames

PySpark DataFrame visualization

Part 1: Create a DataFrame from CSV file

Part 2: SQL Queries on DataFrame

Part 3: Data visualization

PySpark SQL & DataFrames

PySpark MLlib is the Apache Spark scalable machine learning library in Python consisting of common learning algorithms and utilities. Throughout this last chapter, you'll learn important Machine Learning algorithms. You will build a movie recommendation engine and a spam filter, and use k-means clustering.

Panoramica di PySpark MLlib

Librerie ML di PySpark

Algoritmi di PySpark MLlib

Collaborative filtering

Caricare il dataset MovieLens negli RDD

Addestramento del modello e predizioni

Valutazione del modello usando MSE

Classificazione

Caricare dati spam e non-spam

Feature hashing e LabelPoint

Training del modello di Logistic Regression

Clustering

Caricamento e parsing dei dati con 5000 punti

Addestramento K-means

Visualizzare i cluster

Congratulazioni!

Machine Learning with PySpark MLlib

Complete Shakespeare

Movie ratings

5000 points

FIFA 2018

People

Spam

Negli ultimi anni si è parlato molto di Big Data, e ormai sono entrati nel mainstream per molte aziende. Ma cosa sono davvero i Big Data? Questo corso copre i fondamenti dei Big Data tramite PySpark. Spark è un framework di "lightning fast cluster computing" per i Big Data. Fornisce un motore generale per l’elaborazione dei dati e ti permette di eseguire programmi fino a 100 volte più veloci in memoria, o 10 volte più veloci su disco, rispetto a Hadoop. Userai PySpark, un pacchetto Python per programmare in Spark, e le sue potenti librerie di livello superiore come SparkSQL, MLlib (per il Machine Learning), ecc. Esplorerai le opere di William Shakespeare, analizzerai i dati dei Mondiali Fifa 2018 ed eseguirai il clustering su insiemi di dati genomici. Al termine del corso, avrai una comprensione approfondita di PySpark e della sua applicazione all’analisi generale dei Big Data.

Introduction to Python

Impara ad analizzare Big Data con PySpark, SparkSQL e MLlib per la programmazione in Spark.

Fondamenti di Big Data con PySpark

Impara le basi per lavorare con i big data usando PySpark.

Big Data con PySpark

Librerie ML di PySpark

Fondamenti di Big Data con PySpark

Esercizio pratico interattivo