In this chapter, we cover the reasons you need to apply new techniques when data sets are larger than available RAM.  We show that importing and exporting data using the base R functions can be slow and some easy ways to remedy this. Finally, we introduce the bigmemory package. 

What is Scalable Data Processing?

Why is your code slow?

How does processing time vary by data size?

Working with "Out-of-Core" Objects using the Bigmemory Project

Reading a big.matrix object

Attaching a big.matrix object

Creating tables with big.matrix objects

Data summary using bigsummary

References vs. Copies

Copying matrices and big matrices

Working with increasingly large data sets

Now that you've got some experience using bigmemory, we're going to go through some simple data exploration and analysis techniques. In particular, we'll see how to create tables and implement the split-apply-combine approach.

The Bigmemory Suite of Packages

Tabulating using bigtable

Borrower Race and Ethnicity by Year (I)

Split-Apply-Combine

Female Proportion Borrowing

Split

Apply

Combine

Visualize your results using the tidyverse

Visualizing Female Proportion Borrowing

The Borrower Income Ratio

Tidy Big Tables

Limitations of bigmemory

Where should you use bigmemory?

Processing and Analyzing Data with bigmemory

We'll use the iotools package that can process both numeric and string data, and introduce the concept of chunk-wise processing.


Introduction to chunk-wise processing

Can you split-compute-combine it?

Foldable operations (I)

Foldable operations (II)

A first look at iotools: Importing data

Compare read.delim() and read.delim.raw()

Reading raw data and turning it into a data structure

chunk.apply

Reading chunks in as a matrix

Reading chunks in as a data.frame

Parallelizing calls to chunk.apply

Working with iotools

In the previous chapters, we've introduced the housing data and shown how to compute with data that is about as big, or bigger than, the amount of RAM on a single machine. In this chapter, we'll go through a preliminary analysis of the data, comparing various trends over time.

Panoramica dei tipi di analisi per questo capitolo

Rappresentanza razziale ed etnica nei dati sui mutui

Confronto tra razza/etnia dei mutuatari e relative proporzioni

I dati mancano in modo casuale?

Alla ricerca di dati mancanti prevedibili

Qualcosa in più sui dati mancanti

Analizzare i dati sul mercato immobiliare

Razza ed etnia dei mutuatari per anno (II)

Visualizzare le tendenze demografiche corrette

Variazione relativa nella tendenza demografica

Tendenze dei prestiti ai mutuatari: città vs aree rurali

Regione del mutuatario per anno

Chi sta garantendo i prestiti federali?

Congratulazioni!

Case Study: A Preliminary Analysis of the Housing Data

Mortgage data (sample)

Gli insiemi di dati sono spesso più grandi della RAM disponibile, il che crea problemi a chi programma in R, dato che per impostazione predefinita tutte le variabili vengono memorizzate in memoria. Imparerai strumenti per elaborare, esplorare e analizzare i dati direttamente dal disco. Metterai inoltre in pratica l’approccio split-apply-combine e imparerai a scrivere codice scalabile usando i pacchetti bigmemory e iotools. In questo corso userai i dati della Federal Housing Finance Agency, un insieme di dati pubblico che documenta tutti i mutui detenuti o cartolarizzati sia da Federal National Mortgage Association (Fannie Mae) sia da Federal Home Loan Mortgage Corporation (Freddie Mac) dal 2009 al 2015.

Writing Efficient R Code

Impara a scrivere codice scalabile per big data in R con i pacchetti bigmemory e iotools.

Elaborazione scalabile dei dati in R

Impara a scrivere codice scalabile per lavorare con i big data in R usando i pacchetti bigmemory e iotools.

Big Data in R

Assistant Professor, Yale University

Are the data missing at random?

Missing data

Types of Missing Data

MCAR

MNAR

Dealing with missing data in this course

A Quick Check for MAR

MAR Quick Check Example

Let's practice!

Case Study: A Preliminary Analysis of the Housing Data - I dati mancano in modo casuale?

I dati mancano in modo casuale?

Create Your Free Account