In this chapter, we cover the reasons you need to apply new techniques when data sets are larger than available RAM.  We show that importing and exporting data using the base R functions can be slow and some easy ways to remedy this. Finally, we introduce the bigmemory package. 

What is Scalable Data Processing?

Why is your code slow?

How does processing time vary by data size?

Working with "Out-of-Core" Objects using the Bigmemory Project

Reading a big.matrix object

Attaching a big.matrix object

Creating tables with big.matrix objects

Data summary using bigsummary

References vs. Copies

Copying matrices and big matrices

Working with increasingly large data sets

Now that you've got some experience using bigmemory, we're going to go through some simple data exploration and analysis techniques. In particular, we'll see how to create tables and implement the split-apply-combine approach.

The Bigmemory Suite of Packages

Tabulating using bigtable

Borrower Race and Ethnicity by Year (I)

Split-Apply-Combine

Female Proportion Borrowing

Split

Apply

Combine

Visualize your results using the tidyverse

Visualizing Female Proportion Borrowing

The Borrower Income Ratio

Tidy Big Tables

Limitations of bigmemory

Where should you use bigmemory?

Processing and Analyzing Data with bigmemory

We'll use the iotools package that can process both numeric and string data, and introduce the concept of chunk-wise processing.


Pengantar pemrosesan per potongan

Dapatkah Anda melakukan split-compute-combine?

Operasi foldable (I)

Operasi foldable (II)

Sekilas iotools: Mengimpor data

Bandingkan read.delim() dan read.delim.raw()

Membaca data raw dan mengubahnya menjadi struktur data

chunk.apply

Membaca potongan data sebagai matriks

Membaca chunk sebagai data.frame

Memparalelkan pemanggilan chunk.apply

Working with iotools

In the previous chapters, we've introduced the housing data and shown how to compute with data that is about as big, or bigger than, the amount of RAM on a single machine. In this chapter, we'll go through a preliminary analysis of the data, comparing various trends over time.

Overview of types of analysis for this chapter

Race and Ethnic Representation in the Mortgage Data

Comparing the Borrower Race/Ethnicity and their Proportions

Are the data missing at random?

Looking for Predictable Missingness

A little more about missingness

Analyzing the Housing Data

Borrower Race and Ethnicity by Year (II)

Visualizing the Adjusted Demographic Trends

Relative change in demographic trend

Borrower Lending Trends: City vs. Rural

Borrower Region by Year

Who is securing federally guaranteed loans?

Congratulations!

Case Study: A Preliminary Analysis of the Housing Data

Mortgage data (sample)

Sering kali himpunan data lebih besar daripada RAM yang tersedia, yang menimbulkan masalah bagi pemrogram R karena secara default semua variabel disimpan di memori. Anda akan mempelajari alat untuk memproses, mengeksplorasi, dan menganalisis data langsung dari disk. Anda juga akan menerapkan pendekatan split-apply-combine dan mempelajari cara menulis kode yang dapat diskalakan menggunakan paket bigmemory dan iotools. Dalam kursus ini, Anda akan menggunakan data dari Federal Housing Finance Agency, sebuah himpunan data publik yang mencatat semua hipotek yang dipegang atau disekuritisasi oleh Federal National Mortgage Association (Fannie Mae) dan Federal Home Loan Mortgage Corporation (Freddie Mac) dari 2009–2015.

Writing Efficient R Code

Pelajari cara menulis kode skala besar untuk data besar di R dengan paket bigmemory dan iotools.

Pemrosesan Data yang Dapat Diskalakan di R

Pelajari cara menulis kode yang skalabel untuk bekerja dengan big data di R menggunakan paket bigmemory dan iotools.

Data Besar di R

Dapatkah Anda melakukan split-compute-combine?

Pemrosesan Data yang Dapat Diskalakan di R

Latihan interaktif praktis