Collecting data back from Spark

In the exercise 'Exploring the structure of tibbles', back in Chapter 1, you saw that tibbles don't store a copy of the data. Instead, the data stays in Spark, and the tibble simply stores the details of what it would like to retrieve from Spark.

There are lots of reasons that you might want to move your data from Spark to R. You've already seen how some data is moved from Spark to R when you print it. You also need to collect your dataset if you want to plot it, or if you want to use a modeling technique that is not available in Spark. (After all, R has the widest selection of available models of any programming language.)

To collect your data: that is, to move it from Spark to R, you call collect().

A Spark connection has been created for you as spark_conn. A tibble attached to the track metadata stored in Spark has been pre-defined as track_metadata_tbl.

Filter the rows of track_metadata_tbl where artist_familiarity is greater than 0.9, assigning the results to results.
Print the class of results, noting that it is a tbl_lazy (used for remote data).
Collect your results, assigning them to collected.
Print the class of collected, noting that it is a tbl_df (used for local data).

Exercise

Collecting data back from Spark

Instructions