Sorting vs. arranging

So far in this chapter, you've explored some feature transformation functions from Spark's MLlib. sparklyr also provides access to some functions making use of the Spark DataFrame API.

The dplyr way of sorting a tibble is to use arrange(). You can also sort tibbles using Spark's DataFrame API using sdf_sort(). This function takes a character vector of columns to sort on, and currently only sorting in ascending order is supported.

For example, to sort by column x, then (in the event of ties) by column y, then by column z, the following code compares the dplyr and Spark DataFrame approaches.

a_tibble %>%
  arrange(x, y, z)
a_tibble %>%
  sdf_sort(c("x", "y", "z"))

To see which method is faster, try using both arrange(), and sdf_sort(). You can see how long your code takes to run by wrapping it in microbenchmark(), from the package of the same name.

microbenchmark({
  # your code
})

You can learn more about profiling the speed of your code in the Writing Efficient R Code course.