File size optimization
Consider if you're given 2 large data files on a cluster with 10 nodes. Each file contains 10M rows of roughly the same size. While working with your data, the responsiveness is acceptable but the initial read from the files takes a considerable period of time. Note that you are the only one who will use the data and it changes for each run.
Which of the following is the best option to improve performance?
This exercise is part of the course
Cleaning Data with PySpark
Hands-on interactive exercise
Turn theory into action with one of our interactive exercises
