Table Maintenance for Iceberg - Compaction and Abandoned File Cleanup
1. Table Maintenance for Iceberg - Compaction and Abandoned File Cleanup
As you write to a table, you accumulate manifest files. Remember that each snapshot references a manifest list, which references manifests, which ultimately reference data files. If you're doing frequent commits, you can end up with hundreds or thousands of small manifest files. Let's clean those up a little bit. Similar to how we cleaned up the data files, we can clean up the manifests like this. The operation consolidates those small manifests into larger ones, which speeds up query planning. It doesn't affect the actual data scan performance. You're still reading the same number of data files, but it makes the planning phase faster because there are fewer metadata files to open. In addition to metadata compaction, you will need to work with data file compaction at some frequency, where the frequency is dependent on how often new data is ingested into Iceberg. You perform a data compaction procedure like this. But what did this procedure actually do? Data file compaction scans your table for partitions with many small files and rewrites them into optimally sized files based on a size you define. The result is better query performance and significantly reduced metadata overhead. When your query engine scans a partition, it's opening one or two large files instead of 50 small ones, which in many engines is much faster. Compaction of data files generally improves both query performance and planning performance because there's simply less metadata to process. The fewer files Iceberg has to track in its manifests, the faster it can determine which files to scan for a given query. There's one more maintenance operation you should know about, though hopefully you'll rarely need it. It's called remove orphan files. This is important to understand because it's fundamentally different from the others. Remove orphan files is only necessary when Iceberg operations have failed. Under normal circumstances, all Iceberg writes that fail to commit will clean up after themselves. If a commit fails, any written but now unusable data files are cleaned up automatically. But in cases where operations are interrupted or fail in unexpected ways, you might end up with files in your table directory that are no longer referenced by any metadata. We call these files orphaned because there is no longer a way to locate them using Iceberg metadata alone. These orphan files waste storage and can accumulate over time. Remove orphan files takes a list of all the files in the table directory and compares them against what's referenced in the metadata, then removes anything that's orphaned. Here's the critical thing. This procedure is by far the most expensive maintenance operation because its default mode requires that it lists the entire table directory structure in object storage, which can be extremely slow and expensive for large tables. The cost of removing orphan files scales with the number of files in the directory. The larger a table becomes, the more expensive it becomes to run the procedure on. Importantly, it has no effect on table performance. It's purely about reclaiming storage. Some object stores like Amazon's S3 provide an inventory service, which periodically produces a listing of everything in a prefix. This can be used to reduce the listing cost, but the detection of orphans still requires an expensive distributed shuffle, which will scale up as table size does. With that understanding of removing orphan files, it's important to show you how to perform the actual procedure. You should only run remove orphan files sparingly, and only if you've actually experienced failed operations or suspect orphan files exist. Don't make it a part of your regular daily or weekly maintenance routine. Some organizations run it as a safety net on a monthly schedule just to be thorough. But if your writes are stable and you're not seeing unexplained storage growth, you can probably skip it most of the time. The cost of listing millions of objects usually outweighs the benefit unless you have a real orphan problem. When should you run these maintenance operations? Here's a practical rule of thumb I like to follow. Any table being written to continuously will need some kind of data file compaction. The more frequently you write, the more often you need to perform metadata compaction, and the more often you write small files, the more often you should run data compaction. For example, a table receiving streaming writes every 10 minutes might need metadata compaction daily. A table receiving batch writes once per day might only need compaction weekly or monthly. You'll need to monitor your specific tables and adjust based on their write patterns and querying performance. Now, I should mention that many users opt not to run maintenance on their own. There are third-party services and managed iceberg offerings that will handle table maintenance for you. Running compaction and expiration on optimized schedules based on table activity patterns. There are also open-source projects and community tools that provide automated maintenance frameworks. Whether you choose to manage maintenance yourself or use a third-party service, the key is to actually do it. Tables don't maintain themselves. Regardless of what method or methods you choose, be sure to monitor your table health. Track metrics like average file size, number of files per partition, number of manifests, size of metadata.json, and number of snapshots. These metrics will tell you when maintenance is needed and whether your maintenance schedule is working. A well-maintained iceberg table should have consistently sized data files and a manageable number of manifests, and a metadata.json file that stays under a few hundred kilobytes. In the following exercises, you'll practice running each of these maintenance operations on tables with realistic problems. Tables with thousands of small files, bloated metadata, and accumulated snapshot history. You'll see firsthand how each operation affects table performance and structure. This hands-on experience will give you the confidence to maintain production tables effectively.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.