APACHE ICEBERG MIGRATION GUIDE dremio.com/blog/migration-gu…

DREMIO.ICEBERG.DBT.NESSIE.MINIO.POSTGRES.MONGODB If you want to try a deep end-to-end tutorial that will get you hands-on with a variety of popular data tools, try this one out. dremio.com/blog/experience-t… #DataLakehouse #DataEngineering #ApacheIceberg #Dremio #dbt #DataScience #DataAnalytics

RECENT DATA ARCHITECTURE/ENGINEERING/ANALYTICS CONTENT — Apache Iceberg — > What is Data Lakehouse Table Format? dremio.com/blog/apache-icebe… > Comparing Iceberg to Other Lakehouse Solutions dremio.com/blog/comparing-ap… > Iceberg Migration Guide dremio.com/blog/migration-gu… > Hands-on with Managed Polaris Catalog dremio.com/blog/getting-hand… > Hands-on with Self-Managed Polaris dremio.com/blog/getting-hand… — Hybrid Lakehouse — > 3 Dremio Use Cases for On-Prem Data Lakes dremio.com/blog/3-dremio-use… > Hybrid Lakehouse Solution: NetApp dremio.com/blog/hybrid-lakeh… > Hybrid Lakehouse Solution: Minio dremio.com/blog/hybrid-lakeh… > Hybrid Lakehouse Solution: Vast Data dremio.com/blog/hybrid-lakeh… > Hybrid Lakehouse Solution: Pure Storage dremio.com/blog/hybrid-lakeh… — Unified Analytics — > Analysts Guide to JDBC/ODBC, REST, and Arrow Flight dremio.com/blog/a-data-analy… > Unified Lakehouse dremio.com/blog/the-unified-… #DataEngineering #DataLakehouse #DataScience #DataAnalytics #DataArchitecture

HOW ICEBERG CATALOGS WORK Iceberg tables are one part data stored in several parquet files and a second part metadata files that provide context and understanding of that data as a singular table. The metadata entry point is a file called metadata.json which tracks the tables schemas, partition schemes and snapshots. Everytime the table changes a new metadata.json is created. So when there is possibly dozens or hundreds of these metadata.json files, how does an engine like Dremio, Snowflake or Apache Spark know which is the right one to query the table accurately. This is where a catalog comes in like Nessie and Polaris. A catalog acts like a traffic controller maintaining a list of tables along with the file address where the current metadata.json is stored. These references are updated at the end of a transaction after the new metadata.json is created enabling Atomicity guarantees. A catalog directs queries to the right metadata.json and updates that list when writes are complete. If you enjoyed this post, give it a like and a share! Also check out Dremio.com/blog for a lot more Apache Iceberg education resources. #ApacheIceberg #DataLakehouse #DataEngineering

OPTIMIZING ICEBERG TABLES One the things that make Iceberg queries fast is that the metadata can be used eliminate files that don’t need scanning from the scan plan. This is great but if the data is not clustered properly or spread out across many small files, you can still see less than ideal performance. ** Compaction ** When you have more manifests and data files than you need, you are doing more file operations and slowing down performance. By rewriting these files so you can collapse the data into fewer larger files you have the opposite effect. This can be done the REWRITE_DATA_FILES or REWRITE_MANIFESTS procedures in Spark or the OPTIMIZE TABlE command in Dremio. ** Clustering ** If I only am searching for agent in the northwest region, it’d be nice if all those reps where in the same few files, this is known as clustering. When rewriting data files with Spark, there is a “sort” parameter you can pass so it can cluster the data as it rewrites the files. By compacting and clustering you data, the Apache Iceberg metadata becomes even more powerful in skipping data files when executing queries. Read more in my new article on maintaining Apache Iceberg lakehouses here: dremio.com/blog/guide-to-mai… #DataLakehouse #ApacheIceberg #DataEngineering

Do you use an open table format? If so how’s your experience been, vote, reply and share! #ApacheIcberg #DeltaLake #ApacheHudi #DataEngineering #DataLakehouse

Open Tables (Apache Iceberg) + Open (Nessie, Polaris, Gravitino) Catalogs = No Vendor Lock-in Lakehouses Read More: blog.iceberglakehouse.com/op… #DataEngineering #DataLakehouse #DataLake @dremio @SnowflakeDB @ApacheIceberg

NEW MEDIA ON OPEN SOURCE APACHE ICEBERG CATALOGS - New Episode of "Datanation" which you can find on iTunes and Spotify - Substack: open.substack.com/pub/amdata… #ApacheIceberg #DataLakehouse #Dremio #Snowflake #Databricks #DataEngineering

DATA PROFESSIONAL FOLLOW TRAIN - reply to this tweet, I will follow you - follow me and everyone else who replies - retweet to maximize reach of the train #DataEngineering #DataAnalytics #DataScience #BigData #DataLakehouse #DataLake

WHAT IS A DATA LAKEHOUSE? #DataEngineering #DataAnalytics #BigData #ApacheIceberg

8-BIT GAME TRAILER: Quest for the Data Lakehouse #DataLakehouse #DataEngineering #ApacheIceberg #Dremio

FREE YOUR DATA AND GET A FREE BOOK Watch to learn more #DataEngineering #DataAnalytics #DataLakehouse #DataLake #BigData