Q- choosing spark as the execution and picking any dummy data Try to create simple prof…
- User
Q- choosing spark as the execution and picking any dummy data Try to create simple prof of concept of get hands on view of this in action “How Apache Iceberg Came to Be Netflix, in the creation of what became the Apache Iceberg format, concluded that many of the problems with the Hive format stemmed from one simple but fundamental flaw: each table is tracked as directories and subdirectories, limiting the granularity that is necessary to provide consistency guarantees, better concurrency, and several of the features that often are available in data warehouses. With this in mind, Netflix set out to create a new table format with several goals in mind: Consistency If updates to a table occur over multiple partitions, it must not be possible for end users to experience inconsistency in the data they are viewing. An update to a table across multiple partitions should be done quickly and atomically so that the data is consistent to end users. They see the data either before the update or after the update, and not in between. Performance With Hive’s file/directory listing bottleneck, query planning would take excessively long to complete before actually executing the query. The table should provide metadata and avoid excessive file listing so that not only can query planning be a faster process, but also the resulting plans can be executed more quickly since they scan only the files necessary to satisfy the query. Easy to use To get the benefits of techniques such as partitioning, end users should not have to be aware of the physical structure of the table. The table should be able to give users the benefits of partitioning based on naturally intuitive queries and not depend on filtering extra partition columns derived from a column they are already filtering by (e.g., filtering by a month column when you’ve already filtered the timestamp it is derived from). Evolvability Updating schemas of Hive tables could result in unsafe transactions, and updating how a table is partitioned would result in a need to rewrite the entire table. A table should be able to evolve its schema and partitioning scheme safely and without the need for rewriting. Scalability All the preceding goals should be able to be accomplished at the petabyte scale of Netflix’s data. So the team began creating the Iceberg format, which focuses on defining a table as a canonical list of files instead of tracking a table as a list of directories and subdirectories. The Apache Iceberg project is a specification, or a standard of how metadata defining a data lakehouse table should be written across several files. To support the adoption of this standard, Apache Iceberg has many support libraries to help individuals work with the format or compute engines to implement support. Along with these libraries, the project has created implementations for open source compute engines such as Apache Spark and Apache Flink.”
- Assistant
Certainly! Let's create a simple proof of concept (PoC) to demonstrate how Apache Iceberg works with Apache Spark using dummy data. This PoC will help you understand how Iceberg addresses the limitations of traditional Hive tables by providing better consistency, performance, ease of use, evolvability, and scalability.
Table of Contents
- Prerequisites
- Setting Up the Environment
- Creating Dummy Data
- Configuring Apache Spark with Iceberg
- Creating an Iceberg Table
- Performing CRUD Operations
- Demonstrating Schema Evolution
- Querying the Iceberg Table
- Conclusion
Prerequisites
Before we begin, ensure you have the following installed:
- Java 8 or higher
- Apache Spark 3.x
- Python 3.x (if using PySpark)
- Apache Iceberg library
You can use a local machine or a virtual environment like Docker. For simplicity, we'll proceed with a local setup using PySpark.
Setting Up the Environment
1. Install Apache Spark
If you haven't installed Apache Spark, download it from the official website and follow the installation instructions.
2. Install PySpark and Iceberg Dependencies
Install
pysparkandpyarrowusingpip:pip install pyspark pyarrowWe'll also need to include Iceberg's runtime JAR. You can download the latest Iceberg Spark runtime JAR from Maven Central.
For this PoC, we'll assume you have downloaded
iceberg-spark-runtime-3.5.1.jarand placed it in a known directory.Creating Dummy Data
Let's create some dummy data to work with. We'll use PySpark to generate a simple DataFrame.
from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, IntegerType, StringType, TimestampType import datetime # Initialize Spark Session spark = SparkSession.builder \ .appName("IcebergPoC") \ .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \ .config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog") \ .config("spark.sql.catalog.spark_catalog.type", "hive") \ .getOrCreate() # Create Dummy Data data = [ (1, "Alice", datetime.datetime(2023, 1, 10)), (2, "Bob", datetime.datetime(2023, 2, 15)), (3, "Charlie", datetime.datetime(2023, 3, 20)), (4, "Diana", datetime.datetime(2023, 4, 25)), (5, "Ethan", datetime.datetime(2023, 5, 30)), ] schema = StructType([ StructField("id", IntegerType(), False), StructField("name", StringType(), False), StructField("signup_date", TimestampType(), False), ]) df = spark.createDataFrame(data, schema) df.show()Output:
+---+-------+-------------------+ |id |name |signup_date | +---+-------+-------------------+ |1 |Alice |2023-01-10 00:00:00| |2 |Bob |2023-02-15 00:00:00| |3 |Charlie|2023-03-20 00:00:00| |4 |Diana |2023-04-25 00:00:00| |5 |Ethan |2023-05-30 00:00:00| +---+-------+-------------------+Configuring Apache Spark with Iceberg
Ensure that Spark is configured to use Iceberg by specifying the necessary configurations during the Spark session initialization. We've already included these configurations in the Spark session above.
Additionally, you need to include the Iceberg runtime JAR when starting Spark. If you're running this in a script, ensure that the JAR is available in the classpath. Alternatively, you can specify it using the
--jarsoption when submitting a Spark job:spark-submit --jars /path/to/iceberg-spark-runtime-3.5.1.jar your_script.pyCreating an Iceberg Table
We'll create an Iceberg table in a specified database. For simplicity, we'll use the default Spark catalog with Hive support.
# Specify Database and Table Name database = "default" table_name = "iceberg_users" # Create Database if not exists spark.sql(f"CREATE DATABASE IF NOT EXISTS {database}") # Define the Database.Table path (using Hive metastore) fully_qualified_table = f"{database}.{table_name}" # Write DataFrame to Iceberg Table df.writeTo(fully_qualified_table).using("iceberg").createOrReplace()Explanation:
createOrReplace(): Creates the table if it doesn't exist, or replaces it if it does.using("iceberg"): Specifies that we're using Iceberg as the table format.
Performing CRUD Operations
Insert New Data
# New Data to Insert new_data = [ (6, "Fiona", datetime.datetime(2023, 6, 5)), (7, "George", datetime.datetime(2023, 7, 10)), ] new_df = spark.createDataFrame(new_data, schema) new_df.writeTo(fully_qualified_table).append()Read Data from the Iceberg Table
# Read from Iceberg Table iceberg_df = spark.read.table(fully_qualified_table) iceberg_df.show() # Example Output: # +---+-------+-------------------+ # |id |name |signup_date | # +---+-------+-------------------+ # |1 |Alice |2023-01-10 00:00:00| # |2 |Bob |2023-02-15 00:00:00| # |3 |Charlie|2023-03-20 00:00:00| # |4 |Diana |2023-04-25 00:00:00| # |5 |Ethan |2023-05-30 00:00:00| # |6 |Fiona |2023-06-05 00:00:00| # |7 |George |2023-07-10 00:00:00| # +---+-------+-------------------+Update Data
Iceberg supports transactional updates.
from pyspark.sql.functions import when # Update the name where id = 2 updated_df = iceberg_df.withColumn( "name", when(iceberg_df.id == 2, "Bobby").otherwise(iceberg_df.name) ) # Overwrite the Iceberg Table with Updated Data updated_df.writeTo(fully_qualified_table).overwrite()Delete Data
# Delete the record where id = 3 filtered_df = iceberg_df.filter("id != 3") filtered_df.writeTo(fully_qualified_table).overwrite()Demonstrating Schema Evolution
One of Iceberg's strengths is the ability to evolve schemas without rewriting the entire table.
Add a New Column
# Add 'email' column as nullable spark.sql(f""" ALTER TABLE {fully_qualified_table} ADD COLUMN email STRING """)Update Schema with New Data
# New Data with Email updated_data = [ (1, "Alice", datetime.datetime(2023, 1, 10), "alice@example.com"), (2, "Bobby", datetime.datetime(2023, 2, 15), "bobby@example.com"), (4, "Diana", datetime.datetime(2023, 4, 25), "diana@example.com"), (5, "Ethan", datetime.datetime(2023, 5, 30), "ethan@example.com"), (6, "Fiona", datetime.datetime(2023, 6, 5), "fiona@example.com"), (7, "George", datetime.datetime(2023, 7, 10), "george@example.com"), ] updated_schema = StructType([ StructField("id", IntegerType(), False), StructField("name", StringType(), False), StructField("signup_date", TimestampType(), False), StructField("email", StringType(), True), ]) updated_df = spark.createDataFrame(updated_data, updated_schema) updated_df.writeTo(fully_qualified_table).overwrite()Verify Schema and Data
# Read Updated Table updated_iceberg_df = spark.read.table(fully_qualified_table) updated_iceberg_df.printSchema() updated_iceberg_df.show()Schema Output:
root |-- id: integer (nullable = false) |-- name: string (nullable = false) |-- signup_date: timestamp (nullable = false) |-- email: string (nullable = true)Data Output:
+---+-------+-------------------+--------------------+ |id |name |signup_date |email | +---+-------+-------------------+--------------------+ |1 |Alice |2023-01-10 00:00:00|alice@example.com | |2 |Bobby |2023-02-15 00:00:00|bobby@example.com | |4 |Diana |2023-04-25 00:00:00|diana@example.com | |5 |Ethan |2023-05-30 00:00:00|ethan@example.com | |6 |Fiona |2023-06-05 00:00:00|fiona@example.com | |7 |George |2023-07-10 00:00:00|george@example.com | +---+-------+-------------------+--------------------+Querying the Iceberg Table
You can perform various queries to leverage Iceberg's features.
Time Travel: Querying a Previous Snapshot
Iceberg maintains historical snapshots, allowing you to query data as of a specific point in time.
# Get the current timestamp current_time = datetime.datetime.now() # Perform some more operations (e.g., delete a record) filtered_df = updated_iceberg_df.filter("id != 5") filtered_df.writeTo(fully_qualified_table).overwrite() # Query the table as of the previous snapshot previous_snapshot = updated_iceberg_df previous_snapshot.show()Alternatively, you can use SQL to perform time travel queries:
SELECT * FROM default.iceberg_users FOR SYSTEM_TIME AS OF TIMESTAMP '2023-07-01 00:00:00';Partitioning
Iceberg allows efficient partitioning without exposing the partitioning scheme to the user.
# Recreate the table with partitioning on signup_month spark.sql(f""" CREATE OR REPLACE TABLE {fully_qualified_table} USING iceberg PARTITIONED BY (months(signup_date)) """) # Write data with partitioning updated_df.writeTo(fully_qualified_table).overwrite()Filtering and Pruning
Iceberg's metadata allows Spark to prune unnecessary files during queries.
# Query data where signup_month = 01 (January) spark.sql(f""" SELECT * FROM {fully_qualified_table} WHERE months(signup_date) = 1 """).show()Expected Output:
+---+-----+-------------------+------------------+ |id |name |signup_date |email | +---+-----+-------------------+------------------+ |1 |Alice|2023-01-10 00:00:00|alice@example.com | +---+-----+-------------------+------------------+Conclusion
In this proof of concept, we:
- Set Up: Configured Apache Spark with Apache Iceberg support.
- Created Dummy Data: Generated a simple dataset to work with.
- Created an Iceberg Table: Defined a table using Iceberg's format.
- Performed CRUD Operations: Inserted, updated, and deleted data transactionally.
- Demonstrated Schema Evolution: Added a new column without downtime or data rewriting.
- Queried the Table: Leveraged Iceberg's partitioning and time travel features for efficient querying.
Benefits Illustrated
- Consistency: All operations are atomic, ensuring readers see a consistent view of the data.
- Performance: Iceberg's metadata management avoids expensive file system operations, speeding up query planning and execution.
- Ease of Use: Users interact with tables using standard SQL without needing to manage physical partitions.
- Evolvability: Schema changes are handled gracefully without full table rewrites.
- Scalability: Designed to handle petabyte-scale datasets efficiently.
This PoC provides a foundational understanding of how Apache Iceberg integrates with Apache Spark to offer a robust, scalable, and flexible table format suitable for modern data lakehouses.