I've many images in a directory in aws s3 bucket, I want to run an etl job to generate…
- User
I've many images in a directory in aws s3 bucket, I want to run an etl job to generate the metadata of the images in csv format or parquet format. Generate a pyspark code for that and guide me through how to achieve that
- Assistant
Below is a complete, end-to-end recipe that you can copy-paste, parameterise, and run on almost any Spark-based service on AWS (EMR, Glue, Databricks, even a local pyspark shell that has network access to S3).
⚡ What it does
• Scans an S3 prefix recursively
• Extracts for every image
– S3 path
– File size (bytes)
– Last-modified timestamp (from S3)
– Image width, height & format (e.g. JPEG/PNG)
• Saves the resulting dataframe back to S3 in Parquet or CSV
1. Python / PySpark script
Save as
image_metadata_etl.py#!/usr/bin/env python3 # ------------------------------------------------------- # Extract image metadata from an S3 bucket/prefix # ------------------------------------------------------- import argparse, io, sys from pyspark.sql import SparkSession from pyspark.sql.types import * from pyspark.sql.functions import pandas_udf, col import pandas as pd from PIL import Image # Pillow # ------------------------------------------------------------------ # 0. Cmd-line arguments # ------------------------------------------------------------------ parser = argparse.ArgumentParser(description="Generate image metadata") parser.add_argument("--input", required=True, help="s3://bucket/prefix") parser.add_argument("--output", required=True, help="s3://bucket/output_path") parser.add_argument("--format", default="parquet", choices=["parquet", "csv"], help="Output format (parquet|csv)") parser.add_argument("--partitions", type=int, default=None, help="Repartition before write (optional)") args = parser.parse_args() # ------------------------------------------------------------------ # 1. SparkSession # ------------------------------------------------------------------ spark = (SparkSession.builder .appName("image-metadata-etl") # Hadoop-AWS jar is required if you run outside Glue/EMR 6+ # .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.621") .getOrCreate()) # ===> optional: tune S3A hconf = spark._jsc.hadoopConfiguration() hconf.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") hconf.set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.DefaultAWSCredentialsProviderChain") hconf.set("fs.s3a.fast.upload", "true") # ------------------------------------------------------------------ # 2. Schema for the Pandas UDF # ------------------------------------------------------------------ meta_schema = StructType([ StructField("width", IntegerType(), True), StructField("height", IntegerType(), True), StructField("format", StringType(), True) ]) @pandas_udf(meta_schema) # vectorised, runs per batch ⇒ fast def extract_meta(content_series: pd.Series) -> pd.DataFrame: widths, heights, formats = [], [], [] for binary in content_series: try: img = Image.open(io.BytesIO(binary)) w, h = img.size fmt = img.format except Exception as e: # corrupted file? w, h, fmt = None, None, None widths.append(w); heights.append(h); formats.append(fmt) return pd.DataFrame({"width": widths, "height": heights, "format": formats}) # ------------------------------------------------------------------ # 3. Read all images from S3 as binary # ------------------------------------------------------------------ df_raw = (spark.read.format("binaryFile") .option("recursiveFileLookup", "true") .option("pathGlobFilter", "*.jpg,*.jpeg,*.png,*.gif,*.bmp,*tif*") # edit if desired .load(args.input)) # binaryFile schema: path, modificationTime, length, content df_meta = (df_raw .withColumn("meta", extract_meta("content")) .select( col("path").alias("s3_path"), col("modificationTime").cast("timestamp").alias("last_modified"), col("length").alias("size_bytes"), col("meta.width"), col("meta.height"), col("meta.format").alias("image_format") )) if args.partitions: df_meta = df_meta.repartition(args.partitions) # ------------------------------------------------------------------ # 4. Write back to S3 # ------------------------------------------------------------------ if args.format.lower() == "parquet": (df_meta.write.mode("overwrite") .partitionBy("image_format") # helpful for pruning .parquet(args.output)) else: (df_meta.write.mode("overwrite") .option("header", "true") .csv(args.output)) print(f"✅ Finished! {df_meta.count()} records written to {args.output}") spark.stop()
2. Running the job
Below are the two most common ways on AWS; pick whichever service you like.
2.1 AWS Glue 4.0 (Spark 3.3) – no infrastructure to manage
- Create a new Glue Job (Spark, Python).
- Under “Python library path” add
pillow-9.5.0.zip(or newer) uploaded in S3. - (Glue already bundles
hadoop-aws, so no extra jars are needed) - Paste the script above into the job (or point to it in S3).
- Job arguments, e.g.
--input s3://my-bucket/raw_images/ --output s3://my-bucket/image_metadata/ --format parquet --partitions 200 - Attach an IAM role that has
S3:GetObject,S3:PutObject,S3:ListBucket. - Run it → output will appear under the chosen path.
2.2 Amazon EMR 6.x (or 7.x) – if you need full cluster control
# 1. Start a cluster (or use an existing one) aws emr create-cluster \ --name "image-meta" \ --release-label emr-6.15.0 \ --applications Name=Spark \ --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \ --use-default-roles \ --instance-type m6g.xlarge \ --instance-count 3 \ --bootstrap-actions Path="s3://my-bucket/libs/install_pillow.sh" # 2. Example bootstrap script install_pillow.sh #!/bin/bash sudo python3 -m pip install pillow --upgrade # 3. Copy the ETL script to S3 if not already there aws s3 cp image_metadata_etl.py s3://my-bucket/code/ # 4. Submit the Spark job aws emr add-steps --cluster-id j-XXXXXXXXXX --steps '[ { "Name": "Img metadata", "Type": "Spark", "ActionOnFailure": "CONTINUE", "Args": [ "spark-submit", "--deploy-mode","cluster", "s3://my-bucket/code/image_metadata_etl.py", "--input","s3://my-bucket/raw_images/", "--output","s3://my-bucket/image_metadata/", "--format","parquet" ] }]'(You can do the same from the EMR Studio notebook or Databricks workspace—just
pip install pillowin the cluster and run the script interactively.)
3. Verification & usage
spark.read.parquet("s3a://my-bucket/image_metadata/") .printSchema() .show(10, truncate=False)A sample record
+--------------------------------------------------------------+-------------------+----------+-----+------+------------+ |s3_path |last_modified |size_bytes|width|height|image_format| +--------------------------------------------------------------+-------------------+----------+-----+------+------------+ |s3a://my-bucket/raw_images/2024/05/01/cat_01.jpg |2024-05-01 11:22:33| 233728 |1024 | 768 |JPEG |
4. Good practices & tweaks
• Small-file explosion? Consider
spark.hadoop.mapreduce.fileinputformat.input.dir.recursive=trueand/or S3 inventory + Athena if you only need size & timestamps (no width/height).
• Very large datasets → increase--partitionsor callcoalesce()before writing to keep parquet files around 100–200 MB each.
• Want EXIF (camera model, GPS, …)? Extend the UDF withimg._getexif()and parse the dict (wrap intry/except).
• Serverless alternative: use AWS Lambda + S3 event for near-real-time metadata capture; Spark is better for backfills & bulk workloads.
That’s it! 🎉 Drop the script in, press run, and you’ll have a clean Parquet (or CSV) manifest of every image sitting in your S3 bucket, ready to be queried with Athena, Redshift Spectrum, or loaded downstream in your analytics pipeline.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.