Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

I've many images in a directory in aws s3 bucket, I want to run an etl job to generate…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.5-pro’s answer.
  1. User

    I've many images in a directory in aws s3 bucket, I want to run an etl job to generate the metadata of the images in csv format or parquet format. Generate a pyspark code for that and guide me through how to achieve that

  2. Assistant

    Below is a complete, end-to-end recipe that you can copy-paste, parameterise, and run on almost any Spark-based service on AWS (EMR, Glue, Databricks, even a local pyspark shell that has network access to S3).

    ⚡ What it does
    • Scans an S3 prefix recursively
    • Extracts for every image
    – S3 path
    – File size (bytes)
    – Last-modified timestamp (from S3)
    – Image width, height & format (e.g. JPEG/PNG)
    • Saves the resulting dataframe back to S3 in Parquet or CSV


    1. Python / PySpark script

    Save as image_metadata_etl.py

    #!/usr/bin/env python3
    # -------------------------------------------------------
    # Extract image metadata from an S3 bucket/prefix
    # -------------------------------------------------------
    
    import argparse, io, sys
    from pyspark.sql import SparkSession
    from pyspark.sql.types import *
    from pyspark.sql.functions import pandas_udf, col
    import pandas as pd
    from PIL import Image      # Pillow
    
    # ------------------------------------------------------------------
    # 0. Cmd-line arguments
    # ------------------------------------------------------------------
    parser = argparse.ArgumentParser(description="Generate image metadata")
    parser.add_argument("--input",  required=True, help="s3://bucket/prefix")
    parser.add_argument("--output", required=True, help="s3://bucket/output_path")
    parser.add_argument("--format", default="parquet", choices=["parquet", "csv"],
                        help="Output format (parquet|csv)")
    parser.add_argument("--partitions", type=int, default=None,
                        help="Repartition before write (optional)")
    args = parser.parse_args()
    
    # ------------------------------------------------------------------
    # 1. SparkSession
    # ------------------------------------------------------------------
    spark = (SparkSession.builder
             .appName("image-metadata-etl")
             # Hadoop-AWS jar is required if you run outside Glue/EMR 6+
             # .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.621")
             .getOrCreate())
    
    # ===> optional: tune S3A
    hconf = spark._jsc.hadoopConfiguration()
    hconf.set("fs.s3.impl",                 "org.apache.hadoop.fs.s3a.S3AFileSystem")
    hconf.set("fs.s3a.aws.credentials.provider",
              "com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
    hconf.set("fs.s3a.fast.upload", "true")
    
    # ------------------------------------------------------------------
    # 2. Schema for the Pandas UDF
    # ------------------------------------------------------------------
    meta_schema = StructType([
        StructField("width",  IntegerType(), True),
        StructField("height", IntegerType(), True),
        StructField("format", StringType(),  True)
    ])
    
    @pandas_udf(meta_schema)     # vectorised, runs per batch ⇒ fast
    def extract_meta(content_series: pd.Series) -> pd.DataFrame:
        widths, heights, formats = [], [], []
        for binary in content_series:
            try:
                img = Image.open(io.BytesIO(binary))
                w, h       = img.size
                fmt        = img.format
            except Exception as e:            # corrupted file?
                w, h, fmt  = None, None, None
            widths.append(w); heights.append(h); formats.append(fmt)
        return pd.DataFrame({"width": widths, "height": heights, "format": formats})
    
    # ------------------------------------------------------------------
    # 3. Read all images from S3 as binary
    # ------------------------------------------------------------------
    df_raw = (spark.read.format("binaryFile")
              .option("recursiveFileLookup", "true")
              .option("pathGlobFilter", "*.jpg,*.jpeg,*.png,*.gif,*.bmp,*tif*")  # edit if desired
              .load(args.input))
    
    # binaryFile schema: path, modificationTime, length, content
    df_meta = (df_raw
               .withColumn("meta", extract_meta("content"))
               .select(
                   col("path").alias("s3_path"),
                   col("modificationTime").cast("timestamp").alias("last_modified"),
                   col("length").alias("size_bytes"),
                   col("meta.width"),
                   col("meta.height"),
                   col("meta.format").alias("image_format")
               ))
    
    if args.partitions:
        df_meta = df_meta.repartition(args.partitions)
    
    # ------------------------------------------------------------------
    # 4. Write back to S3
    # ------------------------------------------------------------------
    if args.format.lower() == "parquet":
        (df_meta.write.mode("overwrite")
                .partitionBy("image_format")          # helpful for pruning
                .parquet(args.output))
    else:
        (df_meta.write.mode("overwrite")
                .option("header", "true")
                .csv(args.output))
    
    print(f"✅ Finished! {df_meta.count()} records written to {args.output}")
    spark.stop()
    

    2. Running the job

    Below are the two most common ways on AWS; pick whichever service you like.

    2.1 AWS Glue 4.0 (Spark 3.3) – no infrastructure to manage

    1. Create a new Glue Job (Spark, Python).
    2. Under “Python library path” add pillow-9.5.0.zip (or newer) uploaded in S3.
    3. (Glue already bundles hadoop-aws, so no extra jars are needed)
    4. Paste the script above into the job (or point to it in S3).
    5. Job arguments, e.g.
      --input  s3://my-bucket/raw_images/
      --output s3://my-bucket/image_metadata/
      --format parquet
      --partitions 200
      
    6. Attach an IAM role that has S3:GetObject, S3:PutObject, S3:ListBucket.
    7. Run it → output will appear under the chosen path.

    2.2 Amazon EMR 6.x (or 7.x) – if you need full cluster control

    # 1. Start a cluster (or use an existing one)
    aws emr create-cluster \
      --name "image-meta" \
      --release-label emr-6.15.0 \
      --applications Name=Spark \
      --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole \
      --use-default-roles \
      --instance-type m6g.xlarge \
      --instance-count 3 \
      --bootstrap-actions Path="s3://my-bucket/libs/install_pillow.sh"
    
    # 2. Example bootstrap script install_pillow.sh
    #!/bin/bash
    sudo python3 -m pip install pillow --upgrade
    
    # 3. Copy the ETL script to S3 if not already there
    aws s3 cp image_metadata_etl.py s3://my-bucket/code/
    
    # 4. Submit the Spark job
    aws emr add-steps --cluster-id j-XXXXXXXXXX --steps '[
      {
        "Name": "Img metadata",
        "Type": "Spark",
        "ActionOnFailure": "CONTINUE",
        "Args": [
           "spark-submit",
           "--deploy-mode","cluster",
           "s3://my-bucket/code/image_metadata_etl.py",
           "--input","s3://my-bucket/raw_images/",
           "--output","s3://my-bucket/image_metadata/",
           "--format","parquet"
        ]
      }]'
    

    (You can do the same from the EMR Studio notebook or Databricks workspace—just pip install pillow in the cluster and run the script interactively.)


    3. Verification & usage

    spark.read.parquet("s3a://my-bucket/image_metadata/")
          .printSchema()
          .show(10, truncate=False)
    

    A sample record

    +--------------------------------------------------------------+-------------------+----------+-----+------+------------+
    |s3_path                                                      |last_modified     |size_bytes|width|height|image_format|
    +--------------------------------------------------------------+-------------------+----------+-----+------+------------+
    |s3a://my-bucket/raw_images/2024/05/01/cat_01.jpg             |2024-05-01 11:22:33|  233728  |1024 | 768  |JPEG        |
    

    4. Good practices & tweaks

    • Small-file explosion? Consider spark.hadoop.mapreduce.fileinputformat.input.dir.recursive=true and/or S3 inventory + Athena if you only need size & timestamps (no width/height).
    • Very large datasets → increase --partitions or call coalesce() before writing to keep parquet files around 100–200 MB each.
    • Want EXIF (camera model, GPS, …)? Extend the UDF with img._getexif() and parse the dict (wrap in try/except).
    • Serverless alternative: use AWS Lambda + S3 event for near-real-time metadata capture; Spark is better for backfills & bulk workloads.


    That’s it! 🎉 Drop the script in, press run, and you’ll have a clean Parquet (or CSV) manifest of every image sitting in your S3 bucket, ready to be queried with Athena, Redshift Spectrum, or loaded downstream in your analytics pipeline.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.