Skip to content
The Chat Archive
llama-4-maverick-03-26-experimental· Meta LMArena Human Preference 140K

Create a bash script that does the following: - recursively find all video files (e.g.…

  1. User

    Create a bash script that does the following:

    • recursively find all video files (e.g. .avi, .mov, .mp4, .m4v + others video file formats)
    • go through the list of all found video files in alphabetical order and do the following with each file:
    • print the current file including the path
    • convert them with ffmpeg to av1 with crf 25 and opus codec with stereo 96kbps. All metadata should be taken over to the new file. Also the file system timestamps of the original file should be applied to the new converted file.
    • once converted, add the original filename including path to the file "/rpool/ds1/photo/converted.txt". Also move the original file to "/rpool/ds1/photo/converted/ + path of original file + original filename
  2. Assistant

    Here is a bash script that does exactly what you asked for. This script uses find to recursively locate video files, ffmpeg for the conversion, and handles file metadata and movement as specified.

    Before running the script, please ensure you have ffmpeg installed on your system. You can install it on most Linux distributions using your package manager, e.g., sudo apt install ffmpeg on Debian/Ubuntu systems.

    Also, adjust the TARGET_DIR and LOG_FILE variables if your target directory or log file path differs from /rpool/ds1/photo/. The script assumes you have the necessary permissions to read from and write to the specified directories.

    #!/bin/bash
    
    # Directory where converted files' original paths will be logged and moved to
    TARGET_DIR="/rpool/ds1/photo"
    # Log file to store paths of converted files
    LOG_FILE="$TARGET_DIR/converted.txt"
    # Directory to move original files after conversion
    MOVED_ORIGINALS_DIR="$TARGET_DIR/converted"
    
    # Ensure the log file and moved originals directory exist
    mkdir -p "$MOVED_ORIGINALS_DIR"
    touch "$LOG_FILE"
    
    # Find video files recursively from the current directory
    # This list includes common video file extensions, add or remove as needed
    find . -type f \( -iname "*.avi" -o -iname "*.mov" -o -iname "*.mp4" -o -iname "*.m4v" \
                      -o -iname "*.mkv" -o -iname "*.flv" -o -iname "*.wmv" -o -iname "*.webm" \
                      -o -iname "*.mpeg" -o -iname "*.mpg" -o -iname "*.3gp" \) | sort | while read -r file; do
      # Print the current file being processed
      echo "Processing: $file"
    
      # Construct the output filename by inserting '.av1' before the extension
      dirname=$(dirname "$file")
      basename=$(basename "$file")
      filename="${basename%.*}"
      extension="${basename##*.}"
      output_filename="$dirname/$filename.av1.$extension"
    
      # Convert the video file
      # -map_metadata 1 preserves metadata, -c:v libaom-av1 selects AV1 codec, -crf 25 sets quality
      # -c:a libopus selects Opus audio codec, -b:a 96k sets audio bitrate to 96kbps, -ac 2 forces stereo
      ffmpeg -i "$file" -map_metadata 1 -c:v libaom-av1 -crf 25 -c:a libopus -b:a 96k -ac 2 "$output_filename"
    
      if [ $? -eq 0 ]; then
        # Preserve original file's timestamps on the new file
        touch -r "$file" "$output_filename"
    
        # Log the original file path to the log file
        echo "$file" >> "$LOG_FILE"
    
        # Construct the directory structure under MOVED_ORIGINALS_DIR to move the original file into
        relative_path=${file#./}
        moved_originals_path="$MOVED_ORIGINALS_DIR/${relative_path%/*}"
        mkdir -p "$moved_originals_path"
    
        # Move the original file
        mv "$file" "$moved_originals_path"
        echo "Moved original file to: $moved_originals_path/${basename}"
      else
        echo "Error converting $file. Skipping..." >&2
        # Optionally, remove the incomplete output file if conversion failed
        # rm -f "$output_filename"
      fi
    done
    
    echo "Conversion process completed."
    

    How to use this script:

    1. Copy the script into a text file, for example, convert_videos.sh.
    2. Give execute permissions to the script: chmod +x convert_videos.sh.
    3. Run the script from the directory where you want to start searching for video files: ./convert_videos.sh.

    Important Notes:

    • The script searches for files starting from the current directory. If you want to start from a different directory, cd into that directory before running the script, or modify the find command to specify a different starting point (replace . with your desired path).
    • This script assumes your ffmpeg version supports libaom-av1 and libopus. If your ffmpeg was compiled without these codecs, you'll need to recompile or install a different version that includes them.
    • Backup your data before running any mass file conversion and movement script, to prevent loss in case something goes wrong.
    • Depending on the number and size of your video files, this process can take a very long time and require significant disk space (for the converted files and the moved originals).

    Test the script on a small subset of files before running it on your entire collection to ensure everything works as expected. Happy converting!

  3. User

    please use svt av1 and not aom av1

  4. Assistant

    Here's the updated bash script that uses SVT-AV1 (libsvtav1) instead of AOM AV1 (libaom-av1) for video encoding. SVT-AV1 is known for its better performance and faster encoding speeds compared to AOM AV1, especially on multi-core CPUs.

    Before running this script, ensure that your ffmpeg installation is compiled with libsvtav1 support. You can check this by running:

    ffmpeg -encoders | grep libsvtav1
    

    If you don't see libsvtav1 listed, you'll need to install or recompile ffmpeg with SVT-AV1 support. Here's a quick guide for Ubuntu/Debian:

    1. Install required dependencies:
      sudo apt update
      sudo apt install -y nasm build-essential git
      
    2. Build and install SVT-AV1:
      git clone https://github.com/AOMediaCodec/SVT-AV1.git
      cd SVT-AV1
      cd Build/linux
      ./build.sh release
      cd Release
      sudo make install
      
    3. Rebuild ffmpeg with libsvtav1:
      cd ~/
      git clone https://github.com/FFmpeg/FFmpeg.git
      cd FFmpeg
      ./configure --enable-libsvtav1
      make -j$(nproc)
      sudo make install
      sudo ldconfig
      

    Verify ffmpeg now supports libsvtav1 with the command I mentioned earlier.

    Here's the updated script:

    #!/bin/bash
    
    # Directory where converted files' original paths will be logged and moved to
    TARGET_DIR="/rpool/ds1/photo"
    # Log file to store paths of converted files
    LOG_FILE="$TARGET_DIR/converted.txt"
    # Directory to move original files after conversion
    MOVED_ORIGINALS_DIR="$TARGET_DIR/converted"
    
    # Ensure the log file and moved originals directory exist
    mkdir -p "$MOVED_ORIGINALS_DIR"
    touch "$LOG_FILE"
    
    # Find video files recursively from the current directory
    # This list includes common video file extensions, add or remove as needed
    find . -type f \( -iname "*.avi" -o -iname "*.mov" -o -iname "*.mp4" -o -iname "*.m4v" \
                      -o -iname "*.mkv" -o -iname "*.flv" -o -iname "*.wmv" -o -iname "*.webm" \
                      -o -iname "*.mpeg" -o -iname "*.mpg" -o -iname "*.3gp" \) | sort | while read -r file; do
      # Print the current file being processed
      echo "Processing: $file"
    
      # Construct the output filename by inserting '.av1' before the extension
      dirname=$(dirname "$file")
      basename=$(basename "$file")
      filename="${basename%.*}"
      extension="${basename##*.}"
      output_filename="$dirname/$filename.av1.$extension"
    
      # Convert the video file using SVT-AV1
      # -c:v libsvtav1 selects SVT-AV1 codec, -crf 25 sets quality
      # Preset 4-6 is a good balance between speed and quality; adjust as needed (0=slowest/best, 12=fastest/worst)
      # -c:a libopus selects Opus audio codec, -b:a 96k sets audio bitrate to 96kbps, -ac 2 forces stereo
      ffmpeg -i "$file" -map_metadata 1 \
             -c:v libsvtav1 -crf 25 -preset 5 \
             -c:a libopus -b:a 96k -ac 2 \
             "$output_filename"
    
      if [ $? -eq 0 ]; then
        # Preserve original file's timestamps on the new file
        touch -r "$file" "$output_filename"
    
        # Log the original file path to the log file
        echo "$file" >> "$LOG_FILE"
    
        # Construct the directory structure under MOVED_ORIGINALS_DIR to move the original file into
        relative_path=${file#./}
        moved_originals_path="$MOVED_ORIGINALS_DIR/${relative_path%/*}"
        mkdir -p "$moved_originals_path"
    
        # Move the original file
        mv "$file" "$moved_originals_path"
        echo "Moved original file to: $moved_originals_path/${basename}"
      else
        echo "Error converting $file. Skipping..." >&2
        # Optionally, remove the incomplete output file if conversion failed
        # rm -f "$output_filename"
      fi
    done
    
    echo "Conversion process completed."
    

    Key changes:

    1. -c:v libsvtav1 tells ffmpeg to use the SVT-AV1 encoder.
    2. I added -preset 5, which is a reasonable middle ground between encoding speed and quality. Adjust this value based on your needs:
      • Lower presets (0-3) = Slower encoding, better quality.
      • Mid presets (4-6) = Balanced speed and quality (recommended for most users).
      • Higher presets (8-12) = Faster encoding, lower quality.

    How to use:

    1. Save the script as convert_videos.sh.
    2. Run chmod +x convert_videos.sh to make it executable.
    3. Execute it in the directory where your videos are: ./convert_videos.sh.

    Tips:

    • Monitor CPU usage with htop or top during encoding. SVT-AV1 is multithreaded, so it will use all available cores.
    • If your system overheats or slows down, reduce the preset (e.g., from 5 to 6 or 7) for faster (but slightly lower quality) encodes.
    • For 4K videos or high-bitrate sources, consider -crf 23 for visually transparent quality (at the cost of larger file sizes).
    • As always, test on a few files first to ensure compatibility and desired results.

    Enjoy the faster encoding speeds of SVT-AV1! 🚀

  5. User

    the script should stop immediately if the ffmpeg did not finish successful and again print the filename that it was working on

  6. Assistant

    Here's the updated script that:

    1. Exits immediately if ffmpeg fails (non-zero exit code).
    2. Reprints the filename it was working on when the error occurred, for clarity.
    3. Adds a bit more verbosity to help with debugging.

    Here's the modified script:

    #!/bin/bash
    
    # Directory where converted files' original paths will be logged and moved to
    TARGET_DIR="/rpool/ds1/photo"
    # Log file to store paths of converted files
    LOG_FILE="$TARGET_DIR/converted.txt"
    # Directory to move original files after conversion
    MOVED_ORIGINALS_DIR="$TARGET_DIR/converted"
    
    # Ensure the log file and moved originals directory exist
    mkdir -p "$MOVED_ORIGINALS_DIR"
    touch "$LOG_FILE"
    
    # Find video files recursively from the current directory
    # This list includes common video file extensions, add or remove as needed
    find . -type f \( -iname "*.avi" -o -iname "*.mov" -o -iname "*.mp4" -o -iname "*.m4v" \
                      -o -iname "*.mkv" -o -iname "*.flv" -o -iname "*.wmv" -o -iname "*.webm" \
                      -o -iname "*.mpeg" -o -iname "*.mpg" -o -iname "*.3gp" \) | sort | while read -r file; do
      # Print the current file being processed
      echo "===================================================="
      echo "Processing: $file"
    
      # Construct the output filename by inserting '.av1' before the extension
      dirname=$(dirname "$file")
      basename=$(basename "$file")
      filename="${basename%.*}"
      extension="${basename##*.}"
      output_filename="$dirname/$filename.av1.$extension"
    
      # Convert the video file using SVT-AV1
      # -c:v libsvtav1 selects SVT-AV1 codec, -crf 25 sets quality
      # Preset 4-6 is a good balance between speed and quality; adjust as needed (0=slowest/best, 12=fastest/worst)
      # -c:a libopus selects Opus audio codec, -b:a 96k sets audio bitrate to 96kbps, -ac 2 forces stereo
      ffmpeg -i "$file" -map_metadata 1 \
             -c:v libsvtav1 -crf 25 -preset 5 \
             -c:a libopus -b:a 96k -ac 2 \
             "$output_filename"
    
      # Check ffmpeg exit status
      if [ $? -ne 0 ]; then
        echo "===================================================="
        echo "❌ ERROR: ffmpeg FAILED to convert: $file"
        echo "❌ Conversion aborted. No further files will be processed."
        echo "===================================================="
        exit 1 # Exit the script immediately with error code 1
      fi
    
      # If ffmpeg succeeded, proceed with the rest
      echo "✅ Successfully converted: $file"
    
      # Preserve original file's timestamps on the new file
      touch -r "$file" "$output_filename"
    
      # Log the original file path to the log file
      echo "$file" >> "$LOG_FILE"
    
      # Construct the directory structure under MOVED_ORIGINALS_DIR to move the original file into
      relative_path=${file#./}
      moved_originals_path="$MOVED_ORIGINALS_DIR/${relative_path%/*}"
      mkdir -p "$moved_originals_path"
    
      # Move the original file
      mv "$file" "$moved_originals_path"
      echo "✅ Moved original file to: $moved_originals_path/${basename}"
      echo "===================================================="
    done
    
    echo "🎉 Conversion process completed successfully."
    

    What changed:

    1. Immediate Exit on Failure:

      if [ $? -ne 0 ]; then
        echo "❌ ERROR: ffmpeg FAILED to convert: $file"
        echo "❌ Conversion aborted. No further files will be processed."
        exit 1
      fi
      
      • $? checks the exit code of the last command (ffmpeg). If it's not 0, ffmpeg failed.
      • exit 1 stops the script immediately. No more files will be processed.
    2. Reprints the Failing Filename:

      • Notice the echo "❌ ERROR: ffmpeg FAILED to convert: $file" line. This ensures you know exactly which file caused the crash.
    3. Added Visual Separators (====================================================) to make the log output easier to read.

    4. Success Messages (✅ Successfully converted) for happy-path debugging.

    Now, here's what happens if ffmpeg fails:

    ====================================================
    Processing: ./path/to/problematic_video.mp4
    [many lines of ffmpeg output...]
    [ffmpeg crashes or errors out]
    ====================================================
    ❌ ERROR: ffmpeg FAILED to convert: ./path/to/problematic_video.mp4
    ❌ Conversion aborted. No further files will be processed.
    ====================================================
    

    The script stops dead in its tracks. It won't attempt to convert the next file or move anything. This forces you to fix the issue (e.g., bad input file, codec mismatch, disk full, etc.) before retrying.

    Common reasons for ffmpeg failure:

    • Corrupt input video file.
    • Unsupported codec in the input file.
    • ffmpeg compiled without libsvtav1 or libopus support.
    • Disk quota exceeded (no space left on device).
    • Input file permissions issues.

    Once you resolve the problem, re-run the script. It will skip the already converted/moved files (since they're no longer in the find list) and continue from the next unprocessed video.

    Happy (non-crashing) converting! 😊

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.