Create a Python CLI tool to convert text files to different encodings.
- User
Create a Python CLI tool to convert text files to different encodings.
- Assistant
Considering encoding errors
I need to open the input file in binary to prevent any decoding issues due to the operating system's defaults. I’ve got a function here that detects the encoding using a library. If the user doesn’t specify the encoding, we can attempt detection first; if that fails, I’ll default to 'utf-8' with a strict error handling or replace options. I also think about letting users do a
--dry-runfor previews, though it doesn't seem necessary right now since they haven't requested advanced features.Creating a conversion scriptI'm working on a script to convert text files between character encodings. First, I’ll use
argparsefor command-line argument parsing. The functiondetect_encodingchecks for encoding using thecharset-normalizerorchardetif available. For conversion, I read bytes from the input file, decode using the source encoding, and then encode it in the destination format. I also need to handle various error options. I'll ensure the input path exists and check that the output path doesn't, unless the user opts to overwrite.Handling source encodingI need to manage the source encoding, starting with detecting it if the user hasn't specified it. If I can't auto-detect the encoding, I’ll prompt the user to provide it. If the
--show-detectedoption is used, I’ll print the detected encoding and exit. After that, I’ll call the conversion function. It might be useful to add packaging instructions for installation to make things clearer, but I want to keep it simple since the user focused on the CLI tool aspect. Supporting in-place modifications could complicate things unnecessarily, so I'll stick to the output path.Managing file reading methodsUsing
read_bytesis fine for smaller files, but it can lead to high memory usage with larger files. For those, streaming line by line is a better approach, although decoding byte-by-byte can be tricky. We can decode line by line in text mode, making sure to specify the encoding. To avoid reliance on the OS's default encoding, I’ll open the file with the specified source encoding. However, if we detect it, we still process large files by reading them incrementally. Overall, opening with 'r' and the source encoding keeps things simple and efficient.Implementing streaming conversionFor the conversion, I’ll set up a streaming process that opens the input file with specified source encoding and the output file with the target encoding. I’ll read the input in chunks, which helps with memory management. It’s vital to handle newlines correctly: I want to preserve the newline characters as they are in the text, but I also need to manage them properly in Python. While reading with
newline=''helps maintain consistency, we need to be cautious about large files in detection. I plan to limit detection to 1MB for efficiency. Now, let’s create the final script! Below is a self‑contained Python CLI tool that converts text files between encodings.Save it as
textenc.py(or any name you like) and make it executable.#!/usr/bin/env python3 """ textenc.py – Convert text files between character encodings. Basic usage: python textenc.py input.txt output.txt -f utf-8 -t iso-8859-1 Optional automatic detection (if charset-normalizer or chardet is installed): python textenc.py input.txt output.txt -t utf-8 """ import argparse import sys from pathlib import Path from typing import Optional def detect_encoding(path: Path) -> Optional[str]: """ Try to detect the encoding of a file using charset-normalizer or chardet. Returns the encoding name, or None if detection is unavailable or fails. """ # Read at most 1 MB for detection with path.open("rb") as f: raw = f.read(1024 * 1024) # Try charset-normalizer (preferred, pure Python) try: from charset_normalizer import from_bytes # type: ignore except ImportError: pass else: result = from_bytes(raw).best() if result is not None and result.encoding is not None: return result.encoding # Fallback to chardet if available try: import chardet # type: ignore except ImportError: return None res = chardet.detect(raw) enc = res.get("encoding") return enc or None def convert_file( src_path: Path, dst_path: Path, src_encoding: str, dst_encoding: str, errors: str = "strict", ) -> None: """ Convert a text file from src_encoding to dst_encoding. Uses streaming I/O so it works with large files. """ # newline='' preserves existing newline characters with src_path.open( "r", encoding=src_encoding, errors=errors, newline="" ) as fin, dst_path.open( "w", encoding=dst_encoding, errors=errors, newline="" ) as fout: for chunk in iter(lambda: fin.read(8192), ""): fout.write(chunk) def parse_args(argv=None): parser = argparse.ArgumentParser( description="Convert a text file between encodings." ) parser.add_argument( "input", type=Path, help="Path to the input text file.", ) parser.add_argument( "output", type=Path, help="Path to the output text file.", ) parser.add_argument( "-f", "--from-encoding", "--from", dest="src_encoding", help=( "Source encoding (e.g. utf-8, cp1252). " "If omitted, the tool will try to auto-detect it." ), ) parser.add_argument( "-t", "--to-encoding", "--to", dest="dst_encoding", required=True, help="Target encoding (e.g. utf-8, iso-8859-1).", ) parser.add_argument( "-e", "--errors", choices=["strict", "ignore", "replace", "backslashreplace", "namereplace"], default="strict", help=( "Error handling strategy for encoding/decoding problems " "(default: strict)." ), ) parser.add_argument( "--overwrite", action="store_true", help="Allow overwriting the output file if it already exists.", ) parser.add_argument( "--show-detected", action="store_true", help=( "Detect and print the source encoding, then exit. " "Respects --from-encoding if provided." ), ) return parser.parse_args(argv) def main(argv=None) -> int: args = parse_args(argv) src_path: Path = args.input dst_path: Path = args.output if not src_path.is_file(): print(f"Error: input file '{src_path}' does not exist or is not a file.", file=sys.stderr) return 1 if dst_path.exists() and not args.overwrite: print( f"Error: output file '{dst_path}' already exists. " "Use --overwrite to replace it.", file=sys.stderr, ) return 1 # Determine source encoding if args.src_encoding: src_encoding = args.src_encoding else: src_encoding = detect_encoding(src_path) or "" if not src_encoding: print( "Could not auto-detect source encoding. " "Please specify it with --from-encoding.", file=sys.stderr, ) return 1 if args.show_detected: print(src_encoding) return 0 # Perform conversion try: convert_file( src_path=src_path, dst_path=dst_path, src_encoding=src_encoding, dst_encoding=args.dst_encoding, errors=args.errors, ) except LookupError as e: # Invalid codec name print(f"Encoding error: {e}", file=sys.stderr) return 1 except UnicodeError as e: print(f"Unicode conversion error: {e}", file=sys.stderr) if args.errors != "strict": print( "Note: you can try a different error strategy with --errors " "(e.g. 'replace' or 'ignore').", file=sys.stderr, ) return 1 return 0 if __name__ == "__main__": raise SystemExit(main())
Usage examples
- Convert UTF‑8 to Windows‑1252:
python textenc.py input_utf8.txt output_cp1252.txt -f utf-8 -t cp1252- Convert unknown encoding to UTF‑8, using auto‑detection:
python textenc.py legacy.txt utf8.txt -t utf-8(Install a detector first, e.g.
pip install charset-normalizer.)- Convert and replace invalid characters instead of failing:
python textenc.py bad_file.txt fixed.txt -f latin-1 -t utf-8 -e replace- Just detect and print the source encoding:
python textenc.py file.txt out.txt -t utf-8 --show-detected # prints encoding and exits, doesn't write out.txt- Overwrite an existing output file:
python textenc.py in.txt out.txt -f utf-8 -t iso-8859-1 --overwrite
If you want, I can also show how to package this as an installable command (
textenc) usingpyproject.tomlorsetup.py.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.