Develop a Python tool to analyze a 50GB log file using mmap so it doesn't load into RAM.…
- User
Develop a Python tool to analyze a 50GB log file using mmap so it doesn't load into RAM. Parse lines with a compiled regex and stream aggregate counts like error rate per minute to stdout.
- Assistant
Providing memory-mapped log analyzer code
I'm delivering a complete Python script using mmap to efficiently parse a large log file by line, aggregate error counts per minute, and print CSV output with error counts and rates, using a default bytes regex that captures timestamps and levels adjustable via CLI arguments. Below is a simple, self‑contained Python tool that:
- Uses
mmapso the 50 GB file is not read into RAM. - Compiles a regex once and applies it to each line.
- Aggregates totals/errors per minute and prints them to stdout.
By default it expects log lines of the form:
2023-08-01 14:23:45,123 ERROR Something happenedwhere the timestamp is
YYYY-MM-DD HH:MM:SS,msand the level is the next token. You can adapt the regex to your format via a command‑line argument.#!/usr/bin/env python3 import argparse import mmap import re from collections import defaultdict # Default regex: adjust to match your log format. # Captures 'timestamp' and 'level' groups. DEFAULT_PATTERN = ( rb'^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}),\d+\s+' rb'(?P<level>[A-Z]+)' ) def parse_log(file_path, pattern_bytes, error_level=b'ERROR'): """ Stream through a log file via mmap, aggregate counts per minute. Yields tuples: (minute_str, total_count, error_count). """ regex = re.compile(pattern_bytes) counts = defaultdict(lambda: [0, 0]) # minute -> [total, errors] with open(file_path, 'rb') as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm: for line in iter(mm.readline, b''): m = regex.search(line) if not m: continue ts = m.group('timestamp') # bytes like b'2023-08-01 14:23:45' level = m.group('level') # Extract the minute portion: b'YYYY-MM-DD HH:MM' minute = ts[:16] counts[minute][0] += 1 # total if level == error_level: counts[minute][1] += 1 # errors # Yield sorted by minute; assumes lexicographical order matches chronological. for minute in sorted(counts.keys()): total, errors = counts[minute] yield minute.decode('ascii'), total, errors def main(): ap = argparse.ArgumentParser(description="Stream error counts per minute from a large log file using mmap.") ap.add_argument("logfile", help="Path to the log file") ap.add_argument("--pattern", help="Regex pattern in bytes (Python-style, e.g. b'^...')", default=None) ap.add_argument("--error-level", help="Log level to count as error (default: ERROR)", default="ERROR") args = ap.parse_args() pattern_bytes = eval(args.pattern) if args.pattern else DEFAULT_PATTERN error_level = args.error_level.encode('ascii') print("minute,total,error_count,error_rate") for minute, total, errors in parse_log(args.logfile, pattern_bytes, error_level): rate = errors / total if total else 0.0 print(f"{minute},{total},{errors},{rate:.6f}") if __name__ == "__main__": main()Usage
# Default pattern/levels python3 log_analyzer.py /path/to/huge.log # Custom pattern (must capture groups named 'timestamp' and 'level') python3 log_analyzer.py /path/to/huge.log \ --pattern="rb'^(?P<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2})\s+\[(?P<level>[A-Z]+)\]'"The script prints CSV to stdout with
minute,total,error_count,error_rate. Adjust the regex to match your timestamp/level layout; the key is to exposetimestamp(at least up to minute precision) andlevelgroups. The use ofmmapand a compiled regex allows you to process very large files without loading them into RAM. - Uses
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.