Develop a Python tool to analyze and visualize log files using matplotlib.
- User
Develop a Python tool to analyze and visualize log files using matplotlib.
- Assistant
Planning a log analysis tool
The user is looking to develop a Python tool to analyze and visualize log files using matplotlib. I need to clarify the specific log file formats since that wasn’t provided. My approach will include designing the code structure and proposing a generic tool that can adapt via configuration. I think it’ll be a CLI script or small package that can parse various logs, extract useful information like timestamps and count occurrences over time, and plot them.Designing the log analysis tool
I need to include requirements and assumptions for this Python script. The plan is to use argparse, pandas, and matplotlib. I want to create a pluggable parser with regex to read log files, parse them into a DataFrame, and aggregate statistics to produce various plots, such as events over time and counts of different log levels. Since logs vary, I’ll show an example with a generic pattern. It’s essential to allow for custom regex via configuration, but the user didn’t specifically request packaging details.Creating a log analysis script
I’m thinking about providing a single script that accepts a log file path as an argument and includes a configurable regex at the top for users to adjust to match their specific log format. The script would parse log lines into a list of dictionaries with fields like 'timestamp', 'level', and 'message'. It should include analytic operations like counts by level and events per time bin. For visualization using Matplotlib, I could plot the count of log entries over time and counts of each log level, among other things.Considering date parsing and formatting
I’m thinking about how to handle date parsing in the script. We could use
datetime.strptime, but to keep it dependency-free, I might stick with built-indatetimeand accept a format string parameter. To simplify, my base tool can work with logs that start with ISO timestamps, like "2025-11-22 13:45:11,123 - INFO - module - message." I’ll present a regex pattern to help with parsing and note that users will need to adjust theLOG_PATTERNandTIME_FORMATas needed. It sounds like the user prefers a CLI over a GUI for this tool.Finalizing the log analysis toolI need to create a concise yet comprehensive response that outlines my approach and assumptions. The main component will be a script (
log_analyzer.py) that includes necessary imports, a regex pattern for parsing logs, and functions for reading log files and visualizing data. I’ll use Matplotlib and might use Pandas for convenience, although I’ll clarify that it’s optional for users. I’ll also provide the command-line interface details, usage instructions, and notes on adapting the regex for different log formats. Additionally, I'll ensure to mention installing the required libraries. Below is a concrete, ready‑to‑run Python script that:- Parses a log file (with a configurable regex)
- Analyzes it (counts per level, per time window, most common messages)
- Visualizes results using matplotlib (time series, level distribution, top error messages)
You only need
matplotlibandpandas.
1. Example log format (assumption)
The script assumes log lines like:
2025-11-22 13:45:11,123 - INFO - my.module - Something happened 2025-11-22 13:45:12,456 - ERROR - my.module - Something went wrongi.e.:
<timestamp> - <LEVEL> - <module> - <message>You can adjust the regex and time format for your own logs (explained below).
2. The tool:
log_analyzer.py#!/usr/bin/env python3 """ Simple log analysis and visualization tool. Usage: python log_analyzer.py my_app.log --freq 5Min --level ERROR --top 10 """ import re import argparse from datetime import datetime import matplotlib.pyplot as plt import pandas as pd # -------------------------------------------------------------------------------------- # 1) Configure your log format here # -------------------------------------------------------------------------------------- # Example log format: # 2025-11-22 13:45:11,123 - INFO - my.module - Message text... LOG_PATTERN = re.compile( r'^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) - ' r'(?P<level>\w+) - ' r'(?P<module>[\w\.]+) - ' r'(?P<message>.*)$' ) # Python datetime format for the timestamp above TIME_FORMAT = "%Y-%m-%d %H:%M:%S,%f" # -------------------------------------------------------------------------------------- # 2) Parsing # -------------------------------------------------------------------------------------- def parse_log_line(line: str): """Parse a single log line into a dict or return None if it doesn't match.""" match = LOG_PATTERN.match(line) if not match: return None data = match.groupdict() # Convert timestamp string to a datetime object data["timestamp"] = datetime.strptime(data["timestamp"], TIME_FORMAT) return data def load_log_file(path: str) -> pd.DataFrame: """ Load a log file into a pandas DataFrame with columns: timestamp, level, module, message """ records = [] with open(path, "r", encoding="utf-8", errors="ignore") as f: for raw_line in f: line = raw_line.rstrip("\n") parsed = parse_log_line(line) if parsed: records.append(parsed) if not records: raise ValueError( f"No log lines matched the pattern. " f"Check LOG_PATTERN and TIME_FORMAT in the script." ) df = pd.DataFrame(records) return df # -------------------------------------------------------------------------------------- # 3) Analysis & plotting functions # -------------------------------------------------------------------------------------- def plot_events_over_time(df: pd.DataFrame, freq: str = "5Min"): """ Plot the number of log entries over time, aggregated by frequency (e.g. '1Min', '5Min', '1H'). """ ts = df.set_index("timestamp") counts = ts.resample(freq)["message"].count() plt.figure(figsize=(10, 4)) counts.plot() plt.title(f"Log entries over time (freq={freq})") plt.xlabel("Time") plt.ylabel("Count") plt.grid(True, linestyle="--", alpha=0.3) plt.tight_layout() def plot_level_distribution(df: pd.DataFrame): """ Plot a bar chart of log level distribution (INFO, WARNING, ERROR, etc.). """ plt.figure(figsize=(6, 4)) counts = df["level"].value_counts() counts.plot(kind="bar", color="steelblue") plt.title("Log level distribution") plt.xlabel("Level") plt.ylabel("Count") plt.grid(axis="y", linestyle="--", alpha=0.3) plt.tight_layout() def plot_top_messages(df: pd.DataFrame, level: str = "ERROR", n: int = 10): """ Show the top N most frequent messages for a given level (e.g. ERROR). """ subset = df[df["level"].str.upper() == level.upper()] if subset.empty: print(f"No messages with level '{level}' found.") return counts = subset["message"].value_counts().head(n) plt.figure(figsize=(10, 4)) counts.sort_values().plot(kind="barh", color="indianred") plt.title(f"Top {n} '{level.upper()}' messages") plt.xlabel("Count") plt.ylabel("Message") plt.tight_layout() # -------------------------------------------------------------------------------------- # 4) CLI # -------------------------------------------------------------------------------------- def main(): parser = argparse.ArgumentParser( description="Analyze and visualize log files using matplotlib." ) parser.add_argument("logfile", help="Path to the log file to analyze.") parser.add_argument( "--freq", default="5Min", help="Resample frequency for time plot (e.g. '1Min', '5Min', '1H').", ) parser.add_argument( "--level", default="ERROR", help="Log level for the 'top messages' plot (default: ERROR).", ) parser.add_argument( "--top", type=int, default=10, help="Number of top messages to show for the chosen level.", ) parser.add_argument( "--save-prefix", help=( "If provided, save plots as PNG files with this prefix instead of showing " "interactive windows. Example: --save-prefix report_" ), ) args = parser.parse_args() # Load logs df = load_log_file(args.logfile) # Basic text summary print(f"Loaded {len(df)} log entries.") print("\nEntries by level:") print(df["level"].value_counts()) print("\nTime span:") print(f"From {df['timestamp'].min()} to {df['timestamp'].max()}") # 1) Time series plot plot_events_over_time(df, freq=args.freq) if args.save_prefix: plt.savefig(f"{args.save_prefix}time.png", dpi=150) plt.close() else: plt.show() # 2) Level distribution plot_level_distribution(df) if args.save_prefix: plt.savefig(f"{args.save_prefix}levels.png", dpi=150) plt.close() else: plt.show() # 3) Top N messages for a given level plot_top_messages(df, level=args.level, n=args.top) if args.save_prefix: plt.savefig(f"{args.save_prefix}top_{args.level.upper()}.png", dpi=150) plt.close() else: plt.show() if __name__ == "__main__": main()
3. Installation and usage
- Install dependencies:
pip install matplotlib pandas-
Save the script as
log_analyzer.py. -
Run it on a log file:
python log_analyzer.py my_app.logOptional arguments:
# Different time aggregation python log_analyzer.py my_app.log --freq 1Min # Top 20 WARNING messages, save plots to PNG files python log_analyzer.py my_app.log --level WARNING --top 20 --save-prefix analysis_
4. Adapting to your log format
If your logs look different, adjust:
LOG_PATTERN– the regex with named groups:timestamp,level, and optionallymodule,message.TIME_FORMAT– thedatetime.strptimeformat for thetimestampgroup.
Examples:
-
If timestamp is
2025-11-22T13:45:11:TIME_FORMAT = "%Y-%m-%dT%H:%M:%S" -
If your line is:
[2025-11-22 13:45:11] ERROR my.module: Something brokeyou might use:
LOG_PATTERN = re.compile( r'^\[(?P<timestamp>[^]]+)\]\s+' r'(?P<level>\w+)\s+' r'(?P<module>[\w\.]+):\s+' r'(?P<message>.*)$' ) TIME_FORMAT = "%Y-%m-%d %H:%M:%S"
If you show me a few representative log lines, I can tailor the regex and time format for your exact case.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.