Skip to content
TheChatArchiveThe Chat Archive
GPT 5.1· openaiHugging Face Frontier Reasoning Traces

Develop a Python tool to analyze and visualize log files using matplotlib.

  1. User

    Develop a Python tool to analyze and visualize log files using matplotlib.

  2. Assistant

    Planning a log analysis tool

    The user is looking to develop a Python tool to analyze and visualize log files using matplotlib. I need to clarify the specific log file formats since that wasn’t provided. My approach will include designing the code structure and proposing a generic tool that can adapt via configuration. I think it’ll be a CLI script or small package that can parse various logs, extract useful information like timestamps and count occurrences over time, and plot them.Designing the log analysis tool

    I need to include requirements and assumptions for this Python script. The plan is to use argparse, pandas, and matplotlib. I want to create a pluggable parser with regex to read log files, parse them into a DataFrame, and aggregate statistics to produce various plots, such as events over time and counts of different log levels. Since logs vary, I’ll show an example with a generic pattern. It’s essential to allow for custom regex via configuration, but the user didn’t specifically request packaging details.Creating a log analysis script

    I’m thinking about providing a single script that accepts a log file path as an argument and includes a configurable regex at the top for users to adjust to match their specific log format. The script would parse log lines into a list of dictionaries with fields like 'timestamp', 'level', and 'message'. It should include analytic operations like counts by level and events per time bin. For visualization using Matplotlib, I could plot the count of log entries over time and counts of each log level, among other things.Considering date parsing and formatting

    I’m thinking about how to handle date parsing in the script. We could use datetime.strptime, but to keep it dependency-free, I might stick with built-in datetime and accept a format string parameter. To simplify, my base tool can work with logs that start with ISO timestamps, like "2025-11-22 13:45:11,123 - INFO - module - message." I’ll present a regex pattern to help with parsing and note that users will need to adjust the LOG_PATTERN and TIME_FORMAT as needed. It sounds like the user prefers a CLI over a GUI for this tool.Finalizing the log analysis tool

    I need to create a concise yet comprehensive response that outlines my approach and assumptions. The main component will be a script (log_analyzer.py) that includes necessary imports, a regex pattern for parsing logs, and functions for reading log files and visualizing data. I’ll use Matplotlib and might use Pandas for convenience, although I’ll clarify that it’s optional for users. I’ll also provide the command-line interface details, usage instructions, and notes on adapting the regex for different log formats. Additionally, I'll ensure to mention installing the required libraries. Below is a concrete, ready‑to‑run Python script that:

    • Parses a log file (with a configurable regex)
    • Analyzes it (counts per level, per time window, most common messages)
    • Visualizes results using matplotlib (time series, level distribution, top error messages)

    You only need matplotlib and pandas.


    1. Example log format (assumption)

    The script assumes log lines like:

    2025-11-22 13:45:11,123 - INFO - my.module - Something happened
    2025-11-22 13:45:12,456 - ERROR - my.module - Something went wrong
    

    i.e.:

    <timestamp> - <LEVEL> - <module> - <message>

    You can adjust the regex and time format for your own logs (explained below).


    2. The tool: log_analyzer.py

    #!/usr/bin/env python3
    """
    Simple log analysis and visualization tool.
    
    Usage:
        python log_analyzer.py my_app.log --freq 5Min --level ERROR --top 10
    """
    
    import re
    import argparse
    from datetime import datetime
    
    import matplotlib.pyplot as plt
    import pandas as pd
    
    # --------------------------------------------------------------------------------------
    # 1) Configure your log format here
    # --------------------------------------------------------------------------------------
    
    # Example log format:
    # 2025-11-22 13:45:11,123 - INFO - my.module - Message text...
    LOG_PATTERN = re.compile(
        r'^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) - '
        r'(?P<level>\w+) - '
        r'(?P<module>[\w\.]+) - '
        r'(?P<message>.*)$'
    )
    
    # Python datetime format for the timestamp above
    TIME_FORMAT = "%Y-%m-%d %H:%M:%S,%f"
    
    
    # --------------------------------------------------------------------------------------
    # 2) Parsing
    # --------------------------------------------------------------------------------------
    
    def parse_log_line(line: str):
        """Parse a single log line into a dict or return None if it doesn't match."""
        match = LOG_PATTERN.match(line)
        if not match:
            return None
    
        data = match.groupdict()
        # Convert timestamp string to a datetime object
        data["timestamp"] = datetime.strptime(data["timestamp"], TIME_FORMAT)
        return data
    
    
    def load_log_file(path: str) -> pd.DataFrame:
        """
        Load a log file into a pandas DataFrame with columns:
        timestamp, level, module, message
        """
        records = []
        with open(path, "r", encoding="utf-8", errors="ignore") as f:
            for raw_line in f:
                line = raw_line.rstrip("\n")
                parsed = parse_log_line(line)
                if parsed:
                    records.append(parsed)
    
        if not records:
            raise ValueError(
                f"No log lines matched the pattern. "
                f"Check LOG_PATTERN and TIME_FORMAT in the script."
            )
    
        df = pd.DataFrame(records)
        return df
    
    
    # --------------------------------------------------------------------------------------
    # 3) Analysis & plotting functions
    # --------------------------------------------------------------------------------------
    
    def plot_events_over_time(df: pd.DataFrame, freq: str = "5Min"):
        """
        Plot the number of log entries over time, aggregated by frequency (e.g. '1Min', '5Min', '1H').
        """
        ts = df.set_index("timestamp")
        counts = ts.resample(freq)["message"].count()
    
        plt.figure(figsize=(10, 4))
        counts.plot()
        plt.title(f"Log entries over time (freq={freq})")
        plt.xlabel("Time")
        plt.ylabel("Count")
        plt.grid(True, linestyle="--", alpha=0.3)
        plt.tight_layout()
    
    
    def plot_level_distribution(df: pd.DataFrame):
        """
        Plot a bar chart of log level distribution (INFO, WARNING, ERROR, etc.).
        """
        plt.figure(figsize=(6, 4))
        counts = df["level"].value_counts()
        counts.plot(kind="bar", color="steelblue")
        plt.title("Log level distribution")
        plt.xlabel("Level")
        plt.ylabel("Count")
        plt.grid(axis="y", linestyle="--", alpha=0.3)
        plt.tight_layout()
    
    
    def plot_top_messages(df: pd.DataFrame, level: str = "ERROR", n: int = 10):
        """
        Show the top N most frequent messages for a given level (e.g. ERROR).
        """
        subset = df[df["level"].str.upper() == level.upper()]
        if subset.empty:
            print(f"No messages with level '{level}' found.")
            return
    
        counts = subset["message"].value_counts().head(n)
    
        plt.figure(figsize=(10, 4))
        counts.sort_values().plot(kind="barh", color="indianred")
        plt.title(f"Top {n} '{level.upper()}' messages")
        plt.xlabel("Count")
        plt.ylabel("Message")
        plt.tight_layout()
    
    
    # --------------------------------------------------------------------------------------
    # 4) CLI
    # --------------------------------------------------------------------------------------
    
    def main():
        parser = argparse.ArgumentParser(
            description="Analyze and visualize log files using matplotlib."
        )
        parser.add_argument("logfile", help="Path to the log file to analyze.")
        parser.add_argument(
            "--freq",
            default="5Min",
            help="Resample frequency for time plot (e.g. '1Min', '5Min', '1H').",
        )
        parser.add_argument(
            "--level",
            default="ERROR",
            help="Log level for the 'top messages' plot (default: ERROR).",
        )
        parser.add_argument(
            "--top",
            type=int,
            default=10,
            help="Number of top messages to show for the chosen level.",
        )
        parser.add_argument(
            "--save-prefix",
            help=(
                "If provided, save plots as PNG files with this prefix instead of showing "
                "interactive windows. Example: --save-prefix report_"
            ),
        )
    
        args = parser.parse_args()
    
        # Load logs
        df = load_log_file(args.logfile)
    
        # Basic text summary
        print(f"Loaded {len(df)} log entries.")
        print("\nEntries by level:")
        print(df["level"].value_counts())
        print("\nTime span:")
        print(f"From {df['timestamp'].min()} to {df['timestamp'].max()}")
    
        # 1) Time series plot
        plot_events_over_time(df, freq=args.freq)
        if args.save_prefix:
            plt.savefig(f"{args.save_prefix}time.png", dpi=150)
            plt.close()
        else:
            plt.show()
    
        # 2) Level distribution
        plot_level_distribution(df)
        if args.save_prefix:
            plt.savefig(f"{args.save_prefix}levels.png", dpi=150)
            plt.close()
        else:
            plt.show()
    
        # 3) Top N messages for a given level
        plot_top_messages(df, level=args.level, n=args.top)
        if args.save_prefix:
            plt.savefig(f"{args.save_prefix}top_{args.level.upper()}.png", dpi=150)
            plt.close()
        else:
            plt.show()
    
    
    if __name__ == "__main__":
        main()
    

    3. Installation and usage

    1. Install dependencies:
    pip install matplotlib pandas
    
    1. Save the script as log_analyzer.py.

    2. Run it on a log file:

    python log_analyzer.py my_app.log
    

    Optional arguments:

    # Different time aggregation
    python log_analyzer.py my_app.log --freq 1Min
    
    # Top 20 WARNING messages, save plots to PNG files
    python log_analyzer.py my_app.log --level WARNING --top 20 --save-prefix analysis_
    

    4. Adapting to your log format

    If your logs look different, adjust:

    1. LOG_PATTERN – the regex with named groups: timestamp, level, and optionally module, message.
    2. TIME_FORMAT – the datetime.strptime format for the timestamp group.

    Examples:

    • If timestamp is 2025-11-22T13:45:11:

      TIME_FORMAT = "%Y-%m-%dT%H:%M:%S"
      
    • If your line is:

      [2025-11-22 13:45:11] ERROR my.module: Something broke
      

      you might use:

      LOG_PATTERN = re.compile(
          r'^\[(?P<timestamp>[^]]+)\]\s+'
          r'(?P<level>\w+)\s+'
          r'(?P<module>[\w\.]+):\s+'
          r'(?P<message>.*)$'
      )
      TIME_FORMAT = "%Y-%m-%d %H:%M:%S"
      

    If you show me a few representative log lines, I can tailor the regex and time format for your exact case.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.