Skip to content
TheChatArchiveThe Chat Archive
GPT 5.1· openaiHugging Face Frontier Reasoning Traces

Create a Python script to generate synthetic datasets for testing.

  1. User

    Create a Python script to generate synthetic datasets for testing.

  2. Assistant

    Designing a data generation script

    I'm planning to create a script that takes in command line arguments for dataset types like classification, regression, clustering, and timeseries, among others. I'll use argparse for parsing these arguments, including parameters such as sample size, features, and output format.

    I want to include functions for generating different types of data, focusing on simplifying the implementation. I think maintaining a moderate level of explanation will help, especially for users looking to understand the capabilities without overcomplicating things. Keeping it manageable will be key!Planning synthetic data generator

    I’m thinking about creating a synthetic data generator script. While some users may not have certain libraries, I assume they might need to test machine learning code, which suggests they'll need sklearn. I’ll include info on dependencies at the top.

    For timeseries data, I’ll include parameters like start date and frequency but treat samples as timesteps. It seems simpler to use just one feature called 'value' for timeseries. Since they want synthetic datasets for testing, I’ll ensure classification, regression, and clustering options are available.

    I’ll design the code and remember to include help text and docstrings for clarity.Designing the synthetic dataset script

    I'm considering how to keep explanations brief yet informative for the synthetic dataset generator. While we could include a feature for specifying correlation structures, the user didn't ask for an evaluation, just a script.

    So, I’ll provide a clear script along with a short guide on how to run it, mentioning dependencies but keeping it simple. I’ll start the script with a shebang and some usage examples, then import necessary libraries.

    As for the functions, I'll make them generic to accept parameters, while weighing the idea of using a dataclass for configuration—though that might be more than necessary. Let's get outlining!Creating timeseries and data generation functions

    I'm designing the generate_time_series function. It will set up a random number generator based on a seed, create a date index for the specified number of samples, and generate different features. I'll include a trend, seasonal variation, and a noise component for each feature.

    For clustering, I'll utilize make_blobs, while considering the incorporation of mixed tabular data types like numeric, categorical, boolean, and date features. However, I’m wondering if that's too complex. Balancing simplicity with the need for testing various pipelines is key here!Building the dataset generation script

    I’m finalizing the generate_mixed_tabular function, allowing for a mix of numeric, categorical, and datetime features. I think it’s essential to ensure that the generated data meets user needs without being overly complex.

    Next, I’ll set up the command-line interface (CLI) options using argparse, providing default values and clear descriptions for each parameter. I realize different arguments apply to specific dataset types, so I’ll mention that in the help text.

    I need to implement various dataset generation functions depending on the specified type and ensure the script handles output formats properly. Including some basic logging for saved files will help users understand what just happened. I'll also finalize help descriptions, example usage, and neatly format everything for clarity!Finalizing type hints and imports

    I’m considering whether to include from __future__ import annotations. It’s optional, but I think I'll skip it for now. However, adding type hints could enhance the code readability and help during development, even though they aren’t strictly required.

    I’ll ensure the final script has clear type hints for functions to convey what types of data they take and return. That should be helpful for users and anyone reviewing the code later. Now, I'll compose the final answer based on these considerations! ```python #!/usr/bin/env python3 """ synthetic_data_generator.py

    Generate synthetic datasets for testing pipelines, models, or data tooling.

    Supports:

    • classification (tabular, with target column)
    • regression (tabular, with target column)
    • clustering (tabular, with cluster labels)
    • timeseries (time-indexed numeric features)
    • mixed (numeric + categorical + dates + target)

    Dependencies:

    • numpy
    • pandas
    • scikit-learn (for classification/regression/clustering)
    • pyarrow or fastparquet (optional, for Parquet output)

    Example usage: python synthetic_data_generator.py -t classification -n 5000 -f 30 -o data.csv python synthetic_data_generator.py -t timeseries -n 1000 -f 3 --freq H -o ts.parquet python synthetic_data_generator.py -t mixed -n 2000 -o mixed.csv """

    import argparse import os from typing import Optional

    import numpy as np import pandas as pd from sklearn.datasets import make_blobs, make_classification, make_regression

    ---------------------------------------------------------------------------

    Dataset generators

    ---------------------------------------------------------------------------

    def generate_classification_dataset( n_samples: int, n_features: int, n_informative: Optional[int] = None, n_classes: int = 2, class_sep: float = 1.0, noise: float = 0.01, random_state: Optional[int] = None, ) -> pd.DataFrame: """ Binary or multi-class classification dataset.

    noise is used as flip_y (fraction of labels to randomly flip).
    """
    if n_informative is None:
        n_informative = min(n_features, 5)
    
    X, y = make_classification(
        n_samples=n_samples,
        n_features=n_features,
        n_informative=n_informative,
        n_redundant=0,
        n_repeated=0,
        n_classes=n_classes,
        class_sep=class_sep,
        flip_y=noise,
        random_state=random_state,
    )
    
    columns = [f"feature_{i}" for i in range(n_features)]
    df = pd.DataFrame(X, columns=columns)
    df["target"] = y
    return df
    

    def generate_regression_dataset( n_samples: int, n_features: int, n_informative: Optional[int] = None, noise: float = 1.0, random_state: Optional[int] = None, ) -> pd.DataFrame: """ Regression dataset with a continuous target. """ if n_informative is None: n_informative = min(n_features, 10)

    X, y = make_regression(
        n_samples=n_samples,
        n_features=n_features,
        n_informative=n_informative,
        noise=noise,
        random_state=random_state,
    )
    
    columns = [f"feature_{i}" for i in range(n_features)]
    df = pd.DataFrame(X, columns=columns)
    df["target"] = y
    return df
    

    def generate_clustering_dataset( n_samples: int, n_features: int, n_clusters: int = 3, cluster_std: float = 1.0, random_state: Optional[int] = None, ) -> pd.DataFrame: """ Clustering dataset with cluster labels. """ X, y = make_blobs( n_samples=n_samples, n_features=n_features, centers=n_clusters, cluster_std=cluster_std, random_state=random_state, )

    columns = [f"feature_{i}" for i in range(n_features)]
    df = pd.DataFrame(X, columns=columns)
    df["cluster"] = y
    return df
    

    def generate_time_series_dataset( n_samples: int, n_features: int, start_date: str = "2020-01-01", freq: str = "D", noise: float = 0.1, random_state: Optional[int] = None, ) -> pd.DataFrame: """ Simple multi-variate time series with trend + seasonality + noise. Index is a DateTimeIndex named 'timestamp'. """ rng = np.random.RandomState(random_state) index = pd.date_range(start=start_date, periods=n_samples, freq=freq)

    data = {}
    base_freqs = np.linspace(0.5, 2.5, n_features)  # different seasonal frequencies
    for i in range(n_features):
        trend = rng.uniform(-0.05, 0.05) * np.arange(n_samples)
        phase = rng.rand() * 2 * np.pi
        seasonal = np.sin(
            np.linspace(0, base_freqs[i] * 2 * np.pi, n_samples) + phase
        )
        noise_component = rng.normal(scale=noise, size=n_samples)
        data[f"value_{i}"] = trend + seasonal + noise_component
    
    df = pd.DataFrame(data, index=index)
    df.index.name = "timestamp"
    return df
    

    def generate_mixed_tabular_dataset( n_samples: int, n_numeric: int = 5, n_categorical: int = 3, n_datetime: int = 1, random_state: Optional[int] = None, ) -> pd.DataFrame: """ Mixed-type tabular dataset: numeric, categorical, datetime, and a binary target. """ rng = np.random.RandomState(random_state) df = pd.DataFrame()

    # Numeric columns
    for i in range(n_numeric):
        df[f"num_{i}"] = rng.normal(loc=i, scale=1.0 + 0.2 * i, size=n_samples)
    
    # Categorical columns
    for j in range(n_categorical):
        n_categories = rng.randint(2, 6)
        categories = [f"cat{j}_level{k}" for k in range(n_categories)]
        df[f"cat_{j}"] = rng.choice(categories, size=n_samples)
    
    # Datetime columns
    for k in range(n_datetime):
        start = np.datetime64("2020-01-01")
        end = np.datetime64("2020-12-31")
        delta_days = int((end - start).astype(int))
        random_days = rng.randint(0, delta_days, size=n_samples)
        dates = start + random_days.astype("timedelta64[D]")
        df[f"date_{k}"] = dates
    
    # Simple binary target derived from numeric features
    if n_numeric > 0:
        numeric_sum = df.filter(like="num_").sum(axis=1)
        threshold = numeric_sum.median()
        target = (numeric_sum + rng.normal(scale=0.5, size=n_samples)) > threshold
        df["target"] = target.astype(int)
    else:
        df["target"] = rng.randint(0, 2, size=n_samples)
    
    return df
    

    ---------------------------------------------------------------------------

    I/O helpers

    ---------------------------------------------------------------------------

    def save_dataset(df: pd.DataFrame, path: str, file_format: str = "csv") -> None: """ Save DataFrame as CSV or Parquet. """ path = os.path.abspath(path) directory = os.path.dirname(path) if directory and not os.path.exists(directory): os.makedirs(directory, exist_ok=True)

    file_format = file_format.lower()
    if file_format == "csv":
        df.to_csv(path, index=False)
    elif file_format == "parquet":
        df.to_parquet(path, index=False)
    else:
        raise ValueError(f"Unsupported file format: {file_format!r}")
    
    print(f"Saved dataset with shape {df.shape} to {path}")
    

    ---------------------------------------------------------------------------

    CLI

    ---------------------------------------------------------------------------

    def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( description="Generate synthetic datasets for testing.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, )

    parser.add_argument(
        "-t",
        "--type",
        choices=["classification", "regression", "clustering", "timeseries", "mixed"],
        default="classification",
        help="Type of dataset to generate.",
    )
    parser.add_argument(
        "-n",
        "--n-samples",
        type=int,
        default=1000,
        help="Number of samples/rows (or time steps).",
    )
    parser.add_argument(
        "-f",
        "--n-features",
        type=int,
        default=20,
        help="Number of feature columns (where applicable).",
    )
    parser.add_argument(
        "--n-informative",
        type=int,
        default=None,
        help="Number of informative features (classification/regression).",
    )
    parser.add_argument(
        "--n-classes",
        type=int,
        default=2,
        help="Number of classes (classification).",
    )
    parser.add_argument(
        "--n-clusters",
        type=int,
        default=3,
        help="Number of clusters (clustering).",
    )
    parser.add_argument(
        "--noise",
        type=float,
        default=0.1,
        help=(
            "Noise level: label noise for classification, target noise for "
            "regression, cluster spread for clustering, std-dev for timeseries."
        ),
    )
    parser.add_argument(
        "--class-sep",
        type=float,
        default=1.0,
        help="Class separation factor (classification only).",
    )
    parser.add_argument(
        "--random-state",
        type=int,
        default=42,
        help="Random seed for reproducibility.",
    )
    parser.add_argument(
        "-o",
        "--output",
        default="synthetic.csv",
        help="Output file path (.csv or .parquet).",
    )
    parser.add_argument(
        "--format",
        choices=["csv", "parquet"],
        default=None,
        help="Output format (if omitted, inferred from file extension).",
    )
    parser.add_argument(
        "--start-date",
        default="2020-01-01",
        help="Start date for time series index (YYYY-MM-DD).",
    )
    parser.add_argument(
        "--freq",
        default="D",
        help="Pandas frequency string for time series (e.g., D, H, 15T).",
    )
    parser.add_argument(
        "--mixed-n-numeric",
        type=int,
        default=5,
        help="Number of numeric columns for mixed dataset.",
    )
    parser.add_argument(
        "--mixed-n-categorical",
        type=int,
        default=3,
        help="Number of categorical columns for mixed dataset.",
    )
    parser.add_argument(
        "--mixed-n-datetime",
        type=int,
        default=1,
        help="Number of datetime columns for mixed dataset.",
    )
    
    return parser.parse_args()
    

    def infer_format_from_path(path: str) -> str: ext = os.path.splitext(path)[1].lower() if ext in ("", ".csv"): return "csv" if ext in (".parquet", ".pq"): return "parquet" raise ValueError( f"Cannot infer format from extension {ext!r}. " "Use --format csv|parquet explicitly." )

    def main() -> None: args = parse_args()

    if args.type == "classification":
        df = generate_classification_dataset(
            n_samples=args.n_samples,
            n_features=args.n_features,
            n_informative=args.n_informative,
            n_classes=args.n_classes,
            class_sep=args.class_sep,
            noise=args.noise,
            random_state=args.random_state,
        )
    elif args.type == "regression":
        df = generate_regression_dataset(
            n_samples=args.n_samples,
            n_features=args.n_features,
            n_informative=args.n_informative,
            noise=args.noise,
            random_state=args.random_state,
        )
    elif args.type == "clustering":
        df = generate_clustering_dataset(
            n_samples=args.n_samples,
            n_features=args.n_features,
            n_clusters=args.n_clusters,
            cluster_std=max(args.noise, 0.01),
            random_state=args.random_state,
        )
    elif args.type == "timeseries":
        df = generate_time_series_dataset(
            n_samples=args.n_samples,
            n_features=args.n_features,
            start_date=args.start_date,
            freq=args.freq,
            noise=args.noise,
            random_state=args.random_state,
        )
    elif args.type == "mixed":
        df = generate_mixed_tabular_dataset(
            n_samples=args.n_samples,
            n_numeric=args.mixed_n_numeric,
            n_categorical=args.mixed_n_categorical,
            n_datetime=args.mixed_n_datetime,
            random_state=args.random_state,
        )
    else:
        raise ValueError(f"Unsupported dataset type: {args.type}")
    
    file_format = args.format or infer_format_from_path(args.output)
    save_dataset(df, args.output, file_format=file_format)
    

    if name == "main": main()

    
    Basic usage summary:
    
    - Classification CSV:
      ```bash
      python synthetic_data_generator.py -t classification -n 2000 -f 10 -o cls.csv
    
    • Regression Parquet:

      python synthetic_data_generator.py -t regression -n 5000 -f 20 -o reg.parquet
      
    • Clustering:

      python synthetic_data_generator.py -t clustering -n 3000 -f 5 --n-clusters 4 -o clusters.csv
      
    • Time series (hourly, 3 features):

      python synthetic_data_generator.py -t timeseries -n 1000 -f 3 --freq H -o ts.csv
      
    • Mixed tabular:

      python synthetic_data_generator.py -t mixed -n 1500 --mixed-n-numeric 4 --mixed-n-categorical 2 -o mixed.csv
      

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.