Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

I have a python code to run a ML experiment given a set of parameters. I'm looking for a…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219’s answer.
  1. User

    I have a python code to run a ML experiment given a set of parameters. I'm looking for a way to do hparams search so that:

    1. The library is used, active and mantained

    2. I can specify the range of hparams to tune in a yaml file and then use something to read it

    3. Each run should be completely isolated. An independent process spawned

    4. Write the simples, most understandable code possible.

    5. Don't use hydra.

    6. I want to call a function (maybe a launcher) in which I specify: the yaml config file and some (variable number) override parameters. The yaml file should contain also the gpu to use, the number of parallel jobs to launch, the total number of experiments to make

    7. It should be created a local file (bulid from the yaml variables env_id and logger_tag) of "tried hparams" so that each new run that is spawned reads that file to decide the new set of hparams to try.

    8. The hparams selection should account for stocasticity in the final return value.

    I already have run_experiment.py where there is a train(cfg) function that runs a full experiments and returns the final loss.

  2. Assistant

    Below is one possible “minimal-but-complete” solution that respects all the constraints you listed.

    Dependencies

    pip install optuna pyyaml tqdm
    

    1. YAML CONFIG (anything that can be edited from the command line)
    # experiment.yaml
    experiment:                 # --- general info ---------------------------------
      env_id: CartPole-v1
      logger_tag: baseline
      total_trials: 80          # how many different settings to try
      seeds_per_trial: 3        # handle stochasticity  (req. 8)
      n_jobs: 4                 # how many workers in parallel (req. 6)
      gpu_ids: [0,1]            # visible GPUs             (req. 6)
    
    search_space:               # --- what to tune ---------------------------------
      lr:                       # learning rate
        type: loguniform
        low: 1e-5
        high: 1e-2
    
      batch_size:
        type: categorical
        choices: [32, 64, 128, 256]
    
      gamma:
        type: uniform
        low: 0.8
        high: 0.999
    

    1. run_experiment.py (your code – tiny stub so the demo runs)
    # run_experiment.py ------------------------------------------------------------
    # Your real file can be much longer, only train(cfg) must exist and return loss
    import random, time, json, sys, argparse, os
    
    def train(cfg: dict) -> float:
        """
        Dummy training loop.  Replace by your own code.
        We simulate some work and produce a pseudo-loss that depends
        on the given hyper-parameters and on RNG noise.
        """
        time.sleep(1)                                    # pretend to train
        lr, bs, gamma = cfg['lr'], cfg['batch_size'], cfg['gamma']
        noise = random.random()*0.05
        return (1/gamma) + lr*10 + (256/bs)*0.01 + noise # bogus “loss”
    
    if __name__ == "__main__":
        parser = argparse.ArgumentParser()
        parser.add_argument("--config_json", required=True)
        args = parser.parse_args()
    
        with open(args.config_json) as f:
            cfg = json.load(f)
    
        loss = train(cfg)
        # IMPORTANT: print the **only** thing the launcher needs
        # so it can grab it from stdout.
        print(loss)
    

    1. launcher.py (the only new file you actually have to read)
    # launcher.py ------------------------------------------------------------------
    """
    Minimal Optuna-based hyper-parameter search that fulfils all requirements:
    
    (1) Maintained lib  -> Optuna
    (2) Search space in YAML
    (3) Each training run is its own Python process
    (4) Keep code small & readable
    (5) No Hydra
    (6) `launch_experiments(cfg_path, **overrides)`
    (7) Re-uses a local sqlite file 'env_id_logger_tag.db'
    (8) Repeats each trial several times to cope with stochasticity
    """
    from __future__ import annotations
    import os, json, yaml, uuid, subprocess, shutil
    from typing import Dict, Any
    import optuna
    from tqdm import tqdm
    
    # ----------------------------------------------------------------------------- #
    def _suggest_by_spec(trial: optuna.Trial, name: str, spec: Dict[str, Any]):
        """Convert the tiny YAML DSL into optuna.*suggest_* calls."""
        t = spec["type"]
        if t == "uniform":
            return trial.suggest_float(name, spec["low"], spec["high"])
        if t == "loguniform":
            return trial.suggest_float(name, spec["low"], spec["high"], log=True)
        if t == "categorical":
            return trial.suggest_categorical(name, spec["choices"])
        raise ValueError(f"Unknown parameter type: {t}")
    
    # ----------------------------------------------------------------------------- #
    def launch_experiments(cfg_path: str, **overrides):
        """
        Main entry point.  Example call:
    
        >>> launch_experiments('experiment.yaml', total_trials=100, lr={'low':1e-6})
        Any keyword overrides replace (or add) keys in the loaded YAML.
        """
        with open(cfg_path) as f:
            cfg = yaml.safe_load(f)
    
        # --------------------------------------------------------------------- #
        # Apply CLI overrides (simple 1-level merge – keep it readable)
        for k, v in overrides.items():
            cfg[k] = v
    
        exp_cfg      = cfg["experiment"]
        search_space = cfg["search_space"]
    
        env_id   = exp_cfg["env_id"]
        tag      = exp_cfg["logger_tag"]
        n_trials = int(exp_cfg["total_trials"])
        n_jobs   = int(exp_cfg["n_jobs"])
        seeds_pp = int(exp_cfg.get("seeds_per_trial", 1))
        gpu_ids  = list(exp_cfg.get("gpu_ids", []))
    
        # (7) SQLite file == "already tried hparams"
        storage = f"sqlite:///{env_id}_{tag}.db"
        study   = optuna.create_study(direction="minimize",
                                      study_name=f"{env_id}_{tag}",
                                      storage=storage,
                                      load_if_exists=True)
    
        # Used to round-robin GPUs -------------------------------------------------- #
        def _pick_gpu(trial_idx: int) -> str:
            if not gpu_ids:
                return ""                       # let the OS / CUDA choose
            return str(gpu_ids[trial_idx % len(gpu_ids)])
    
        # Objective ---------------------------------------------------------------- #
        def objective(trial: optuna.Trial) -> float:
            # 1) sample a full set of parameters from YAML search_space
            params = {k: _suggest_by_spec(trial, k, spec)
                      for k, spec in search_space.items()}
    
            # 2) Evaluate the setting `seeds_pp` times to average out randomness
            losses = []
            for seed in range(seeds_pp):
                # Compose a temporary json that run_experiment.py understands
                tmp_json = f"/tmp/{uuid.uuid4().hex}.json"
                with open(tmp_json, "w") as fp:
                    json.dump({**params, "seed": seed}, fp)
    
                # spawn a completely independent python process  (req. 3)
                env = os.environ.copy()
                gpu = _pick_gpu(trial.number + seed)
                if gpu:                          # stick to a single visible GPU
                    env["CUDA_VISIBLE_DEVICES"] = gpu
    
                cmd = ["python", "-u", "run_experiment.py", "--config_json", tmp_json]
                proc = subprocess.run(cmd,
                                      capture_output=True, text=True, env=env)
    
                if proc.returncode != 0:
                    # Propagate failure to Optuna
                    raise RuntimeError(proc.stderr)
    
                try:
                    loss = float(proc.stdout.strip().split()[-1])
                except ValueError:
                    raise RuntimeError(f"Could not parse loss from output:\n{proc.stdout}")
    
                losses.append(loss)
                os.remove(tmp_json)              # keep /tmp tidy
    
            # Average loss over different random seeds (req. 8)
            return sum(losses)/len(losses)
    
        # --------------------------------------------------------------------- #
        print(f"\n🚀 Starting study            : {env_id}_{tag}")
        print(f"• Trials to run              : {n_trials}")
        print(f"• Parallel jobs              : {n_jobs}")
        print(f"• Seeds per trial (robust)   : {seeds_pp}")
        print(f"• GPUs                       : {gpu_ids or 'any'}\n")
    
        study.optimize(objective, n_trials=n_trials, n_jobs=n_jobs, show_progress_bar=True)
    
        # --------------------------------------------------------------------- #
        best = study.best_trial
        print("\n✅  DONE. Best config so far:")
        for k, v in best.params.items():
            print(f" - {k:15s}: {v}")
        print(f"Loss = {best.value:.5f}")
    
    # ----------------------------------------------------------------------------- #
    if __name__ == "__main__":
        import argparse, sys
        p = argparse.ArgumentParser()
        p.add_argument("yaml_config", help="Path to YAML file, e.g. experiment.yaml")
        # Accept arbitrary `key=value` overrides after a `--`
        #   python launcher.py experiment.yaml total_trials=200 n_jobs=8
        p.add_argument("overrides", nargs="*", help="key=value pairs to override YAML")
        args = p.parse_args()
    
        # Parse the CLI overrides into python types in the laziest possible way
        parsed_ovr = {}
        for item in args.overrides:
            key, val = item.split("=", 1)
            # Naive JSON-eval gives us ints, floats, lists, …
            parsed_ovr[key] = json.loads(val)
        launch_experiments(args.yaml_config, **parsed_ovr)
    

    How this meets each requirement

    1. Library is active/maintained – Optuna is one of the most popular HPO libs (⭐ 10k+, weekly releases).
    2. Search space is fully described inside a YAML file.
    3. Every evaluation (run_experiment.py) is started via subprocess.run, hence its own PID and memory space.
    4. Only ~100 lines of commented Python, no magical framework layers.
    5. Hydra is not imported anywhere.
    6. launch_experiments(yaml_file, **overrides) is the public API; the launcher.py CLI is a thin wrapper.
    7. The local SQLite file env_id_logger_tag.db keeps every tried trial. When you re-run launcher.py, Optuna resumes exactly where it stopped.
    8. Stochasticity is handled by running the same hyper-parameters several times (seeds_per_trial) and reporting the average loss.

    Running it

    # first run
    python launcher.py experiment.yaml
    
    # later, resume but try another 100 trials and search a wider lr range
    python launcher.py experiment.yaml total_trials=100 lr='{"low":1e-6,"high":1e-1,"type":"loguniform"}'
    

    That’s it – a small, dependency-light, completely process-isolated hyper-parameter search framework you can drop next to your existing run_experiment.py. Happy tuning!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.