I have a python code to run a ML experiment given a set of parameters. I'm looking for a…
- User
I have a python code to run a ML experiment given a set of parameters. I'm looking for a way to do hparams search so that:
-
The library is used, active and mantained
-
I can specify the range of hparams to tune in a yaml file and then use something to read it
-
Each run should be completely isolated. An independent process spawned
-
Write the simples, most understandable code possible.
-
Don't use hydra.
-
I want to call a function (maybe a launcher) in which I specify: the yaml config file and some (variable number) override parameters. The yaml file should contain also the gpu to use, the number of parallel jobs to launch, the total number of experiments to make
-
It should be created a local file (bulid from the yaml variables
env_idandlogger_tag) of "tried hparams" so that each new run that is spawned reads that file to decide the new set of hparams to try. -
The hparams selection should account for stocasticity in the final return value.
I already have run_experiment.py where there is a train(cfg) function that runs a full experiments and returns the final loss.
-
- Assistant
Below is one possible “minimal-but-complete” solution that respects all the constraints you listed.
Dependencies
pip install optuna pyyaml tqdm
- YAML CONFIG (anything that can be edited from the command line)
# experiment.yaml experiment: # --- general info --------------------------------- env_id: CartPole-v1 logger_tag: baseline total_trials: 80 # how many different settings to try seeds_per_trial: 3 # handle stochasticity (req. 8) n_jobs: 4 # how many workers in parallel (req. 6) gpu_ids: [0,1] # visible GPUs (req. 6) search_space: # --- what to tune --------------------------------- lr: # learning rate type: loguniform low: 1e-5 high: 1e-2 batch_size: type: categorical choices: [32, 64, 128, 256] gamma: type: uniform low: 0.8 high: 0.999
- run_experiment.py (your code – tiny stub so the demo runs)
# run_experiment.py ------------------------------------------------------------ # Your real file can be much longer, only train(cfg) must exist and return loss import random, time, json, sys, argparse, os def train(cfg: dict) -> float: """ Dummy training loop. Replace by your own code. We simulate some work and produce a pseudo-loss that depends on the given hyper-parameters and on RNG noise. """ time.sleep(1) # pretend to train lr, bs, gamma = cfg['lr'], cfg['batch_size'], cfg['gamma'] noise = random.random()*0.05 return (1/gamma) + lr*10 + (256/bs)*0.01 + noise # bogus “loss” if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--config_json", required=True) args = parser.parse_args() with open(args.config_json) as f: cfg = json.load(f) loss = train(cfg) # IMPORTANT: print the **only** thing the launcher needs # so it can grab it from stdout. print(loss)
- launcher.py (the only new file you actually have to read)
# launcher.py ------------------------------------------------------------------ """ Minimal Optuna-based hyper-parameter search that fulfils all requirements: (1) Maintained lib -> Optuna (2) Search space in YAML (3) Each training run is its own Python process (4) Keep code small & readable (5) No Hydra (6) `launch_experiments(cfg_path, **overrides)` (7) Re-uses a local sqlite file 'env_id_logger_tag.db' (8) Repeats each trial several times to cope with stochasticity """ from __future__ import annotations import os, json, yaml, uuid, subprocess, shutil from typing import Dict, Any import optuna from tqdm import tqdm # ----------------------------------------------------------------------------- # def _suggest_by_spec(trial: optuna.Trial, name: str, spec: Dict[str, Any]): """Convert the tiny YAML DSL into optuna.*suggest_* calls.""" t = spec["type"] if t == "uniform": return trial.suggest_float(name, spec["low"], spec["high"]) if t == "loguniform": return trial.suggest_float(name, spec["low"], spec["high"], log=True) if t == "categorical": return trial.suggest_categorical(name, spec["choices"]) raise ValueError(f"Unknown parameter type: {t}") # ----------------------------------------------------------------------------- # def launch_experiments(cfg_path: str, **overrides): """ Main entry point. Example call: >>> launch_experiments('experiment.yaml', total_trials=100, lr={'low':1e-6}) Any keyword overrides replace (or add) keys in the loaded YAML. """ with open(cfg_path) as f: cfg = yaml.safe_load(f) # --------------------------------------------------------------------- # # Apply CLI overrides (simple 1-level merge – keep it readable) for k, v in overrides.items(): cfg[k] = v exp_cfg = cfg["experiment"] search_space = cfg["search_space"] env_id = exp_cfg["env_id"] tag = exp_cfg["logger_tag"] n_trials = int(exp_cfg["total_trials"]) n_jobs = int(exp_cfg["n_jobs"]) seeds_pp = int(exp_cfg.get("seeds_per_trial", 1)) gpu_ids = list(exp_cfg.get("gpu_ids", [])) # (7) SQLite file == "already tried hparams" storage = f"sqlite:///{env_id}_{tag}.db" study = optuna.create_study(direction="minimize", study_name=f"{env_id}_{tag}", storage=storage, load_if_exists=True) # Used to round-robin GPUs -------------------------------------------------- # def _pick_gpu(trial_idx: int) -> str: if not gpu_ids: return "" # let the OS / CUDA choose return str(gpu_ids[trial_idx % len(gpu_ids)]) # Objective ---------------------------------------------------------------- # def objective(trial: optuna.Trial) -> float: # 1) sample a full set of parameters from YAML search_space params = {k: _suggest_by_spec(trial, k, spec) for k, spec in search_space.items()} # 2) Evaluate the setting `seeds_pp` times to average out randomness losses = [] for seed in range(seeds_pp): # Compose a temporary json that run_experiment.py understands tmp_json = f"/tmp/{uuid.uuid4().hex}.json" with open(tmp_json, "w") as fp: json.dump({**params, "seed": seed}, fp) # spawn a completely independent python process (req. 3) env = os.environ.copy() gpu = _pick_gpu(trial.number + seed) if gpu: # stick to a single visible GPU env["CUDA_VISIBLE_DEVICES"] = gpu cmd = ["python", "-u", "run_experiment.py", "--config_json", tmp_json] proc = subprocess.run(cmd, capture_output=True, text=True, env=env) if proc.returncode != 0: # Propagate failure to Optuna raise RuntimeError(proc.stderr) try: loss = float(proc.stdout.strip().split()[-1]) except ValueError: raise RuntimeError(f"Could not parse loss from output:\n{proc.stdout}") losses.append(loss) os.remove(tmp_json) # keep /tmp tidy # Average loss over different random seeds (req. 8) return sum(losses)/len(losses) # --------------------------------------------------------------------- # print(f"\n🚀 Starting study : {env_id}_{tag}") print(f"• Trials to run : {n_trials}") print(f"• Parallel jobs : {n_jobs}") print(f"• Seeds per trial (robust) : {seeds_pp}") print(f"• GPUs : {gpu_ids or 'any'}\n") study.optimize(objective, n_trials=n_trials, n_jobs=n_jobs, show_progress_bar=True) # --------------------------------------------------------------------- # best = study.best_trial print("\n✅ DONE. Best config so far:") for k, v in best.params.items(): print(f" - {k:15s}: {v}") print(f"Loss = {best.value:.5f}") # ----------------------------------------------------------------------------- # if __name__ == "__main__": import argparse, sys p = argparse.ArgumentParser() p.add_argument("yaml_config", help="Path to YAML file, e.g. experiment.yaml") # Accept arbitrary `key=value` overrides after a `--` # python launcher.py experiment.yaml total_trials=200 n_jobs=8 p.add_argument("overrides", nargs="*", help="key=value pairs to override YAML") args = p.parse_args() # Parse the CLI overrides into python types in the laziest possible way parsed_ovr = {} for item in args.overrides: key, val = item.split("=", 1) # Naive JSON-eval gives us ints, floats, lists, … parsed_ovr[key] = json.loads(val) launch_experiments(args.yaml_config, **parsed_ovr)
How this meets each requirement
- Library is active/maintained – Optuna is one of the most popular HPO libs (⭐ 10k+, weekly releases).
- Search space is fully described inside a YAML file.
- Every evaluation (
run_experiment.py) is started viasubprocess.run, hence its own PID and memory space. - Only ~100 lines of commented Python, no magical framework layers.
- Hydra is not imported anywhere.
launch_experiments(yaml_file, **overrides)is the public API; thelauncher.pyCLI is a thin wrapper.- The local SQLite file
env_id_logger_tag.dbkeeps every tried trial. When you re-runlauncher.py, Optuna resumes exactly where it stopped. - Stochasticity is handled by running the same hyper-parameters several times (
seeds_per_trial) and reporting the average loss.
Running it
# first run python launcher.py experiment.yaml # later, resume but try another 100 trials and search a wider lr range python launcher.py experiment.yaml total_trials=100 lr='{"low":1e-6,"high":1e-1,"type":"loguniform"}'That’s it – a small, dependency-light, completely process-isolated hyper-parameter search framework you can drop next to your existing
run_experiment.py. Happy tuning!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.