What's the best way to set up different jupyter environments for ipynb notebooks.…
- User
What's the best way to set up different jupyter environments for ipynb notebooks. Currently we have jupyterhub running on our VM with dockerswarm. I have one library that has pinned a ton of other libraries to a really old version. Is there a way to have the default to one without that library and another with pinned libraries.
- Assistant
Yes, there are excellent ways to manage different environments for
.ipynbnotebooks in a JupyterHub setup, especially when you're using Docker Swarm. Since you want flexibility in isolating environments (specifically: one standard, and another with a pinned older library), you can achieve this using Docker-based multi-environment kernel setups within JupyterHub.Below is a guide to achieve a clean and robust multi-environment setup on JupyterHub running on Docker Swarm.
✅ Objective
- Have multiple Jupyter notebook environments, e.g.:
- A standard "default" environment (modern libraries).
- An "old-deps" environment (with pinned/old library versions).
- Run them on JupyterHub using DockerSpawner (ideal for Swarm).
- Allow users to choose between environments via the Jupyter interface.
🔧 Step-by-Step Setup
1. Use JupyterHub with DockerSpawner
Since you're using Docker Swarm, the most compatible setup is to use the
DockerSpawnerin Swarm mode, using different Docker images representing different environments.Your setup likely already resembles this. Each user session is spawned in its own Docker container, using a Docker image.
2. Build Separate Docker Images for Each Environment
Create two (or more) Docker images, each representing a fully isolated environment—with different Python packages installed.
a. Default Environment Dockerfile
FROM jupyter/base-notebook:python-3.11 # Example: Default modern packages RUN pip install numpy pandas matplotlib seaborn \ && fix-permissions "${HOME}"Tag this as something like:
myorg/jupyter-default-envb. Old/Pinned Dependencies Dockerfile
FROM jupyter/base-notebook:python-3.8 # Example: Installing a problematic pinned dependency (like TensorFlow 1.x) COPY requirements-old.txt ./ # Contains pinned versions RUN pip install -r requirements-old.txt \ && fix-permissions "${HOME}"Example
requirements-old.txt:tensorflow==1.13.1 numpy==1.16.4 pandas==0.24.2Tag this as:
myorg/jupyter-legacy-env
3. Register Multiple Docker Images in JupyterHub
You can use
DockerSpawner.ProfileSpawnerto allow users to select the environment from a dropdown menu when they log in.In
jupyterhub_config.py:from dockerspawner import DockerSpawner from dockerspawner import SwarmSpawner from jupyterhub.spawner import Spawner c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner' # This enables the profile list c.Spawner.cmd = ['jupyter-labhub'] c.SwarmSpawner.use_internal_ip = True c.SwarmSpawner.network_name = 'jupyterhub-network' # Define different environments as spawner profiles: c.SwarmSpawner.profile_list = [ { 'display_name': 'Default Environment', 'description': 'Latest Python + scientific libraries', 'docker_image': 'myorg/jupyter-default-env:latest', }, { 'display_name': 'Legacy Pinned Environment', 'description': 'Pinned dependencies for compatibility', 'docker_image': 'myorg/jupyter-legacy-env:latest', }, ]This will present a dropdown to the user in the JupyterHub interface when they start or restart their server.
4. Optional: Use
nb_conda_kernelsfor intra-kernel switching (if not using DockerSpawner)If you're not using Swarm or DockerSpawner, and instead using conda environments on a shared VM, you can install:
conda install nb_conda_kernelsThen each conda environment that has
ipykernelinstalled will show up in the Jupyter Notebook interface as a new kernel.But, for DockerSwarm setups, separate Docker images are more isolated and cleaner.
5. Version-Control Dockerfiles
Put all the Dockerfiles in a Git repo under version control. This makes it easier to manage reproducibility, rollbacks, and audits.
✅ Result
- When users log into JupyterHub, they choose between:
- "Default Environment" for day-to-day work.
- "Legacy Environment" for compatibility with older code/libs.
- You are not polluting one environment with conflicting versions.
- Docker ensures isolation, security, and reproducibility.
🚀 Pro Tips
- ✅ Use shared persistent volumes (e.g., NFS or S3-like storage) so notebooks are retained across environments.
- ✅ Automate builds using GitHub Actions or GitLab CI to push new Docker images to your registry.
- ✅ Keep environment-specific notebooks or markdown docs explaining which versions are installed in each image to help your team.
Let me know if you want a working example Dockerfile or a sample Swarm config!
- Have multiple Jupyter notebook environments, e.g.:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.