Skip to content
TheChatArchiveThe Chat Archive
chatgpt-4o-latest-20250326· OpenAIHugging Face LMArena Human Preference 140K

What's the best way to set up different jupyter environments for ipynb notebooks.…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Amazon Novaamazon.nova-pro-v1:0’s answer.
  1. User

    What's the best way to set up different jupyter environments for ipynb notebooks. Currently we have jupyterhub running on our VM with dockerswarm. I have one library that has pinned a ton of other libraries to a really old version. Is there a way to have the default to one without that library and another with pinned libraries.

  2. Assistant

    Yes, there are excellent ways to manage different environments for .ipynb notebooks in a JupyterHub setup, especially when you're using Docker Swarm. Since you want flexibility in isolating environments (specifically: one standard, and another with a pinned older library), you can achieve this using Docker-based multi-environment kernel setups within JupyterHub.

    Below is a guide to achieve a clean and robust multi-environment setup on JupyterHub running on Docker Swarm.


    ✅ Objective

    • Have multiple Jupyter notebook environments, e.g.:
      • A standard "default" environment (modern libraries).
      • An "old-deps" environment (with pinned/old library versions).
    • Run them on JupyterHub using DockerSpawner (ideal for Swarm).
    • Allow users to choose between environments via the Jupyter interface.

    🔧 Step-by-Step Setup

    1. Use JupyterHub with DockerSpawner

    Since you're using Docker Swarm, the most compatible setup is to use the DockerSpawner in Swarm mode, using different Docker images representing different environments.

    Your setup likely already resembles this. Each user session is spawned in its own Docker container, using a Docker image.


    2. Build Separate Docker Images for Each Environment

    Create two (or more) Docker images, each representing a fully isolated environment—with different Python packages installed.

    a. Default Environment Dockerfile

    FROM jupyter/base-notebook:python-3.11
    
    # Example: Default modern packages
    RUN pip install numpy pandas matplotlib seaborn \
        && fix-permissions "${HOME}"
    

    Tag this as something like: myorg/jupyter-default-env

    b. Old/Pinned Dependencies Dockerfile

    FROM jupyter/base-notebook:python-3.8
    
    # Example: Installing a problematic pinned dependency (like TensorFlow 1.x)
    COPY requirements-old.txt ./  # Contains pinned versions
    RUN pip install -r requirements-old.txt \
        && fix-permissions "${HOME}"
    

    Example requirements-old.txt:

    tensorflow==1.13.1
    numpy==1.16.4
    pandas==0.24.2
    

    Tag this as: myorg/jupyter-legacy-env


    3. Register Multiple Docker Images in JupyterHub

    You can use DockerSpawner.ProfileSpawner to allow users to select the environment from a dropdown menu when they log in.

    In jupyterhub_config.py:

    from dockerspawner import DockerSpawner
    from dockerspawner import SwarmSpawner
    from jupyterhub.spawner import Spawner
    
    c.JupyterHub.spawner_class = 'dockerspawner.SwarmSpawner'
    
    # This enables the profile list
    c.Spawner.cmd = ['jupyter-labhub']
    c.SwarmSpawner.use_internal_ip = True
    c.SwarmSpawner.network_name = 'jupyterhub-network'
    
    # Define different environments as spawner profiles:
    c.SwarmSpawner.profile_list = [
        {
            'display_name': 'Default Environment',
            'description': 'Latest Python + scientific libraries',
            'docker_image': 'myorg/jupyter-default-env:latest',
        },
        {
            'display_name': 'Legacy Pinned Environment',
            'description': 'Pinned dependencies for compatibility',
            'docker_image': 'myorg/jupyter-legacy-env:latest',
        },
    ]
    

    This will present a dropdown to the user in the JupyterHub interface when they start or restart their server.


    4. Optional: Use nb_conda_kernels for intra-kernel switching (if not using DockerSpawner)

    If you're not using Swarm or DockerSpawner, and instead using conda environments on a shared VM, you can install:

    conda install nb_conda_kernels
    

    Then each conda environment that has ipykernel installed will show up in the Jupyter Notebook interface as a new kernel.

    But, for DockerSwarm setups, separate Docker images are more isolated and cleaner.


    5. Version-Control Dockerfiles

    Put all the Dockerfiles in a Git repo under version control. This makes it easier to manage reproducibility, rollbacks, and audits.


    ✅ Result

    • When users log into JupyterHub, they choose between:
      • "Default Environment" for day-to-day work.
      • "Legacy Environment" for compatibility with older code/libs.
    • You are not polluting one environment with conflicting versions.
    • Docker ensures isolation, security, and reproducibility.

    🚀 Pro Tips

    • ✅ Use shared persistent volumes (e.g., NFS or S3-like storage) so notebooks are retained across environments.
    • ✅ Automate builds using GitHub Actions or GitLab CI to push new Docker images to your registry.
    • ✅ Keep environment-specific notebooks or markdown docs explaining which versions are installed in each image to help your team.

    Let me know if you want a working example Dockerfile or a sample Swarm config!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.