Skip to content
TheChatArchiveThe Chat Archive
Human-written demonstration (OpenAssistant volunteers)· OpenAssistant (human-written)Hugging Face OpenAssistant Conversations (oasst2)

Do you have anything to add to this problem resolving around a scheduler calling…

In OpenAssistant data, “assistant” turns were written by human volunteers as demonstrations. They are not model outputs.
  1. User

    Do you have anything to add to this problem resolving around a scheduler calling multiple dockerized clients that handle AI requests?

    ML Parallelity Problem (Docker)

    • some models need all the hardware
    • we want the same hardware to work on multiple jobs
    • -> models need to be switched

    Possible Solutions

    • Put all models in same container
    • Scheduler sends load and unload signal to container
  2. Assistant

    I have some things to add regarding your solutions

    • As you have mentioned, you can put all the models in the same container and switch between models in the same container. However, you could run into the issue of conflicts, due to the possibility of requiring different versions of the same library.
    • Having the scheduler load and unload different containers is definitely feasible.

    If you want to be able to run multiple tasks at once where possible, such as when the combined required resources of multiple models are less than what the hardware can handle. Then you will likely need to implement a queue system and a way of checking if multiple tasks in the queue can be run simultaneously. If you implement this, you may also want to implement a priority system, so that important tasks can be run alone to ensure they are run quickly.

    There are also preexisting docker orchestration that may prove helpful.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.