Project: AI Toolchain is a collection of tools for quickly building and deploying…
- User
Project: AI Toolchain is a collection of tools for quickly building and deploying machine learning models for various use cases. Currently, the toolchain includes a text translation model, and more models may be added in the future. It abstracts the dirty details of how a model works similar to Huggingface and gives a clean API that you can orchestrate at aenter code here BFF level.
Features to be implemented Abstract the layer of deployment for AI Tools. Anyone should be easily add a new model to the stack without thinking about deployments. We should be able to deploy AI Tools in such a way where each model (every model can be packaged as a container) should be independently scalable. As a user, I should be able to access APIs associated with any model.
Acceptance Criteria Document to add a new model and test it on local Document to deploy a new model Document to accesss APIs asscoiated with model
- Assistant
AI Toolchain Implementation Guide
This guide will help you understand how to add a new model to the AI Toolchain, test it locally, deploy it, and access the associated APIs. By following these steps, you'll be able to easily integrate new models into the AI Toolchain and make them available for use.
1. Adding a New Model to the AI Toolchain
-
First, create a new directory for your model within the AI Toolchain repository and name it according to your model's purpose. For example, if you're adding a sentiment analysis model, the folder should be named
sentiment_analysis. -
Within the new directory, add the following files:
-
model.py: This file should contain the implementation of the model using a popular machine learning framework like TensorFlow, PyTorch, or scikit-learn. It should include methods for training, evaluating, and predicting. -
requirements.txt: List any Python dependencies required by your model.
-
-
Add a
Dockerfilein the model's directory to containerize the model. The file should include instructions to install dependencies and copy relevant files to the container. Here's an example:FROM python:3.8-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY model.py . CMD ["python", "model.py"]
2. Testing the Model Locally
- Navigate to your model's directory:
cd sentiment_analysis - Build the Docker image:
docker build -t sentiment_analysis . - Run the Docker container:
docker run -p 8000:8000 sentiment_analysis - Test your model's API locally using a tool like curl or Postman.
3. Deploying the Model
-
Push your model's Docker image to a container registry like Docker Hub or Google Container Registry:
docker push sentiment_analysis -
Create a Kubernetes deployment configuration file,
deployment.yaml, with the following parameters:- name: sentiment_analysis
- replicas: 2 (or any desired number for scalability)
- container image: your container registry URL
- container port: 8000
-
Apply the Kubernetes deployment to your cluster:
kubectl apply -f deployment.yaml -
Create a Kubernetes service configuration file,
service.yaml, for accessing your model's API:- name: sentiment_analysis
- selector: with the same label as your model's deployment
- port: 8000
- targetPort: 8000
- type: LoadBalancer (or any other desired service type)
-
Apply the Kubernetes service to your cluster:
kubectl apply -f service.yaml
4. Accessing the Model's APIs
After deploying the model using Kubernetes, you can access its APIs through the assigned external IP address or domain name.
- Retrieve the external IP address or domain name:
kubectl get svc sentiment_analysis - Use the IP address or domain name to access the model's API using a tool like curl or Postman.
By following these steps, you can add new models to the AI Toolchain, test them locally, deploy them, and access their APIs for various use cases. This will enable the rapid deployment and scalability of AI tools for a wide range of applications.
-
- User
Use docker and docker compose for above also as a user, I should be able to access APIs associated with any model
- Assistant
To use Docker and Docker Compose for the AI Toolchain's setup, follow these steps to add a new model, test it locally, and deploy it using Docker Compose. This will allow users to access APIs associated with any model.
1. Adding a New Model to the AI Toolchain
Follow the steps in the previous answer to create the necessary files and the Dockerfile for your new model.
2. Testing the Model Locally with Docker Compose
-
Create a
docker-compose.ymlfile in the root directory of the AI Toolchain repository with the following content:version: "3.9" services: sentiment_analysis: build: ./sentiment_analysis ports: - "8000:8000"Replace
./sentiment_analysiswith the path to your model's directory and adjust the ports if necessary. -
Run
docker-compose upto build and run the container for your model. -
Test your model's API locally using a tool like curl or Postman, connecting to
localhost:8000.
3. Deploying the Model using Docker Compose
-
Push your model's Docker image to a container registry:
docker-compose build sentiment_analysis docker tag sentiment_analysis:latest your-container-registry-url/sentiment_analysis:latest docker push your-container-registry-url/sentiment_analysis:latest -
Update the
docker-compose.ymlfile in your deployment environment to include your new model:version: "3.9" services: sentiment_analysis: image: your-container-registry-url/sentiment_analysis:latest ports: - "8000:8000" -
Deploy the model using Docker Compose in the deployment environment by running
docker-compose up -d.
4. Accessing the Model's APIs
Once the Docker Compose deployment is complete, users can access the APIs associated with each model using the assigned server IP address or domain name and the specified port number.
For example, if the model is deployed on a server with the IP address
192.168.1.100, and the model's API listens on port 8000, users can access the API by sending requests to192.168.1.100:8000.By using Docker and Docker Compose, the AI Toolchain can efficiently add new models, test them locally, and deploy them so that users can access the associated APIs for various use cases.
-
- User
Im not understand how in the above solution anyone should be easily add a new model to the stack without thinking about deployments and as a user, I should be able to access APIs associated with any model?
-
Github for current AI tools contains: .github/workflows Lower case repo name for GH Packages 2 months ago benchmarks Fix add test 3 weeks ago flake8 Fix: module/folder names 2 months ago src Merge pull request #100 from rishav-eulb/rishav-eulb-patch-2 last week .flake8 Fixes: #29 2 months ago .gitignore Initial Commit 2 months ago .gitpod.Dockerfile Included installation of Git last week .gitpod.yml Corrected .gitpod.yml last week Dockerfile Added Gitpod last week README.md docs: readme updated last month api.py Moved restart and watch functions to separate file last week contribution.md Corrected Gitpod button last week deploy.sh deploy.sh_error last month docker-compose.yml Added Gitpod last week poetry.lock Feat: Reload debug server on changes to src folder last week prometheus.yml add prometheus.yml 2 weeks ago pyproject.toml Feat: Reload debug server on changes to src folder last week repository_data.json Feat: Azure translation 3 weeks ago sample.env Fix: Add sample env 2 months ago template_batch_model.py added repo structure and translation model 2 months ago template_model_request.py added repo structure and translation model 2 months ago test.py Fix add test 3 weeks ago watch_folder.py Moved restart and watch functions to separate file last week
-
README.md AI Toolchain AI Toolchain is a collection of tools for quickly building and deploying machine learning models for various use cases. Currently, the toolchain includes a text translation model, and more models may be added in the future.
How to Run To deploy all models, simply execute the deploy.sh script located in the root folder. This script calls the deployment files of each model. Note that the toolchain may switch to using Docker in the future for deployment.
To create a new model class, use the template_batch_model.py file as a starting point. Your new model class should implement the method mentioned in the template file.
To create a new request class, use the template_model_request.py file as a starting point. This class is used to map the incoming request to the data needed by the model.
To add your new model and request to the API, modify the repository dictionary in api.py.
Repository The repository is structured as follows
Setup To set up the AI Toolchain environment, follow these steps:
python3 -m venv venv source venv/bin/activate pip install poetry poetry install quart --app api --debug run Poetry Fixes poetry lock --no-update Contributing Contributions to AI Toolchain are welcome! To contribute, please follow these guidelines:
Fork the repository and create a new branch for your feature or bug fix. Write tests for your changes. Submit a pull request describing your changes and why they are needed. Thank you for considering contributing to AI Toolchain!
- Current dockerfile in github:
Use an official Python runtime as a parent image
FROM python:3.9-slim
Set the working directory to /app
WORKDIR /app
Copy the poetry.lock and pyproject.toml files to the working directory
COPY poetry.lock pyproject.toml /app/
Install dependencies using Poetry
RUN pip install poetry &&
poetry config virtualenvs.create false &&
poetry install --no-dev --no-rootCopy the rest of the application code to the working directory
COPY . /app/
Set the entrypoint for the container
ENTRYPOINT ["hypercorn", "api", "-b", "0.0.0.0"]
Expose additional ports for Gitpod
EXPOSE 8888
- Current docker-compose file: version: '3'
services: app: build: context: . dockerfile: Dockerfile ports: - "8000:8000" restart: always environment: - PYTHONUNBUFFERED=1 - PYTHONDONTWRITEBYTECODE=1
- Current api.py: import importlib
from quart import Quart, g, request, jsonify, abort, Response from markupsafe import escape import json import aiohttp import os from functools import wraps from dotenv import load_dotenv from quart_compress import Compress import time from watch_folder import watch_src_folder import asyncio
load_dotenv()
app = Quart(name) Compress(app)
watch_dir = "src" debug_mode = app.debug
async def create_client_session(): """ Create an aiohttp ClientSession. """ async with aiohttp.ClientSession() as session: return session
extra_dirs = ['src'] extra_files = extra_dirs[:] for extra_dir in extra_dirs: for dirname, dirs, files in os.walk(extra_dir): for filename in files: filename = os.path.join(dirname, filename) if os.path.isfile(filename): extra_files.append(filename)
app.config.update(extra_files=extra_files)
with open('repository_data.json') as f: repository_data = json.load(f)
AUTH_HEADER = os.getenv("AUTH_HEADER") AUTH_HEADER_KEY = os.getenv("AUTH_HEADER_KEY")
def verify_auth_header(auth_header_key, expected_value): def decorator(f): @wraps(f) async def decorated_function(*args, **kwargs): auth_header = request.headers.get(auth_header_key) if not auth_header or auth_header != expected_value: print("Unauthorized access"); abort(401) # Unauthorized return await f(*args, **kwargs)
return decorated_function return decorator@app.route("/") def welcome(): return "Welcome!"
@app.route("/repository") def repository(): """ Returns the repository data, which contains the available models and their configurations""" return jsonify(repository_data)
def json_to_object(request_class, json_str): """Converts a JSON string to an object of the given class at level 1.""" data = json.loads(json_str) return request_class(**data)
def get_model_config(use_case, provider, mode): """ Returns the model config for the given use case, provider and mode """ use_case_data = repository_data.get('use_cases').get(use_case) if use_case_data is None: return f'{escape(use_case)} Use case is not available', 400
provider_data = use_case_data.get(provider) if provider_data is None: return f'{escape(provider)} Provider is not available', 400 mode = provider_data.get(mode) if mode is None: return f'{escape(mode)} Mode is not available', 400 return mode, 200@app.route("/<use_case>//", methods=['POST']) @verify_auth_header(AUTH_HEADER_KEY, AUTH_HEADER) async def transformer(use_case, provider, mode): """ Returns the translation for the given tex; provider and mode are as mentioned in the repository""" start_time = time.time()
model_config = get_model_config(use_case, provider, mode) if model_config[1] != 200: return model_config model_class_name = model_config[0].get('model_class') model_request_class_name = model_config[0].get('request_class') module = importlib.import_module("src" + "." + use_case + "." + provider + "." + mode) model = getattr(module, model_class_name)(app) model_request = getattr(module, model_request_class_name) request_class = json_to_object(model_request, json.dumps(await request.json)) if model_config[0].get("__is_async"): response = await model.inference(request_class) else: response = model.inference(request_class) end_time = time.time() response_time_ms = int((end_time - start_time) * 1000) headers = {"ai-tools-response-time": str(response_time_ms)} return Response(response, headers=headers)@app.before_serving async def startup(): """ Startup function called before serving requests. """ app.client = await create_client_session() # monitor src if in debug mode if debug_mode: asyncio.get_event_loop().create_task(watch_src_folder(app, watch_dir))
quart --app api --debug run
hypercorn api -b 0.0.0.0:8000
- OPENAI_API_KEY=${OPENAI_API_KEY} - AUTH_HEADER=${AUTH_HEADER} - AUTH_HEADER_KEY=${AUTH_HEADER_KEY}prometheus: image: prom/prometheus ports: - "9090:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml gitpod: image: gitpod/workspace-full ports: - "8888:8888" volumes: - ./workspace:/workspace
-
- Assistant
Based on the provided Github files, I will provide an updated implementation guide for adding a new model to the AI Toolchain, testing it locally, and deploying it using Docker Compose. This will allow users to access APIs associated with any model.
1. Adding a New Model to the AI Toolchain
-
First, create a new directory for your model within the
srcfolder in the AI Toolchain repository and name it according to your model's purpose. For example, if you're adding a sentiment analysis model with a specificproviderandmode, the folder should be located atsrc/sentiment_analysis/provider/mode/. -
Within the new directory, add the following files:
-
model.py: This file should contain the implementation of the model using a popular machine learning framework like TensorFlow, PyTorch, or scikit-learn. It should include methods for training, evaluating, and predicting. -
requirements.txt: List any Python dependencies required by your model.
-
-
Create a new
Dockerfilein your model's directory to containerize the model. The file should include instructions to install dependencies and copy relevant files to the container. Here's an example:FROM python:3.9-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY model.py . CMD ["python", "model.py"] -
Update the
repository_data.jsonfile to include an entry for your new model with the corresponding information.
2. Testing the Model Locally with Docker Compose
-
Add a new service entry for your model in the existing
docker-compose.ymlfile by following this example:sentiment_analysis: build: context: ./src/sentiment_analysis/provider/mode/ dockerfile: Dockerfile ports: - "8001:8000" environment: - PYTHONUNBUFFERED=1 - PYTHONDONTWRITEBYTECODE=1Replace
./src/sentiment_analysis/provider/mode/with the path to your model's directory and adjust the port numbers accordingly. -
Run
docker-compose upto build and run the containers for the main app and your new model. -
Test your model's API locally using a tool like curl or Postman, connecting to
http://localhost:8000/use_case/provider/mode.
3. Deploying with Docker Compose
When you're ready to deploy your model, you can use the existing
docker-compose.ymlfile to deploy the entire AI Toolchain stack, including your new model.-
Push your model's Docker image to a container registry:
docker-compose build sentiment_analysis docker tag sentiment_analysis:latest your-container-registry-url/sentiment_analysis:latest docker push your-container-registry-url/sentiment_analysis:latest -
Update the
docker-compose.ymlfile in your deployment environment with the new service entries for your model, as was done for testing locally. -
Run
docker-compose up -din your deployment environment to start all services, including your new model.
4. Accessing the Model's APIs
Once the AI Toolchain stack is deployed using Docker Compose, users can access the APIs associated with each model by calling the main app's API at
http://your.deployment.url/use_case/provider/mode.For example, if the AI Toolchain is deployed at
http://your.deployment.url, and your new model is a sentiment analysis model at the endpoint/sentiment_analysis/provider/mode, users can access the API by sending requests tohttp://your.deployment.url/sentiment_analysis/provider/mode.By following these steps, you can add new models to the AI Toolchain, test them locally, and deploy them using Docker Compose, allowing users to access the APIs associated with any model.
-
- User
Now as a user i have a my model please explain how im supose the get the api without thinking about deployments using above method
- Assistant
As a user, once your model is added to the AI Toolchain following the above method, you can access the API without worrying about deployments by using the main app's API at
http://your.deployment.url/use_case/provider/mode.Here's a step-by-step guide to access the API for your added model:
-
Ensure that the model is added to the
repository_data.jsonfile as mentioned in step 4 of "Adding a New Model to the AI Toolchain". This makes your model accessible through the main app's endpoint. -
When the AI Toolchain is deployed using the provided Docker Compose configuration, the main app's API is exposed at port 8000 (
http://your.deployment.url:8000). The API accepts requests at the endpoint/use_case/provider/modeand directs the processing of the request to the corresponding model added to the toolchain. -
Now, you can send a request to your model's API using the main app's endpoint. For example, if your model is a sentiment analysis model with a specific
providerandmode, you would access the API by sending requests tohttp://your.deployment.url:8000/sentiment_analysis/provider/mode.You can use any HTTP client library, curl, or tools like Postman to send requests and interact with your model's API.
The AI Toolchain provides a unified API to access various models without worrying about deployments. Once your model is integrated into the AI Toolchain, you just need to send requests to the main app's endpoint with the correct path parameters to access your model's API.
-