Skip to content
TheChatArchiveThe Chat Archive
grok-3-preview-02-24· xAIHugging Face LMArena Human Preference 140K

How to use embedings in ollama cli? How can it read pdfs?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-5-sonnet-20241022’s answer.
  1. User

    How to use embedings in ollama cli? How can it read pdfs?

  2. Assistant

    Using embeddings in Ollama CLI and enabling it to read PDFs involves a few steps. Ollama itself is primarily a tool for running LLMs locally, but it doesn't natively handle embeddings or PDF parsing. You'll need to integrate additional tools to achieve your goal. Here's a concise guide:


    1. Using Embeddings in Ollama CLI

    Ollama recently added support for embeddings via its API, but this isn't directly available through the CLI. You'll need to use the API or a framework like LangChain to generate embeddings.

    Steps for Embeddings

    1. Install Ollama
      Ensure Ollama is installed and running locally. Download a model that supports embeddings (e.g., nomic-embed-text or mxbai-embed-large):

      ollama pull nomic-embed-text
      
    2. Generate Embeddings via API
      Use the Ollama API to generate embeddings for text. Example using Python:

      import requests
      
      url = "http://localhost:11434/api/embeddings"
      payload = {
          "model": "nomic-embed-text",
          "prompt": "Your text here"
      }
      response = requests.post(url, json=payload)
      embedding = response.json()["embedding"]
      print(embedding)
      
    3. Store Embeddings
      Use a local vector database like ChromaDB or FAISS to store embeddings for later retrieval.


    2. Reading PDFs

    Ollama doesn't natively read PDFs, so you'll need to extract text from PDFs using a library like PyPDF2, pdfplumber, or pymupdf.

    Steps to Read PDFs

    1. Install a PDF Parser
      Install a Python library for PDF parsing, e.g., pdfplumber:

      pip install pdfplumber
      
    2. Extract Text from PDFs
      Example Python script to extract text:

      import pdfplumber
      
      def extract_text_from_pdf(pdf_path):
          with pdfplumber.open(pdf_path) as pdf:
              text = ""
              for page in pdf.pages:
                  text += page.extract_text()
          return text
      
      pdf_text = extract_text_from_pdf("path/to/your/file.pdf")
      print(pdf_text)
      
    3. Chunk the Text
      Split the extracted text into smaller chunks (e.g., paragraphs or sentences) to make embeddings manageable:

      def chunk_text(text, chunk_size=500):
          words = text.split()
          return [' '.join(words[i:i + chunk_size]) for i in range(0, len(words), chunk_size)]
      
      chunks = chunk_text(pdf_text)
      

    3. Putting It Together (RAG Pipeline)

    To build a Retrieval-Augmented Generation (RAG) system with Ollama, follow these steps:

    1. Extract and Chunk PDF Text
      Use the above code to extract and chunk text from your PDFs.

    2. Generate Embeddings for Chunks
      Use the Ollama API to generate embeddings for each chunk:

      import requests
      
      def get_embedding(text, model="nomic-embed-text"):
          url = "http://localhost:11434/api/embeddings"
          payload = {"model": model, "prompt": text}
          response = requests.post(url, json=payload)
          return response.json()["embedding"]
      
      chunk_embeddings = [get_embedding(chunk) for chunk in chunks]
      
    3. Store Embeddings in a Vector Database
      Use a local vector database like ChromaDB to store embeddings:

      pip install chromadb
      

      Example:

      import chromadb
      
      client = chromadb.Client()
      collection = client.create_collection("pdf_chunks")
      collection.add(
          embeddings=chunk_embeddings,
          documents=chunks,
          ids=[f"chunk_{i}" for i in range(len(chunks))]
      )
      
    4. Query Processing
      When you ask a question, generate an embedding for the query and retrieve relevant chunks:

      query = "What is the main finding of the research?"
      query_embedding = get_embedding(query)
      results = collection.query(query_embeddings=[query_embedding], n_results=3)
      relevant_chunks = results["documents"][0]
      
    5. Generate Answer with Ollama CLI
      Use the retrieved chunks as context and pass them to an LLM via Ollama CLI:

      ollama run llama3 "Here is the context: {relevant_chunks}. Now answer: {query}"
      

      Alternatively, use the API for programmatic access:

      def generate_answer(context, query, model="llama3"):
          url = "http://localhost:11434/api/generate"
          prompt = f"Context: {context}\n\nQuestion: {query}"
          payload = {"model": model, "prompt": prompt}
          response = requests.post(url, json=payload)
          return response.json()["response"]
      
      answer = generate_answer("\n".join(relevant_chunks), query)
      print(answer)
      

    4. Automating with Frameworks

    To simplify this process, use frameworks like LangChain or LlamaIndex, which integrate Ollama, PDF parsing, embeddings, and vector databases seamlessly.

    Example with LangChain

    1. Install LangChain:

      pip install langchain langchain-community
      
    2. Example script:

      from langchain_community.document_loaders import PyPDFLoader
      from langchain_community.embeddings import OllamaEmbeddings
      from langchain_community.vectorstores import Chroma
      from langchain_community.llms import Ollama
      from langchain.chains import RetrievalQA
      
      # Load PDF
      loader = PyPDFLoader("path/to/your/file.pdf")
      documents = loader.load_and_split()
      
      # Generate embeddings
      embeddings = OllamaEmbeddings(model="nomic-embed-text")
      
      # Store in vector database
      vectorstore = Chroma.from_documents(documents, embeddings)
      
      # Set up LLM
      llm = Ollama(model="llama3")
      
      # Create RAG chain
      qa_chain = RetrievalQA.from_chain_type(llm, retriever=vectorstore.as_retriever())
      
      # Ask a question
      query = "What is the main finding of the research?"
      answer = qa_chain.run(query)
      print(answer)
      

    5. Notes

    • Model Selection: Use nomic-embed-text for embeddings and a larger model like llama3 or mistral for answer generation.
    • Hardware: Ensure you have enough RAM/GPU to run both the embedding model and the LLM locally.
    • Performance: For large PDFs, consider pre-processing and indexing embeddings once, then reusing the vector database.

    This setup lets you build a local, PDF-reading chatbot with embeddings and RAG using Ollama! 🚀

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.