Skip to content
TheChatArchiveThe Chat Archive
Qwenqwen3-235b-a22b· Alibaba (Qwen)Hugging Face LMArena Human Preference 140K

an agent that can search the internet, read documents and chunk the documents to…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Mistralmistral-medium-2505’s answer.
  1. User

    an agent that can search the internet, read documents and chunk the documents to embeddings

  2. Qwen Assistant

    To create an agent that can search the web, process documents, and generate embeddings, you can combine several tools and libraries. Below is a structured approach with code examples and implementation guidance:


    1. Internet Search

    Use APIs or web scraping to retrieve documents:

    Tools:

    • Google Programmable Search Engine (PSE) or SerpAPI: For reliable search results.
    • Requests + BeautifulSoup: For basic web scraping (handle paywalls/CAPTCHAs carefully).
    • Playwright/Puppeteer: For dynamic JavaScript-rendered content.

    Example (SerpAPI):

    import os
    from serpapi import GoogleSearch
    
    def search_web(query):
        params = {
            "q": query,
            "api_key": os.getenv("SERPAPI_KEY")
        }
        search = GoogleSearch(params)
        results = search.get_dict()
        return [link["link"] for link in results.get("organic_results", [])]
    

    2. Document Parsing

    Extract text from URLs or files (PDF, DOCX, HTML, etc.):

    Tools:

    • PyPDF2: PDF parsing.
    • python-docx: Word documents.
    • BeautifulSoup: HTML parsing.
    • Unstructured.io: Multi-format document parsing.

    Example (HTML + PDF Parsing):

    import requests
    from bs4 import BeautifulSoup
    from PyPDF2 import PdfReader
    from io import BytesIO
    
    def parse_document(url):
        response = requests.get(url)
        if url.endswith(".pdf"):
            pdf = PdfReader(BytesIO(response.content))
            return "\n".join(page.extract_text() for page in pdf.pages)
        else:
            soup = BeautifulSoup(response.text, "html.parser")
            return soup.get_text()
    

    3. Text Chunking

    Split documents into smaller, context-aware chunks:

    Tools:

    • LangChain’s RecursiveTextSplitter: Splits text while preserving sentence boundaries.

    Example:

    from langchain.text_splitter import RecursiveCharacterTextSplitter
    
    def chunk_text(text, chunk_size=1000, chunk_overlap=200):
        splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size,
            chunk_overlap=chunk_overlap
        )
        return splitter.split_text(text)
    

    4. Embeddings Generation

    Convert chunks into vector embeddings:

    Tools:

    • OpenAI API: High-quality embeddings (text-embedding-ada-002).
    • Sentence Transformers: Open-source models (e.g., all-MiniLM-L6-v2).

    Example (OpenAI):

    import openai
    
    def get_embeddings(chunks, model="text-embedding-ada-002"):
        embeddings = []
        for chunk in chunks:
            response = openai.Embedding.create(input=chunk, model=model)
            embeddings.append(response["data"][0]["embedding"])
        return embeddings
    

    Example (Local, Free):

    from sentence_transformers import SentenceTransformer
    
    model = SentenceTransformer("all-MiniLM-L6-v2")
    def get_embeddings_local(chunks):
        return model.encode(chunks)
    

    5. Putting It All Together

    Integrate components into a pipeline:

    def run_agent(query):
        # Step 1: Search the web
        urls = search_web(query)
        
        all_embeddings = []
        for url in urls:
            try:
                # Step 2: Parse document
                text = parse_document(url)
                
                # Step 3: Chunk text
                chunks = chunk_text(text)
                
                # Step 4: Generate embeddings
                embeddings = get_embeddings_local(chunks)  # or get_embeddings()
                all_embeddings.extend(embeddings)
            except Exception as e:
                print(f"Error processing {url}: {e}")
        
        return all_embeddings
    

    6. Optional Enhancements

    • Storage: Use FAISS, ChromaDB, or Pinecone to store embeddings.
    • Caching: Cache search results or parsed documents to avoid redundant work.
    • Error Handling: Add retries for failed URLs or rate-limited APIs.
    • Security: Use local embeddings for sensitive data (avoid sending to OpenAI).

    Example Usage

    query = "climate change research 2023"
    embeddings = run_agent(query)
    print(f"Generated {len(embeddings)} embeddings.")
    

    Key Considerations

    • Chunk Size: Adjust based on use case (smaller for precision, larger for context).
    • API Costs: OpenAI embeddings cost ~$0.0004/1K tokens.
    • Ethics: Respect robots.txt and terms of service when scraping.

    This agent can power applications like question-answering systems, RAG (Retrieval-Augmented Generation), or semantic search tools.

  3. User

    I have access to model in bedrock by AWS

  4. Qwen Assistant

    If you have access to AWS Bedrock, you can leverage its foundation models (e.g., Amazon Titan, Anthropic Claude, Meta Llama, etc.) to build your agent. Below is an updated implementation that integrates AWS Bedrock for embeddings and text generation, while retaining the core pipeline:


    Key Changes for AWS Bedrock

    1. Use Amazon Titan Embeddings for document chunk embeddings.
    2. Use Claude or Llama for text generation (optional).
    3. Replace OpenAI/Sentence Transformers with Bedrock API calls.
    4. Use AWS SDK (boto3) for Bedrock integration.

    Step-by-Step Implementation

    1. Install Dependencies

    pip install boto3 langchain unstructured[all-docs] requests beautifulsoup4 pypdf
    

    2. Configure AWS Credentials

    Ensure your AWS credentials are set up via environment variables or ~/.aws/credentials.

    3. Bedrock Embeddings Setup

    import boto3
    import json
    
    def get_bedrock_client():
        return boto3.client(
            service_name="bedrock-runtime",
            region_name="us-east-1"  # Replace with your AWS region
        )
    
    def get_titan_embeddings(chunks, client):
        embeddings = []
        for chunk in chunks:
            body = json.dumps({
                "inputText": chunk
            })
            response = client.invoke_model(
                modelId="amazon.titan-embed-text-v1",
                body=body
            )
            embeddings.append(json.loads(response["body"].read())["embedding"])
        return embeddings
    

    4. Updated Pipeline

    def run_bedrock_agent(query):
        # Step 1: Search the web
        urls = search_web(query)  # Reuse the search_web function from earlier
        
        # Step 2: Initialize Bedrock client
        bedrock_client = get_bedrock_client()
        
        all_embeddings = []
        for url in urls:
            try:
                # Step 3: Parse document
                text = parse_document(url)  # Reuse parse_document from earlier
                
                # Step 4: Chunk text
                chunks = chunk_text(text)  # Reuse chunk_text from earlier
                
                # Step 5: Generate embeddings using Amazon Titan
                embeddings = get_titan_embeddings(chunks, bedrock_client)
                all_embeddings.extend(embeddings)
            except Exception as e:
                print(f"Error processing {url}: {e}")
        
        return all_embeddings
    

    Example Usage

    query = "climate change research 2023"
    embeddings = run_bedrock_agent(query)
    print(f"Generated {len(embeddings)} embeddings.")
    

    Key Benefits of Using AWS Bedrock

    1. No API Key Management: Integrated with AWS IAM for secure access.
    2. Scalability: Fully managed service with automatic scaling.
    3. Cost-Effective: Pay only for what you use, with no upfront costs.
    4. Model Flexibility: Switch between models (Titan, Claude, Llama, etc.) via API.

    Optional: Use Bedrock for Text Generation

    If you need to summarize or process text (e.g., answer questions), use models like Claude:

    def generate_with_claude(prompt, client):
        body = json.dumps({
            "prompt": prompt,
            "max_tokens_to_sample": 300
        })
        response = client.invoke_model(
            modelId="anthropic.claude-v2",
            body=body
        )
        return json.loads(response["body"].read())["completion"]
    

    Important Notes

    • Model IDs: Replace "amazon.titan-embed-text-v1" with the correct ID for your use case.
    • Rate Limits: AWS Bedrock has quotas; handle throttling with retries.
    • Text Length: Titan Embeddings supports up to 8192 tokens per input.

    Next Steps

    • Store embeddings in a vector database (e.g., FAISS, Pinecone, or Amazon OpenSearch).
    • Build a RAG (Retrieval-Augmented Generation) pipeline using Bedrock models.
    • Add caching for search results or embeddings to reduce costs.

    Let me know if you need help integrating Bedrock with a vector database or building a full RAG application!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.