an agent that can search the internet, read documents and chunk the documents to…
- User
an agent that can search the internet, read documents and chunk the documents to embeddings
Assistant
To create an agent that can search the web, process documents, and generate embeddings, you can combine several tools and libraries. Below is a structured approach with code examples and implementation guidance:
1. Internet Search
Use APIs or web scraping to retrieve documents:
Tools:
- Google Programmable Search Engine (PSE) or SerpAPI: For reliable search results.
- Requests + BeautifulSoup: For basic web scraping (handle paywalls/CAPTCHAs carefully).
- Playwright/Puppeteer: For dynamic JavaScript-rendered content.
Example (SerpAPI):
import os from serpapi import GoogleSearch def search_web(query): params = { "q": query, "api_key": os.getenv("SERPAPI_KEY") } search = GoogleSearch(params) results = search.get_dict() return [link["link"] for link in results.get("organic_results", [])]
2. Document Parsing
Extract text from URLs or files (PDF, DOCX, HTML, etc.):
Tools:
- PyPDF2: PDF parsing.
- python-docx: Word documents.
- BeautifulSoup: HTML parsing.
- Unstructured.io: Multi-format document parsing.
Example (HTML + PDF Parsing):
import requests from bs4 import BeautifulSoup from PyPDF2 import PdfReader from io import BytesIO def parse_document(url): response = requests.get(url) if url.endswith(".pdf"): pdf = PdfReader(BytesIO(response.content)) return "\n".join(page.extract_text() for page in pdf.pages) else: soup = BeautifulSoup(response.text, "html.parser") return soup.get_text()
3. Text Chunking
Split documents into smaller, context-aware chunks:
Tools:
- LangChain’s RecursiveTextSplitter: Splits text while preserving sentence boundaries.
Example:
from langchain.text_splitter import RecursiveCharacterTextSplitter def chunk_text(text, chunk_size=1000, chunk_overlap=200): splitter = RecursiveCharacterTextSplitter( chunk_size=chunk_size, chunk_overlap=chunk_overlap ) return splitter.split_text(text)
4. Embeddings Generation
Convert chunks into vector embeddings:
Tools:
- OpenAI API: High-quality embeddings (
text-embedding-ada-002). - Sentence Transformers: Open-source models (e.g.,
all-MiniLM-L6-v2).
Example (OpenAI):
import openai def get_embeddings(chunks, model="text-embedding-ada-002"): embeddings = [] for chunk in chunks: response = openai.Embedding.create(input=chunk, model=model) embeddings.append(response["data"][0]["embedding"]) return embeddingsExample (Local, Free):
from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") def get_embeddings_local(chunks): return model.encode(chunks)
5. Putting It All Together
Integrate components into a pipeline:
def run_agent(query): # Step 1: Search the web urls = search_web(query) all_embeddings = [] for url in urls: try: # Step 2: Parse document text = parse_document(url) # Step 3: Chunk text chunks = chunk_text(text) # Step 4: Generate embeddings embeddings = get_embeddings_local(chunks) # or get_embeddings() all_embeddings.extend(embeddings) except Exception as e: print(f"Error processing {url}: {e}") return all_embeddings
6. Optional Enhancements
- Storage: Use FAISS, ChromaDB, or Pinecone to store embeddings.
- Caching: Cache search results or parsed documents to avoid redundant work.
- Error Handling: Add retries for failed URLs or rate-limited APIs.
- Security: Use local embeddings for sensitive data (avoid sending to OpenAI).
Example Usage
query = "climate change research 2023" embeddings = run_agent(query) print(f"Generated {len(embeddings)} embeddings.")
Key Considerations
- Chunk Size: Adjust based on use case (smaller for precision, larger for context).
- API Costs: OpenAI embeddings cost ~$0.0004/1K tokens.
- Ethics: Respect robots.txt and terms of service when scraping.
This agent can power applications like question-answering systems, RAG (Retrieval-Augmented Generation), or semantic search tools.
- User
I have access to model in bedrock by AWS
Assistant
If you have access to AWS Bedrock, you can leverage its foundation models (e.g., Amazon Titan, Anthropic Claude, Meta Llama, etc.) to build your agent. Below is an updated implementation that integrates AWS Bedrock for embeddings and text generation, while retaining the core pipeline:
Key Changes for AWS Bedrock
- Use Amazon Titan Embeddings for document chunk embeddings.
- Use Claude or Llama for text generation (optional).
- Replace OpenAI/Sentence Transformers with Bedrock API calls.
- Use AWS SDK (
boto3) for Bedrock integration.
Step-by-Step Implementation
1. Install Dependencies
pip install boto3 langchain unstructured[all-docs] requests beautifulsoup4 pypdf2. Configure AWS Credentials
Ensure your AWS credentials are set up via environment variables or
~/.aws/credentials.3. Bedrock Embeddings Setup
import boto3 import json def get_bedrock_client(): return boto3.client( service_name="bedrock-runtime", region_name="us-east-1" # Replace with your AWS region ) def get_titan_embeddings(chunks, client): embeddings = [] for chunk in chunks: body = json.dumps({ "inputText": chunk }) response = client.invoke_model( modelId="amazon.titan-embed-text-v1", body=body ) embeddings.append(json.loads(response["body"].read())["embedding"]) return embeddings4. Updated Pipeline
def run_bedrock_agent(query): # Step 1: Search the web urls = search_web(query) # Reuse the search_web function from earlier # Step 2: Initialize Bedrock client bedrock_client = get_bedrock_client() all_embeddings = [] for url in urls: try: # Step 3: Parse document text = parse_document(url) # Reuse parse_document from earlier # Step 4: Chunk text chunks = chunk_text(text) # Reuse chunk_text from earlier # Step 5: Generate embeddings using Amazon Titan embeddings = get_titan_embeddings(chunks, bedrock_client) all_embeddings.extend(embeddings) except Exception as e: print(f"Error processing {url}: {e}") return all_embeddings
Example Usage
query = "climate change research 2023" embeddings = run_bedrock_agent(query) print(f"Generated {len(embeddings)} embeddings.")
Key Benefits of Using AWS Bedrock
- No API Key Management: Integrated with AWS IAM for secure access.
- Scalability: Fully managed service with automatic scaling.
- Cost-Effective: Pay only for what you use, with no upfront costs.
- Model Flexibility: Switch between models (Titan, Claude, Llama, etc.) via API.
Optional: Use Bedrock for Text Generation
If you need to summarize or process text (e.g., answer questions), use models like Claude:
def generate_with_claude(prompt, client): body = json.dumps({ "prompt": prompt, "max_tokens_to_sample": 300 }) response = client.invoke_model( modelId="anthropic.claude-v2", body=body ) return json.loads(response["body"].read())["completion"]
Important Notes
- Model IDs: Replace
"amazon.titan-embed-text-v1"with the correct ID for your use case. - Rate Limits: AWS Bedrock has quotas; handle throttling with retries.
- Text Length: Titan Embeddings supports up to 8192 tokens per input.
Next Steps
- Store embeddings in a vector database (e.g., FAISS, Pinecone, or Amazon OpenSearch).
- Build a RAG (Retrieval-Augmented Generation) pipeline using Bedrock models.
- Add caching for search results or embeddings to reduce costs.
Let me know if you need help integrating Bedrock with a vector database or building a full RAG application!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.