app.py import os import json import streamlit as st try: from google import genai from…
- User
app.py
import os import json import streamlit as st
try: from google import genai from google.genai import types except ImportError: genai = None types = None
--------------------- Gemini Client ---------------------
class GeminiClient: def init(self): self.project = "" self.location = ""
if not self.project: st.error("Set GOOGLE_CLOUD_PROJECT environment variable before running") st.stop() if genai is None: st.error("Please install the google-generativeai package to use Gemini model") st.stop() self.client = genai.Client( vertexai=True, project=self.project, location=self.location, ) def generate(self, system_prompt: str, user_content: str) -> str: """ Generate chat completion using Vertex AI Gemini model. Since Vertex AI Gemini doesn't support 'chat.completions' yet as in OpenAI, we simulate a chat prompt by concatenating system and user messages. """ # Combine prompts into one message full_message = system_prompt + "\n\n" + user_content # Call the model: use generate_content, note the model name may require update response = self.client.models.generate_content( model='gemini-2.0-flash-001', contents=full_message ) # Return the generated text return response.candidates[0].content.parts[0].text--------------------- FUNCTIONS ---------------------
def generate_summary(client: GeminiClient, data: dict) -> str:
system_prompt = """Analyze the JSON order activity log below and create a structured summary. Focus on:
1. Final Outcome: Clearly state if the order was Cancelled or Returned or anything else (with reasons from the data and outcome and last recorded activity).
2. Order Timeline Overview (creation timestamp, final status, key users/teams involved)
3. Critical Events: Chronological breakdown of(give in the sturcture as provided):
- Fraud checks (declines, locks, emails)
- Payment actions (authorization, voids)
- Cancellation/return triggers (e.g., "Order Cancelled" descriptions)
- Post-resolution access (e.g., coaching/reviews).
4. Key Notes (patterns, automation flags, unusual access patterns, or notable reasons)
5. Key Evidence: Specific JSON entries proving the outcome (e.g., "Cancelled due to DECLINED: Fraud").
Structure Requirements:
- Use markdown headers and bullet points.
- Differentiate between system automation (e.g., "DOT COM") and human actions.
- Group repeated events (e.g., multiple "Coaching" logs).
- Highlight the exact reason for cancellation/return from the "text" or "description" fields.
Rules:
- If "Cancelled" appears in any "description" or "text" field, declare it as the outcome.
- If "Returned" is not explicitly mentioned, assume cancellation.
- Never guess – use only the provided JSON for conclusions."""
system_prompt = """Analyze the JSON order activity log below and produce a structured, detailed summary report strictly based on the data provided. Your analysis must follow these guidelines and structure:1. Final Outcome
- Clearly state the final status of the order: Cancelled, Returned, or another status.
- Base the conclusion solely on explicit mentions of "Cancelled" or "Returned" found in any "description" or "text" fields.
- If "Returned" appears, specify the exact reason for the return as stated in the JSON.
- If "Cancelled" is found and "Returned" is not, declare the order Cancelled.
- Do not infer or guess beyond the explicit data.
2. Order Timeline Overview
- Provide the order creation timestamp, formatted as MM/DD/YYYY.
- State the final status of the order.
- List the key users, teams, or system automation agents involved, differentiating human actors from automated systems (e.g., "DOT COM").
- Briefly summarize the main flow of actions/events across the order lifecycle.
3. Critical Events (Chronological Breakdown)
Organize and list events chronologically by date (MM/DD/YYYY), grouped as follows:
-
Fraud Checks
- Include all declines, fraud holds, account locks, fraud alert emails, or related system flags.
- Specify whether each event was initiated by system automation or a human user.
-
Payment Actions
- List events such as authorizations, voids, failed transactions, chargebacks, or refunds.
-
Customer Issues Reported
- Document all customer-reported issues, including dates and descriptions as recorded in the JSON.
-
Claims and Risk Reviews
- Detail any claims raised, risk assessment activities, or reviews by risk or claims teams.
-
Cancellation/Return Triggers
- Highlight the exact events that triggered cancellation or return, quoting relevant "description" or "text" fields.
-
Post-resolution Access
- Include coaching, user reviews, follow-up notes, or other post-resolution activity logs.
- Group multiple coaching or review events logically together.
4. Key Notes
- Identify any suspicious patterns such as:
- Frequent or unusual automated system flags.
- Multiple coaching sessions or repeated reviews.
- Any unusual or notable user behaviors or system interventions.
- Call out any automation flags and note if events are clearly system-generated vs. human actions.
5. Key Evidence
- Provide specific JSON entries verbatim (quoting "description" or "text") that directly support your conclusion about the final outcome.
- Clearly highlight the cancellation or return reason stated within these entries.
Formatting Requirements:
- Use Markdown headers and bullet points for readability.
- Differentiate system automation actions (e.g., "DOT COM," "System") from human operator actions (specific users or team names).
- Use normal date format (MM/DD/YYYY) consistently for all timestamps.
- Group repeated events under combined bullet points with summaries.
Important Rules:
- If any "description" or "text" field contains "Cancelled", classify the order as Cancelled.
- If "Returned" is present with reasons, classify order as Returned and provide the explicit reason.
- Avoid assumptions and only conclude based on explicit JSON data.
Use this prompt verbatim when processing JSON order activity logs to generate insightful, fact-based summaries highlighting order lifecycle, customer interactions, claims and risk assessments, fraud and payment activities, and the conclusive final outcome. """
user_content = json.dumps(data, indent=2) try: return client.generate(system_prompt=system_prompt, user_content=user_content) except Exception as e: return f"Error generating summary: {str(e)}"--------------------- STREAMLIT UI ---------------------
st.set_page_config("Macy's Notes Analyzer with Vertex AI Gemini", layout="wide") st.title("📑 Macy's Notes Analysis — Direct LLM Processing (Vertex AI Gemini)")
uploaded_file = st.file_uploader("Upload Macy's JSON Notes", type="json")
if uploaded_file: try: data = json.load(uploaded_file)
client = GeminiClient() # Initialize Vertex AI client with st.spinner("🔍 Analyzing documents..."): summary = generate_summary(client, data) st.success("✅ Analysis Complete!") # Display formatted summary st.markdown("## 📝 Comprehensive Summary") st.markdown(summary) except json.JSONDecodeError: st.error("❌ Invalid JSON file format") except Exception as e: st.error(f"❌ Processing error: {str(e)}")Optional: Add original data preview
if uploaded_file and st.checkbox("Show raw data preview"): uploaded_file.seek(0) st.json(json.load(uploaded_file))
in the code can you tell me how many token we are sending and how many tokes we are getting on frontend itself also tell me time to get the response
- Assistant
To measure tokens sent/received and timing in your Streamlit app with the Gemini API, you need to:
- Calculate token counts for your prompt (
system_prompt + user_content) before sending it. - Count tokens in the response.
- Measure the time taken for the API call.
How to do it?
Your current code uses Google Gemini API via
genai.Client. Unfortunately, Google's Gemini API doesn't provide token usage info directly in its response metadata, unlike the OpenAI API.So, you have to estimate tokens yourself client-side, both in prompt and response, using a tokenizer compatible with Gemini's model (likely similar to PaLM or BPE tokenizers), or approximate tokens by text length / word count.
Step 1: Add timing measurement
import timeAround the call:
start_time = time.time() response = self.client.models.generate_content( model='gemini-2.0-flash-001', contents=full_message ) end_time = time.time() self.last_response_time = end_time - start_timeStore the duration to show in Streamlit.
Step 2: Estimate tokens usage (approximate)
Without an official tokenizer from Google, simplest approximation:
- Use the number of words or characters / 4 (roughly 4 characters = 1 token).
- Or install tiktoken or similar for better estimates.
For example:
def estimate_tokens(text: str) -> int: # Approximate: 1 token = 4 chars return max(1, len(text) // 4)
Step 3: Modify GeminiClient to store tokens info
In
GeminiClient:class GeminiClient: def __init__(self): ... self.last_request_tokens = 0 self.last_response_tokens = 0 self.last_response_time = 0.0 def generate(self, system_prompt: str, user_content: str) -> str: full_message = system_prompt + "\n\n" + user_content # Estimate tokens in prompt self.last_request_tokens = estimate_tokens(full_message) start_time = time.time() response = self.client.models.generate_content( model='gemini-2.0-flash-001', contents=full_message ) end_time = time.time() self.last_response_time = end_time - start_time # Extract generated text text = response.candidates[0].content.parts[0].text # Estimate tokens in response self.last_response_tokens = estimate_tokens(text) return text
Step 4: Show token counts and timing in Streamlit UI
In your main Streamlit flow, after
summary = generate_summary(client, data):st.markdown(f"**Prompt tokens:** {client.last_request_tokens}") st.markdown(f"**Response tokens:** {client.last_response_tokens}") st.markdown(f"**Response time:** {client.last_response_time:.2f} seconds")
Summary
- Gemini API does not expose real token counts, so estimate with a simple heuristic (chars/4).
- Add timing around the API call with
time.time(). - Store tokens and timing inside the client for display.
- Show these metrics next to the summary in Streamlit frontend.
If you want better token counting, you can try using the
tiktokenpackage's tokenizer for an OpenAI-compatible model or find Google's official tokenizer for Gemini (if available). But heuristic counting is often good enough for basic monitoring. - Calculate token counts for your prompt (
- User
give me full code
- Assistant
Certainly! Here's your full updated
app.pywith added token counting (approximate) and timing measurement displayed in the Streamlit UI.# app.py import os import json import time import streamlit as st try: from google import genai from google.genai import types except ImportError: genai = None types = None # Simple approximate token counter: 1 token ≈ 4 chars def estimate_tokens(text: str) -> int: return max(1, len(text) // 4) # --------------------- Gemini Client --------------------- class GeminiClient: def __init__(self): self.project = "" self.location = "" # You can alternatively read your project and location from env vars here: if not self.project: self.project = os.getenv("GOOGLE_CLOUD_PROJECT", "") if not self.location: self.location = os.getenv("GOOGLE_CLOUD_LOCATION", "us-central1") # Provide a default if desired if not self.project: st.error("Set GOOGLE_CLOUD_PROJECT environment variable before running") st.stop() if genai is None: st.error("Please install the google-generativeai package to use Gemini model") st.stop() self.client = genai.Client( vertexai=True, project=self.project, location=self.location, ) # Store last request/response tokens and time for frontend display self.last_request_tokens = 0 self.last_response_tokens = 0 self.last_response_time = 0.0 def generate(self, system_prompt: str, user_content: str) -> str: """ Generate chat completion using Vertex AI Gemini model. Since Vertex AI Gemini doesn't support 'chat.completions' yet as in OpenAI, we simulate a chat prompt by concatenating system and user messages. """ # Combine prompts into one message full_message = system_prompt + "\n\n" + user_content # Estimate tokens in prompt self.last_request_tokens = estimate_tokens(full_message) # Call the model: start_time = time.time() response = self.client.models.generate_content( model='gemini-2.0-flash-001', contents=full_message ) end_time = time.time() self.last_response_time = end_time - start_time # Extract generated text text = response.candidates[0].content.parts[0].text # Estimate tokens in response self.last_response_tokens = estimate_tokens(text) return text # --------------------- FUNCTIONS --------------------- def generate_summary(client: GeminiClient, data: dict) -> str: system_prompt = """Analyze the JSON order activity log below and produce a structured, detailed summary report strictly based on the data provided. Your analysis must follow these guidelines and structure: # 1. Final Outcome - Clearly state the final status of the order: **Cancelled**, **Returned**, or another status. - Base the conclusion solely on explicit mentions of "Cancelled" or "Returned" found in any "description" or "text" fields. - If "Returned" appears, specify the exact reason for the return as stated in the JSON. - If "Cancelled" is found and "Returned" is not, declare the order Cancelled. - Do not infer or guess beyond the explicit data. # 2. Order Timeline Overview - Provide the **order creation timestamp**, formatted as MM/DD/YYYY. - State the **final status** of the order. - List the **key users, teams, or system automation agents** involved, differentiating human actors from automated systems (e.g., "DOT COM"). - Briefly summarize the main flow of actions/events across the order lifecycle. # 3. Critical Events (Chronological Breakdown) Organize and list events chronologically by date (MM/DD/YYYY), grouped as follows: - **Fraud Checks** - Include all declines, fraud holds, account locks, fraud alert emails, or related system flags. - Specify whether each event was initiated by system automation or a human user. - **Payment Actions** - List events such as authorizations, voids, failed transactions, chargebacks, or refunds. - **Customer Issues Reported** - Document all customer-reported issues, including dates and descriptions as recorded in the JSON. - **Claims and Risk Reviews** - Detail any claims raised, risk assessment activities, or reviews by risk or claims teams. - **Cancellation/Return Triggers** - Highlight the exact events that triggered cancellation or return, quoting relevant "description" or "text" fields. - **Post-resolution Access** - Include coaching, user reviews, follow-up notes, or other post-resolution activity logs. - Group multiple coaching or review events logically together. # 4. Key Notes - Identify any suspicious patterns such as: - Frequent or unusual automated system flags. - Multiple coaching sessions or repeated reviews. - Any unusual or notable user behaviors or system interventions. - Call out any automation flags and note if events are clearly system-generated vs. human actions. # 5. Key Evidence - Provide specific JSON entries verbatim (quoting "description" or "text") that directly support your conclusion about the final outcome. - Clearly highlight the cancellation or return reason stated within these entries. --- # Formatting Requirements: - Use Markdown headers and bullet points for readability. - Differentiate system automation actions (e.g., "DOT COM," "System") from human operator actions (specific users or team names). - Use normal date format (MM/DD/YYYY) consistently for all timestamps. - Group repeated events under combined bullet points with summaries. --- # Important Rules: - If any "description" or "text" field contains **"Cancelled"**, classify the order as **Cancelled**. - If "Returned" is present with reasons, classify order as **Returned** and provide the explicit reason. - Avoid assumptions and only conclude based on explicit JSON data. --- Use this prompt verbatim when processing JSON order activity logs to generate insightful, fact-based summaries highlighting order lifecycle, customer interactions, claims and risk assessments, fraud and payment activities, and the conclusive final outcome. """ user_content = json.dumps(data, indent=2) try: return client.generate(system_prompt=system_prompt, user_content=user_content) except Exception as e: return f"Error generating summary: {str(e)}" # --------------------- STREAMLIT UI --------------------- st.set_page_config("Macy's Notes Analyzer with Vertex AI Gemini", layout="wide") st.title("📑 Macy's Notes Analysis — Direct LLM Processing (Vertex AI Gemini)") uploaded_file = st.file_uploader("Upload Macy's JSON Notes", type="json") if uploaded_file: try: data = json.load(uploaded_file) client = GeminiClient() # Initialize Vertex AI client with st.spinner("🔍 Analyzing documents..."): summary = generate_summary(client, data) st.success("✅ Analysis Complete!") # Display tokens and time info st.markdown(f"**Prompt tokens (approx.):** {client.last_request_tokens}") st.markdown(f"**Response tokens (approx.):** {client.last_response_tokens}") st.markdown(f"**Response time:** {client.last_response_time:.2f} seconds") # Display formatted summary st.markdown("## 📝 Comprehensive Summary") st.markdown(summary) except json.JSONDecodeError: st.error("❌ Invalid JSON file format") except Exception as e: st.error(f"❌ Processing error: {str(e)}") # Optional: Add original data preview if uploaded_file and st.checkbox("Show raw data preview"): uploaded_file.seek(0) st.json(json.load(uploaded_file))
What changed?
- Added
estimate_tokens()function for approximate counting. - In
GeminiClient.generate(), added:- Token count estimate for prompt and response.
- Timing of the request.
- Displayed these stats right before summary.
- Reading
GOOGLE_CLOUD_PROJECTand optional location from environment, fallback if needed. - Kept the rest of your app intact.
Notes
- This is an approximation since Gemini currently does not provide direct token usage info.
- You can refine token estimation by integrating suitable tokenizer libraries if you find Google's or a compatible tokenizer.
- Adjust environment variable usage as needed for your deployment.
Let me know if you want the code for precise token counting or any other additions!
- Added
- User
use this from vertexai.preview import tokenization
- Assistant
Got it! You want to replace the approximate token counting with the tokenizer from
vertexai.preview.tokenizationwhich is the official way to count tokens for Vertex AI Gemini models.Here’s your full
app.pyupdated to use that tokenizer for precise token counting of prompt and response, and still measure response time.
# app.py import os import json import time import streamlit as st try: from google import genai from google.genai import types from vertexai.preview import tokenization except ImportError: genai = None types = None tokenization = None # --------------------- Gemini Client --------------------- class GeminiClient: def __init__(self): self.project = os.getenv("GOOGLE_CLOUD_PROJECT", "") self.location = os.getenv("GOOGLE_CLOUD_LOCATION", "us-central1") if not self.project: st.error("Set GOOGLE_CLOUD_PROJECT environment variable before running") st.stop() if genai is None: st.error("Please install the google-generativeai package to use Gemini model") st.stop() if tokenization is None: st.error("Please install vertexai package to use tokenizer") st.stop() self.client = genai.Client( vertexai=True, project=self.project, location=self.location, ) # Initialize tokenizer for model gemini-2.0-flash-001 # Adjust model name if using different model self.tokenizer = tokenization.Tokenizer.for_model("gemini-2.0-flash-001") # Store last request/response tokens and time self.last_request_tokens = 0 self.last_response_tokens = 0 self.last_response_time = 0.0 def generate(self, system_prompt: str, user_content: str) -> str: # Combine prompts into one message full_message = system_prompt + "\n\n" + user_content # Count tokens precisely self.last_request_tokens = len(self.tokenizer.encode(full_message)) start_time = time.time() response = self.client.models.generate_content( model='gemini-2.0-flash-001', contents=full_message ) end_time = time.time() self.last_response_time = end_time - start_time text = response.candidates[0].content.parts[0].text # Count tokens in response self.last_response_tokens = len(self.tokenizer.encode(text)) return text # --------------------- FUNCTIONS --------------------- def generate_summary(client: GeminiClient, data: dict) -> str: system_prompt = """Analyze the JSON order activity log below and produce a structured, detailed summary report strictly based on the data provided. Your analysis must follow these guidelines and structure: # 1. Final Outcome - Clearly state the final status of the order: **Cancelled**, **Returned**, or another status. - Base the conclusion solely on explicit mentions of "Cancelled" or "Returned" found in any "description" or "text" fields. - If "Returned" appears, specify the exact reason for the return as stated in the JSON. - If "Cancelled" is found and "Returned" is not, declare the order Cancelled. - Do not infer or guess beyond the explicit data. # 2. Order Timeline Overview - Provide the **order creation timestamp**, formatted as MM/DD/YYYY. - State the **final status** of the order. - List the **key users, teams, or system automation agents** involved, differentiating human actors from automated systems (e.g., "DOT COM"). - Briefly summarize the main flow of actions/events across the order lifecycle. # 3. Critical Events (Chronological Breakdown) Organize and list events chronologically by date (MM/DD/YYYY), grouped as follows: - **Fraud Checks** - Include all declines, fraud holds, account locks, fraud alert emails, or related system flags. - Specify whether each event was initiated by system automation or a human user. - **Payment Actions** - List events such as authorizations, voids, failed transactions, chargebacks, or refunds. - **Customer Issues Reported** - Document all customer-reported issues, including dates and descriptions as recorded in the JSON. - **Claims and Risk Reviews** - Detail any claims raised, risk assessment activities, or reviews by risk or claims teams. - **Cancellation/Return Triggers** - Highlight the exact events that triggered cancellation or return, quoting relevant "description" or "text" fields. - **Post-resolution Access** - Include coaching, user reviews, follow-up notes, or other post-resolution activity logs. - Group multiple coaching or review events logically together. # 4. Key Notes - Identify any suspicious patterns such as: - Frequent or unusual automated system flags. - Multiple coaching sessions or repeated reviews. - Any unusual or notable user behaviors or system interventions. - Call out any automation flags and note if events are clearly system-generated vs. human actions. # 5. Key Evidence - Provide specific JSON entries verbatim (quoting "description" or "text") that directly support your conclusion about the final outcome. - Clearly highlight the cancellation or return reason stated within these entries. --- # Formatting Requirements: - Use Markdown headers and bullet points for readability. - Differentiate system automation actions (e.g., "DOT COM," "System") from human operator actions (specific users or team names). - Use normal date format (MM/DD/YYYY) consistently for all timestamps. - Group repeated events under combined bullet points with summaries. --- # Important Rules: - If any "description" or "text" field contains **"Cancelled"**, classify the order as **Cancelled**. - If "Returned" is present with reasons, classify order as **Returned** and provide the explicit reason. - Avoid assumptions and only conclude based on explicit JSON data. --- Use this prompt verbatim when processing JSON order activity logs to generate insightful, fact-based summaries highlighting order lifecycle, customer interactions, claims and risk assessments, fraud and payment activities, and the conclusive final outcome. """ user_content = json.dumps(data, indent=2) try: return client.generate(system_prompt=system_prompt, user_content=user_content) except Exception as e: return f"Error generating summary: {str(e)}" # --------------------- STREAMLIT UI --------------------- st.set_page_config("Macy's Notes Analyzer with Vertex AI Gemini", layout="wide") st.title("📑 Macy's Notes Analysis — Direct LLM Processing (Vertex AI Gemini)") uploaded_file = st.file_uploader("Upload Macy's JSON Notes", type="json") if uploaded_file: try: data = json.load(uploaded_file) client = GeminiClient() # Initialize Vertex AI client with st.spinner("🔍 Analyzing documents..."): summary = generate_summary(client, data) st.success("✅ Analysis Complete!") # Display tokens and time info st.markdown(f"**Prompt tokens:** {client.last_request_tokens}") st.markdown(f"**Response tokens:** {client.last_response_tokens}") st.markdown(f"**Response time:** {client.last_response_time:.2f} seconds") # Display formatted summary st.markdown("## 📝 Comprehensive Summary") st.markdown(summary) except json.JSONDecodeError: st.error("❌ Invalid JSON file format") except Exception as e: st.error(f"❌ Processing error: {str(e)}") # Optional: Add original data preview if uploaded_file and st.checkbox("Show raw data preview"): uploaded_file.seek(0) st.json(json.load(uploaded_file))
Notes:
- Used
tokenization.Tokenizer.for_model("gemini-2.0-flash-001")to get the proper tokenizer. - Token counts for both prompt and response are exact according to the tokenizer.
- Response time measured with
time.time(). - Added error handling if
vertexai.preview.tokenizationis not installed. - Make sure you have installed the right package:
pip install vertexai google-generativeai
Let me know if you want me to help adapt this to handle streaming or multiple messages or anything else!
- Used
- User
is it counting token again have some costing to it
- Assistant
Great question! Here's the info about token counting and cost in your scenario:
Does token counting with
vertexai.preview.tokenizationincur any cost?No, token counting by itself is a local, offline operation performed by your client code. The tokenizer just processes your prompt and response strings locally — it does NOT make any API calls and does not incur any usage cost.
So, using
tokenization.Tokenizer.for_model()to count tokens is free and purely client-side.
Where do costs come from?
Costs are incurred only when you actually call the LLM API service, i.e., when you call:
response = self.client.models.generate_content( model='gemini-2.0-flash-001', contents=full_message )This sends data to the Vertex AI model and results in billed usage based on tokens processed by the model.
Summary
Action Token counting cost? API usage cost? Using tokenization.Tokenizer.encode()No (local, free) None Calling generate_content(...)No Yes (based on tokens)
Additional notes
- Token counting locally is very fast and efficient.
- You should count tokens locally before making calls if you want to monitor or limit usage.
- Using this method won’t add latency or cost beyond what the API call does.
- The token counts you obtain from the tokenizer are estimates of what the API will count for billing.
If you want, I can show you an example of limiting inputs based on token counts or logging usage for cost monitoring!