Skip to content
TheChatArchiveThe Chat Archive
gpt-4.1-mini-2025-04-14· OpenAIAi2 WildChat-4.8M (AI2)

app.py import os import json import streamlit as st try: from google import genai from…

  1. User

    app.py

    import os import json import streamlit as st

    try: from google import genai from google.genai import types except ImportError: genai = None types = None

    --------------------- Gemini Client ---------------------

    class GeminiClient: def init(self): self.project = "" self.location = ""

        if not self.project:
            st.error("Set GOOGLE_CLOUD_PROJECT environment variable before running")
            st.stop()
    
        if genai is None:
            st.error("Please install the google-generativeai package to use Gemini model")
            st.stop()
    
        self.client = genai.Client(
            vertexai=True,
            project=self.project,
            location=self.location,
        )
    
    def generate(self, system_prompt: str, user_content: str) -> str:
        """
        Generate chat completion using Vertex AI Gemini model.
        Since Vertex AI Gemini doesn't support 'chat.completions' yet as in OpenAI,
        we simulate a chat prompt by concatenating system and user messages.
        """
        # Combine prompts into one message
        full_message = system_prompt + "\n\n" + user_content
        
        # Call the model: use generate_content, note the model name may require update
        response = self.client.models.generate_content(
            model='gemini-2.0-flash-001',
            contents=full_message
        )
        # Return the generated text
        return response.candidates[0].content.parts[0].text
    

    --------------------- FUNCTIONS ---------------------

    def generate_summary(client: GeminiClient, data: dict) -> str:

    system_prompt = """Analyze the JSON order activity log below and create a structured summary. Focus on:

    1. Final Outcome: Clearly state if the order was Cancelled or Returned or anything else (with reasons from the data and outcome and last recorded activity).

    2. Order Timeline Overview (creation timestamp, final status, key users/teams involved)

    3. Critical Events: Chronological breakdown of(give in the sturcture as provided):

    - Fraud checks (declines, locks, emails)

    - Payment actions (authorization, voids)

    - Cancellation/return triggers (e.g., "Order Cancelled" descriptions)

    - Post-resolution access (e.g., coaching/reviews).

    4. Key Notes (patterns, automation flags, unusual access patterns, or notable reasons)

    5. Key Evidence: Specific JSON entries proving the outcome (e.g., "Cancelled due to DECLINED: Fraud").

    Structure Requirements:

    - Use markdown headers and bullet points.

    - Differentiate between system automation (e.g., "DOT COM") and human actions.

    - Group repeated events (e.g., multiple "Coaching" logs).

    - Highlight the exact reason for cancellation/return from the "text" or "description" fields.

    Rules:

    - If "Cancelled" appears in any "description" or "text" field, declare it as the outcome.

    - If "Returned" is not explicitly mentioned, assume cancellation.

    - Never guess – use only the provided JSON for conclusions."""

    system_prompt = """Analyze the JSON order activity log below and produce a structured, detailed summary report strictly based on the data provided. Your analysis must follow these guidelines and structure:
    

    1. Final Outcome

    • Clearly state the final status of the order: Cancelled, Returned, or another status.
    • Base the conclusion solely on explicit mentions of "Cancelled" or "Returned" found in any "description" or "text" fields.
    • If "Returned" appears, specify the exact reason for the return as stated in the JSON.
    • If "Cancelled" is found and "Returned" is not, declare the order Cancelled.
    • Do not infer or guess beyond the explicit data.

    2. Order Timeline Overview

    • Provide the order creation timestamp, formatted as MM/DD/YYYY.
    • State the final status of the order.
    • List the key users, teams, or system automation agents involved, differentiating human actors from automated systems (e.g., "DOT COM").
    • Briefly summarize the main flow of actions/events across the order lifecycle.

    3. Critical Events (Chronological Breakdown)

    Organize and list events chronologically by date (MM/DD/YYYY), grouped as follows:

    • Fraud Checks

      • Include all declines, fraud holds, account locks, fraud alert emails, or related system flags.
      • Specify whether each event was initiated by system automation or a human user.
    • Payment Actions

      • List events such as authorizations, voids, failed transactions, chargebacks, or refunds.
    • Customer Issues Reported

      • Document all customer-reported issues, including dates and descriptions as recorded in the JSON.
    • Claims and Risk Reviews

      • Detail any claims raised, risk assessment activities, or reviews by risk or claims teams.
    • Cancellation/Return Triggers

      • Highlight the exact events that triggered cancellation or return, quoting relevant "description" or "text" fields.
    • Post-resolution Access

      • Include coaching, user reviews, follow-up notes, or other post-resolution activity logs.
      • Group multiple coaching or review events logically together.

    4. Key Notes

    • Identify any suspicious patterns such as:
      • Frequent or unusual automated system flags.
      • Multiple coaching sessions or repeated reviews.
      • Any unusual or notable user behaviors or system interventions.
    • Call out any automation flags and note if events are clearly system-generated vs. human actions.

    5. Key Evidence

    • Provide specific JSON entries verbatim (quoting "description" or "text") that directly support your conclusion about the final outcome.
    • Clearly highlight the cancellation or return reason stated within these entries.

    Formatting Requirements:

    • Use Markdown headers and bullet points for readability.
    • Differentiate system automation actions (e.g., "DOT COM," "System") from human operator actions (specific users or team names).
    • Use normal date format (MM/DD/YYYY) consistently for all timestamps.
    • Group repeated events under combined bullet points with summaries.

    Important Rules:

    • If any "description" or "text" field contains "Cancelled", classify the order as Cancelled.
    • If "Returned" is present with reasons, classify order as Returned and provide the explicit reason.
    • Avoid assumptions and only conclude based on explicit JSON data.

    Use this prompt verbatim when processing JSON order activity logs to generate insightful, fact-based summaries highlighting order lifecycle, customer interactions, claims and risk assessments, fraud and payment activities, and the conclusive final outcome. """

    user_content = json.dumps(data, indent=2)
    
    try:
        return client.generate(system_prompt=system_prompt, user_content=user_content)
    except Exception as e:
        return f"Error generating summary: {str(e)}"
    

    --------------------- STREAMLIT UI ---------------------

    st.set_page_config("Macy's Notes Analyzer with Vertex AI Gemini", layout="wide") st.title("📑 Macy's Notes Analysis — Direct LLM Processing (Vertex AI Gemini)")

    uploaded_file = st.file_uploader("Upload Macy's JSON Notes", type="json")

    if uploaded_file: try: data = json.load(uploaded_file)

        client = GeminiClient()  # Initialize Vertex AI client
        
        with st.spinner("🔍 Analyzing documents..."):
            summary = generate_summary(client, data)
            
        st.success("✅ Analysis Complete!")
        
        # Display formatted summary
        st.markdown("## 📝 Comprehensive Summary")
        st.markdown(summary)
                
    except json.JSONDecodeError:
        st.error("❌ Invalid JSON file format")
    except Exception as e:
        st.error(f"❌ Processing error: {str(e)}")
    

    Optional: Add original data preview

    if uploaded_file and st.checkbox("Show raw data preview"): uploaded_file.seek(0) st.json(json.load(uploaded_file))

    in the code can you tell me how many token we are sending and how many tokes we are getting on frontend itself also tell me time to get the response

  2. Assistant

    To measure tokens sent/received and timing in your Streamlit app with the Gemini API, you need to:

    1. Calculate token counts for your prompt (system_prompt + user_content) before sending it.
    2. Count tokens in the response.
    3. Measure the time taken for the API call.

    How to do it?

    Your current code uses Google Gemini API via genai.Client. Unfortunately, Google's Gemini API doesn't provide token usage info directly in its response metadata, unlike the OpenAI API.

    So, you have to estimate tokens yourself client-side, both in prompt and response, using a tokenizer compatible with Gemini's model (likely similar to PaLM or BPE tokenizers), or approximate tokens by text length / word count.


    Step 1: Add timing measurement

    import time
    

    Around the call:

    start_time = time.time()
    response = self.client.models.generate_content(
        model='gemini-2.0-flash-001',
        contents=full_message
    )
    end_time = time.time()
    self.last_response_time = end_time - start_time
    

    Store the duration to show in Streamlit.


    Step 2: Estimate tokens usage (approximate)

    Without an official tokenizer from Google, simplest approximation:

    • Use the number of words or characters / 4 (roughly 4 characters = 1 token).
    • Or install tiktoken or similar for better estimates.

    For example:

    def estimate_tokens(text: str) -> int:
        # Approximate: 1 token = 4 chars
        return max(1, len(text) // 4)
    

    Step 3: Modify GeminiClient to store tokens info

    In GeminiClient:

    class GeminiClient:
        def __init__(self):
            ...
            self.last_request_tokens = 0
            self.last_response_tokens = 0
            self.last_response_time = 0.0
    
        def generate(self, system_prompt: str, user_content: str) -> str:
            full_message = system_prompt + "\n\n" + user_content
            
            # Estimate tokens in prompt
            self.last_request_tokens = estimate_tokens(full_message)
            
            start_time = time.time()
            response = self.client.models.generate_content(
                model='gemini-2.0-flash-001',
                contents=full_message
            )
            end_time = time.time()
            self.last_response_time = end_time - start_time
            
            # Extract generated text
            text = response.candidates[0].content.parts[0].text
            
            # Estimate tokens in response
            self.last_response_tokens = estimate_tokens(text)
            
            return text
    

    Step 4: Show token counts and timing in Streamlit UI

    In your main Streamlit flow, after summary = generate_summary(client, data):

    st.markdown(f"**Prompt tokens:** {client.last_request_tokens}")
    st.markdown(f"**Response tokens:** {client.last_response_tokens}")
    st.markdown(f"**Response time:** {client.last_response_time:.2f} seconds")
    

    Summary

    • Gemini API does not expose real token counts, so estimate with a simple heuristic (chars/4).
    • Add timing around the API call with time.time().
    • Store tokens and timing inside the client for display.
    • Show these metrics next to the summary in Streamlit frontend.

    If you want better token counting, you can try using the tiktoken package's tokenizer for an OpenAI-compatible model or find Google's official tokenizer for Gemini (if available). But heuristic counting is often good enough for basic monitoring.

  3. User

    give me full code

  4. Assistant

    Certainly! Here's your full updated app.py with added token counting (approximate) and timing measurement displayed in the Streamlit UI.

    # app.py
    
    import os
    import json
    import time
    import streamlit as st
    
    try:
        from google import genai
        from google.genai import types
    except ImportError:
        genai = None
        types = None
    
    
    # Simple approximate token counter: 1 token ≈ 4 chars
    def estimate_tokens(text: str) -> int:
        return max(1, len(text) // 4)
    
    
    # --------------------- Gemini Client ---------------------
    class GeminiClient:
        def __init__(self):
            self.project = ""
            self.location = ""
    
            # You can alternatively read your project and location from env vars here:
            if not self.project:
                self.project = os.getenv("GOOGLE_CLOUD_PROJECT", "")
            if not self.location:
                self.location = os.getenv("GOOGLE_CLOUD_LOCATION", "us-central1")  # Provide a default if desired
    
            if not self.project:
                st.error("Set GOOGLE_CLOUD_PROJECT environment variable before running")
                st.stop()
    
            if genai is None:
                st.error("Please install the google-generativeai package to use Gemini model")
                st.stop()
    
            self.client = genai.Client(
                vertexai=True,
                project=self.project,
                location=self.location,
            )
    
            # Store last request/response tokens and time for frontend display
            self.last_request_tokens = 0
            self.last_response_tokens = 0
            self.last_response_time = 0.0
    
        def generate(self, system_prompt: str, user_content: str) -> str:
            """
            Generate chat completion using Vertex AI Gemini model.
            Since Vertex AI Gemini doesn't support 'chat.completions' yet as in OpenAI,
            we simulate a chat prompt by concatenating system and user messages.
            """
            # Combine prompts into one message
            full_message = system_prompt + "\n\n" + user_content
    
            # Estimate tokens in prompt
            self.last_request_tokens = estimate_tokens(full_message)
    
            # Call the model:
            start_time = time.time()
            response = self.client.models.generate_content(
                model='gemini-2.0-flash-001',
                contents=full_message
            )
            end_time = time.time()
            self.last_response_time = end_time - start_time
    
            # Extract generated text
            text = response.candidates[0].content.parts[0].text
    
            # Estimate tokens in response
            self.last_response_tokens = estimate_tokens(text)
    
            return text
    
    
    # --------------------- FUNCTIONS ---------------------
    def generate_summary(client: GeminiClient, data: dict) -> str:
        system_prompt = """Analyze the JSON order activity log below and produce a structured, detailed summary report strictly based on the data provided. Your analysis must follow these guidelines and structure:
    
    # 1. Final Outcome
    - Clearly state the final status of the order: **Cancelled**, **Returned**, or another status.
    - Base the conclusion solely on explicit mentions of "Cancelled" or "Returned" found in any "description" or "text" fields.
    - If "Returned" appears, specify the exact reason for the return as stated in the JSON.
    - If "Cancelled" is found and "Returned" is not, declare the order Cancelled.
    - Do not infer or guess beyond the explicit data.
    
    # 2. Order Timeline Overview
    - Provide the **order creation timestamp**, formatted as MM/DD/YYYY.
    - State the **final status** of the order.
    - List the **key users, teams, or system automation agents** involved, differentiating human actors from automated systems (e.g., "DOT COM").
    - Briefly summarize the main flow of actions/events across the order lifecycle.
    
    # 3. Critical Events (Chronological Breakdown)
    Organize and list events chronologically by date (MM/DD/YYYY), grouped as follows:
    
    - **Fraud Checks**
      - Include all declines, fraud holds, account locks, fraud alert emails, or related system flags.
      - Specify whether each event was initiated by system automation or a human user.
    
    - **Payment Actions**
      - List events such as authorizations, voids, failed transactions, chargebacks, or refunds.
    
    - **Customer Issues Reported**
      - Document all customer-reported issues, including dates and descriptions as recorded in the JSON.
    
    - **Claims and Risk Reviews**
      - Detail any claims raised, risk assessment activities, or reviews by risk or claims teams.
    
    - **Cancellation/Return Triggers**
      - Highlight the exact events that triggered cancellation or return, quoting relevant "description" or "text" fields.
    
    - **Post-resolution Access**
      - Include coaching, user reviews, follow-up notes, or other post-resolution activity logs.
      - Group multiple coaching or review events logically together.
    
    # 4. Key Notes
    - Identify any suspicious patterns such as:
      - Frequent or unusual automated system flags.
      - Multiple coaching sessions or repeated reviews.
      - Any unusual or notable user behaviors or system interventions.
    - Call out any automation flags and note if events are clearly system-generated vs. human actions.
    
    # 5. Key Evidence
    - Provide specific JSON entries verbatim (quoting "description" or "text") that directly support your conclusion about the final outcome.
    - Clearly highlight the cancellation or return reason stated within these entries.
    
    ---
    
    # Formatting Requirements:
    - Use Markdown headers and bullet points for readability.
    - Differentiate system automation actions (e.g., "DOT COM," "System") from human operator actions (specific users or team names).
    - Use normal date format (MM/DD/YYYY) consistently for all timestamps.
    - Group repeated events under combined bullet points with summaries.
    
    ---
    
    # Important Rules:
    - If any "description" or "text" field contains **"Cancelled"**, classify the order as **Cancelled**.
    - If "Returned" is present with reasons, classify order as **Returned** and provide the explicit reason.
    - Avoid assumptions and only conclude based on explicit JSON data.
    
    ---
    
    Use this prompt verbatim when processing JSON order activity logs to generate insightful, fact-based summaries highlighting order lifecycle, customer interactions, claims and risk assessments, fraud and payment activities, and the conclusive final outcome. """
    
        user_content = json.dumps(data, indent=2)
    
        try:
            return client.generate(system_prompt=system_prompt, user_content=user_content)
        except Exception as e:
            return f"Error generating summary: {str(e)}"
    
    
    # --------------------- STREAMLIT UI ---------------------
    st.set_page_config("Macy's Notes Analyzer with Vertex AI Gemini", layout="wide")
    st.title("📑 Macy's Notes Analysis — Direct LLM Processing (Vertex AI Gemini)")
    
    uploaded_file = st.file_uploader("Upload Macy's JSON Notes", type="json")
    
    if uploaded_file:
        try:
            data = json.load(uploaded_file)
    
            client = GeminiClient()  # Initialize Vertex AI client
    
            with st.spinner("🔍 Analyzing documents..."):
                summary = generate_summary(client, data)
    
            st.success("✅ Analysis Complete!")
    
            # Display tokens and time info
            st.markdown(f"**Prompt tokens (approx.):** {client.last_request_tokens}")
            st.markdown(f"**Response tokens (approx.):** {client.last_response_tokens}")
            st.markdown(f"**Response time:** {client.last_response_time:.2f} seconds")
    
            # Display formatted summary
            st.markdown("## 📝 Comprehensive Summary")
            st.markdown(summary)
    
        except json.JSONDecodeError:
            st.error("❌ Invalid JSON file format")
        except Exception as e:
            st.error(f"❌ Processing error: {str(e)}")
    
    # Optional: Add original data preview
    if uploaded_file and st.checkbox("Show raw data preview"):
        uploaded_file.seek(0)
        st.json(json.load(uploaded_file))
    

    What changed?

    • Added estimate_tokens() function for approximate counting.
    • In GeminiClient.generate(), added:
      • Token count estimate for prompt and response.
      • Timing of the request.
    • Displayed these stats right before summary.
    • Reading GOOGLE_CLOUD_PROJECT and optional location from environment, fallback if needed.
    • Kept the rest of your app intact.

    Notes

    • This is an approximation since Gemini currently does not provide direct token usage info.
    • You can refine token estimation by integrating suitable tokenizer libraries if you find Google's or a compatible tokenizer.
    • Adjust environment variable usage as needed for your deployment.

    Let me know if you want the code for precise token counting or any other additions!

  5. User

    use this from vertexai.preview import tokenization

  6. Assistant

    Got it! You want to replace the approximate token counting with the tokenizer from vertexai.preview.tokenization which is the official way to count tokens for Vertex AI Gemini models.

    Here’s your full app.py updated to use that tokenizer for precise token counting of prompt and response, and still measure response time.


    # app.py
    
    import os
    import json
    import time
    import streamlit as st
    
    try:
        from google import genai
        from google.genai import types
        from vertexai.preview import tokenization
    except ImportError:
        genai = None
        types = None
        tokenization = None
    
    
    # --------------------- Gemini Client ---------------------
    class GeminiClient:
        def __init__(self):
            self.project = os.getenv("GOOGLE_CLOUD_PROJECT", "")
            self.location = os.getenv("GOOGLE_CLOUD_LOCATION", "us-central1")
    
            if not self.project:
                st.error("Set GOOGLE_CLOUD_PROJECT environment variable before running")
                st.stop()
    
            if genai is None:
                st.error("Please install the google-generativeai package to use Gemini model")
                st.stop()
    
            if tokenization is None:
                st.error("Please install vertexai package to use tokenizer")
                st.stop()
    
            self.client = genai.Client(
                vertexai=True,
                project=self.project,
                location=self.location,
            )
    
            # Initialize tokenizer for model gemini-2.0-flash-001
            # Adjust model name if using different model
            self.tokenizer = tokenization.Tokenizer.for_model("gemini-2.0-flash-001")
    
            # Store last request/response tokens and time
            self.last_request_tokens = 0
            self.last_response_tokens = 0
            self.last_response_time = 0.0
    
        def generate(self, system_prompt: str, user_content: str) -> str:
            # Combine prompts into one message
            full_message = system_prompt + "\n\n" + user_content
    
            # Count tokens precisely
            self.last_request_tokens = len(self.tokenizer.encode(full_message))
    
            start_time = time.time()
            response = self.client.models.generate_content(
                model='gemini-2.0-flash-001',
                contents=full_message
            )
            end_time = time.time()
            self.last_response_time = end_time - start_time
    
            text = response.candidates[0].content.parts[0].text
    
            # Count tokens in response
            self.last_response_tokens = len(self.tokenizer.encode(text))
    
            return text
    
    
    # --------------------- FUNCTIONS ---------------------
    def generate_summary(client: GeminiClient, data: dict) -> str:
        system_prompt = """Analyze the JSON order activity log below and produce a structured, detailed summary report strictly based on the data provided. Your analysis must follow these guidelines and structure:
    
    # 1. Final Outcome
    - Clearly state the final status of the order: **Cancelled**, **Returned**, or another status.
    - Base the conclusion solely on explicit mentions of "Cancelled" or "Returned" found in any "description" or "text" fields.
    - If "Returned" appears, specify the exact reason for the return as stated in the JSON.
    - If "Cancelled" is found and "Returned" is not, declare the order Cancelled.
    - Do not infer or guess beyond the explicit data.
    
    # 2. Order Timeline Overview
    - Provide the **order creation timestamp**, formatted as MM/DD/YYYY.
    - State the **final status** of the order.
    - List the **key users, teams, or system automation agents** involved, differentiating human actors from automated systems (e.g., "DOT COM").
    - Briefly summarize the main flow of actions/events across the order lifecycle.
    
    # 3. Critical Events (Chronological Breakdown)
    Organize and list events chronologically by date (MM/DD/YYYY), grouped as follows:
    
    - **Fraud Checks**
      - Include all declines, fraud holds, account locks, fraud alert emails, or related system flags.
      - Specify whether each event was initiated by system automation or a human user.
    
    - **Payment Actions**
      - List events such as authorizations, voids, failed transactions, chargebacks, or refunds.
    
    - **Customer Issues Reported**
      - Document all customer-reported issues, including dates and descriptions as recorded in the JSON.
    
    - **Claims and Risk Reviews**
      - Detail any claims raised, risk assessment activities, or reviews by risk or claims teams.
    
    - **Cancellation/Return Triggers**
      - Highlight the exact events that triggered cancellation or return, quoting relevant "description" or "text" fields.
    
    - **Post-resolution Access**
      - Include coaching, user reviews, follow-up notes, or other post-resolution activity logs.
      - Group multiple coaching or review events logically together.
    
    # 4. Key Notes
    - Identify any suspicious patterns such as:
      - Frequent or unusual automated system flags.
      - Multiple coaching sessions or repeated reviews.
      - Any unusual or notable user behaviors or system interventions.
    - Call out any automation flags and note if events are clearly system-generated vs. human actions.
    
    # 5. Key Evidence
    - Provide specific JSON entries verbatim (quoting "description" or "text") that directly support your conclusion about the final outcome.
    - Clearly highlight the cancellation or return reason stated within these entries.
    
    ---
    
    # Formatting Requirements:
    - Use Markdown headers and bullet points for readability.
    - Differentiate system automation actions (e.g., "DOT COM," "System") from human operator actions (specific users or team names).
    - Use normal date format (MM/DD/YYYY) consistently for all timestamps.
    - Group repeated events under combined bullet points with summaries.
    
    ---
    
    # Important Rules:
    - If any "description" or "text" field contains **"Cancelled"**, classify the order as **Cancelled**.
    - If "Returned" is present with reasons, classify order as **Returned** and provide the explicit reason.
    - Avoid assumptions and only conclude based on explicit JSON data.
    
    ---
    
    Use this prompt verbatim when processing JSON order activity logs to generate insightful, fact-based summaries highlighting order lifecycle, customer interactions, claims and risk assessments, fraud and payment activities, and the conclusive final outcome. """
    
        user_content = json.dumps(data, indent=2)
    
        try:
            return client.generate(system_prompt=system_prompt, user_content=user_content)
        except Exception as e:
            return f"Error generating summary: {str(e)}"
    
    
    # --------------------- STREAMLIT UI ---------------------
    st.set_page_config("Macy's Notes Analyzer with Vertex AI Gemini", layout="wide")
    st.title("📑 Macy's Notes Analysis — Direct LLM Processing (Vertex AI Gemini)")
    
    uploaded_file = st.file_uploader("Upload Macy's JSON Notes", type="json")
    
    if uploaded_file:
        try:
            data = json.load(uploaded_file)
    
            client = GeminiClient()  # Initialize Vertex AI client
    
            with st.spinner("🔍 Analyzing documents..."):
                summary = generate_summary(client, data)
    
            st.success("✅ Analysis Complete!")
    
            # Display tokens and time info
            st.markdown(f"**Prompt tokens:** {client.last_request_tokens}")
            st.markdown(f"**Response tokens:** {client.last_response_tokens}")
            st.markdown(f"**Response time:** {client.last_response_time:.2f} seconds")
    
            # Display formatted summary
            st.markdown("## 📝 Comprehensive Summary")
            st.markdown(summary)
    
        except json.JSONDecodeError:
            st.error("❌ Invalid JSON file format")
        except Exception as e:
            st.error(f"❌ Processing error: {str(e)}")
    
    # Optional: Add original data preview
    if uploaded_file and st.checkbox("Show raw data preview"):
        uploaded_file.seek(0)
        st.json(json.load(uploaded_file))
    

    Notes:

    • Used tokenization.Tokenizer.for_model("gemini-2.0-flash-001") to get the proper tokenizer.
    • Token counts for both prompt and response are exact according to the tokenizer.
    • Response time measured with time.time().
    • Added error handling if vertexai.preview.tokenization is not installed.
    • Make sure you have installed the right package:
      pip install vertexai google-generativeai
      

    Let me know if you want me to help adapt this to handle streaming or multiple messages or anything else!

  7. User

    is it counting token again have some costing to it

  8. Assistant

    Great question! Here's the info about token counting and cost in your scenario:


    Does token counting with vertexai.preview.tokenization incur any cost?

    No, token counting by itself is a local, offline operation performed by your client code. The tokenizer just processes your prompt and response strings locally — it does NOT make any API calls and does not incur any usage cost.

    So, using tokenization.Tokenizer.for_model() to count tokens is free and purely client-side.


    Where do costs come from?

    Costs are incurred only when you actually call the LLM API service, i.e., when you call:

    response = self.client.models.generate_content(
        model='gemini-2.0-flash-001',
        contents=full_message
    )
    

    This sends data to the Vertex AI model and results in billed usage based on tokens processed by the model.


    Summary

    ActionToken counting cost?API usage cost?
    Using tokenization.Tokenizer.encode()No (local, free)None
    Calling generate_content(...)NoYes (based on tokens)

    Additional notes

    • Token counting locally is very fast and efficient.
    • You should count tokens locally before making calls if you want to monitor or limit usage.
    • Using this method won’t add latency or cost beyond what the API call does.
    • The token counts you obtain from the tokenizer are estimates of what the API will count for billing.

    If you want, I can show you an example of limiting inputs based on token counts or logging usage for cost monitoring!