Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-mini-2024-07-18· OpenAIAi2 WildChat-4.8M (AI2)

Questions "1. Introduction and touch up on gen ai use cases worked 2. Given an example…

  1. User

    Questions "1. Introduction and touch up on gen ai use cases worked 2. Given an example and output schema, Give me the steps that youll follow to get the output 3. How will you reduce latency 4. How will you calculate accuracy of System & make sure that its adheres to schema" "1. Introduction 2. A free text input containing hashtags, emoticons, and mixed content was provided. The task for the Large Language Model (LLM) was to extract specific information (such as product names) and conduct sentiment analysis. The output had to be formatted in a structured JSON format that included the product name and sentiment, categorized as positive (+), negative (-), or neutral.

    • prompt
    • system
    • instructions
    • examples
    • parameters/settings
    • systematic experimentation
    • force format adherence" "1. Introduction and touch up on gen ai use cases worked
    1. Project specific question - Challenges faced
    2. Give sample product review, desired output. 3.1 Experiments to find best value for temp, max o/p, top p, top k 3.2 Evaluation metrics of the output 3.3. Output Validation"
    3. Introduction 2. How do you enforce json schema on the output of LLM 3. How do you measure latency of LLM ?
    4. Introduction 2. How will you finalize a model for your use case (based on one of the project i have worked on) 3. Yesterdays question on how we can get the desired schema 4. How can you do text extraction from PDF especially tables and images from PDF 5. How do you measure latency of LLMs? "1. Introduction
    5. How you will evaluate 2 set of embeddings
    6. How would you extract the tables from the PDF document?
    7. How do you measure the latency of LLMs?" "1. Introduction
    8. How to get the desired schema in the output?
    9. How would you extract the tables from the PDF document?
    10. How do you measure the latency of LLMs?"
    11. Intro 2.Asked to me explain the projects in details 3.How you will evaluate 2 set of embeddings from 2 different model for a flipkart where a creatin item is listed on 1 page and from 2nd model on 4th page , how is the recommendation working? "1. Intro
    12. Project Related Questions (model deployment, Embedding search, evaluation)
    13. Write a logic to get data from kafka transaction stream and find the traffic for particular zipcode in lasr 30 minutes
    14. Write a logic to find a score of varienty of content user is watching based on his recent behaviour
    15. ever worked on streamlit?" "1. Introduction
    16. Started with Traditional ML project with detailed explanation - What was the problem? , Which Model we have used?
    17. Gen AI project discussion - Architecture Discussion
    18. Which technologies you have used?
    19. How would you evaluate RAG pipeline?
    20. What all the metrics are there in RAGAS? 6.1 Explain Context_precision metric.
    21. What is Reciprocal Rank Fusion ?
    22. How would you extract the Graphs and tables from the PDF document?
    23. How will you write a prompt that can generate exactly 2000 Words for an essay on a given topic Eg. “Bertrand Russel’s - a history of western philosophy” Not the complete prompt just restrict LLM to write essay exaclty with 2000 words" "1. Introduction
    24. Most challenging technical task faced
    25. Question related to feature engineering:

    Feature Creation Challenge: Content Variety Score for Churn Prediction You are working on a churn prediction model for a streaming TV service. You suspect that users who watch a wider variety of content are less likely to churn. You have access to the following ""Watch Data"":

    Field NameData TypeDescription
    client_idIntegerUnique identifier for the customer
    start_timeIntegerUnix timestamp of the viewing session start
    program_idStringUnique identifier for the program watched
    minutes_watchedIntegerNumber of minutes watched
    thumbs_upStringUser feedback ('Y' for yes, 'N' for no, null for no feedback)

    Your task: Design and describe a single feature, called content_variety_score, that quantifies the diversity of content watched by each user. This feature should be designed in such a way that a higher score indicates greater variety in viewing habits.

    Consider the following in your design:

    Time Decay: Recent viewing patterns should be weighted more heavily than older viewing patterns. How will your feature incorporate the start_time information to reflect this? Scale and Normalization: The feature should be comparable across users, regardless of how much they watch overall. How will you ensure the score is appropriately scaled and normalized?

    Deliverables:

    A precise definition of the content_variety_score. Include a formula or pseudocode that clearly demonstrates how it is calculated. " "1. Introduction. 2. Explained project. 3. How to compare two LLM's Question 1: Example Data Stream (Illustrative - Freeform Text Snippets):

    Freeform Text:

    Context: ""OMG, the new ✨SparklePhone Pro✨ is AMAZING! 😍 The camera is insane. #SparklePhone #BestPhoneEver"" ""Just got my SparklePhone Pro, and the battery life is terrible. 😠 Two hours and it's dead. Don't buy! #SparklePhone #TechFail"" ""Anyone know if the SparkleWatch has GPS? 🤔 Thinking of getting one for running. #SparkleWatch #FitnessTracker"" ""SparkleTab is on sale! 🎉 $299 for the 128GB model. Such a steal! #SparkleTab #TabletDeals"" ""The SparklePhone Pro's camera is 🔥🔥🔥. So much better than my old phone. #SparklePhone #Photography"" ""SparkleWatch is great for tracking my workouts! 💪 #SparkleWatch #FitnessMotivation"" ""Hate the SparkleTab's keyboard. So clunky. 😒 #SparkleTab #TechProblems"" …

    Desired Output: [ {""product"": ""SparklePhone Pro"", ""sentiment"": ""positive"", ""mentions"": 2}, {""product"": ""SparklePhone Pro"", ""sentiment"": ""negative"", ""mentions"": 1}, {""product"": ""SparkleWatch"", ""sentiment"": ""neutral"", ""mentions"": 1}, {""product"": ""SparkleTab"", ""sentiment"": ""positive"", ""mentions"": 1}, {""product"": ""SparklePhone Pro"", ""sentiment"": ""positive"", ""mentions"": 3}, {""product"": ""SparkleWatch"", ""sentiment"": ""positive"", ""mentions"": 2}, {""product"": ""SparkleTab"", ""sentiment"": ""negative"", ""mentions"": 1} ]

    Output Rules: Must be List of JSON / Dicts of products Each product must have the fields sentiment, mentions, product Sentiment must be [positive, negative or neutral] Mentions must be an Integer , must be >= 0 List length can be variable Product cannot be made up

    Goal is to Format & rules must be adhered to at all costs reduce Perceived latency to the least ‘ms’ possible

    PROMPT:

    • SYSTEM : You are expert in extracting information from text
    • INSTRUCTIONS: Carefully read the paragraph and extract product name and sentiment. Output format should be :

    [ {""product"": """", ""sentiment"": ""<positive, Negative, Neutral>"", ""mentions"": ''},

    ]

    • EXAMPLES : This will be the example of input format . ""OMG, the new ✨SparklePhone Pro✨ is AMAZING! 😍 The camera is insane. #SparklePhone #BestPhoneEver"" ""Just got my SparklePhone Pro, and the battery life is terrible. 😠 Two hours and it's dead. Don't buy! #SparklePhone #TechFail"" ""Anyone know if the SparkleWatch has GPS? 🤔 Thinking of getting one for running. #SparkleWatch #FitnessTracker"" ""SparkleTab is on sale! 🎉 $299 for the 128GB model. Such a steal! #SparkleTab #TabletDeals"" ""The SparklePhone Pro's camera is 🔥🔥🔥. So much better than my old phone. #SparklePhone #Photography"" ""SparkleWatch is great for tracking my workouts! 💪 #SparkleWatch #FitnessMotivation"" ""Hate the SparkleTab's keyboard. So clunky. 😒 #SparkleTab #TechProblems"" …

    based on this output should be like: [ {""product"": ""SparklePhone Pro"", ""sentiment"": ""positive"", ""mentions"": 2}, ] Based on user input

    • PARAMETERS / SETTINGS:

    • Temp : 0

    • Max_Tokens 1024

    • Top_P : 0.9

    • Top_K: 0.1

    • Freq/Presence_Penalty ....

    • SYSTEMATIC EXPERIMENTATION

    • FORCE FORMAT ADHERENCE={ ""product"":{ type:str}, sentiment:{type:str}, mentions:

    }

    import ast ast.litral_eval(response)

    " " 1) Explain the brief into in this fashion: Projects undertaken -> Specify if they were Gen AI project -> Technical details about the project -> Challenges faces -> Steps taken to mitigate them 2) Write a validation function in function which can validate the citations generated from the raw text given below. The function should perform two tasks generated percentage match in the raw text chunk and validations of how truthful the groundnesses is.( without using LLM)" "1)Projects undertaken -> Technical details about the project -> Challenges faces -> Steps taken to mitigate them 2)How to evaluate 2 llm on latency 3) how to restrict llm to output in specified format and how to validate that" "1) Intoruction 2) Brief explanantion on the projects undertaken and follow up questions on those. 3) How to measure latency of LLMs 4) How to evaluate LLMs? 5) code/psudeo code for question 3 &4 from the doc https://docs.google.com/document/d/15zhD4xt5PKImhlNuYylTbxmrme4-FQ1n-mI7ZqwvniQ/edit"

    "1) Introduction 2) What were the projects ? 3) how did you convert words in to embeddings for KMeans 4) Code for content Variety Score for Churn Prediction" "1)brief regarding projects, 2) challenges faced in projects and how did you overcome challenges 3) write code for llm response validation using pydantic 4) how to evaluate rag model 5) metrics to evaluate latency of 2 llms" "1) Introduction 2) Projects 3) Question related to reson behind the tech stack for the projects 4) How to Evaluate RAG 5) Formula ro context precison 6) Write a code to evaluate citation and the answers given by the LLM based on the raw context"

    this was the questions asked in an interview answer them and also generate similar questions from the above questions

  2. Assistant

    Answers to Interview Questions

    1. Introduction

    In the realm of artificial intelligence, particularly in the application of generative AI (Gen AI), I've had the opportunity to work on various projects that leverage the power of large language models (LLMs) for tasks like text extraction, sentiment analysis, and data validation. These projects have not only enhanced my understanding of AI technologies but have also equipped me with the skills necessary to handle complex challenges, from model deployment to optimizing performance metrics.

    2. How to Get the Desired Schema in the Output?

    To enforce a specific output schema for LLM-generated responses, I would utilize several strategies:

    • Prompt Engineering: Clearly define the expected output structure in the prompt. For example, specifying that the output should strictly adhere to a JSON format with set fields like "product," "sentiment," and "mentions."
    • Post-Processing: Implement a validation function that checks if the output matches the expected schema before usage. This function would parse the response and ensure that all fields meet the required specifications.
    • Feedback Loops: Use reinforcement learning techniques where the model is penalized for outputs that do not follow the desired schema during training or fine-tuning phases.

    3. How Would You Extract Tables from the PDF Document?

    To extract tables from a PDF document, I would follow these steps:

    1. Select a Library: Utilize libraries such as PyPDF2, pdfplumber, or Tabula to handle the extraction process.
    2. Load the PDF: Open the PDF file using the selected library to create a document object.
    3. Identify Table Structures: Use the library methods to detect tables. For instance:
      • Using pdfplumber, leverage the extract_table method that identifies the structure based on the PDF layout.
    4. Extract Data: Convert the identified tables' contents into a structured format, such as a list of dictionaries or a DataFrame.
    5. Cleanup: Post-process the extracted data to address any formatting issues or inaccuracies.

    4. How Do You Measure Latency of LLMs?

    Measuring the latency of LLMs involves the following steps:

    • Define Latency Metrics: Identify what latency means for your application (e.g., response time, processing time).
    • Create a Benchmark Test: Set up a controlled environment where you can input queries into the LLM and record the time taken for responses.
    • Use Timer Functions: Implement timer functions in your code (e.g., time.time() in Python) to record the time just before sending a request and immediately after receiving a response.
    • Analyze Results: Collect latency data over multiple runs to analyze average response times, variance, and identify any performance bottlenecks.

    Generated Questions Inspired by Responses

    1. Introduction:

      • Can you explain your experience with Gen AI and any projects you have undertaken in this domain?
    2. Project Challenges:

      • What specific challenges did you encounter during your AI projects, and what steps did you take to overcome them?
    3. Output Validation:

      • How do you ensure the validity of the outputs generated by an LLM in terms of format and content accuracy?
    4. Extraction Techniques:

      • What approaches would you recommend for extracting images and graphs from PDF documents alongside tables?
    5. Latency Measurement in Practice:

      • What tools or methodologies do you use to measure the real-time performance of LLMs in a production environment?
    6. Evaluating LLMs:

      • How would you compare the performance of two LLMs based on their response time and output quality?
    7. Feature Engineering:

      • Can you design a feature that quantifies the diversity of viewing habits for a recommendation system?
    8. Prompt Optimization:

      • How can you refine prompts to ensure they generate outputs that meet strict formatting requirements?
    9. Embedding Comparison:

      • How do you evaluate and compare embeddings generated from two different models in a recommendation system?
    10. Technical Stack Justification:

      • Why did you choose a specific technology stack for your projects, and how did it facilitate your project goals?
  3. User

    male a list of the questions in the prompt and the generated questions

  4. Assistant

    Certainly! Here’s a consolidated list of the original interview questions from your prompt and the newly generated questions:

    Original Interview Questions

    1. Introduction and touch up on gen ai use cases worked.
    2. Given an example and output schema, give me the steps that you'll follow to get the output.
    3. How will you reduce latency?
    4. How will you calculate accuracy of the system & make sure that it adheres to schema?
    5. A free text input containing hashtags, emoticons, and mixed content was provided. The task for the LLM was to extract specific information (such as product names) and conduct sentiment analysis. The output had to be formatted in a structured JSON format that included the product name and sentiment, categorized as positive (+), negative (-), or neutral.
    6. How do you enforce JSON schema on the output of LLM?
    7. How do you measure latency of LLMs?
    8. How will you finalize a model for your use case (based on one of the projects I have worked on)?
    9. How can you do text extraction from PDF, especially tables and images from PDF?
    10. How do you measure latency of LLMs?
    11. How will you evaluate 2 sets of embeddings?
    12. Write a logic to get data from Kafka transaction streams and find the traffic for a particular zipcode in the last 30 minutes.
    13. Write a logic to find a score of the variety of content a user is watching based on their recent behavior.
    14. Ever worked on Streamlit?
    15. Which technologies have you used?
    16. How would you evaluate RAG (Retrieval-Augmented Generation) pipeline?
    17. What metrics are there in RAGAS?
    18. Explain Context Precision metric.
    19. What is Reciprocal Rank Fusion?
    20. How would you extract Graphs and tables from the PDF document?
    21. How will you write a prompt that can generate exactly 2000 words for an essay on a given topic, e.g., “Bertrand Russell’s - A History of Western Philosophy”?
    22. Projects undertaken -> Specify if they were Gen AI projects -> Technical details about the project -> Challenges faced -> Steps taken to mitigate them.
    23. Write a validation function in Python which can validate the citations generated from the raw text given below.
    24. How to evaluate 2 LLMs on latency?
    25. How to restrict LLM to output in a specified format and how to validate that?
    26. What projects have you undertaken?
    27. How did you convert words into embeddings for KMeans?
    28. Code for content variety score for churn prediction.
    29. What metrics are there to evaluate the latency of 2 LLMs?

    Generated Questions

    1. Can you explain your experience with Gen AI and any projects you have undertaken in this domain?
    2. What specific challenges did you encounter during your AI projects, and what steps did you take to overcome them?
    3. How do you ensure the validity of the outputs generated by an LLM in terms of format and content accuracy?
    4. What approaches would you recommend for extracting images and graphs from PDF documents alongside tables?
    5. What tools or methodologies do you use to measure the real-time performance of LLMs in a production environment?
    6. How would you compare the performance of two LLMs based on their response time and output quality?
    7. Can you design a feature that quantifies the diversity of viewing habits for a recommendation system?
    8. How can you refine prompts to ensure they generate outputs that meet strict formatting requirements?
    9. How do you evaluate and compare embeddings generated from two different models in a recommendation system?
    10. Why did you choose a specific technology stack for your projects, and how did it facilitate your project goals?

    Feel free to reach out if you need further assistance or additional information!

  5. User

    answer all the questions

  6. Assistant

    Sure! Here are comprehensive answers to all the original interview questions:

    Original Interview Questions and Answers

    1. Introduction and touch up on gen AI use cases worked.

      • I have worked on various generative AI use cases, including sentiment analysis, automated content generation, and information extraction from unstructured text. One notable project involved developing a sentiment analysis model that assessed user reviews for products, extracting relevant features such as product names and corresponding sentiments.
    2. Given an example and output schema, give me the steps that you'll follow to get the output.

      • To produce a structured output from a free-form text:
        1. Preprocessing: Clean the text by removing irrelevant characters (hashtags, emojis, etc.).
        2. Model Input: Feed the cleaned text into the LLM with a properly structured prompt specifying the desired output format.
        3. Extraction Logic: Implement logic to extract product names and sentiments.
        4. Output Formatting: Ensure the results adhere to the specified schema (JSON).
        5. Validation: Post-process the output to check format adherence and accuracy.
    3. How will you reduce latency?

      • Latency can be reduced by:
        • Optimizing Model Architecture: Use a model that balances performance with complexity.
        • Batching Requests: Process multiple requests simultaneously to reduce per-query overhead.
        • Caching Responses: Store frequent outputs to avoid recomputation.
        • Using Asynchronous Requests: Implement non-blocking I/O operations in API calls.
    4. How will you calculate accuracy of System & make sure that it adheres to schema?

      • Accuracy can be assessed by comparing model outputs against a ground truth dataset using metrics such as precision, recall, and F1-score. To ensure adherence to the schema:
        • Implement schema validation rules in the post-processing stage.
        • Use libraries like jsonschema to validate the output structure.
    5. The task for the LLM was to extract specific information (such as product names) and conduct sentiment analysis. Output must be formatted in a structured JSON format.

      • An LLM can be instructed through prompt engineering to extract items like product names and sentiments, incorporating contextual understanding of positives and negatives as defined by common expressions and associated keywords.
    6. How do you enforce JSON schema on the output of LLM?

      • I would enforce JSON schema by implementing validation checks with libraries like jsonschema or writing custom validation functions that confirm the output matches the expected structure, types, and rules outlined in the schema.
    7. How do you measure latency of LLMs?

      • Latency can be measured using a benchmarking approach that involves:
        • Recording timestamps before and after model inference.
        • Averaging the results across multiple requests to account for variability.
        • Measuring both cold starts (first request) and warm starts (subsequent requests).
    8. How will you finalize a model for your use case (based on one of the projects I worked on)?

      • Finalizing a model involves:
        • Evaluating model performance against validation datasets using appropriate metrics.
        • Fine-tuning based on specific use cases.
        • Conducting A/B testing to compare against baseline models.
        • Incorporating feedback loops from real system interactions.
    9. How can you do text extraction from PDF, especially tables and images?

      • Text extraction can be accomplished using libraries such as:
        • pdfplumber: for text and table extraction.
        • Pytesseract: for OCR on images.
        • Tabula: for structured table extractions. This involves importing the PDF, identifying text and table structures, and converting them into a structured format like CSV or DataFrame.
    10. How do you measure latency of LLMs?

      • See answer to question 7.
    11. How will you evaluate 2 sets of embeddings?

      • To evaluate embeddings:
        • Cosine Similarity: Measure how similar the embeddings are by calculating the cosine distance.
        • Downstream Task Performance: Compare model performance on a specific task, such as classification or clustering, using these embeddings.
        • Visualization: Use techniques like PCA or t-SNE for visualizing clusters formed by the embeddings.
    12. Write a logic to get data from Kafka transaction stream and find the traffic for a particular zipcode in the last 30 minutes.

      • You could employ a Spark or Kafka consumer in Python to:
        from kafka import KafkaConsumer
        from datetime import datetime, timedelta
        import json
        
        consumer = KafkaConsumer('traffic_topic')
        current_time = datetime.now()
        time_threshold = current_time - timedelta(minutes=30)
        
        for message in consumer:
            data = json.loads(message.value)
            if data['zipcode'] == 'desired_zipcode' and datetime.fromisoformat(data['timestamp']) >= time_threshold:
                # Process traffic data
        
    13. Write a logic to find a score of the variety of content a user is watching based on their recent behavior.

    • You could define a content_variety_score based on diversity in program types:
      def content_variety_score(watch_data):
          unique_programs = set(d['program_id'] for d in watch_data)
          return len(unique_programs)  # Score based on unique program count
      
    1. Ever worked on Streamlit?

      • Yes, I have utilized Streamlit for creating interactive web applications that showcase machine learning models and visualizations. It allows rapid prototyping and deployment without extensive front-end experience.
    2. Which technologies have you used?

      • I have used Python with libraries such as TensorFlow and PyTorch for model training, along with tools like Flask for web hosting, Pandas for data manipulation, and Streamlit for deployment.
    3. How would you evaluate the RAG pipeline?

      • The evaluation of a RAG pipeline involves:
        • Assessing retrieval effectiveness (precision, recall of retrieved documents).
        • Analyzing the quality of generated content using human evaluations or automatic metrics like BLEU, ROUGE.
        • Measuring end-to-end latency for the pipeline.
    4. What metrics are there in RAGAS?

      • RAGAS employs various metrics such as:
        • Precision/Recall: For evaluating the retrieval component.
        • F1-score: A balance between precision and recall.
        • Contextual Relevance: Measuring how relevant the generated response is to the query context.
    5. Explain Context Precision metric.

      • Context Precision measures the accuracy of information retrieval in terms of how contextually relevant the retrieved content is to the user’s query. It can be calculated as the proportion of retrieved items that are relevant within the context specified in the query.
    6. What is Reciprocal Rank Fusion?

      • Reciprocal Rank Fusion is a method used to combine the outputs of multiple retrieval systems, where the rank positions of the documents from each system are combined to produce a single ranking. Each document's final score is determined by the reciprocal of its rank position across all systems.
    7. How would you extract graphs and tables from the PDF document?

      • For graphs, one would use OCR scraping techniques (with tools like Pytesseract) to interpret images. For tables, libraries like pdfplumber can be used to identify table structures and extract them into a structured format.
    8. How will you write a prompt that can generate exactly 2000 words for an essay on a given topic?

      • Create a prompt that specifies:
        "Write an essay about 'Bertrand Russell’s - A History of Western Philosophy' that is exactly 2000 words long. Ensure to cover key philosophical themes and thinkers concisely while adhering to the word limit."
        
    9. Projects undertaken -> Specify if they were Gen AI projects -> Technical details about the project -> Challenges faced -> Steps taken to mitigate them.

      • One project involved developing a chatbot using LLMs (Gen AI). The challenge was to ensure natural conversation flow, which we mitigated through iterative testing and refining prompts based on user feedback.
    10. Write a validation function in Python that can validate the citations generated from raw text.

      • A simple function could look like this:
        def validate_citation(raw_text, citation):
            found = citation in raw_text
            match_percentage = (raw_text.count(citation) / len(raw_text.split())) * 100
            return found, match_percentage
        
    11. How to evaluate 2 LLMs on latency?

      • Latency evaluation for 2 LLMs can be done by following similar steps as previously described: measure the time taken for each model to respond to the same set of queries and compare the average response times.
    12. How to restrict LLM to output in a specified format and how to validate that?

      • Restricting the output can be accomplished through clear prompt instructions. Validation can involve implementing JSON schema validation or post-processing checks on the generated text to verify format and content.
    13. What projects have you undertaken?

      • I have undertaken several projects, including a real-time sentiment analysis tool for social media, a recommendation system for e-commerce, and a chatbot for customer support.
    14. How did you convert words into embeddings for KMeans?

      • Words can be converted to embeddings using pre-trained models like Word2Vec or BERT. These embeddings serve as input to KMeans for clustering similar words or phrases based on their contextual meanings.
    15. Code for content variety score for churn prediction.

      • The code might look like:
        def content_variety_score(data):
            user_content = {}
            for entry in data:
                user_id = entry['client_id']
                if user_id not in user_content:
                    user_content[user_id] = set()
                user_content[user_id].add(entry['program_id'])
            variety_scores = {user_id: len(content_ids) for user_id, content_ids in user_content.items()}
            return variety_scores
        
    16. What metrics are there to evaluate the latency of 2 LLMs?

      • Common metrics include:
        • Average Response Time: Average time taken to generate responses.
        • 99th Percentile Latency: Time taken for the slowest 1% of requests.
        • Throughput: Number of requests handled per unit time.

    I hope these answers provide a comprehensive coverage of the questions! If you need any more elaboration or adjustments, feel free to ask.