help me to write summary in PBI. summary is intended to my colleagues, but tone should…
- User
help me to write summary in PBI. summary is intended to my colleagues, but tone should not too formal, just normal human like, but not too casual. The task was this: The "apply markdown formatting" prompt is currently part of the LLM call flow and appears to be a significant contributor to latency (e.g., ~4.5 seconds in the 99th percentile). This SPIKE aims to investigate whether this step is:
Analyze the logic and necessity of the "apply markdown formatting" step.
Determine if LLM formatting can be delayed, skipped, or handled differently for first-prompt responses.
Identify any downstream dependencies or user experience impacts of removing this prompt. The results: though in most cases markdown formatting looked acceptable already after 'rag prompt' though there are several cases where rag prompt produces broken markdown, which is fixed by "apply markdown formatting" prompt. if we would use current rag prompt without changes, it produces around 6 cases out of 90 of broken markdown, with some additional changes (better example, some additional instructions) it produces 2 cases of broken markdown. Looks like with current gpt-o-mini model it is too early to drop that additional formatting step
- Assistant
PBI: Summary of Markdown Formatting Spike Investigation
Title: Investigate and Optimize "Apply Markdown Formatting" Step for Latency Reduction
Description:
Hey team, I wanted to share a quick recap of that spike I ran on the "apply markdown formatting" step in our LLM call flow. As you know, this step is a bit of a bottleneck, adding noticeable latency—like around 4.5 seconds in the 99th percentile—which could be impacting user experience. The goal was to dig into whether we can simplify or remove it without causing issues, so I focused on analyzing its necessity, exploring options to delay or skip it for first-prompt responses, and checking for any downstream dependencies or UX risks.From the investigation, the results were mixed but insightful. In most cases, the markdown coming out of the 'rag prompt' looks solid and doesn't need extra fixes. However, there are still some edge cases where it breaks down—without any changes, we saw about 6 out of 90 test cases with broken markdown. I tried some tweaks, like adding better examples and more detailed instructions to the rag prompt, which brought that down to just 2 broken cases. That said, with the current GPT-o-mini model, it's still not reliable enough to drop the formatting step entirely, as those remaining issues could lead to poor user experience or downstream problems (e.g., inconsistent rendering in our UI).
Overall, this isn't ready to cut yet, but it's a step in the right direction. We could look at refining the rag prompt further or finding ways to make the formatting less latency-heavy, like handling it asynchronously. Let me know your thoughts—happy to discuss next steps in our next sync!
Acceptance Criteria:
- Documented findings shared with the team.
- Recommendations for potential optimizations outlined.
- No immediate changes to production until further validation.
Priority: Medium (since latency is a concern, but stability comes first). Assignee: [Your Name or Me] Estimated Effort: 2-4 hours for review and follow-up.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.