One of the client i consult for wants to build a enterprise customer facing support…
- User
One of the client i consult for wants to build a enterprise customer facing support agent which would be able to talk to at least 30 different APIs using tools to answer customer queries. Also has multi level workflows like check this field from this API then follow this path and check this API and respond like this to the user. Tried llama, gemma, qwen3. So far best results we got was with llama3.3:70B hosted on a beefy machine. Cannot go to proprietary models for data concerns. Are open source models at a stage for using at this scale and complexity?
- Assistant
Open source models have made significant progress, but what you're describing - an enterprise support agent handling 30+ APIs with complex conditional workflows - is pushing the boundaries of what current open source models can reliably deliver.
Your experience with Llama 3.3 70B aligns with the current state: it's among the best open source options, but there are several challenges at this scale:
Tool calling reliability: Even the best open source models struggle with consistent tool use, especially when dealing with dozens of APIs. They often hallucinate API responses, miss required parameters, or fail to chain multiple API calls correctly. Error rates compound quickly when you need multi-step workflows.
Context management: With 30+ APIs, you'll need extensive system prompts describing each API's schema, parameters, and usage patterns. This eats into context windows rapidly. While models like Llama 3.3 have decent context lengths, maintaining accuracy across long contexts with complex instructions remains challenging.
Workflow execution: The conditional logic you described ("if this field equals X, then call API Y") requires strong reasoning capabilities. Open source models can handle simple branching, but complex decision trees with multiple conditions often lead to errors or the model getting "lost" in the workflow.
Some strategies that might help:
-
Hybrid approach: Use the LLM primarily for understanding intent and generating responses, but handle the actual API orchestration and workflow logic in traditional code. The LLM decides what needs to be done, your system executes it deterministically.
-
Specialized fine-tuning: Fine-tune specifically on your API calling patterns and workflows. This can significantly improve reliability for your specific use case.
-
Ensemble methods: Use multiple smaller models for different tasks - one for intent classification, another for parameter extraction, etc.
-
Aggressive prompt engineering: Break down complex workflows into smaller, more manageable chunks with very explicit instructions.
The reality is that while open source models are improving rapidly, this level of complexity typically requires either significant engineering around the model's limitations or acceptance of a higher error rate than you might want for customer-facing applications. You might need to constrain the scope initially - perhaps starting with 5-10 critical APIs and simpler workflows, then expanding as you refine the system.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.