LLM Function Calls Hit a Wall; Code Orchestration Offers a Scalable Solution

---

When it comes to working with Model-Composed Programs (MCPs) and integrating them into real-world applications, one common practice is to feed the outputs from tool calls back into the Large Language Model (LLM) as messages. The idea is that the model will interpret this data and determine the next steps. This approach can be effective for small datasets, but it quickly becomes problematic with larger, more complex data.

The Problem with LLM Function Calls

Let's dive into why LLM function calls don't scale well:

• Large JSON Blobs: When using MCP servers, such as those from Linear and Intercom, the tool calls often return large JSON blobs. For example, when we asked Linear's MCP to list issues in our project, it returned 50 issues, which amounted to approximately 70k characters or 25k tokens.
• Token Overhead: These JSON responses are bloated with id fields and other metadata that take up many tokens but offer little semantic value. This inefficiency can quickly exhaust the token limit of LLMs, leading to slower processing times and higher costs.
• Data Reproduction: If you want the AI to perform operations like sorting issues by due date, it would need to reproduce all the issues verbatim as output tokens. This is not only slow but also prone to errors, especially when dealing with large datasets.

Real-World Examples

To illustrate this, consider our use case with Linear and Intercom:

• Linear MCP: Listing 50 issues resulted in a 70k character response, which is roughly 25k tokens. This includes a lot of metadata and id fields that are not semantically meaningful.
• Intercom MCP: Similarly, the tool calls returned large JSON blobs without predefined schemas, making it difficult to parse the data efficiently.

When using Claude with these MCPs, the entire JSON blob is sent back to the model verbatim. This approach can lead to significant performance issues and data loss, as the model might fail to accurately reproduce or process all the data.

Data vs Orchestration

The core issue here is that we are conflating orchestration (managing the workflow) with data processing (handling the actual data). This confusion leads to inefficiencies and scalability problems. Here’s how code orchestration can help:

• Structured Data Processing: Instead of feeding large JSON blobs back into the LLM, we can parse the data and operate on structured arrays. For example, if you need to sort issues by due date, you can perform a sort operation directly on the parsed data.
• Avoiding Hallucinations: By separating orchestration from data processing, you reduce the risk of hallucinations (where the model generates incorrect or irrelevant information). This is particularly important when dealing with complex datasets that contain detailed information like steps to reproduce issues or user instructions.

Code Orchestration in Action

Let’s walk through a simple example to see how code orchestration can be more effective:

1. Fetch Data: Use an API call to fetch the data from Linear or Intercom.
2. Parse JSON: Parse the JSON response into a structured array of issues.
3. Perform Operations: Use standard programming constructs (like sort) to process the data as needed.
4. Generate Output: Feed the processed data back into the LLM for further orchestration.

By following this approach, you can handle large datasets more efficiently and avoid the pitfalls of token overhead and data reproduction.

Conclusion

While LLM function calls are a powerful tool, they hit a wall when dealing with large, real-world datasets. By separating orchestration from data processing using code orchestration, we can achieve better performance, reduce costs, and minimize the risk of errors. This approach is more scalable and aligns well with the structured nature of modern APIs.