Optimizing Table Data Formats for LLMs: Token Efficiency and Accuracy

When it comes to building reliable AI systems, especially those involving large language models (LLMs), one often overlooked aspect is the format used to pass tabular data. Whether you’re using markdown tables, CSV, JSON, or something else entirely, your choice can significantly impact both the accuracy of your system and the cost associated with token usage.

Why This Matters

System Accuracy

If the data isn’t formatted in a way that’s easy for an LLM to consume, you might be unnecessarily reducing the accuracy of your entire pipeline. For instance, if the model has trouble parsing the structure of the data, it may misinterpret or miss key information.

Token Costs

Different formats can vary widely in terms of token usage. Some formats use several times more tokens than others to represent the same data. Since many LLM providers charge based on token consumption, your choice of format can directly affect your inference costs.

Methodology

To understand which format works best, we conducted a controlled experiment using GPT-4.1-nano, a popular and powerful LLM. Here’s how we set it up:

• Dataset: 1,000 synthetic employee records with 8 attributes each (ID, name, age, city, department, salary, experience, project count).
• Questions: 1,000 randomized queries about specific data points.
• Model: GPT-4.1-nano.
• Formats Tested: 11 different data representation formats.

Example Question-Answer Pairs

Q: "How many years of experience does Grace X413 have? (Return just the number, e.g., '12'.)"
A: "15"

Q: "What is Alice W204's salary? (Return just the number, e.g., '85200'.)"
A: "131370"

Notes on Methodology

We passed a relatively large number of records to the LLM to test its limits. In practice, with large structured datasets, you might want to chunk the data and/or query it to extract only the most relevant information before passing it to the model.

For formats like CSV, HTML tables, and markdown tables that include headers, repeating those headers periodically (e.g., every 100 records) can help with understanding. However, for simplicity, we didn’t do this in our tests.

Results: How Well Did the LLM Understand Each Format?

We evaluated the accuracy of the LLM’s answers across the 11 tested formats:

• Markdown-KV: 60.7%
• XML: 56.0%
• INI: 55.7%
• YAML: 54.7%
• HTML: 53.6%
• JSON: 52.3%
• Markdown-Table: 51.9%
• Natural-Language: 49.6%
• JSONL: 45%

Key Takeaways

• Markdown-KV emerged as the most accurate format, with a 60.7% accuracy rate.
• JSONL performed the worst, with only a 45% accuracy rate.
• Formats like Markdown-Table and Natural-Language were less effective, possibly due to their more complex structures.

Token Efficiency

While not explicitly tested in our experiment, it’s worth noting that formats like CSV and JSON tend to be more token-efficient compared to markdown tables or natural language. For example, a simple CSV record might look like this:

ID,name,age,city,department,salary,experience,project_count
X413,Grace,32,San Francisco,Engineering,150000,15,5

In contrast, the same data in a markdown table would use more tokens due to the additional formatting:

| ID   | Name  | Age | City          | Department | Salary  | Experience | Project Count |
|------|-------|-----|---------------|------------|---------|------------|---------------|
| X413 | Grace | 32  | San Francisco | Engineering| 150000