GPT-5 Tackles EVTX Parsing in Zig: A Real-World Benchmark

GPT-5 Tackles EVTX Parsing in Zig: A Real-World Benchmark

Introduction

Over the past few days, I've been putting GPT-5 through its paces by benchmarking it on a real-world programming task. The goal? To write a parser for the EVTX format (a binary format used for Windows Event Logs) in Zig. This is a medium-sized problem that requires a few thousand lines of code and has a well-defined spec, making it an excellent test case for evaluating AI-generated code.

Why This Problem?

I chose this task for several reasons:

• Not Too Easy: It's not a trivial problem; you need to write a significant amount of code to get a working parser.
• Not Too Hard: While challenging, it's not as ambitious as writing a full-fledged system with 20k+ lines of code.
• Deterministic: There's a clear spec and reference parsers available, making it easier to verify correctness.
• Personal Struggle: I've worked on this problem before and know which parts are particularly tricky.

The Setup

I maintain an EVTX parser in Rust called evtx, which is still quite popular. When I originally wrote it, I used the spec from the Windows XML Event Log (EVTX) documentation.

The task for GPT-5 was straightforward:

1. Take the Spec and Write a Parser: Start with the spec and write a parser in Zig.
2. Match Output to Rust: Ensure the output of the parser matches that of my existing Rust implementation.

For an added challenge, I initially withheld access to the actual Rust code to see how well GPT-5 could interpret the spec on its own. Eventually, I allowed it to peek at the Rust code to refine its solution.

Why Zig?

I chose Zig for a few reasons:

• Interesting Technology: Zig is a modern systems programming language that's particularly good for writing binary parsers.
• No Cloning: By using Zig, I ensured that GPT-5 couldn't simply clone an existing implementation from memory or web searches. Public implementations exist in Go, Python, C, and C++.
• Personal Interest: I'm not very proficient in Zig, so this was a chance to see how well the model could handle it.
• Minimal Dependencies: Zig doesn't have a large ecosystem, which simplifies the task of writing a standalone parser.
• Out-of-Distribution: This is where smaller models and non-frontier ones typically struggle.

The Results

GPT-5 delivered impressive results:

• Complete Parser: It wrote a complete parser for the EVTX format in Zig.
• Optimization: The model optimized the code until it was faster than my carefully written Rust implementation, which was already the fastest available.

Here are some key points:

• Performance: GPT-5's Zig parser outperformed the existing Rust parser. This is significant because Rust is known for its performance and safety features.
• Correctness: The output of the parser matched the expected results from the Rust implementation, ensuring correctness.
• Code Quality: The generated code was clean and well-structured, making it easy to read and maintain.

Comparison with Previous Models

I've attempted this task with other models before, including Sonnet 3.5. Here's how GPT-5 compares:

• Sonnet 3.5: While capable, Sonnet 3.5 struggled to produce a fully working parser. It had issues with complex data structures and optimization.
• GPT-5: GPT-5 not only produced a working parser but also optimized it for performance, demonstrating a significant improvement in code generation capabilities.

Conclusion

GPT-5's ability to write and optimize a complex binary parser in Zig is a testament to its advanced code-generation capabilities. This real-world benchmark shows that GPT-5 can handle medium-sized programming tasks with high accuracy and performance.

If you're interested in the final result, the parser is