I have done a fair amount of low level performance optimization with Opus 5 and its reasoning is still very poor. Like why is CRC so slow and going through loops until I ask it if is using hardware instructions and it tells me it is using its own hand coded implementation poor. Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.
> Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.
I suspect a lot of the training set for this sort of thing is people online speculating about cache performance incorrectly.
Speaking as somebody who is a performance geek, my knowledge came from relentless experimenting over the years (starting in the 8 bit era). Beyond the basics, I haven’t seen much on high-performance engineering online. To learn, you need to do the hard yards and I think performance tweaking becomes almost instinctive rather than something driven by a hard set of rules.
At a low enough level, every performance tweak becomes unique and bespoke.
Of course, you could find people online talking about how to write high-performance code, but beyond a few basic techniques, their advice may not work for you — nobody can write a generalist article about performance engineering that will definitely solve the problem you have right now.
Arguably, there are fewer patterns for an LLM to infer as highly optimised code tends to become more and more opaque in the search for a nanosecond here or there.
I don't disagree, but IMO, a lot of code doesn't get to the point where those very low level techniques drive performance. Like, yes, if you are doing some heavy floating point math then that's where you end up needing it. However, in a lot of code finding hot paths and often simply switching out a O(n^2) for an O(n log n) or faster.
Getting and using tools to find hotpaths is generally the most important performance tweaking skill.
I would think it's that the kinds of places which value this kind of knowledge often have major disincentive to share it. I'm thinking of HFT firms as one example.
I see the same thing, except I was working on high level performance reasoning. Whenever a piece of code has multiple steps that require multiple algorithms to work, AI almost always fails to guess which step is the slowest and what causes that step to be slow. Even Fable makes wrong guesses. You definitely need to give them a benchmark feedback loop.
Are there actually any humans who can reason about things like cache performance from first principles? I know there are some people who think they can, and I suspect they're fooling themselves. The one iron principle of micro-optimization at the level of cache hits is "measure, measure, measure", you just cannot think your way to the right answer on the first try. Processors and instruction sets today are too complicated, and tips that worked on one generation might be neutral or worse on the very next revision, making all the cargo cult knowledge passed around on this topic at best useless. I'll echo one of the sibling commenters here and say that LLMs probably bullshit their way to answers on questions like this because that's what humans online do as well.
If you give an LLM a proper testing harness and feedback loop to actually generate hypotheses, test them and revise them, I suspect it will do much better.
can LLMs write better code if you keep asking them to “write better code”?
One of the keys for me was the use of types. Typestate when functions mint witnesses that can only come from it and are required to proceed and newtypes where you use custom types instead of strings so the agent can't forget. You can also use it to force the agent to use the implementation rather than reinvent the wheel by simulating linear types. Types are a much smaller target to optimize and provide constraints that fail loudly at compile time.
I've been having great results with this kind of thing. I find that really just need a sensible framework within which the optimization can take place. Essentially just providing the measurement harness, and some sort of motivation for what I'm doing.
The great thing about LLM is that it seems to have the checklist for everything. If I rattle off a few things like "don't allocate on the hot path" and "remember to pin the cores" it will come up with a few items of its own that I might have forgotten.
Eventually, it will have gone through the whole list with me, while having documented all the measurements along the way.
But it's still guided by experience. If I see unusual numbers, I might say "hey did you forget to compile it in release mode?" and it will apologize and fix that. If I don't, it may just continue exploring without realising everything is wrong.
Once I had repo commands that could dump `sample` results and a cpu profiler/trace and then a benchmark tool that let me A/A + ABBA/BAAB-test the current modified git workspace against HEAD or any commit, the LLMs could just do their thing.
And that's how my homemade terminal uses much less memory than ghostty/kitty/iterm yet has more throughput.
AI is going to increasingly unmask people and companies who don't care about correct and performant software now that it's become so trivial to guarantee both. It used to at least be expensive and time-consuming and expertise-demanding to do those things.
Agree it's amazing how much low-hanging performance fruit AI can trivially find. On the other hand though, once you get through the obvious no-brainer stuff, there's a lot of non-trivial tradeoffs in performance and I think that still demands a good amount of expertise to guide the AI in the right direction. Obviously AI will continue working it's way up the value chain, but I think there's a glass ceiling for AI where the right macro tradeoffs and perspectives on how software should work will bump into the hard and often articulated reality that different stakeholders want different things and often have either magical thinking or even self-deception about how those desires can co-exist with what everyone else wants.
This isn't a new problem by any means, but now that code is cheap, it means instead of getting frustrated with engineering and their pesky unimportant details, people will get frustrated with the AI and it's pesky unimportant details.
Yeah, the biggest example is performance optimizations that sacrifice your data model to the point that you'd never accept them.
I think it's one reason why ADRs are an important of a software project, especially with LLMs. You need a place were you can document invariants, why you have them + the rejected ideas and acceptable risks.
It helps smart agents like Fable help you decide on trade-offs and it's kind of incredible to witness that happening.
I think they can be useful for quickly iterating through benchmarks and trying lots of ideas, but they won't come up with them on their own. Also, I'm not sure why, maybe some mean reversion thing, but they will never, ever suggest writing a tool to make their own life easier, get more accurate information, or anything. Once I point it at a tool, it can be ok at using it (I say ok because they seem to skim the help docs, which is truly ironic, considering I seem to read it more thoroughly even though I'm 100x slower at it. I assume this is some token saving system prompt), but they won't suggest it for you.
This is why I'm not worried about being replaced for now or the forseeable future. For all of the improvements they've made, this part just never seems to change. They could slap another heuristic prompt for the edge case, but eventually it'll revert to the mean again.
I think there is a way to use LLMs to help with programming, but not when I'm not the driver in the seat writing the tests and deciding the architecture. Also I would never ship code written by them as the final product for anything I care about. Since I, like most people, find reading code to be arduous. The more fun thing to do is to force yourself to rewrite it all, treating the LLM's work as a rough draft.
> they will never, ever suggest writing a tool to make their own life easier, get more accurate information, or anything
Make sure you process doesn't depend on anyone reading your mind.
When I run into things like this, it becomes a one-liner in my instructions/harness or in the canned prompt/skill I use that sets off a process.
In this case, I instruct agents to proactively build/improve diagnostic tooling if it would help them with their task + if it meets a bar of generalization/reusability (else it should be an ephemeral probe that gets abandoned at the end of the solution).
> If it can be measured, then LLMs can optimize it
Then they can start attempting to optimize it. They can also spin round and round making the numbers worse because they don't actually know what to do.
I had this experience at work trying to optimize a little high level Pytorch. It can't really get better than it already was, but LLMs were quite willing to pretend they will. The real solution is I need to open a PR for one of Pytorch's tracking issues.
Yeah, but that's just the scientific process of hypothesis -> evidence -> conclusion.
You need a measurement that can falsify hypotheses and reject branches that won't work.
Also, if all you have left in your project are performance issues that are hard to identify without flailing around (even with Fable/Astra) despite sampler/profiler reports, then you're doing really well and I wouldn't assume you're going to fare much better than the sota models in terms of stabs in the dark.
The point of this post is that this is explicitly not the case. If the metric is measured, the agent finds a way eventually (around 5 total tries typically unless it gets stuck), and learns from iterations where changes caused a regression after a revert.
In one case I used a made-up metric (since I didn't know the exact name or if it existed) and it somehow optimized that too.
Absolutely agree. "Make this faster" even with a solid way to measure progress is often fraught. It kept doing some very strange micro improvements that never improved the performance in the large but greatly improved complexity. I was _very_ much in the loop on a recent performance improvement I made. I really tried to walk away and let it find me the answer, but in the end I strongly guided it. As a lab assistant it was a huge help, but it messed up finding the improvement a number of times until I finally strongly intervened.
I keep the benchmark harness checked in and read-only to the agent, and let it touch everything else. Once it can't rewrite the measuring stick, the gaming just stops.
They're quite good at just iterating different "ideas" on a performance metric with an objective measure. They can use tools like `perf` and do some analysis on the output. Sometimes they go off in the weeds unproductively, and sometimes they give up because your goal was too high, but as long as you're sort of babysitting the process, you can make pretty rapid improvement to naive code.
"Rank the top findings/solutions by impact vs confidence" continues to be one of my best quickwins to add to all sorts of prompts. Bam, now you have a reasoned priority list.
As the author of Tera mentioned in the article, I am curious how it can get 2x faster. Was the benchmark using tera v2 with the `fast` feature enabled?
They were with tera 2.1.0, however using the default settings and not enabling the `fast` feature (which I did not know about, TIL).
The comparison benchmark results in the post were older and prior to further passes. After updating to tera 2.4.0 and enabling the `fast` feature set, I reran the benchmarks against the current codebase and the difference between my Rust crate and Tera is now about ~1.5x.
Some of the technical differences are (which in disclosure were surfaced by GPT 5.6 Sol): context and loop values are borrowed while Tera clones, cached static positions/lookups, less per-render heap setup, escaping chunks directly into the final String instead of bytes, and more granular bytecode functions which avoid stacks.
one thing nobody mentioned here, once the agent is looping against the same benchmark it will happily optimize for the benchmark itself and not the real workload. worth rerunning the win against a slightly different input shape after, just to check it did not memorize the harness instead of actually fixing anything.
I suspect a lot of the training set for this sort of thing is people online speculating about cache performance incorrectly.
At a low enough level, every performance tweak becomes unique and bespoke.
Of course, you could find people online talking about how to write high-performance code, but beyond a few basic techniques, their advice may not work for you — nobody can write a generalist article about performance engineering that will definitely solve the problem you have right now.
Arguably, there are fewer patterns for an LLM to infer as highly optimised code tends to become more and more opaque in the search for a nanosecond here or there.
Getting and using tools to find hotpaths is generally the most important performance tweaking skill.
If you give an LLM a proper testing harness and feedback loop to actually generate hypotheses, test them and revise them, I suspect it will do much better.
One of the keys for me was the use of types. Typestate when functions mint witnesses that can only come from it and are required to proceed and newtypes where you use custom types instead of strings so the agent can't forget. You can also use it to force the agent to use the implementation rather than reinvent the wheel by simulating linear types. Types are a much smaller target to optimize and provide constraints that fail loudly at compile time.
The great thing about LLM is that it seems to have the checklist for everything. If I rattle off a few things like "don't allocate on the hot path" and "remember to pin the cores" it will come up with a few items of its own that I might have forgotten.
Eventually, it will have gone through the whole list with me, while having documented all the measurements along the way.
But it's still guided by experience. If I see unusual numbers, I might say "hey did you forget to compile it in release mode?" and it will apologize and fix that. If I don't, it may just continue exploring without realising everything is wrong.
Once I had repo commands that could dump `sample` results and a cpu profiler/trace and then a benchmark tool that let me A/A + ABBA/BAAB-test the current modified git workspace against HEAD or any commit, the LLMs could just do their thing.
And that's how my homemade terminal uses much less memory than ghostty/kitty/iterm yet has more throughput.
AI is going to increasingly unmask people and companies who don't care about correct and performant software now that it's become so trivial to guarantee both. It used to at least be expensive and time-consuming and expertise-demanding to do those things.
This isn't a new problem by any means, but now that code is cheap, it means instead of getting frustrated with engineering and their pesky unimportant details, people will get frustrated with the AI and it's pesky unimportant details.
I think it's one reason why ADRs are an important of a software project, especially with LLMs. You need a place were you can document invariants, why you have them + the rejected ideas and acceptable risks.
It helps smart agents like Fable help you decide on trade-offs and it's kind of incredible to witness that happening.
This is why I'm not worried about being replaced for now or the forseeable future. For all of the improvements they've made, this part just never seems to change. They could slap another heuristic prompt for the edge case, but eventually it'll revert to the mean again.
I think there is a way to use LLMs to help with programming, but not when I'm not the driver in the seat writing the tests and deciding the architecture. Also I would never ship code written by them as the final product for anything I care about. Since I, like most people, find reading code to be arduous. The more fun thing to do is to force yourself to rewrite it all, treating the LLM's work as a rough draft.
They can, in fact, generate plausible performance optimization ideas on their own.
Make sure you process doesn't depend on anyone reading your mind.
When I run into things like this, it becomes a one-liner in my instructions/harness or in the canned prompt/skill I use that sets off a process.
In this case, I instruct agents to proactively build/improve diagnostic tooling if it would help them with their task + if it meets a bar of generalization/reusability (else it should be an ephemeral probe that gets abandoned at the end of the solution).
Then they can start attempting to optimize it. They can also spin round and round making the numbers worse because they don't actually know what to do.
You need a measurement that can falsify hypotheses and reject branches that won't work.
Also, if all you have left in your project are performance issues that are hard to identify without flailing around (even with Fable/Astra) despite sampler/profiler reports, then you're doing really well and I wouldn't assume you're going to fare much better than the sota models in terms of stabs in the dark.
In one case I used a made-up metric (since I didn't know the exact name or if it existed) and it somehow optimized that too.
"Claude, if this idea doesn't measure as an improvement (use X benchmark and a T-test), discard it and try the next idea."
Unless its an easy memory/parallel/algorithmic win, its not worth it.
The comparison benchmark results in the post were older and prior to further passes. After updating to tera 2.4.0 and enabling the `fast` feature set, I reran the benchmarks against the current codebase and the difference between my Rust crate and Tera is now about ~1.5x.
Some of the technical differences are (which in disclosure were surfaced by GPT 5.6 Sol): context and loop values are borrowed while Tera clones, cached static positions/lookups, less per-render heap setup, escaping chunks directly into the final String instead of bytes, and more granular bytecode functions which avoid stacks.