Software performance is a contentious topic among programmers. On one hand, you'll see Jonathan Blow arguing programmers should spend their valuable time bikeshedding small details to cut down a few CPU instructions, and on the other you'll see people running an electron app that runs on a hypervisor that runs on a VM, claiming everything's just fine.

Ultimately, a large portion of this discussion comes to opportunity costs and incentives, which are domain and case specific, which I believe isn’t particularly fruitful to rehash in this essay. I will, however, propose a framework for discussing what to optimize, and why. I have heard similar arguments in discussions before, so there's certainly some folk knowledge on this, although I don't have any formal citations, Martin Fowler seems to have influenced some of these ideas, so I thank him for that.

The core thesis I propose is that, performance, and performance requirements, fundamentally change the way software is used. The faster we can run a program, we can run it more. A program that takes a fraction of a second can be run without disrupting the user at all, while a program that takes a few seconds must require explicit user approval and action to run. If it takes a minute, the user will probably run it while checking their social media in the background, a 10 minute program will only run at coffee breaks, a 30 minute program will probably run in the CI or overnight. The best example of this is perhaps textual analysis of code. State of the art type checkers are fast and incremental, and they can provide almost immediate feedback to the users in most cases. These systems are now fundamental parts of the activity of programming, to the point it is expected for new languages to have LSP integrations and type inference to be counted as "usable".

Another important example, which is in fact related to my research, is in random testing. Random testing has many flavors, the most common is no doubt fuzz testing, which is roughly defined as generating random bytes as inputs to the system under test, using many heuristics, mainly coverage information acquired by instrumenting the system, in an effort to find crashes. My research focuses on Property-Based Testing, which, in contrast to fuzz testing, focuses on very short time horizons in the order of milliseconds, seconds, or at most a few minutes; while you'll see fuzz testing running in continuous integration systems for hours or days. It is no doubt both approaches are useful on their own, and they have been used interchangeably in many cases, but the focus, the incentives, the evaluations of PBT and fuzz testing differ vastly, which I believe the budget plays a big role in, because the budget changes the trade-offs we need to make, hence the maneuvers we can take to explore new possibilities in the design space.

This view of performance gives us another avenue to think of, other than the classical "faster is better" mantra. Of course, given everything is constant, faster is better, but you can also think of what are the interesting possibilities with a larger time budget? Imagine a game engine that can run 60 FPS in real time for a given configuration. For anything that does not require real time processing, you can increase the FPS, or give it a much better rendering quality that you wouldn't have the opportunity to do otherwise. Same thinking follows the other way. For any given software, and what you do with it, you can at many times conjure new possibilities with a lower time budget. Today, we're used to grammar checkers or type checkers running in the background, giving us instant feedback, but our unit tests are running only when we explicitly run them, because they are not instant. What if you had an environment where any relevant test would automatically run as you type, could that lead to higher-quality software?

Our core assumptions on what we can do with software are limited by our assumptions on how fast we can run it. Coming from the land of WYISYG (what-you-see-is-what-you-get) editors, it is actually very weird to realize that writing LaTex documents require you to explicitly compile them, because LaTex has not been designed with the goal to be that fast. I see the problem from the other direction in many cases. People see how slow something is, and they asssume that is the normal speed, that's the way things were, are, and will be; but that doesn't have to be the case, the underlying hardware we use are immensely powerful, to the degree that software engineers have been blessed with ignorance that they can just not care about performance, but those that care can create great things. Figma has broken the mold of what a web app can do, and it is today worth 70 billion dollars, although I'm sure for reasons far beyond what I understand, or could discuss. This type of thinking gives us a framework to think about software optimization that exceeds serving more users with less resources, it lets us think about how we can run a process more often, or how we can run more processes at the same time. It, on the other hand, gives us a simple way of cost reduction and preservation of resources. If you want to spend less money or compute, you can just do less, and you will compute less. For instance you can take your realtime system that updates 100 times per second and turn it into a system that updates every 2 minutes, you will be running 1/200th of what you previously did. Although this type of thinking is very very simple, I find having to remind myself that user experience is not set in stone, and every part of your software is a trade-off between implementation time, execution time, and user experience. You can either spend more time to optimize, or more time to run, or you can just degrade the user experience by running less often or with less quality.

Before concluding, I will mention LLMs as they seem to consume so much of our daily discussions. I think of LLMs today a bit similar to LaTex in their usage. I think we have taken their current velocity as given, but they are slow. I think so many new avenues would explore if they were 10x, or perhaps a 100x more performant. You could instantly generate 5 different potential results to a user and allow them to choose, as if they are picking options from a dropdown in a form. You could feed much larger data, increase the amount o information both the system providess, and also consumes, creating a real assistant that can run seamlessly alongside your tasks instead of constantly breaking your flow, marketed by "let your agent run for 30 minutes while drinking your coffee" that is the common sentiment today. Faster internet has given us constant connectivity, streaming, real-time video calls, even real-time robotic control in medical operations, I imagine faster models will change the way technology is used in similar ways, and we are still at the beginning of what is possible.

Perhaps I was naive, thinking most software engineers were as clueless as I am on thinking about the ramifications of performance and how it can change the systems we build, but if not, I hope this essay has given you a new perspective on how to think about performance, and how it can be used as a constraint to design better software. The next time you are faced with a performance problem, I encourage you to think not just about how to make it faster, but also about what new possibilities a different time budget could open up for your software.