AlgoTune

Can Language Models Speed Up General-Purpose Numerical Programs?
Can language models optimize the runtime of popular algorithms like gzip compression, AES encryption or SVD? To answer this, we built AlgoTune, a benchmark consisting of more than one hundred widely used math, physics, and computer science functions. For each function, the goal is to write code that is faster than the reference implementation while producing the same outputs as the reference, on a held-out test set of inputs. In addition to the benchmark, we also developed AlgoTuner, an agent which enables language models to iteratively optimize code.
This site contains AlgoTuner trajectories for all AlgoTune tasks. Each entry shows the complete conversation between the model and the AlgoTune environment, including code edits, timing evaluations, and the iterative optimization process.
The AlgoTune score for each model is the harmonic mean of its speedups across all AlgoTune tasks. In the table at the bottom of this page, you can find the speedups achieved by each model on each AlgoTune task.
To measure speedups for the algorithms in AlgoTune, we implement a class containing three functions for each algorithm. One generates problem instances (i.e. in the case of PCA this a matrix and number of components), one method checks that the problem has been solved (i.e. for PCA, we check that the matrix is orthonormal), and the last function is a reference solver (for the PCA task, we just use a PCA solver from scikit-learn).