This only pertains to benchmarking 'speedup' type scenarios. Most of the lessons are obvious in retrospect, but hard earned. This is not about profiling, but rather about setting up scripting so that one has stable end to end speedup.

§ What Not To Benchmark

• If you plan to measure a speedup that will be less than, say 1.2x, then maybe go back and find something better to do instead, since such speedups can often just be system noise. The scariest paper that explains this is Producing wrong data without doing anything obviously wrong , where things like link order of object files, and the executable name (which affects alignment) can cause upto 20% performance deltas.
• This eventually led me to decide to only work on projects where I know I have asymptotic gains, or incredible reasons to believe I will see performance (e.g. a better parallel algorithm for a sequential thing, or some such).

§ Things Not To Read Because I Have Read Them For You

• The SIGPLAN empirical evaluation guidelines are utterly unactionable. Thus, I dislike them intensely, since reading them provides one with no information on what to do next.
• System Benchmarking Crimes By Gernot Heiser is good, but IMO easy to condense into (a) don't use percentages, and (b) don't lie.
• How not to lie with statistics: The correct way to summarize benchmark results is good, but is also easy to condence into (a) use geomean.
• Scientific Benchmarking of Parallel Computing Systems is good as well, but wasn't relevant to me since it's about benching parallel/distributed algorithms, while I have only really needed to bench single-machine algorithms.

§ What To Benchmark For

• If you have a real good reason for why you expect a speedup (for example, an algorithmic, asymptotic improvement, or a real win through mechanical sympathy), then go ahead and benchmark it.
• In this case, you need to be able to measure both (a) the speedup, as well as (b) the causal reason for the speedup.
• For example, if one is implementing an asymptotically better algorithm, then one should see the right asymptotics upon plotting.
• If one builds an algorithm with much better cache locality, then measure cache hits using approrpiate tooling (e.g. Intel VTune or linux perf), and report this.
• If you do something, and see speedups with no explanation , then you are not doing science, and the speedup can be safely ignored !

§ How to Benchmark What We're Looking For

• Begin by deploying the project at hand as a binary (NOT a shell script, a single binary). The binary prints minimal info to stdout, which is captured, and explains success/failure by exit code. This makes it trivial to parse, capture output, and to use tools like runlim to fix space and time usage.
• Run the tool on each problem, with K runs per (problem, tool) variant. Have each run write its data into a folder corresponding to the run as a JSONL file. JSONL can be trivially catenated by concatenating all the files, which makes it great to aggregate,
• This raw data is processed by first catenating into a single jsonl file by a trivial cat runs/latest/data/**.jsonl > runs/latest/aggregate.jsonl. This can then be loaded by polars (in python) to compute statistics and plot with.
• I always plot using a cactus plot for solvers See 'Benchmarking Solvers, SAT style , since it provides a nice way to compare speedups. I don't know of a similarly nice way to calculate speedups for things like compiler optimizations. In theory, one can use the same methodology, but in practice, I haven't seen anyone do so.
• To compute overall speedups, use geomeans. The key motivator for me is that we typically want to measure the speedup of our algorithm against a baseline, and the nice thing about geomeans is that the geomean of the speedup is the speedup of the geomean, which ensures that we can just take the ratio of the geomeans to measure speedup. This tells us what what we are measuring is 'correct', spritually speaking.
• Whatever 'plotting' you choose to use, don't throw away data. Also, print your data also as a table, not just a figure, becase eyeballing a table makes it easy to see aggregate information.
• Don't be clever. Setup stupid systems, and parallelize on a large machine by spawing the runs in parallel with time and memory limits.

§ Emails With Mate Soos, A Person Who Regularly Benchmarks Computers

• Mate Soos is an expert at developing solvers, and currently owns the world's fastest randomized sharpsat solver, so I take what he says somewhat seriously.
• I use ulimit to limit each process's memory etc limits.
• I make sure none of them can use more memory than K, where the CPU has at least B*K memory, where B is the number of processes I will be running on the CPU.
• I make sure the CPU has at least B cores. Not threads, cores.
• I make sure that I do the copying of all data to the local HDD of the machine before I start the process
• I run all processes under /usr/bin/time -v and write its output to a separate file with -o FILE and save, and parse it It tells me about max memory usage, USER and SYSTEM time, wallclock time, memory pages etc. This is essential data, and can be fully relied on.
• I build all my systems to have a single binary. If you have multiple binaries that call each other, you are gonna be in a world of pain and have to use runlim it's good but annoying to use.
• I make sure my systems are useable, single binary, no shell script or stuff like that. Then I don't have to use runlim.
• In general, if you control what you are running, you can make your life a lot easier.
• I then get all the data off the system to my local machine, and process it with a python script into a CSV which I then import into an SQL database.
• I then query this SQL database to generate gnuplot files, and to generate summarized data

get_data.py gets the data from the files, create_graphs_ganak.py generates graphs, summarised tables, jupyter notebook, etc. Note that I used to write bash scripts. I'm actually okay at bash scripting, but python is a LOT more robust and a LOT easier to maintain and improve. Don't forget to add lots of checking and error-outs and asserts into that script, so you don't accidentally parse wrong data.

Even if you do all the above, there'll be variation. Quite a bit of it, maybe up to 5-10% in some cases, on a single file. That's life. Computers have CPU power limits and dynamic clocks and sleep states and shared CPU caches, and other processes running and network latency etc. If you are not prepared to deal with that, then you are gonna be sad. You need to run at least 400 instances every time and then the variation evens out.

In general, if you are spending more than 5 minutes to schedule a cluster run, or you are spending more than 5 minutes getting the data and crunching it from the cluster, you are doing it wrong. You should dedicate at least a 2-3 days to writing the initial scripts, and examining failures, etc. Then keep improving the script every time. Every single time. Just spend 10-20 minutes improving it when you do a run and want more data, more summaries, better tables, etc. I once saw a PhD student spending 8h hand-crunching the data after a cluster run, and he only got about 3-4 data points. I sometimes do 30-40 cluster runs in a week, and get 40+ data points from each. It would have taken the guy a year what took me about 1 hour. Then they are surprised they can't win a competition, even though they spent weeks on the cluster data. It turns out, it's not how much you work, it's how efficiently you work.

§ Conclusion