How to avoid mistakes in benchmarks

I once saw a post on LinkedIn with a clickbait title: ”.NET 9 is slower than .NET 8”. A strong claim. Of course, I had to verify it — I’m a big fan of benchmarking myself. Let’s skip straight to what’s wrong with the benchmark.

❌ Methods don’t return a result

Modern compilers are smart. They can detect when code doesn’t affect the program and just remove it. This optimization is called dead-code elimination (DCE). In this case, the DoSomeThing method essentially does nothing, so the compiler removes it in the For method. But in ForEach_Linq, it can’t do that because a delegate is created. So we end up comparing two methods with different behavior (see Figure 2).

✅ Always return something from your methods

The fixed version might look like this. We don’t care what the sum variable holds. The key is that using it prevents DCE and doesn’t significantly affect benchmark results.

❌ Blindly comparing foreach

As we know, foreach is designed for collections that return an enumerator. But even if the collection supports an enumerator, that doesn’t mean the compiler will use it. See this example in Figure 3:

• foreach over an array turns into a while loop with an indexer.
• foreach over a List becomes a while loop with List<int>.Enumerator.
• foreach over ICollection<int> becomes a while loop with IEnumerator<int>.

✅ Compare foreach performance across different types

Since foreach compiles into different code depending on the collection type, performance can vary a lot. That’s why it’s important to compare each case separately. This behavior might change in the future, though.

❌ Passing methods as parameters

Passing a method as a parameter creates a delegate. In our example — an Action<int> (see Figure 4). A delegate is a reference type, and creating one means memory allocation, which affects benchmark results.

✅ When passing a method, pre-create the delegate

It’s better to create the delegate in the GlobalSetup method, which runs before the actual benchmarks.

Conclusion

Now let’s look at the results of the corrected benchmark. “Bad” refers to the original benchmark, and “Better” is after fixes.

In the Bad benchmark (run on my laptop), the ForEach_LinqParallel method was actually a bit faster on .NET 9 than .NET 8. The difference mentioned by the author was within the margin of error.

In the Better version, ForEach_LinqParallel was 90 microseconds slower on .NET 9 than on .NET 8. I would still consider that a negligible difference and wouldn’t make a big deal out of it.

Overall, results across all three versions of .NET seem more or less the same. So, I wouldn’t go so far as to say that ”.NET 9 is slower than .NET 8.”