Single-file C# benchmarks with BenchmarkDotNet

If you ever wondered if it’s possible to run a benchmark from a single file, then the answer is yes. Starting from .NET 10, there is a possibility to create C# applications within a single *.cs file. The problem is that BenchmarkDotNet (BDN) doesn’t support this scenario with its default setup. In this article I will show how to use the InProcess toolchain as a workaround.

What is file-based C# application?

A file-based app is a single *.cs file that you can run directly:

// HelloWorld.cs
Console.WriteLine("Hello, world!");

Run it like this:

dotnet HelloWorld.cs
# or
dotnet run HelloWorld.cs

Why doesn’t BenchmarkDotNet work out of the box?

By default, it uses process-level isolation by generating, building, and running a separate console app for each of your benchmarks. Process-level isolation makes measurements accurate, stable, and not affected by the host process. In file-based apps, the .NET SDK uses a generated project and builds outputs to a temp location, so there’s no normal *.csproj for BDN to discover.

The workaround: run benchmarks in-process

If you’re okay with in-process benchmarking, you can bypass the project-generation step by forcing BDN to use the InProcess toolchain. But keep in mind that in-process benchmarks are less isolated and the results can be affected by the host process.

Here is the minimal working example:

#:package BenchmarkDotNet@0.15.8
#:property Optimize=true
#:property Configuration=Release
#:property PublishAot=false

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<Benchmarks>();

[InProcess]
public class Benchmarks
{
    [Benchmark]
    public Task Example() => Task.Delay(100);
}

You might have noticed that we used Optimize=true, Configuration=Release and PublishAot=false.

The Optimize and Configuration flags are required because otherwise .NET will execute your application in Debug mode without optimizations. And, as we know, we should never use Debug for benchmarking because it will spoil the results.

The PublishAot flag disables AOT compilation, which is enabled by default when publishing file-based C# applications. AOT might be useful if you want to reduce the application startup time. BDN supports AOT, but at the same time it requires an additional configuration for BDN.

Running the benchmark

To run the benchmark, simply execute:

dotnet run Benchmark.cs

If you’re Linux or macOS user, you will probably want to add a shebang. But first, you need to find the path to your dotnet executable:

which dotnet

Then add the shebang line to the top of your Benchmark.cs file. In my case, the path is /usr/bin/dotnet, so the shebang line will look like this:

#!/usr/bin/dotnet run
#:package BenchmarkDotNet@0.15.8
// ... rest of the code

Then make the file executable:

chmod +x Benchmark.cs

And run it directly:

./Benchmark.cs

Whatever way you choose to run it, you should see the benchmark results like this:

| Method  |     Mean |   Error |  StdDev |
| ------- | -------: | ------: | ------: |
| Example | 100.4 ms | 0.31 ms | 0.26 ms |

Conclusion

In short, file-based C# apps can run benchmarks, but only if you use the InProcess toolchain. It may be OK for your case, but don’t forget that in-process benchmarks are not isolated from the host process, so the results can be affected by it.