`TarFile.ExtractToDirectory` may be Significantly Slower than Windows `tar.exe`

[nagilson]

Description

Working on dotnetup we realized dotnetup was slower than the .NET Install Script by a wide margin, but only on Windows 11 when downloading .NET 11. Otherwise, it was a tie or marginally faster. Note that performance tests were conducted on two machines from one dev (me) and include variability; multiple steps were repeated each time, and variation was considered.

NET 11 moved to use windows tarballs with hardlinks. The .NET SDK is an archive with many directories and small files. We realized the Install Script used tar.exe instead of dotnetup using TarFile functions.

We implemented our own wrapper around tar.exe instead of using the .NET implementation because it was so much slower. This essentially caused the 2 products to be tied in install time for the .NET SDK.

Configuration

x64 .NET 11 on Windows 11 - On WSL2, the performance comparison is not an issue.
The compared implementation is a NAOT, SelfContained, PublishTrimmed executable, with one implementation running TarFile source code and the other implementing a custom wrapper around the relatively recent built-in Windows tar.exe from bsdtar.

See
https://github.com/nagilson/dotnet-win-tar-compare

An Interface is implemented using .NET's TarFile and Windows tar.exe

Regression?

No, I don't believe this is.

Data

Product	Version	Format	.NET implementation median (ms)	.NET Wrapper on Windows tar.exe (bsdtar) median (ms)	Faster	Difference
.NET SDK	11.0.100-rc.1.26425.128	tar.gz	10,875.8	3,938.2	Windows tar.exe (bsdtar)	63.8%
.NET SDK	11.0.100-rc.1.26425.128	tar	10,891.6	2,989.2	Windows tar.exe (bsdtar)	72.6%
.NET Runtime	11.0.0-rc.1.26425.128	tar.gz	656.2	358.6	Windows tar.exe (bsdtar)	45.4%
.NET Runtime	11.0.0-rc.1.26425.128	tar	358.1	191.5	Windows tar.exe (bsdtar)	46.5%

Product	Format	Extractor	Minimum (ms)	Median (ms)	Maximum (ms)
.NET Runtime	tar	.NET implementation	339.6	358.1	395.0
.NET Runtime	tar	Windows tar.exe (bsdtar)	188.3	191.5	263.4
.NET Runtime	tar.gz	.NET implementation	595.8	656.2	678.1
.NET Runtime	tar.gz	Windows tar.exe (bsdtar)	354.2	358.6	380.2
.NET SDK	tar	.NET implementation	10,796.9	10,891.6	11,494.1
.NET SDK	tar	Windows tar.exe (bsdtar)	2,893.4	2,989.2	3,283.2
.NET SDK	tar.gz	.NET implementation	10,785.6	10,875.8	11,378.7
.NET SDK	tar.gz	Windows tar.exe (bsdtar)	3,877.4	3,938.2	3,963.3

Analysis

A series of hypotheses for the performance gap are available:
https://github.com/nagilson/dotnet-win-tar-compare/blob/main/PERFORMANCE-HYPOTHESES.md

⚠️ Hypothesis Investigations were Gen-AI assisted.

1. Verified prefixes can be cached if the cache is invalided after symlink expansion to reduce work.

https://github.com/libarchive/libarchive/blob/v3.8.9/libarchive/archive_write_disk_windows.c#L2124-L2251

https://github.com/dotnet/dotnet/blob/3551975be08744f0418857c5bed8ab1545c5dd47/src/runtime/src/libraries/System.Formats.Tar/src/System/Formats/Tar/TarEntry.cs#L363-L421

cc @JeremyKuhne @BrennanConroy @danroth27

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.