zstd compression for packages

Hey folks,

We had a coding day in Foundations last week and Balint and Julian added support for zstd compression to dpkg [1] and apt [2].

[1] <a href="" title=""></a>
[2] <a href="" title=""></a>

Zstd is a compression algorithm developed by Facebook that offers far
higher decompression speeds than xz or even gzip (at roughly constant
speed and memory usage across all levels), while offering 19 compression
levels ranging from roughly comparable to gzip in size (but much faster)
to 19, which is roughly comparable to xz -6:

In our configuration, we run zstd at level 19. For bionic main amd64,
this causes a size increase of about 6%, from roughly 5.6 to 5.9 GB.
Installs speed up by about 10%, or, if eatmydata is involved, by up to
40% - user time generally by about 50%.

Our implementations for apt and dpkg support multiple frames as used by
pzstd, so packages can be compressed and decompressed in parallel

We are considering requesting a FFe for that - the features are not
invasive, and it allows us to turn it on by default in 18.10.

Balint and Julian

Raw Measurements
All measurements where performed on a cloud instance of bionic, in a basic bionic schroot with overlay, on an ssd.

Kernel install (eatmydata, perf report, time spent in compression)
Kernel install (eatmydata)
12.49user 3.04system 0:12.57elapsed 123%CPU (0avgtext+0avgdata 68720maxresident)k
0inputs+1056712outputs (0major+159306minor)pagefaults 0swaps

5.60user 2.33system 0:07.07elapsed 112%CPU (0avgtext+0avgdata 81388maxresident)k
0inputs+1108720outputs (0major+171171minor)pagefaults 0swaps

4.52user 3.30system 0:33.14elapsed 23%CPU (0avgtext+0avgdata 25152maxresident)k
0inputs+544560outputs (0major+386394minor)pagefaults 0swaps

firefox eatmydata
11.34user 6.66system 1:18.04elapsed 23%CPU (0avgtext+0avgdata 64676maxresident)k
16inputs+1370112outputs (0major+1024989minor)pagefaults 0swaps

libreoffice eatmydata
10.86user 5.78system 0:17.70elapsed 94%CPU (0avgtext+0avgdata 64800maxresident)k
0inputs+1370112outputs (0major+1043637minor)pagefaults 0swaps


Re: zstd compression for packages

By Julian Andres Klode at 03/12/2018 - 06:58

On Mon, Mar 12, 2018 at 11:06:11AM +0100, Julian Andres Klode wrote:
More links:

PPA: <a href="" title=""></a>
APT merge request: <a href="" title=""></a>
dpkg patches: <a href="" title=""></a>

I'd also like to talk a bit more about libzstd itself: The package is
currently in universe, but btrfs recently gained support for zstd,
so we already have a copy in the kernel and we need to MIR it anyway
for btrfs-progs.