snapcraft prime takes considerably more time on unpacking stage packages than before

[Konstantin Pavlov]

Since the upgrade to version 2.40 in xenial, snapcraft prime takes considerably more time (in our case, from ~35 minutes to ~65 minutes on average) to build packages.

Most of the time is spent on seemingly unpacking packages, with builds get "stuck" on "Preparing to build vlc", with snapcraft process using 100% CPU for minutes, apparently reading every file there is.

I've tried upgrading to 2.41 per IRC discussion, which didnt help. The version that did not exhibit the problem (builds were faster) is 2.39.3+really2.35.

We have around ~400 stage packages.

The IRC discussion followed:

17:15 < sergiusens> seb128: the only potential change we have is we went from calling dpkg-deb to using a python implementationdebian.arfile
17:15 < sergiusens> thresh: ^
17:16 < sergiusens> I will run some numbers against that, but I cannot do that today
17:16 < thresh> I do see python proccess doing a lot of read/write under strace, yes
17:16 < seb128> the python thing might be much slower than dpkg-deb?
17:17 < sergiusens> seb128: we do not have a need for that feature anymore given the last PR I am working on, so we could potentially move back to dpkg-deb without much thought

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/2111