Snap builds on arm are extremely slow causing them to fail

[Pat McGowan]

A build that should take a few minutes is taking two hours then timing out and failing.
See https://code.launchpad.net/~ubuntu-sdk-team/+snap/ubuntu-app-platform
Other builds are slow but complete
See https://code.launchpad.net/~pat-mcgowan/+snap/testdemo
Earlier builds on the last project were fast as of 2/14.

It seems the get requests may be taking an unexpected amount of time, almost as if they are throttled.

[Timo Jyrinki (timo-jyrinki)]

The last successful build of ubuntu-app-platform was on 2017-02-08, and started failing with the 2 hour timeout by ~2017-02-20.

[William Grant (wgrant)]

This was probably a temporary network problem. Is it still reproducible?

[Timo Jyrinki (timo-jyrinki)]

It has happened since February and continues to happen.

[Timo Jyrinki (timo-jyrinki)]

Continuing in the question instead.

[Timo Jyrinki (timo-jyrinki)]

(it seems the status is reflected in the question so better to keep the bug open)

[William Grant (wgrant)]

No, this isn't a code bug.

[Pat McGowan (pat-mcgowan)]

This seems related to the change at
https://github.com/snapcore/snapcraft/commit/20b422459119000640fa4103fe097eca5440f920

[Pat McGowan (pat-mcgowan)]

specifically https://github.com/snapcore/snapcraft/commit/20b422459119000640fa4103fe097eca5440f920#diff-e8d25d56f5ed513cad7c55002a709f32R316

The issue arises for arm builders because they are located in the US so the latency is much higher, whereas for amd64 the latency within London is always <2ms

[William Grant (wgrant)]

The majority of the slowdown was a combination of snapcraft's lack of connection preservation and slow DNS resolution of the builder proxy due to a possible BIND bug. IS has applied a workaround for the BIND problem, fixing much of the latency and most of the build slowdown.

US-based builds (ARM, POWER and z) should now succeed, though they won't be perfectly fast. We can't fix the remaining latency or bandwidth issues without the restoration of connection preservation in snapcraft.

[Kyle Fazzari (kyrofa)]

Thanks wgrant, that's good to hear. While this is of course still an issue, Pat can you verify that your builds are succeeding once again?

[Leo Arias (elopio)]

I disabled my apt-cacher-ng and configured my sources to use the archive from the UK.
I built Pat's snap locally without the code to handle the stage packages cache, and it took 10 minutes. Then I got the caching code, and it took 20 minutes. My connection is not really stable, so the measurements are not very useful, but taking twice the time is too much. I've marked this bug as confirmed.

[Sergio Schvezov (sergiusens)]

Pat you are indeed correct, this indeed creates a new connection per package to download https://github.com/snapcore/snapcraft/blob/master/snapcraft/internal/repo/_deb.py#L303

[Pat McGowan (pat-mcgowan)]

IS provided a fix to speed up DNS lookups and the build now completes in around 16 mins (vs original 7 mins)
https://code.launchpad.net/~ubuntu-sdk-team/+snap/ubuntu-app-platform/+build/30005

"The majority of the slowdown was a combination of snapcraft's lack of connection preservation and slow DNS resolution of the builder proxy due to a possible BIND bug. IS has applied a workaround for the BIND problem, fixing much of the latency and most of the build slowdown."

So while the lack of a persistent connection slows initial builds at least they now complete in a reasonable time on LP.

[Timo Jyrinki (timo-jyrinki)]

I was away last week, but I'm happy to find that the builds are now working.

For ubuntu-app-platform, the build aborted at 2 hours due to timeout before, versus now succeeding in 16 minutes.