Snap fails to build, 503 proxy error

[Alan Pope 🍺🐧🐱 🦄]

[16/Jul/2018:10:18:15 +0000] "CONNECT git.xiph.org:443 HTTP/1.1" 503 1789 "-" "git/2.7.4"
fatal: unable to access 'https://git.xiph.org/celt.git/': Received HTTP code 503 from proxy after CONNECT
fatal: clone of 'https://git.xiph.org/celt.git/' into submodule path 'celt-0.7.0' failed
Failed to recurse into submodule path '3rdparty/sbcelt-src'

https://launchpadlibrarian.net/378715573/buildlog_snap_ubuntu_xenial_armhf_96015d27dc6daea80ecdd7c5bac64b6c-xenial_BUILDING.txt.gz

[Lauri Ojansivu (xet7)]

Downloading 'node-v8.11.3-linux-x64.tar.gz' 22%
[30/Jul/2018:14:50:49 +0000] "CONNECT nodejs.org:443 HTTP/1.0" 200 17978314 "-" "-"
[30/Jul/2018:14:50:54 +0000] "CONNECT registry.npmjs.org:443 HTTP/1.1" 503 1777 "-" "-"
[30/Jul/2018:14:51:04 +0000] "CONNECT registry.npmjs.org:443 HTTP/1.1" 503 1777 "-" "-"
[30/Jul/2018:14:52:05 +0000] "CONNECT registry.npmjs.org:443 HTTP/1.1" 503 1777 "-" "-"
npm ERR! code E503
npm ERR! 503 Service Unavailable: npm@latest

https://launchpadlibrarian.net/380841345/buildlog_snap_ubuntu_xenial_amd64_wekan-devel_BUILDING.txt.gz

[Jeff Dickey (heroku-jdxcode)]

I'm getting this as well. It seems to have started sometime between July 19th and July 25th based on our last successful release. It happens on every push and we haven't made any changes to our snapcraft.yml.

[Jeff Dickey (heroku-jdxcode)]

https://build.snapcraft.io/user/heroku/cli/313487

Setting up yarn (1.9.4-1) ...
+ cd packages/cli
+ yarn
yarn install v1.9.4
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
(node:10675) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
[17/Aug/2018:03:42:14 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:14 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:14 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:14 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:14 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:14 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:14 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:14 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
info There appears to be trouble with your network connection. Retrying...
[17/Aug/2018:03:42:17 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:17 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:17 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:17 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:17 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:17 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:17 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:17 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
info There appears to be trouble with your network connection. Retrying...
[17/Aug/2018:03:42:20 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:20 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:20 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:20 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-" "yarn/1.9.4 npm/? node/v10.9.0 linux x64"
[17/Aug/2018:03:42:20 +0000] "CONNECT registry.yarnpkg.com:443 HTTP/1.1" 503 2010 "-...

[Lauri Ojansivu (xet7)]

I currently build Wekan package using Xubuntu 16.04 desktop, with command `sudo snapcraft`
https://github.com/wekan/wekan-snap/issues/58

I don't know is there plans to fix build service.

[Colin Watson (cjwatson)]

The proxy seems to be failing consistently for that site (according to logs), but working for others.

From 15 minutes' last-thing-on-a-Friday investigation, my guess is that the problem may be related to IPv6: both the failing sites listed here have AAAA records. It's possible that our squid cache is misconfigured in a way that doesn't play well with IPv6; but I'll need to experiment further on a staging site.

[Martin Wimpress  (flexiondotorg)]

This issue is also affecting builds of Node-RED:

  * https://launchpad.net/~build.snapcraft.io/+snap/b69a088aa70edd446e36c2072b30e222-xenial

[Evan (ev)]

A few more:
https://launchpad.net/~build.snapcraft.io/+snap/785f7d0b2ea1ee249c5a34ec6a877a4a-xenial/+build/326213
https://launchpad.net/~build.snapcraft.io/+snap/f597555d8070c49765daf314640444e0-xenial/+build/326180

[DaveCJ (ceejay-7)]

Any idea when this may actually hit the public build server ?
Still 503 out here...

[Colin Watson (cjwatson)]

I filed a ticket with our sysadmins a few days ago to roll out the fix, and I'm waiting for it to get to the top of their queue. I'll mark this bug Fix Released once that's done.

[Colin Watson (cjwatson)]

This should be fixed now.