DNS for security.ubuntu.com and archive.ubuntu.com resolves to IP address that gives 404 when installing packages.

[Alastair Growcott]

For the last few weeks I have been getting "404 Not found" when Docker is installing packages when building an image.

On several completely different machines, nslookup on security.ubuntu.com gives:

Server: 37.235.1.174
Address: 37.235.1.174#53

Non-authoritative answer:
Name: security.ubuntu.com
Address: 91.189.88.161
Name: security.ubuntu.com
Address: 91.189.88.162
Name: security.ubuntu.com
Address: 91.189.88.149
Name: security.ubuntu.com
Address: 91.189.91.26
Name: security.ubuntu.com
Address: 91.189.88.152
Name: security.ubuntu.com
Address: 91.189.91.23

I have flushed the cache on my DNS server, which feeds off Google's, and also tried directly accessing other DNS servers as above. All give exactly the same results.

Browsing http://91.189.88.161/ubuntu does not work (404).
Browsing http://91.189.88.162/ubuntu does not work (404).
Browsing http://91.189.88.149/ubuntu does not work (404).
Browsing http://91.189.91.26/ubuntu works.
Browsing http://91.189.88.152/ubuntu does not work (404).
Browsing http://91.189.91.23/ubuntu works.

Here is the output for the relevant step of building the docker image:

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  binutils bzip2 cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc gcc-5
  libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl
  libasan2 libatomic1 libc-dev-bin libc6-dev libcc1-0 libcilkrts5 libdpkg-perl
  libfakeroot libfile-fcntllock-perl libgcc-5-dev libgomp1 libisl15 libitm1
  liblsan0 libmpc3 libmpfr4 libmpx0 libquadmath0 libstdc++-5-dev libtsan0
  libubsan0 linux-libc-dev make manpages manpages-dev xz-utils
Suggested packages:
  binutils-doc bzip2-doc cpp-doc gcc-5-locales debian-keyring g++-multilib
  g++-5-multilib gcc-5-doc libstdc++6-5-dbg gcc-multilib autoconf automake
  libtool flex bison gdb gcc-doc gcc-5-multilib libgcc1-dbg libgomp1-dbg
  libitm1-dbg libatomic1-dbg libasan2-dbg liblsan0-dbg libtsan0-dbg
  libubsan0-dbg libcilkrts5-dbg libmpx0-dbg libquadmath0-dbg glibc-doc
  libstdc++-5-doc make-doc man-browser
The following NEW packages will be installed:
  binutils build-essential bzip2 cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc
  gcc-5 libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libasan2 libatomic1 libc-dev-bin libc6-dev libcc1-0
  libcilkrts5 libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-5-dev
  libgomp1 libisl15 libitm1 liblsan0 libmpc3 libmpfr4 libmpx0 libquadmath0
  libstdc++-5-dev libtsan0 libubsan0 linux-libc-dev make manpages manpages-dev
  xz-utils
0 upgraded, 40 newly installed, 0 to remove and 6 not upgraded.
Need to get 40.0 MB of archives.
After this operation, 146 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmpfr4 amd64 3.1.4-1 [191 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmpc3 amd64 1.0.3-1 [39.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 bzip2 amd64 1.0.6-8 [32.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 manpages all 4.04-2 [1087 kB]
Err:5 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 binutils amd64 2.26.1-1ubuntu1~16.04.5
  404 Not Found [IP: 91.189.88.162 80]
Err:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 libc-dev-bin amd64 2.23-0ubuntu9
  404 Not Found [IP: 91.189.88.162 80]
Ign:7 http://security.ubuntu.com/ubuntu xenial-security/main amd64 linux-libc-dev amd64 4.4.0-101.124
Ign:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 libc6-dev amd64 2.23-0ubuntu9
Get:9 http://archive.ubuntu.com/ubuntu xenial/main amd64 libisl15 amd64 0.16.1-1 [524 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 cpp-5 amd64 5.4.0-6ubuntu1~16.04.5 [7786 kB]

The same 4 problem addresses above also appear when doing an nslookup of archive.ubuntu.com and cause the same error.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1745129/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Jacek Nykis (jacekn)]

Hello,

Thank you for the report.

Browsing to IP addresses directly does not work because of virtual hosting:
https://en.wikipedia.org/wiki/Virtual_hosting

Virtual hosting requires the "Host" header. You can see that this request returns 404:
$ curl http://91.189.88.152/ubuntu/

but if I send the header it works fine:
$ curl -H "Host: security.ubuntu.com" http://91.189.88.152/ubuntu/

It's hard to say for sure what's causing the problem but my guess would be lack of "apt update" before "apt install".

Can I also suggest that you reach out to the docker community for help? As far as I can see security.ubuntu.com servers are working as expected.

I will mark this bug as invalid but if I got something wrong please reopen.

[Alastair Growcott (agrowcott)]

Hello and thanks for the response.

I am doing an "apt update" before the install, but it's a separate step so may not be being executed recently enough (due to Docker caching images). I updated the docker file to do it on the same line and that seems to work. It is running now but seems ok so far.

I have already "reached out to the docker community". See the question this bug is linked to.

Thanks again for youe help.