ISST-KVM:CTE:R3-0:raing12: Base system install fails with "Debootstrap Error :Invalid Release signature (key id 40976EAF437D05B5) " using Ubuntu 15.10 latest daily build (20150805)

[bugproxy]

== Comment: #0 - Pratiba Joshi Ms <email address hidden> - 2015-08-06 06:24:58 ==
Problem description:
--------------------------

--- I have tried installing a guest with Ubuntu 15.10 using latest daily build (20150805) .

--- Installation failed at "Install the base system" step with "debootstrap error".

--- But if we go back to main menu and tried it proceeded with successful isntallation.

DEFECT : Ubuntu15.10
TYPE OF DEFECT:Installation

      ?????????????????? [!!] Install the base system ???????????????????
      ? ?
  ????? Debootstrap Error ? ???
  ? ? The following error occurred: ? ?
  ? ? ? ?
  ? ? Invalid Release signature (key id 40976EAF437D05B5) ? ?
  ? Ch? ? ?
  ? ? Check /var/log/syslog or see virtual console 4 for the details. ? ?
  ????? ? ???
      ? <Go Back> <Continue> ?
      ? ?
      ???????????????????????????????????????????????????????????????????

    ?????????????????? [!!] Install the base system ???????????????????
      ? ?
      ? Failed to install the base system ?
      ? The base system installation into /target/ failed. ?
      ? ?
      ? Check /var/log/syslog or see virtual console 4 for the details. ?
      ? ?
      ? <Go Back> <Continue> ?
      ? ?
      ???????????????????????????????????????????????????????????????????

     ??????????????????? [!!] Install the base system ????????????????????
     ? ?
     ? Installation step failed ?
     ? An installation step failed. You can try to run the failing item ?
     ? again from the menu, or skip it and choose something else. The ?
     ? failing step is: Install the base system ?
     ? ?
     ? <Continue> ?
     ? ?
     ?????????????????????????????????????????????????????????????????????

I have attached /var/log/syslog file with this bug .It allowed to proceed with installation so taking back the system and attaching the logs for debugging .

contact info: <email address hidden>

== Comment: #1 - Pratiba Joshi Ms <email address hidden> - 2015-08-06 06:27:10 ==
Login info : -

NOTE:-
----------

System is on a private network. Access the private network via SSH to "banner.isst.aus.stglabs.ibm.com" using your GSA ID and password.
(Banner itself is behind a BSO, so must authenticate through that first.)

Login details :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry ]

Host login:-

rain-kvm.isst.aus.stglabs.ibm.com has address 10.33.23.121
[root/don2rry]

virsh console raing12 --force

IPMI Login :-
-------------
From banner machine run the following :
ssh banner.isst.aus.stglabs.ibm.com [debug/don2rry ]

ipmitool -I lanplus -H 10.33.23.107 -P don2rry sol deactivate
ipmitool -I lanplus -H 10.33.23.107 -P don2rry sol activate

-----------------------------------------------------------------------------
                             TESTING INFORMATION
-----------------------------------------------------------------------------

SYSTEM INFORMATION
------------------
  HOST NAME or NETWORK ADDRESS: rain-kvm.isst.aus.stglabs.ibm.com [10.33.23.121]

   FSP NAME and FSP ip fsp-rain-kvm.isst.aus.stglabs.ibm.com [ 10.33.23.107 ]

  KVM BUILD LEVEL: - GA3 KVM 3.1.0 build 25

-----------------------------------------------------------------------------
                            DEBUGGING INFORMATION
-----------------------------------------------------------------------------

DEBUG / LOGIN INFORMATION
-------------------------
  ERROR LOG: on local machine
  FSP LOGIN: see name above dev/FipSdev

RECENT SYSTEM CHANGES : none
-----------------------------------------
  none

== Comment: #3 - Pratiba Joshi Ms <email address hidden> - 2015-08-06 07:16:41 ==
From the installation logs i could see some errors related to bootstrap as below:

/var/log # cat syslog | grep -i bootstrap
Aug 6 04:37:26 anna[1590]: DEBUG: retrieving bootstrap-base 1.144ubuntu2
Aug 6 04:37:26 anna[1590]: 2015-08-06 04:37:26 URL:http://ports.ubuntu.com/ubuntu-ports//pool/main/b/base-installer/bootstrap-base_1.144ubuntu2_ppc64el.udeb [126338/126338] -> "/var/cache/anna/bootstrap-base_1.144ubuntu2_ppc64el.udeb" [1]
Aug 6 04:37:27 anna[1590]: DEBUG: retrieving debootstrap-udeb 1.0.72
Aug 6 04:37:27 anna[1590]: 2015-08-06 04:37:27 URL:http://ports.ubuntu.com/ubuntu-ports//pool/main/d/debootstrap/debootstrap-udeb_1.0.72_all.udeb [18514/18514] -> "/var/cache/anna/debootstrap-udeb_1.0.72_all.udeb" [1]
Aug 6 04:38:28 main-menu[214]: INFO: Menu item 'bootstrap-base' selected
Aug 6 04:38:30 debootstrap: gpgv: Signature made Thu Aug 6 09:51:40 2015 UTC using DSA key ID 437D05B5
Aug 6 04:38:30 debootstrap: gpgv:
Aug 6 04:38:30 debootstrap: BAD signature from "Ubuntu Archive Automatic Signing Key <email address hidden>"
Aug 6 04:38:30 debootstrap:
Aug 6 04:38:35 base-installer: error: exiting on error base-installer/debootstrap-failed
Aug 6 04:38:38 main-menu[214]: WARNING **: Configuring 'bootstrap-base' failed with error code 30
Aug 6 04:38:38 main-menu[214]: WARNING **: Menu item 'bootstrap-base' failed.
/var/log #

== Comment: #4 - Pratiba Joshi Ms <email address hidden> - 2015-08-11 06:41:44 ==
I am seeing same issue in Ubuntu 15.10 daily latest build (20150810) also.

I am attaching installer logs also.

== Comment: #5 - Pratiba Joshi Ms <email address hidden> - 2015-08-11 06:47:53 ==
same kind of errors seen in logs also.

/var/log # cat syslog | grep -i bootstrap
Aug 11 10:33:16 anna[1472]: DEBUG: retrieving bootstrap-base 1.144ubuntu2
Aug 11 10:33:16 anna[1472]: 2015-08-11 10:33:16 URL:http://ports.ubuntu.com/ubuntu-ports//pool/main/b/base-installer/bootstrap-base_1.144ubuntu2_ppc64el.udeb [126338/126338] -> "/var/cache/anna/bootstrap-base_1.144ubuntu2_ppc64el.udeb" [1]
Aug 11 10:33:17 anna[1472]: DEBUG: retrieving debootstrap-udeb 1.0.72
Aug 11 10:33:17 anna[1472]: 2015-08-11 10:33:17 URL:http://ports.ubuntu.com/ubuntu-ports//pool/main/d/debootstrap/debootstrap-udeb_1.0.72_all.udeb [18514/18514] -> "/var/cache/anna/debootstrap-udeb_1.0.72_all.udeb" [1]
Aug 11 10:38:06 main-menu[274]: INFO: Menu item 'bootstrap-base' selected
Aug 11 10:38:09 debootstrap: gpgv: Signature made Tue Aug 11 06:30:35 2015 UTC using DSA key ID 437D05B5
Aug 11 10:38:09 debootstrap: gpgv: BAD signature from "Ubuntu Archive Automatic Signing Key <email address hidden>"
Aug 11 10:41:24 base-installer: error: exiting on error base-installer/debootstrap-failed
Aug 11 10:41:28 main-menu[274]: WARNING **: Configuring 'bootstrap-base' failed with error code 30
Aug 11 10:41:28 main-menu[274]: WARNING **: Menu item 'bootstrap-base' failed.
/var/log #

== Comment: #10 - Pratiba Joshi Ms <email address hidden> - 2015-08-13 02:38:22 ==
same Issue is recreated with 20150812 builld also.

== Comment: #12 - David Heller <email address hidden> - 2015-08-13 17:43:29 ==
Please verify your netboot installer kernel and initrd match the latest on the upstream mirror, here:

$ wget -qO- http://ports.ubuntu.com/dists/wily/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/vmlinux | md5sum -
17af525c48054a5e6b746f11e1bf9afb -
$ wget -qO- http://ports.ubuntu.com/dists/wily/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/initrd.gz | md5sum -
c3eb76bebb2169b59ab44bdd3c0a6a65 -

== Comment: #13 - Pratiba Joshi Ms <email address hidden> - 2015-08-14 06:31:21 ==
(In reply to comment #12)
> Please verify your netboot installer kernel and initrd match the latest on
> the upstream mirror, here:
>
> $ wget -qO-
> http://ports.ubuntu.com/dists/wily/main/installer-ppc64el/current/images/
> netboot/ubuntu-installer/ppc64el/vmlinux | md5sum -
> 17af525c48054a5e6b746f11e1bf9afb -
> $ wget -qO-
> http://ports.ubuntu.com/dists/wily/main/installer-ppc64el/current/images/
> netboot/ubuntu-installer/ppc64el/initrd.gz | md5sum -
> c3eb76bebb2169b59ab44bdd3c0a6a65 -

I have verified our netboot installer kernel and initrd match the latest on the upstream mirror, its the same.

Please find as below:

[pratijos@kte ppc64el]$ pwd
/distros/ubuntu-1510-netboot/ubuntu-installer/ppc64el
[pratijos@kte ppc64el]$

[pratijos@kte ppc64el]$ md5sum vmlinux
17af525c48054a5e6b746f11e1bf9afb vmlinux

[pratijos@kte ppc64el]$ md5sum initrd.gz
c3eb76bebb2169b59ab44bdd3c0a6a65 initrd.gz

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1485511/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-08-17 15:01 EDT-------
As of today (17 Aug) there has been no change in the netboot kernel & initrd, or in the version of the "debootstrap" package, so if there is an issue I suppose there would no change in the result over the last week?

$ wget -qO- http://ports.ubuntu.com/dists/wily/main/binary-ppc64el/Packages.bz2 | bunzip2 | grep debootstrap | grep Filename
Filename: pool/main/d/debootstrap/debootstrap_1.0.72_all.deb

Canonical, can you look at this please? Submitters claim this is reproducible.. Thx.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-08-26 06:15 EDT-------
This issue is seen frequently in many daily builds.

Recreated in latest 20150824 Ubuntu 15.10 build in SGC1 guest.

I have observed and analyzed logs found same error as below:

BusyBox v1.22.1 (Ubuntu 1:1.22.0-9ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

~ # cd /var/log
/var/log # ls
partman syslog
/var/log # cat syslog | grep -i bootstrap
Aug 25 21:40:22 anna[1481]: DEBUG: retrieving bootstrap-base 1.144ubuntu2
Aug 25 21:40:22 anna[1481]: 2015-08-25 21:40:22 URL:http://ports.ubuntu.com/ubuntu-ports//pool/main/b/base-installer/bootstrap-base_1.144ubuntu2_ppc64el.udeb [126338/126338] -> "/var/cache/anna/bootstrap-base_1.144ubuntu2_ppc64el.udeb" [1]
Aug 25 21:40:23 anna[1481]: DEBUG: retrieving debootstrap-udeb 1.0.72
Aug 25 21:40:23 anna[1481]: 2015-08-25 21:40:23 URL:http://ports.ubuntu.com/ubuntu-ports//pool/main/d/debootstrap/debootstrap-udeb_1.0.72_all.udeb [18514/18514] -> "/var/cache/anna/debootstrap-udeb_1.0.72_all.udeb" [1]
Aug 25 21:47:19 main-menu[290]: INFO: Menu item 'bootstrap-base' selected
Aug 25 21:47:25 debootstrap: gpgv: Signature made Tue Aug 25 09:31:36 2015 UTC using DSA key ID 437D05B5
Aug 25 21:47:25 debootstrap: gpgv:
Aug 25 21:47:25 debootstrap: BAD signature from "Ubuntu Archive Automatic Signing Key <email address hidden>"
Aug 25 21:47:25 debootstrap:
Aug 25 21:49:07 base-installer: error: exiting on error base-installer/debootstrap-failed
Aug 25 21:49:10 main-menu[290]: WARNING **: Configuring 'bootstrap-base' failed with error code 30
Aug 25 21:49:10 main-menu[290]: WARNING **: Menu item 'bootstrap-base' failed.
/var/log #

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-09-24 07:52 EDT-------
Any updates on this bug??

------- Comment From <email address hidden> 2015-09-24 07:53 EDT-------
we are seeing this issue in latest build 20150923 also.

Please let me know on this.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-10-08 16:42 EDT-------
*** Bug 131251 has been marked as a duplicate of this bug. ***

------- Comment From <email address hidden> 2015-10-08 16:45 EDT-------
This bug may be related to Launchpad 24234.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-10-08 17:35 EDT-------
Expanding on LTC bug 131251, which was just duped to this bug. The bug submitter from LTC 131251 can successfully install and then get a signing key failure. He reports success every other install attempt. This behaviour may imply one of the servers in ports.ubuntu.com is out of sync.

Canonical: please investigate.
Thanks, Gary

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2015-11-05 23:29 EDT-------
I also believe this is related to https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/24061

Which happens when using mirrors, which is also the case here.

I don't understand the different states in Launchpad though (not clear as to which builds have a fix)

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[Steve Langasek (vorlon)]

This bug should only happen due to a buggy proxy on the user's side, or infrequently if the download check happens to take place right as the ports.ubuntu.com site is being updated.

Ubuntu now supports the 'InRelease' signature format on its mirrors, which ensures the signature and signed data are downloaded atomically and are therefore always in sync. debian-installer should use this file in place of the Release{,.gpg} resource pair for verifying mirror integrity. Assigning this bug to the debootstrap package, as it looks like this verification happens in debootstrap-udeb.

[Mathieu Trudel-Lapierre (cyphermox)]

Ok, I retrieved the old code that was handling InRelease support in debootstrap 1.0.47. I will test this locally, including adding a dependency on gpg-udeb so that this works correctly in d-i.

One of the reasons it was removed earlier was that gpg was not made available in d-i, probably due to space concerns (it's almost twice the size).

Now I see we don't even have gpg-udeb in xenial, since it was dropped earlier in Debian, and in later releases gpgv-udeb is moved to gnupg2.

In light of this, I'll start with re-adding gpg-udeb in xenial so that we can perhaps change the shipping gpg binary in d-i to it for both wily and xenial; then I'll be able to re-add InRelease support in debootstrap.

gnupg2 requires some additional libraries though, so that would further increase the size impact on d-i images.

[Steve Langasek (vorlon)]

FWIW I don't understand how InRelease support in debootstrap requires any changes to gpg-related udebs in d-i. If debootstrap is currently throwing a signature verification error in d-i, that means it has some way of verifying signatures... which means it has gpg support. Why does a change to *which* file we're verifying (InRelease, vs. Release{.gpg}) imply having to add new gpg udebs?

[Mathieu Trudel-Lapierre (cyphermox)]

gpgv currently in d-i can only do signature verification, and presumably it only does so with detached signatures.

TBH, I haven't dug in that much more than what's in the comment yet: I looked up code for InRelease support that was previously in Debian, and it was requiring a more complete gpg implementation for some reason.

This is on my list to fix today, I'll write the results here (or just upload things directly if it just works).

[Mathieu Trudel-Lapierre (cyphermox)]

I discussed this on Friday with infinity, and after some testing it looks like gpgv should be sufficient. The issue with it is that it doesn't extract the signed data to a file so that it will be succesfully parsed by whatever uses the file afterwards. At a glance, it looks like that may just be grep though.

Regardless, I'm able to extract the relevant data with sed and grep; I'll test that in a PPA to verify that there are no regressions, from which point we can test "in the wild", if the original reporter for this bug is able to reproduce this issue easily enough.

[Kevin W. Rudd (kevinr)]

We are continuing to get reports of this issue via the mirrored bug attached to bug #1544339 . I think the test teams can reproduce easily enough, so we would appreciate any status update or test suggestions.

Thanks.

[Mathieu Trudel-Lapierre (cyphermox)]

There is a package here available for testing:
https://launchpad.net/~mathieu-tl/+archive/ubuntu/installer-dev/+sourcepub/6167724/+listing-archive-extra

So far it looks fine to me, but I've also been in the process of making sure this fix also gets to Debian, and to do things right I wanted to get it in Debian first and then into Ubuntu. If there is more delay and this update has more good marks in testing, we'll "skip" Debian (and keep working on that in parallel) and upload to Ubuntu for it to make it in 16.04

[Launchpad Janitor (janitor)]

This bug was fixed in the package debootstrap - 1.0.78+nmu1ubuntu1

---------------
debootstrap (1.0.78+nmu1ubuntu1) xenial; urgency=medium

  * Reinstate InRelease file support to better deal with proxies where Release
    files may be cached and thus out of sync. InRelease is atomic and therefore
    not subject to failing validation because of proxy servers. (LP: #1485511)

 -- Mathieu Trudel-Lapierre <email address hidden> Fri, 11 Mar 2016 13:03:43 -0500

[Breno Leitão (breno-leitao)]

Tested on our side, and the fix works. Thank you!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-16 10:13 EDT-------
*** Bug 136852 has been marked as a duplicate of this bug. ***

[Gianfranco Costamagna (costamagnagianfranco)]

Hello bugproxy, can you please try my ppa?
 https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/locutusofborg-ppa
there is a new debootstrap version, and I would like to be sure the proxy issue is still fixed

[Gianfranco Costamagna (costamagnagianfranco)]

I uploaded the new version on artful, please ping if you find any regression