[Upgrade][9.1] Upgrade unsuccessful. GPG error: http://mirror.fuel-infra.org.

[Mykola Stolyarenko]

Actual Behaviour:

Cluster by Fuel 9.0 deployed.

Upgrade doing by https://mirantis.jira.com/wiki/pages/viewpage.action?spaceKey=QA&title=MOS+9.1+installation manual

history: (snapshot last from http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/)

yum-config-manager --add-repo http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-09-11-182323/x86_64/
rpm --import http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-09-11-182323/RPM-GPG-KEY-mos9.0
update-prepare prepare master
fuel env
update-prepare prepare env 1

[root@nailgun ~]# update-prepare prepare env 1
Warning: Permanently added '10.109.0.3' (ECDSA) to the list of known hosts.
W: GPG error: http://mirror.fuel-infra.org mos9.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
Reading package lists...
Building dependency tree...
Reading state information...
nailgun-agent is already the newest version.
nailgun-mcagents is already the newest version.
The following packages were automatically installed and are no longer required:
  cloud-guest-utils eatmydata python-cheetah python-oauth python-serial
  python3-pycurl python3-software-properties software-properties-common
  unattended-upgrades
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Warning: Permanently added '10.109.0.5' (ECDSA) to the list of known hosts.
W: GPG error: http://mirror.fuel-infra.org mos9.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
Reading package lists...
Building dependency tree...
Reading state information...
nailgun-agent is already the newest version.
nailgun-mcagents is already the newest version.
The following packages were automatically installed and are no longer required:
  cloud-guest-utils eatmydata python-cheetah python-configobj python-oauth
  python-serial python3-pycurl python3-software-properties
  software-properties-common unattended-upgrades
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Warning: Permanently added '10.109.0.4' (ECDSA) to the list of known hosts.
W: GPG error: http://mirror.fuel-infra.org mos9.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
Reading package lists...
Building dependency tree...
Reading state information...
nailgun-agent is already the newest version.
nailgun-mcagents is already the newest version.
The following packages were automatically installed and are no longer required:
  cloud-guest-utils eatmydata python-cheetah python-oauth python-serial
  python3-pycurl python3-software-properties software-properties-common
  unattended-upgrades
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

[root@nailgun ~]# fuel2 env redeploy --noop 1
Deployment task with id 7 for the environment 1 has been started.
[root@nailgun ~]# fuel2 task show 7
+------------+------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------+
| id | 7 |
| uuid | 471500bf-d7f3-4384-ae21-0aec3db6ec57 |
| status | error |
| name | deploy |
| graph_type | None |
| cluster | 1 |
| result | {} |
| dry_run | True |
| progress | 100 |
| message | Error occurred when running task: [Errno 111] Connection refused |
+------------+------------------------------------------------------------------+

Expected behaviour:

Upgrade should be successful

[Roman Vyalov (r0mikiam)]

moved to fuel-sustaining team.
W: GPG error: http://mirror.fuel-infra.org mos9.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
it is warning not a error.
as i know we dont have check for signing repos

[Mykola Stolyarenko (mstolyarenko)]

[root@nailgun ~]# fuel2 env redeploy --noop 1
Deployment task with id 7 for the environment 1 has been started.
[root@nailgun ~]# fuel2 task show 7
+------------+------------------------------------------------------------------+
| Field | Value |
+------------+------------------------------------------------------------------+
| id | 7 |
| uuid | 471500bf-d7f3-4384-ae21-0aec3db6ec57 |
| status | error |
| name | deploy |
| graph_type | None |
| cluster | 1 |
| result | {} |
| dry_run | True |
| progress | 100 |
| message | Error occurred when running task: [Errno 111] Connection refused |
+------------+------------------------------------------------------------------+

[Maksim Malchuk (mmalchuk)]

You forgot the #10 step (add the repo on UI).
And, also, please provide the diagnostic snapshot, to solve the task error status.

[Maksim Malchuk (mmalchuk)]

2016-09-20 11:17:05.584 ERROR [7fa4693c6880] (manager) Error occurred when running task
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nailgun/task/manager.py", line 266, in _execute_async
    self._execute_async_content(supertask, **kwargs)
  File "/usr/lib/python2.7/site-packages/nailgun/task/manager.py", line 503, in _execute_async_content
    rpc.cast('naily', task_messages)
  File "/usr/lib/python2.7/site-packages/nailgun/rpc/__init__.py", line 97, in cast
    publish()
  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 172, in publish
    routing_key, mandatory, immediate, exchange, declare)
  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 177, in _publish
    channel = self.channel
  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 194, in _get_channel
    channel = self._channel = channel()
  File "/usr/lib/python2.7/site-packages/kombu/utils/__init__.py", line 425, in __call__
    value = self.__value__ = self.__contract__()
  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 209, in <lambda>
    channel = ChannelPromise(lambda: connection.default_channel)
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 756, in default_channel
    self.connection
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
    self._connection = self._establish_connection()
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
    conn = self.Connection(**opts)
  File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
2016-09-20 11:17:05.592 DEBUG [7fa4693c6880] (task) Updating task: 471500bf-d7f3-4384-ae21-0aec3db6ec57
2016-09-20 11:17:05.596 DEBUG [7fa4693c6880] (task) Updating cluster status: 471500bf-d7f3-4384-ae21-0aec3db6ec57 cluster_id: 1 status: error
2016-09-20 11:17:05.605 DEBUG [7fa4693c6880] (task) Updating cluster (TestMuranoPluginBvt (id=1, mode=ha_compact)) status: from operational to error

Mykola, you trying to upgrade the cluster with detach-murano plugin installed, the problem can be caused by the plugin, and actually the environment successfully deployed, but nailgun logs contain this error.

[Tatsuro Makita (makita-tatsuro)]

I try to update the Mirantis OpenStack environment from 9.0 to 9.1.

Procedure：7
https://docs.mirantis.com/openstack/fuel/fuel-9.1/release-notes/update-product.html

It has caused the following problems.
Is this would be similar events?

This environment was using 'Neutron VMware DVS ML2 plugin' plugin.

```
[root@fuel ~]# fuel env
id | status | name | release_id
---+-------------+------+-----------
1 | operational | DVS | 2
[root@fuel ~]#
[root@fuel ~]# update-prepare prepare env 1
Warning: Permanently added '192.168.114.103' (ECDSA) to the list of known hosts.
W: GPG error: http://mirror.fuel-infra.org mos9.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
Reading package lists...
Building dependency tree...
Reading state information...
nailgun-agent is already the newest version.
nailgun-mcagents is already the newest version.
The following packages were automatically installed and are no longer required:
  cloud-guest-utils eatmydata python-cheetah python-oauth python-serial
  python3-pycurl python3-software-properties software-properties-common
  unattended-upgrades
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 53 not upgraded.
Warning: Permanently added '192.168.114.104' (ECDSA) to the list of known hosts.
W: GPG error: http://mirror.fuel-infra.org mos9.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
Reading package lists...
Building dependency tree...
Reading state information...
nailgun-agent is already the newest version.
nailgun-mcagents is already the newest version.
The following packages were automatically installed and are no longer required:
  cloud-guest-utils eatmydata python-cheetah python-oauth python-serial
  python3-pycurl python3-software-properties software-properties-common
  unattended-upgrades
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 52 not upgraded.
Warning: Permanently added '192.168.114.102' (ECDSA) to the list of known hosts.
W: GPG error: http://mirror.fuel-infra.org mos9.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos9.0-security Release: The following signatures couldn't be verified because the public key ...

[Denis Meltsaykin (dmeltsaykin)]

Tatsuro, could you please show us the output of the "apt-cache policy nailgun-mcagents" command from any slave node (no matter controller or compute)?

[Tatsuro Makita (makita-tatsuro)]

Hi Denis

Thank you for your comment.

I resolved this problem.
I report you that how to resolve in my environment.

Resolve procedure:
1. FuelSlaveNode login
2. gpg --recv-keys BCE5CC461FA22B08
3. apt-key list
--> Check: The [fuel-infra (Example key) <email address hidden>] Key does not exsist.
4. gpg -a --export BCE5CC461FA22B08 | apt-key add -
5. apt-key list
--> Check: The [fuel-infra (Example key) <email address hidden>] Key does exsist.
6. apt-get update
--> Check: To no output "GPG error" of the message.
7. Repeat 1-6 procedure in the other Fuel slave node.

I could update from 9.0 to 9.1.

And I tried to execute your command.
show you following.

----
[root@fuel ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---+-----------+------------------+---------+---------------+-------------------+------------+---------------+--------+---------
 1 | deploying | Untitled (a4:6c) | 1 | 192.168.114.3 | 00:0c:29:1e:a4:6c | controller | | 1 | 1
 2 | deploying | Untitled (d1:52) | 1 | 192.168.114.2 | 00:0c:29:d9:d1:52 | compute | | 1 | 1
[root@fuel ~]# ssh 192.168.114.3
Warning: Permanently added '192.168.114.3' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-98-generic x86_64)

 * Documentation: https://help.ubuntu.com/
New release '16.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Thu Oct 20 05:12:53 2016 from 192.168.114.100
root@node-1:~# apt-cache policy nailgun-mcagents
nailgun-mcagents:
  Installed: 9.0.0-1~u14.04+mos774
  Candidate: 9.0.0-1~u14.04+mos774
  Version table:
 *** 9.0.0-1~u14.04+mos774 0
       1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     9.0.0-1~u14.04+mos750 0
       1050 http://192.168.114.100:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
root@node-1:~#
----

[Tatsuro Makita (makita-tatsuro)]

Hi, everyone.

I posted the procedure.
This information will do that useful for everyone.

Login Slave node.
And ....
---
root@node-1:~# gpg --recv-keys BCE5CC461FA22B08
gpg: requesting key 1FA22B08 from hkp server keys.gnupg.net
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key 1FA22B08: public key "fuel-infra (Example key) <email address hidden>" imported
gpg: Total number processed: 1
gpg: imported: 1 (RSA: 1)
root@node-1:~# apt-key list
/etc/apt/trusted.gpg
--------------------
pub 1024D/437D05B5 2004-09-12
uid Ubuntu Archive Automatic Signing Key <email address hidden>
sub 2048g/79164387 2004-09-12

pub 1024D/FBB75451 2004-12-30
uid Ubuntu CD Image Automatic Signing Key <email address hidden>

pub 4096R/C0B21F32 2012-05-11
uid Ubuntu Archive Automatic Signing Key (2012) <email address hidden>

pub 4096R/EFE21092 2012-05-11
uid Ubuntu CD Image Automatic Signing Key (2012) <email address hidden>

root@node-1:~# gpg -a --export BCE5CC461FA22B08 | apt-key add -
OK
root@node-1:~# apt-key list
/etc/apt/trusted.gpg
--------------------
pub 1024D/437D05B5 2004-09-12
uid Ubuntu Archive Automatic Signing Key <email address hidden>
sub 2048g/79164387 2004-09-12

pub 1024D/FBB75451 2004-12-30
uid Ubuntu CD Image Automatic Signing Key <email address hidden>

pub 4096R/C0B21F32 2012-05-11
uid Ubuntu Archive Automatic Signing Key (2012) <email address hidden>

pub 4096R/EFE21092 2012-05-11
uid Ubuntu CD Image Automatic Signing Key (2012) <email address hidden>

pub 2048R/1FA22B08 2015-07-21
uid fuel-infra (Example key) <email address hidden>
sub 2048R/6273AA09 2015-07-21

root@node-1:~#
----

Please execute all Slave node.

[Denis Meltsaykin (dmeltsaykin)]

The root cause of the issue is not connected with GPG errors. It is caused by wrong repository preferences on slave nodes.

[Denis Meltsaykin (dmeltsaykin)]

The fix is on review: https://review.fuel-infra.org/#/c/27654/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/27654
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: db8a09ed4f9ee25c64b1a399526a7b80d26081ae
Author: Denis V. Meltsaykin <email address hidden>
Date: Thu Oct 20 14:57:19 2016

Fix repository pins

Before 9.1 was released the mos9.0-updates repository was unusable
and contained wrong data. Since prioirities for the repositories
calculated during deployment all environments deployed before 9.1
releasing have the error in their repository pins and the unusable
update repository. This is fixed with sed during a preparation
procedure. Also restarting of mcollective is added, since the nature of
ruby classes it might not refresh its agents.

Change-Id: I10ce8e2e9ef3221a41fbdcfcdb066dd2b8ef92c0
Closes-Bug: #1625611

[TatyanaGladysheva (tgladysheva)]

Verified on 9.2 snapshot #465.