unattended-upgrade broke system

Bug #1664152 reported by Aleksey Zvyagintsev
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Aleksey Zvyagintsev
Newton
Fix Released
Critical
Aleksey Zvyagintsev

Bug Description

In fuel10 we have enabled "unattended-upgrades"[0] job for ubuntu-xenial

systemctl status unattended-upgrades.service
● unattended-upgrades.service - Unattended Upgrades Shutdown
   Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:unattended-upgrade(8)

Since we use unconfigured (with default suit configuration) service, it cause over-installing Canonical packages over MOS :

cat /var/log/unattended-upgrades/unattended-upgrades.log
2017-02-11 09:54:35,412 INFO Initial blacklisted packages:
2017-02-11 09:54:35,413 INFO Initial whitelisted packages:
2017-02-11 09:54:35,413 INFO Starting unattended upgrades script
2017-02-11 09:54:35,413 INFO Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security']
2017-02-11 09:54:39,881 INFO Packages that will be upgraded: haproxy libjs-bootswatch libjs-jquery-tablesorter python-pbr ruby-rethtool
2017-02-11 09:54:39,881 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2017-02-11 09:55:06,361 INFO All upgrades installed

Which mean, especially in my case: broken nailgun-agent, due downgrading ruby-rethtool package:

/var/log/nailgun-agent.log
W, [2017-02-13T10:32:10.317023 #31471] WARN -- : Can't get data from lspci. Reason: lspci exited with status 1
E, [2017-02-13T10:32:10.327470 #31471] ERROR -- : Error 'uninitialized constant Rethtool::EthtoolCmdDriver
Did you mean? Rethtool::EthtoolCmd' while obtaining enp3s0f1 driver info.
/usr/bin/lspci: option requires an argument -- 's'
Usage: lspci [<switches>]
.....
W, [2017-02-13T10:32:10.332541 #31471] WARN -- : Can't get data from lspci. Reason: lspci exited with status 1
E, [2017-02-13T10:32:10.520426 #31471] ERROR -- : {"message": "deque(['meta', 'interfaces', 0, 'driver']): None is not of type 'string'", "errors": []}

From my POV i can see 2 possible solutions :
1) Disable unattended-upgrades.service
2) Configure service to consider mos repositories

[0]https://help.ubuntu.com/community/AutomaticSecurityUpdates

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 10.x-updates → 10.1
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Marking as critical because it breaks default configuration

Changed in fuel:
assignee: nobody → MOS Linux (mos-linux)
importance: High → Critical
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

This bug is a bug in unattended upgrades package from debian/ubuntu.

Here are the bugs:

https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1251228
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838920

We can disable u-a and provide a script similar to u-a or patch it and send the patch to upstream and wait for upstream to accept it (while having it disabled during this period).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/433063

Changed in fuel:
assignee: MOS Linux (mos-linux) → Oleksiy Molchanov (omolchanov)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/433064

Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Aleksey Zvyagintsev (azvyagintsev)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/newton)

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/433064

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/433063

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/433071

tags: added: sla1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-agent (master)

Reviewed: https://review.openstack.org/433071
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=6f6fd2fa428e1ecdaaf36853679be00bd45aa108
Submitter: Jenkins
Branch: master

commit 6f6fd2fa428e1ecdaaf36853679be00bd45aa108
Author: azvyagintsev <email address hidden>
Date: Mon Feb 13 16:09:29 2017 +0200

    Disable Recommends and Suggests packages

      Disable Recommends and Suggests package installs,
      inside Ubuntu target-image

    Change-Id: Ie8a450a960b9b00cd4d2c0545a19678baed147d6
    Co-Authored-By: Ivan Suzdal <email address hidden>
    Closes-Bug: 1664152

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/434318

Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: 10.1 → 11.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (master)

Fix proposed to branch: master
Review: https://review.openstack.org/434770

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-agent (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/434802

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-agent (stable/newton)

Change abandoned by Aleksey Zvyagintsev (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/434802

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-agent (master)

Reviewed: https://review.openstack.org/434770
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=9f40d27bbecbee5ddf55fb69a2fb4cadf205ce25
Submitter: Jenkins
Branch: master

commit 9f40d27bbecbee5ddf55fb69a2fb4cadf205ce25
Author: azvyagintsev <email address hidden>
Date: Thu Feb 16 11:37:01 2017 +0200

    Add rsync dependency for fuel-agent

    Change-Id: I05e0b33420058772216220bb9fcd5f61c40f3e27
    Co-Authored-By: Ivan Suzdal <email address hidden>
    Partial-Bug: 1664152

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-agent (stable/newton)

Reviewed: https://review.openstack.org/434802
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=99b0cd6b3e46b75b1a9d7810bb81de1f2ff0a866
Submitter: Jenkins
Branch: stable/newton

commit 99b0cd6b3e46b75b1a9d7810bb81de1f2ff0a866
Author: azvyagintsev <email address hidden>
Date: Thu Feb 16 11:37:01 2017 +0200

    Add rsync dependency for fuel-agent

    Change-Id: I05e0b33420058772216220bb9fcd5f61c40f3e27
    Co-Authored-By: Ivan Suzdal <email address hidden>
    Partial-Bug: 1664152

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/434318
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=a3efbff41f8a0987225a12fc08a962cd3a77228d
Submitter: Jenkins
Branch: stable/newton

commit a3efbff41f8a0987225a12fc08a962cd3a77228d
Author: azvyagintsev <email address hidden>
Date: Mon Feb 13 16:09:29 2017 +0200

    Disable Recommends and Suggests packages

      Disable Recommends and Suggests package installs,
      inside Ubuntu target-image

    Change-Id: Ie8a450a960b9b00cd4d2c0545a19678baed147d6
    Co-Authored-By: Ivan Suzdal <email address hidden>
    Closes-Bug: 1664152
    (cherry picked from commit 6f6fd2fa428e1ecdaaf36853679be00bd45aa108)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-agent 11.0.0.0rc1

This issue was fixed in the openstack/fuel-agent 11.0.0.0rc1 release candidate.

Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on 10.0 #1543. No unattended-upgrades package found.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.