Security hardening YUM package add/remove task sometimes corrupts yum database on target hosts

Bug #1851954 reported by Jeff Albert
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Undecided
Jeff Albert

Bug Description

The ansible-hardening role contains in its RedHat 7 implementation of STIG a single play which both adds and removes packages:

https://github.com/openstack/ansible-hardening/blob/master/tasks/rhel7stig/packages.yml#L16

This task frequently results in RPM database corruption on the target hosts: it looks as if ansible splits the task into multiple calls to yum and triggers them simultaneously, resulting in lock conflicts that break the RPM database:

2019-11-09 18:10:01,305 p=43314 u=root | failed: [redacted_hostname] (item=absent) => {"changed": false, "item": "absent", "module_stderr": "Warning: Permanently added 'redacted_ip' (E
CDSA) to the list of known hosts.\r\n------------------------------------------------------------------------------\n* WARNING
                *\n* You are accessing a secured system and your actions will be logged along *\n* with identifying information. Disconnect immediately if you are not an *\
n* authorized user of this system. *\n------------------------------------------------------------------------------\nerror: rpmdb: BDB
0113 Thread/process 17868/140288263792704 failed: BDB1507 Thread died in Berkeley DB library\nerror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, r
un database recovery\nerror: cannot open Packages index using db5 - (-30973)\nerror: cannot open Packages database in /var/lib/rpm\nTraceback (most recent call last):\n File \"
/tmp/ansible_8P4c67/ansible_module_yum.py\", line 1411, in <module>\n main()\n File \"/tmp/ansible_8P4c67/ansible_module_yum.py\", line 1373, in main\n my.conf\n File \"/
usr/lib/python2.7/site-packages/yum/__init__.py\", line 1079, in <lambda>\n conf = property(fget=lambda self: self._getConfig(),\n File \"/usr/lib/python2.7/site-packages/yum
/__init__.py\", line 349, in _getConfig\n startupconf = config.readStartupConfig(fn, root, releasever)\n File \"/usr/lib/python2.7/site-packages/yum/config.py\", line 1115, i
n readStartupConfig\n startupconf.distroverpkg)\n File \"/usr/lib/python2.7/site-packages/yum/config.py\", line 1260, in _getsysver\n raise Errors.YumBaseError(\"Error: \"
 + str(e))\nyum.Errors.YumBaseError: Error: rpmdb open failed\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}

In my environment, where this issue has plagued our upgrade and scaling tasks, I have broken the task up into two separate plays; one for adding packages and one for removing them. This approach does not result in RPM database corruption. I recommend adopting this approach and backporting it to supported OSA releases.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-hardening (master)

Fix proposed to branch: master
Review: https://review.opendev.org/693614

Changed in openstack-ansible:
assignee: nobody → Jeff Albert (jralbert)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-hardening (master)

Reviewed: https://review.opendev.org/693614
Committed: https://git.openstack.org/cgit/openstack/ansible-hardening/commit/?id=8db1a33cbf636865fd876f2e312972c0f1d28c04
Submitter: Zuul
Branch: master

commit 8db1a33cbf636865fd876f2e312972c0f1d28c04
Author: Jeff Albert <email address hidden>
Date: Sat Nov 9 11:33:07 2019 -0800

    Splits STIG yum add/removes

    In order to prevent RPM database corruption on the target hosts, this
    change splits the STIG yum add/remove tasks into two separate plays.

    Change-Id: I68751339d5b4cbfb61b8e3cf4ffbfeb47ea5fd76
    Closes-Bug: #1851954

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-hardening (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/695406

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-hardening (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/695407

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ansible-hardening (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/695410

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-hardening (stable/stein)

Reviewed: https://review.opendev.org/695407
Committed: https://git.openstack.org/cgit/openstack/ansible-hardening/commit/?id=2d9ad40d0f2203010d126e3194c526dd7bfab89c
Submitter: Zuul
Branch: stable/stein

commit 2d9ad40d0f2203010d126e3194c526dd7bfab89c
Author: Jeff Albert <email address hidden>
Date: Sat Nov 9 11:33:07 2019 -0800

    Splits STIG yum add/removes

    In order to prevent RPM database corruption on the target hosts, this
    change splits the STIG yum add/remove tasks into two separate plays.

    Change-Id: I68751339d5b4cbfb61b8e3cf4ffbfeb47ea5fd76
    Closes-Bug: #1851954
    (cherry picked from commit 8db1a33cbf636865fd876f2e312972c0f1d28c04)

tags: added: in-stable-stein
tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-hardening (stable/train)

Reviewed: https://review.opendev.org/695406
Committed: https://git.openstack.org/cgit/openstack/ansible-hardening/commit/?id=56f11dbb5f8cd1744b868b7f7b9f86457b8a3509
Submitter: Zuul
Branch: stable/train

commit 56f11dbb5f8cd1744b868b7f7b9f86457b8a3509
Author: Jeff Albert <email address hidden>
Date: Sat Nov 9 11:33:07 2019 -0800

    Splits STIG yum add/removes

    In order to prevent RPM database corruption on the target hosts, this
    change splits the STIG yum add/remove tasks into two separate plays.

    Change-Id: I68751339d5b4cbfb61b8e3cf4ffbfeb47ea5fd76
    Closes-Bug: #1851954
    (cherry picked from commit 8db1a33cbf636865fd876f2e312972c0f1d28c04)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ansible-hardening (stable/rocky)

Reviewed: https://review.opendev.org/695410
Committed: https://git.openstack.org/cgit/openstack/ansible-hardening/commit/?id=013542e42bf99067c4db6a32a2aee1a30134bcba
Submitter: Zuul
Branch: stable/rocky

commit 013542e42bf99067c4db6a32a2aee1a30134bcba
Author: Jeff Albert <email address hidden>
Date: Sat Nov 9 11:33:07 2019 -0800

    Splits STIG yum add/removes

    In order to prevent RPM database corruption on the target hosts, this
    change splits the STIG yum add/remove tasks into two separate plays.

    Change-Id: I68751339d5b4cbfb61b8e3cf4ffbfeb47ea5fd76
    Closes-Bug: #1851954
    (cherry picked from commit 8db1a33cbf636865fd876f2e312972c0f1d28c04)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening rocky-eol

This issue was fixed in the openstack/ansible-hardening rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening stein-eol

This issue was fixed in the openstack/ansible-hardening stein-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening train-eol

This issue was fixed in the openstack/ansible-hardening train-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening ussuri-eol

This issue was fixed in the openstack/ansible-hardening ussuri-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening yoga-eom

This issue was fixed in the openstack/ansible-hardening yoga-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening victoria-eom

This issue was fixed in the openstack/ansible-hardening victoria-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening wallaby-eom

This issue was fixed in the openstack/ansible-hardening wallaby-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening xena-eom

This issue was fixed in the openstack/ansible-hardening xena-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ansible-hardening zed-eom

This issue was fixed in the openstack/ansible-hardening zed-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.