Feature039 is failing during repo setup, Error: "Failed to set locale, defaulting to C , RPM: error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch"

[Sandeep Yadav]

Description:

Feature039 is failing during repo setup, Error: "Failed to set locale, defaulting to C , RPM: error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch"

Looks like issue started ~12:00 23rd Aug'21
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039

Log Snippet:-

~~~
2021-08-24 01:46:13.985690 | primary | TASK [repo-setup : Setup repos on live host] ***********************************
2021-08-24 01:46:13.985704 | primary | Tuesday 24 August 2021 01:46:13 +0000 (0:00:00.089) 0:00:40.295 ********
2021-08-24 01:55:46.588900 | primary | fatal: [supplemental]: FAILED! => {
2021-08-24 01:55:46.588982 | primary | "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result",
2021-08-24 01:55:46.588996 | primary | "changed": true
2021-08-24 01:55:46.589006 | primary | }
~~~

~~~
2021-08-23 15:56:17 | Failed to set locale, defaulting to C
2021-08-23 15:56:17 | RPM: error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch
2021-08-23 15:56:17 | RPM: error: cannot open Packages index using db5 - (-30969)
2021-08-23 15:56:17 | RPM: error: cannot open Packages database in /var/lib/rpm
2021-08-23 15:56:17 | The downloaded packages were saved in cache until the next successful transaction.
2021-08-23 15:56:17 | You can remove cached packages by executing 'dnf clean packages'.
2021-08-23 15:56:17 | Error: Could not run transaction.
~~~

Logs url:-

Master:-
https://logserver.rdoproject.org/98/804598/8/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/93831bb/logs/undercloud/home/zuul/repo_setup.sh.1629775791.log.txt.gz

Wallaby:-
https://logserver.rdoproject.org/59/805559/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/45677bf/logs/undercloud/home/zuul/repo_setup.sh.1629769448.log.txt.gz

[chandan kumar (chkumar246)]

Based on google search, here is workaround: https://github.com/WhitewaterFoundry/Fedora-Remix-for-WSL/issues/47

```
cd /var/lib/rpm/
sudo rpm --rebuilddb
sudo rpm -qa rpm
```

Is it coming for other feature set also? or Just fs039?

[chandan kumar (chkumar246)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master got skipped today and was passing earlier. It is specific to check only and only coming in jobs which is runned more than 45 mins.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/805803

[Rabi Mishra (rabi)]

Just noticed a green run https://review.rdoproject.org/zuul/build/aa0e177ded6a4ed7978530bd1fa2e135

Probably it's been automagically sorted or the error is intermittent.

[chandan kumar (chkumar246)]

Since the issue on happened on supplemental node:
https://logserver.rdoproject.org/98/804598/8/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/93831bb/logs/supplemental/home/centos/repo_setup.log.txt.gz

```
2021-08-24 03:35:55 | Failed to download metadata for repo 'centos-opstools'
2021-08-24 03:35:55 | Error: Failed to download metadata for repo 'centos-opstools'
```

Yes it might be a temporary issue.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/805803