Migration from percona cluster charm in blocked state

Bug #1936467 reported by Diko Parvanov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Barbican Charm
Invalid
Undecided
Unassigned

Bug Description

2 out of 3 units in blocked state with error 'Services not running that should be: barbican-worker'

Logs for barbican-worker show:

2021-07-15 13:41:14.466 492690 ERROR barbican.model.repositories [-] Error configuring registry database with supplied sql_connection. Got error: (pymysql.err.OperationalError) (1045, "Access denied for user 'barbican'@'10.254.4.59' (using password: NO)")
(Background on this error at: http://sqlalche.me/e/e3q8): sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1045, "Access denied for user 'barbican'@'10.254.4.59' (using password: NO)")

/etc/barbican/barbican.conf has the same values for all 3 units, sql_connection is properly set to ues

sql_connection = mysql+pymysql://barbican:REDACTED@127.0.0.1:3306/barbican

barbican user exists in mysql innodb cluster and using mysql -u barbican -h 127.0.0.1 -p works on the units.

Strange things is report using password: NO

Revision history for this message
Diko Parvanov (dparv) wrote :

seem that [database] is missing from /etc/barbican/barbican.conf and the sql connection is not properly

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-barbican (master)
Changed in charm-barbican:
status: New → In Progress
Revision history for this message
David Ames (thedac) wrote :

This is an interesting bug.

Diko was able to add the [database] header and get things working.

However, when I updated the template to use section-database which includes the [database] header the db sync (barbican-manage db upgrade) fails as it attempts to use SQLite.

Upstream documentation even for the latest version of Barbican seems to indicate sql_connection in the [DEFAULT] header space [1].

It is theoretically possible the barbican-worker expects something different from what barbican-mage expects. It may not hurt to have both in the template. More investigation is necessary.

[0] https://review.opendev.org/c/openstack/charm-barbican/+/801361
[1] https://docs.openstack.org/barbican/latest/sample_config.html

Revision history for this message
Liam Young (gnuoy) wrote (last edit ):

I think what we are misinterpreting what we saw when sql_connection was moved into the [database] section. When sql_connection was in the DEFAULT section the parameter was picked up and for some reason yet to be determined the connection fails. When the sql_connection is moved into the [database] section it fails to be picked up at all and for some reason the barbican worker does not error, I assume because it drops back to sqllite. Basically moving sql_connection parameter into the database section is the equivalent of commenting it out.

 I validated this by:
1) Having a valid sql_connection string in the default section of the config and confirming there was a corresponding connection in the mysql db.
2) Moved the sql_connection into the [database] section and confirmed the barbican-worker started fine but there was no connection in the mysql db.

Revision history for this message
Liam Young (gnuoy) wrote :

fwiw I'm monitoring the barbican db connections with:

select * from information_schema.processlist where user='barbican';

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-barbican (master)

Change abandoned by "David Ames <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-barbican/+/801361
Reason: Root cause was out of date packages. Change is not required.

Revision history for this message
David Ames (thedac) wrote :

It is my understanding that package updates to the 2/3 machines that could not connect were the ultimate fix.

It would be very good to document the apt log entry here to determine which package was at fault. i.e. was it an oslo.db update?

Abandoned the PR and marking this bug invalid.

Changed in charm-barbican:
status: In Progress → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.