Comment 3 for bug 1918366

Revision history for this message
Javier Peña (jpena-c) wrote :

I have been checking the logs for the neutron db sync issue. While checking the logs, we see that jobs running on host-id 144026dc3a6267aa9593dce15cccee121a718779ba75fb6ce23b65e7 tend to exhibit this behavior. Here are some examples, all on the same host-id:

- https://logserver.rdoproject.org/51/29351/23/check/periodic-tripleo-ci-centos-8-scenario010-standalone-master/cc0c0bf/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log.txt.gz
- https://logserver.rdoproject.org/59/32159/4/check/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri/28b571c/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log.txt.gz
- https://logserver.rdoproject.org/59/32159/4/check/periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri/ac89ed2/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log.txt.gz
- https://logserver.rdoproject.org/60/31460/8/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria/0799ef8/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log.txt.gz
- https://logserver.rdoproject.org/62/29562/5/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/731a24c/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log.txt.gz
- https://logserver.rdoproject.org/89/32289/1/check/rdoinfo-tripleo-master-testing-centos-8-containers-multinode/7a6705d/logs/undercloud/var/log/containers/stdouts/neutron_db_sync.log.txt.gz

When examining the VM behavior during the neutron db-sync operation, we find that the disk usage is very high, but performance is low (99%busy, avq > 6 ms, avio > 10ms, but < 1.0 MB/s written). In another job, running on a different host (host-id cd94de9976e8f8b3f2212dccfe55af90ebe4613f70f21e858f6b7d20), the operation finished in about 20 seconds, and disk performance was normal (15%busy, avq < 5ms, avio < 1ms, > 2 MB/s written). No other VM parameter was abnormal (CPU and memory usage were good).

I've opened a ticket to our cloud provider so they can check the host.