I have been checking the logs for the neutron db sync issue. While checking the logs, we see that jobs running on host-id 144026dc3a6267aa9593dce15cccee121a718779ba75fb6ce23b65e7 tend to exhibit this behavior. Here are some examples, all on the same host-id:
When examining the VM behavior during the neutron db-sync operation, we find that the disk usage is very high, but performance is low (99%busy, avq > 6 ms, avio > 10ms, but < 1.0 MB/s written). In another job, running on a different host (host-id cd94de9976e8f8b3f2212dccfe55af90ebe4613f70f21e858f6b7d20), the operation finished in about 20 seconds, and disk performance was normal (15%busy, avq < 5ms, avio < 1ms, > 2 MB/s written). No other VM parameter was abnormal (CPU and memory usage were good).
I've opened a ticket to our cloud provider so they can check the host.
I have been checking the logs for the neutron db sync issue. While checking the logs, we see that jobs running on host-id 144026dc3a6267a a9593dce15cccee 121a718779ba75f b6ce23b65e7 tend to exhibit this behavior. Here are some examples, all on the same host-id:
- https:/ /logserver. rdoproject. org/51/ 29351/23/ check/periodic- tripleo- ci-centos- 8-scenario010- standalone- master/ cc0c0bf/ logs/undercloud /var/log/ containers/ stdouts/ neutron_ db_sync. log.txt. gz /logserver. rdoproject. org/59/ 32159/4/ check/periodic- tripleo- ci-centos- 8-scenario010- ovn-provider- standalone- ussuri/ 28b571c/ logs/undercloud /var/log/ containers/ stdouts/ neutron_ db_sync. log.txt. gz /logserver. rdoproject. org/59/ 32159/4/ check/periodic- tripleo- ci-centos- 8-scenario010- standalone- ussuri/ ac89ed2/ logs/undercloud /var/log/ containers/ stdouts/ neutron_ db_sync. log.txt. gz /logserver. rdoproject. org/60/ 31460/8/ check/periodic- tripleo- ci-centos- 8-standalone- full-tempest- api-victoria/ 0799ef8/ logs/undercloud /var/log/ containers/ stdouts/ neutron_ db_sync. log.txt. gz /logserver. rdoproject. org/62/ 29562/5/ check/periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp-featurese t001-ussuri/ 731a24c/ logs/undercloud /var/log/ containers/ stdouts/ neutron_ db_sync. log.txt. gz /logserver. rdoproject. org/89/ 32289/1/ check/rdoinfo- tripleo- master- testing- centos- 8-containers- multinode/ 7a6705d/ logs/undercloud /var/log/ containers/ stdouts/ neutron_ db_sync. log.txt. gz
- https:/
- https:/
- https:/
- https:/
- https:/
When examining the VM behavior during the neutron db-sync operation, we find that the disk usage is very high, but performance is low (99%busy, avq > 6 ms, avio > 10ms, but < 1.0 MB/s written). In another job, running on a different host (host-id cd94de9976e8f8b 3f2212dccfe55af 90ebe4613f70f21 e858f6b7d20) , the operation finished in about 20 seconds, and disk performance was normal (15%busy, avq < 5ms, avio < 1ms, > 2 MB/s written). No other VM parameter was abnormal (CPU and memory usage were good).
I've opened a ticket to our cloud provider so they can check the host.