| 2025-07-18 19:00:30 |
Alexey Stupnikov |
description |
openstacksdk-functional-devstack-networking-ext is quite unstable: almost half of runs fail. From logs it looks like at some point keystone starts shooting HTTP 500 because of various reasons. But after a second look it looks like OOM kills are to blame.
https://zuul.opendev.org/t/openstack/build/83578f702aed4ff48460bed81454a5bb/artifacts:
- first failed test [1]
- mysql was restarted [2]
- OOM killed few processes [3]
[1]
https://de339b72b688803b5a43-80f76e47f83bff14c764cd8b2b7f1a08.ssl.cf2.rackcdn.com/openstack/83578f702aed4ff48460bed81454a5bb/job-output.txt
2025-07-18 11:08:43.078479 | controller | {0} openstack.tests.functional.network.v2.test_floating_ip.TestFloatingIP.test_get_tags [12.887117s] ... FAILED
2025-07-18 11:08:43.527669 | controller | keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
[2]
https://de339b72b688803b5a43-80f76e47f83bff14c764cd8b2b7f1a08.ssl.cf2.rackcdn.com/openstack/83578f702aed4ff48460bed81454a5bb/controller/logs/mysql/error_log.txt
2025-07-18T10:21:49.192932Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.42-0ubuntu0.24.04.2' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu).
2025-07-18T11:08:44.436067Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.42-0ubuntu0.24.04.2) starting as process 127538
[3]
https://de339b72b688803b5a43-80f76e47f83bff14c764cd8b2b7f1a08.ssl.cf2.rackcdn.com/openstack/83578f702aed4ff48460bed81454a5bb/controller/logs/syslog.txt
Jul 18 11:08:41 npc27e2a2bfa514 kernel: Out of memory: Killed process 688 (systemd) total-vm:20156kB, anon-rss:768kB, file-rss:4608kB, shmem-rss:0kB, UID:1000 pgtables:84kB oom_score_adj:100
Jul 18 11:08:41 npc27e2a2bfa514 kernel: Out of memory: Killed process 689 ((sd-pam)) total-vm:20988kB, anon-rss:0kB, file-rss:1152kB, shmem-rss:0kB, UID:1000 pgtables:76kB oom_score_adj:100
Jul 18 11:08:41 npc27e2a2bfa514 kernel: Out of memory: Killed process 47491 (mysqld) total-vm:2851636kB, anon-rss:222960kB, file-rss:7808kB, shmem-rss:0kB, UID:112 pgtables:1912kB oom_score_adj:0 |
openstacksdk-functional-devstack-networking-ext is quite unstable: almost half of runs fail. From logs it looks like at some point keystone starts shooting HTTP 500 because of various reasons. But after a second look it looks like OOM kills are to blame.
https://zuul.opendev.org/t/openstack/build/83578f702aed4ff48460bed81454a5bb/artifacts:
- first failed test [1]
- mysql was restarted [2]
- OOM killed few processes [3]
It looks like fails are pretty consistent, but OOM may kill different processes. So errors in test output may look different.
[1]
https://de339b72b688803b5a43-80f76e47f83bff14c764cd8b2b7f1a08.ssl.cf2.rackcdn.com/openstack/83578f702aed4ff48460bed81454a5bb/job-output.txt
2025-07-18 11:08:43.078479 | controller | {0} openstack.tests.functional.network.v2.test_floating_ip.TestFloatingIP.test_get_tags [12.887117s] ... FAILED
2025-07-18 11:08:43.527669 | controller | keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
[2]
https://de339b72b688803b5a43-80f76e47f83bff14c764cd8b2b7f1a08.ssl.cf2.rackcdn.com/openstack/83578f702aed4ff48460bed81454a5bb/controller/logs/mysql/error_log.txt
2025-07-18T10:21:49.192932Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.42-0ubuntu0.24.04.2' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu).
2025-07-18T11:08:44.436067Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.42-0ubuntu0.24.04.2) starting as process 127538
[3]
https://de339b72b688803b5a43-80f76e47f83bff14c764cd8b2b7f1a08.ssl.cf2.rackcdn.com/openstack/83578f702aed4ff48460bed81454a5bb/controller/logs/syslog.txt
Jul 18 11:08:41 npc27e2a2bfa514 kernel: Out of memory: Killed process 688 (systemd) total-vm:20156kB, anon-rss:768kB, file-rss:4608kB, shmem-rss:0kB, UID:1000 pgtables:84kB oom_score_adj:100
Jul 18 11:08:41 npc27e2a2bfa514 kernel: Out of memory: Killed process 689 ((sd-pam)) total-vm:20988kB, anon-rss:0kB, file-rss:1152kB, shmem-rss:0kB, UID:1000 pgtables:76kB oom_score_adj:100
Jul 18 11:08:41 npc27e2a2bfa514 kernel: Out of memory: Killed process 47491 (mysqld) total-vm:2851636kB, anon-rss:222960kB, file-rss:7808kB, shmem-rss:0kB, UID:112 pgtables:1912kB oom_score_adj:0 |
|
