Glance

OOM kills python / mysqld in various nova devstack jobs

Bug #2002951 reported by Balazs Gibizer
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
New
Undecided
Unassigned
OpenStack Compute (nova)
Fix Released
High
Unassigned
tempest
Fix Released
Undecided
Unassigned

Bug Description

The following tests exited without returning a status
and likely segfaulted or crashed Python:

 * tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive[id-777e468f-17ca-4da4-b93d-b7dbf56c0494]

And in the syslog: https://zuul.opendev.org/t/openstack/build/f5aa5edd4d354c2685fc1f3e13d0ef77/log/controller/logs/syslog.txt#3688

Jan 13 22:31:13 np0032729364 kernel: Out of memory: Killed process 114509 (python) total-vm:4966188kB, anon-rss:3914748kB, file-rss:5080kB, shmem-rss:0kB, UID:1002 pgtables:9764kB oom_score_adj:0

Example run: https://zuul.opendev.org/t/openstack/build/f5aa5edd4d354c2685fc1f3e13d0ef77

I see this happening in multiple jobs in the last 10 days:
* nova-ceph-multistore 14x
* nova-multi-cell 1x
* nova-next 1x

$ logsearch log --result FAILURE --project openstack/nova --branch master --file controller/logs/syslog.txt 'kernel: Out of memory: Killed process' --days 10
[..snip..]
Searching logs:
ece0cf2ce71c4a8790a0a36529dd0a8e:/home/gibi/.cache/logsearch/ece0cf2ce71c4a8790a0a36529dd0a8e/controller/logs/syslog.txt:3774:Jan 14 22:57:33 np0032733292 kernel: Out of memory: Killed process 115024 (python) total-vm:4981004kB, anon-rss:3904068kB, file-rss:5320kB, shmem-rss:0kB, UID:1002 pgtables:9376kB oom_score_adj:0

f5aa5edd4d354c2685fc1f3e13d0ef77:/home/gibi/.cache/logsearch/f5aa5edd4d354c2685fc1f3e13d0ef77/controller/logs/syslog.txt:3688:Jan 13 22:31:13 np0032729364 kernel: Out of memory: Killed process 114509 (python) total-vm:4966188kB, anon-rss:3914748kB, file-rss:5080kB, shmem-rss:0kB, UID:1002 pgtables:9764kB oom_score_adj:0

1447c6274e924e068578ca260c9ac2a6:/home/gibi/.cache/logsearch/1447c6274e924e068578ca260c9ac2a6/controller/logs/syslog.txt:3824:Jan 13 21:34:13 np0032729237 kernel: Out of memory: Killed process 114489 (python) total-vm:4975072kB, anon-rss:3954804kB, file-rss:5312kB, shmem-rss:0kB, UID:1002 pgtables:9400kB oom_score_adj:0

446a5a73b22d432295820e5b8083a2f9:/home/gibi/.cache/logsearch/446a5a73b22d432295820e5b8083a2f9/controller/logs/syslog.txt:5103:Jan 13 10:04:25 np0032720733 kernel: Out of memory: Killed process 48920 (mysqld) total-vm:5233384kB, anon-rss:300872kB, file-rss:0kB, shmem-rss:0kB, UID:116 pgtables:2652kB oom_score_adj:0

fae1fbe258134dd8ba060cb743707247:/home/gibi/.cache/logsearch/fae1fbe258134dd8ba060cb743707247/controller/logs/syslog.txt:6686:Jan 13 09:44:04 np0032720410 kernel: Out of memory: Killed process 47404 (mysqld) total-vm:5208828kB, anon-rss:278080kB, file-rss:0kB, shmem-rss:0kB, UID:116 pgtables:2572kB oom_score_adj:0

1bbcaa703b7d42c7a266fde3a6acca65:/home/gibi/.cache/logsearch/1bbcaa703b7d42c7a266fde3a6acca65/controller/logs/syslog.txt:3717:Jan 13 03:41:39 np0032719591 kernel: Out of memory: Killed process 114777 (python) total-vm:4954352kB, anon-rss:4001500kB, file-rss:5124kB, shmem-rss:0kB, UID:1002 pgtables:9416kB oom_score_adj:0

7d9ca42edc5e4bdeb17be8e8045c6468:/home/gibi/.cache/logsearch/7d9ca42edc5e4bdeb17be8e8045c6468/controller/logs/syslog.txt:3828:Jan 12 22:06:40 np0032716841 kernel: Out of memory: Killed process 114731 (python) total-vm:4964792kB, anon-rss:4055532kB, file-rss:5072kB, shmem-rss:0kB, UID:1002 pgtables:9212kB oom_score_adj:0

bcb7bcbbbb3b478586906c31c6558b13:/home/gibi/.cache/logsearch/bcb7bcbbbb3b478586906c31c6558b13/controller/logs/syslog.txt:3769:Jan 12 20:17:35 np0032714959 kernel: Out of memory: Killed process 114973 (python) total-vm:4971976kB, anon-rss:3855572kB, file-rss:5356kB, shmem-rss:0kB, UID:1002 pgtables:9696kB oom_score_adj:0

7572c2bf5e6547c0a1fc6b0f180a2e1f:/home/gibi/.cache/logsearch/7572c2bf5e6547c0a1fc6b0f180a2e1f/controller/logs/syslog.txt:3805:Jan 12 17:44:16 ubuntu-focal-ovh-gra1-0032713996 kernel: Out of memory: Killed process 114616 (python) total-vm:4974804kB, anon-rss:3949084kB, file-rss:5176kB, shmem-rss:0kB, UID:1002 pgtables:9604kB oom_score_adj:0

aa5cf699f8d04995b43d009e55a1accd:/home/gibi/.cache/logsearch/aa5cf699f8d04995b43d009e55a1accd/controller/logs/syslog.txt:3796:Jan 12 16:23:26 ubuntu-focal-inmotion-iad3-0032713625 kernel: Out of memory: Killed process 114640 (python) total-vm:4964156kB, anon-rss:4310768kB, file-rss:5340kB, shmem-rss:0kB, UID:1002 pgtables:9628kB oom_score_adj:0

8bc71a0ec0d34373bd25d4f691136084:/home/gibi/.cache/logsearch/8bc71a0ec0d34373bd25d4f691136084/controller/logs/syslog.txt:3794:Jan 12 15:27:35 ubuntu-focal-rax-dfw-0032712709 kernel: Out of memory: Killed process 114830 (python) total-vm:4968664kB, anon-rss:3861940kB, file-rss:5140kB, shmem-rss:0kB, UID:1002 pgtables:9380kB oom_score_adj:0

81d7cef2e0b240f89fcfa727304d8e8d:/home/gibi/.cache/logsearch/81d7cef2e0b240f89fcfa727304d8e8d/controller/logs/syslog.txt:3785:Jan 12 14:50:02 ubuntu-focal-rax-ord-0032711683 kernel: Out of memory: Killed process 116102 (python) total-vm:4975108kB, anon-rss:4059012kB, file-rss:5316kB, shmem-rss:0kB, UID:1002 pgtables:9644kB oom_score_adj:0

c75eb700717b4d3c9942be1385cd45bf:/home/gibi/.cache/logsearch/c75eb700717b4d3c9942be1385cd45bf/controller/logs/syslog.txt:3777:Jan 11 21:01:17 ubuntu-focal-rax-iad-0032702258 kernel: Out of memory: Killed process 114917 (python) total-vm:4969648kB, anon-rss:3886448kB, file-rss:5236kB, shmem-rss:0kB, UID:1002 pgtables:9732kB oom_score_adj:0

fa2d7bea85ad4d29acc37d78d2adb3c3:/home/gibi/.cache/logsearch/fa2d7bea85ad4d29acc37d78d2adb3c3/controller/logs/syslog.txt:3737:Jan 09 18:25:11 ubuntu-focal-rax-ord-0032676791 kernel: Out of memory: Killed process 114623 (python) total-vm:4965012kB, anon-rss:3819224kB, file-rss:5068kB, shmem-rss:0kB, UID:1002 pgtables:9372kB oom_score_adj:0

ba57680eb8de4bf2841ed5f6b2d8b3cc:/home/gibi/.cache/logsearch/ba57680eb8de4bf2841ed5f6b2d8b3cc/controller/logs/syslog.txt:3830:Jan 09 18:18:09 ubuntu-focal-inmotion-iad3-0032676865 kernel: Out of memory: Killed process 114140 (python) total-vm:4963936kB, anon-rss:3869684kB, file-rss:5308kB, shmem-rss:0kB, UID:1002 pgtables:8856kB oom_score_adj:0

4e19ddc2b0064e548093ad06205a7d67:/home/gibi/.cache/logsearch/4e19ddc2b0064e548093ad06205a7d67/controller/logs/syslog.txt:3839:Jan 09 18:07:16 ubuntu-focal-ovh-bhs1-0032676704 kernel: Out of memory: Killed process 114230 (python) total-vm:4974020kB, anon-rss:3923744kB, file-rss:5276kB, shmem-rss:0kB, UID:1002 pgtables:9396kB oom_score_adj:0

Builds with matching logs 16/409:
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+
| uuid | finished | pipeline | review | job |
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+
| ece0cf2ce71c4a8790a0a36529dd0a8e | 2023-01-14T23:17:56 | check | https://review.opendev.org/866218 | nova-ceph-multistore |
| f5aa5edd4d354c2685fc1f3e13d0ef77 | 2023-01-13T23:10:05 | gate | https://review.opendev.org/869900 | nova-ceph-multistore |
| 1447c6274e924e068578ca260c9ac2a6 | 2023-01-13T22:03:25 | gate | https://review.opendev.org/866218 | nova-ceph-multistore |
| 446a5a73b22d432295820e5b8083a2f9 | 2023-01-13T10:55:20 | check | https://review.opendev.org/869950 | nova-multi-cell |
| fae1fbe258134dd8ba060cb743707247 | 2023-01-13T10:47:16 | check | https://review.opendev.org/855654 | nova-next |
| 1bbcaa703b7d42c7a266fde3a6acca65 | 2023-01-13T04:17:19 | check | https://review.opendev.org/867978 | nova-ceph-multistore |
| 7d9ca42edc5e4bdeb17be8e8045c6468 | 2023-01-12T22:36:25 | check | https://review.opendev.org/863918 | nova-ceph-multistore |
| bcb7bcbbbb3b478586906c31c6558b13 | 2023-01-12T20:41:13 | check | https://review.opendev.org/869900 | nova-ceph-multistore |
| 7572c2bf5e6547c0a1fc6b0f180a2e1f | 2023-01-12T18:22:51 | check | https://review.opendev.org/869950 | nova-ceph-multistore |
| aa5cf699f8d04995b43d009e55a1accd | 2023-01-12T16:52:01 | check | https://review.opendev.org/869950 | nova-ceph-multistore |
| 8bc71a0ec0d34373bd25d4f691136084 | 2023-01-12T15:52:19 | check | https://review.opendev.org/870012 | nova-ceph-multistore |
| 81d7cef2e0b240f89fcfa727304d8e8d | 2023-01-12T15:39:13 | check | https://review.opendev.org/670213 | nova-ceph-multistore |
| c75eb700717b4d3c9942be1385cd45bf | 2023-01-11T21:30:33 | check | https://review.opendev.org/863916 | nova-ceph-multistore |
| fa2d7bea85ad4d29acc37d78d2adb3c3 | 2023-01-09T18:43:44 | check | https://review.opendev.org/863918 | nova-ceph-multistore |
| ba57680eb8de4bf2841ed5f6b2d8b3cc | 2023-01-09T18:50:43 | check | https://review.opendev.org/863920 | nova-ceph-multistore |
| 4e19ddc2b0064e548093ad06205a7d67 | 2023-01-09T18:37:37 | check | https://review.opendev.org/863915 | nova-ceph-multistore |
+----------------------------------+---------------------+----------+-----------------------------------+----------------------+

Tags: gate-failure
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :
Download full text (4.0 KiB)

in all the 16 nova-ceph-multistore jobs that failed in the last 10 days the same test case got killed

$ grep Killed oom-kills.txt | cut -d ':' -f2 | sed 's|controller/logs/syslog.txt|job-output.txt|g' | xargs grep inprogress
/home/gibi/.cache/logsearch/358f5a3fda114764b794323a2f5d2312/job-output.txt:2023-01-17 09:32:24.979704 | controller | {3} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/8587e6a4b6de4edb93de5c064928788e/job-output.txt:2023-01-16 12:00:25.032670 | controller | {2} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/ece0cf2ce71c4a8790a0a36529dd0a8e/job-output.txt:2023-01-14 23:15:05.647826 | controller | {1} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/f5aa5edd4d354c2685fc1f3e13d0ef77/job-output.txt:2023-01-13 23:07:03.726600 | controller | {0} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/1447c6274e924e068578ca260c9ac2a6/job-output.txt:2023-01-13 22:00:29.545457 | controller | {0} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/1bbcaa703b7d42c7a266fde3a6acca65/job-output.txt:2023-01-13 04:13:16.809645 | controller | {2} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/7d9ca42edc5e4bdeb17be8e8045c6468/job-output.txt:2023-01-12 22:32:21.335537 | controller | {3} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/bcb7bcbbbb3b478586906c31c6558b13/job-output.txt:2023-01-12 20:38:12.837475 | controller | {2} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/7572c2bf5e6547c0a1fc6b0f180a2e1f/job-output.txt:2023-01-12 18:17:48.522025 | controller | {0} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/aa5cf699f8d04995b43d009e55a1accd/job-output.txt:2023-01-12 16:48:59.307916 | controller | {1} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/8bc71a0ec0d34373bd25d4f691136084/job-output.txt:2023-01-12 15:49:35.341633 | controller | {0} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/81d7cef2e0b240f89fcfa727304d8e8d/job-output.txt:2023-01-12 15:35:03.978880 | controller | {1} tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive [] ... inprogress
/home/gibi/.cache/logsearch/c75eb700717b4d3c9942be1385cd4...

Read more...

tags: added: gate-failure
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

I took 6 of these runs and extracted which test cases ran in the killed worker. Then I checked for common test cases and I only found tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive in the intersection of the test case sets. That is also the one that got killed by OOM.

Sylvain Bauza (sylvain-bauza)
Changed in nova:
status: New → Confirmed
importance: Undecided → High
importance: High → Critical
Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

One idea from Dan to move this to the end of the test case list by renaming it to see if there are a set of test cases triggering OOM not just this one. That would help finding a pattern.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/870924

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :
Download full text (4.8 KiB)

In the run https://zuul.opendev.org/t/openstack/build/a52064f718904b33a7b7136bf63a1458/logs the test test_attach_scsi_disk_with_config_drive was moved to the end of the execution and it still triggered an OOM during the test execution.

OOM happened at
Jan 18 15:55:13 np0032779705 kernel: Out of memory: Killed process 127344 (python) total-vm:5232640kB, anon-rss:4173792kB, file-rss:5404kB, shmem-rss:0kB, UID:1002 pgtables:8964kB oom_score_adj:0

dstat shows that there is a sudden increase in memory starting at 15:55:02:

Jan 18 15:55:00.704919 np0032779705 dstat.sh[85428]: 18-01 15:55:00| 28 12 59 1 0|6225M 142M 256M 990M| 569M 569M| 171M 21M|1453 458 |9600 10k|1.65 2.42 4.12|2.0 0 25|40.0 5343 |2297M 1998M| 31 494 0 132 8

Jan 18 15:55:01.702533 np0032779705 dstat.sh[85428]: 18-01 15:55:01| 14 0 85 0 1|6224M 143M 256M 991M| 79k 79k| 265k 148k|17.0 10.0 |2051 4493 |1.65 2.42 4.12|1.0 0 0| 0 0 |2297M 1998M| 31 492 0 101 10

Jan 18 15:55:02.703908 np0032779705 dstat.sh[85428]: 18-01 15:55:02| 14 2 84 0 0|6339M 118M 256M 902M| 39k 39k| 492k 15M|19.0 592 |4253 6209 |1.65 2.42 4.12|1.0 0 0|4.00 2178 |2306M 1989M| 31 492 0 101 10

Jan 18 15:55:03.704748 np0032779705 dstat.sh[85428]: 18-01 15:55:03| 10 11 79 0 0|6965M 128M 227M 300M|3700 3700 | 480k 91M|39.0 1237 |9253 5032 |1.68 2.41 4.11|3.0 0 1.0|2.00 23.0k|2399M 1896M| 31 492 0 95 10

Jan 18 15:55:04.712489 np0032779705 dstat.sh[85428]: 18-01 15:55:04| 10 14 74 1 0|7437M 125M 21M 55M| 34k 34k|6923k 203M| 371 1489 | 16k 7296 |1.68 2.41 4.11|3.0 0 5.0|30.8 50.8k|2605M 1690M| 31 492 0 95 10

Jan 18 15:55:05.711668 np0032779705 dstat.sh[85428]: 18-01 15:55:05| 10 19 67 4 0|7488M 103M 2560k 49M|1318 1318 | 50M 513M|1645 2102 | 12k 9149 |1.68 2.41 4.11|4.0 0 6.0| 623 131k|3139M 1156M| 31 492 0 90 10

Jan 18 15:55:06.702850 np0032779705 dstat.sh[85428]: 18-01 15:55:06| 12 17 54 16 0|7415M 126M 6088k 95M| 97k 97k| 100M 258M|4687 6623 | 21k 23k|1.68 2.41 4.11|1.0 2.0 434|10.2k 64.3k|3371M 924M| 31 492 0 91 10

Jan 18 15:55:07.705732 np0032779705 dstat.sh[85428]: 18-01 15:55:07| 5 9 75 10 0|7427M 120M 5272k 90M|7606 7606 | 65M 153M|3704 3302 | 15k 13k|1.68 2.41 4.11|4.0 0 2.0|13.4k 39.3k|3489M 806M| 31 492 0 90 10

Jan 18 15:55:08.703123 np0032779705 dstat.sh[85428]: 18-01 15:55:08| 5 10 76 9 0|7434M 121M 4320k 85M| 35k 35k| 60M 193M|3134 6494 | 16k 10k|1.62 2.39 4.09|3.0 0 0|14.2k 48.8k|3641M 654M| 31 492 0 90 10

Jan 18 15:55:09.704144 np0032779705 dstat.sh[85428]: 18-01 15:55:09| 8 12 56 24 0|7434M 122M 4120k 83M| 46k 46k| 116M 197M|13.6k 6780 | 30k 37k|1.62 2.39 4.09|4.0 1.0 0|27.9k 50.6k|3762M 533M| 31 492 0 90 10

Jan 18 15:55:10.701265 np0032779705 dstat.sh[85428]: 18-01 15:55:10| 5 10 75 9 1|7442M 122M 3784k 76M| 33k 33k| 63M 199M|3784 4171 | 16k 13k|1.62 2.39 4.09|1.0 1.0 1.0|15.7k 50.8k|3920M 375M| 31 492 0 90 10

Jan 18 15:55:11.702276 np0032779705 dstat.sh[85428]: 18-01 15:55:11| 5 10...

Read more...

Revision history for this message
Balazs Gibizer (balazs-gibizer) wrote :

The suspected code part is called from our test here:
The https://github.com/openstack/tempest/blob/6fa213cc0fcab744b46109e5f07cb58b0df4a314/tempest/api/compute/admin/test_volume.py#L84

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

FWIW, I created another change that was running this test *earlier*, and it worked :

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_362/870924/2/check/nova-ceph-multistore/3626391/testr_results.html

That being said, this test tooked more than 181secs so I created a new revision for knowing how it takes for creating the cached image and how large this cached image is using the memory :

https://review.opendev.org/c/openstack/tempest/+/870913/2/tempest/api/compute/admin/test_aaa_volume.py#90

Still waiting the results but here I think we need to modify this test to maybe not caching this way if we can, or maybe to be run differently.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tempest (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/870974

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Glance default cirros image increased on November from 16MB to 1GB. Shouldn't be in theory a problem but since this test is blindly copying the full image data into memory, it's becoming a very greedy run.

This test necessarly needs to be refactored so that only metadata is copied in memory or if data has to be compared, this has necessarly has to be done by chunks.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/871000

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tempest (master)

Reviewed: https://review.opendev.org/c/openstack/tempest/+/870974
Committed: https://opendev.org/openstack/tempest/commit/a6b7e334cf27eddacb9ec28fb728dbfa30e4a8c6
Submitter: "Zuul (22348)"
Branch: master

commit a6b7e334cf27eddacb9ec28fb728dbfa30e4a8c6
Author: Balazs Gibizer <email address hidden>
Date: Wed Jan 18 17:43:01 2023 +0100

    Skip test_attach_scsi_disk_with_config_drive

    This test causes excessive memory usage in the test executor as it
    downloads glance image data in memory. This currently causes OOM issues
    in the gate so the test case is skipped until the image data copying is
    made smarter.

    Related-Bug: #2002951
    Change-Id: I0bfac0310fdd6ef44a633d40422cfcfdfc85a426

Revision history for this message
Lukas Piwowarski (lukas-piwowarski) wrote :

I'm moving this to Confirmed as we agreed in this patch that this is an issue [1]. We should close this once the test_attach_scsi_disk_with_config_drive is unskipped.

[1] https://opendev.org/openstack/tempest/commit/a6b7e334cf27eddacb9ec28fb728dbfa30e4a8c6

Changed in tempest:
status: New → Confirmed
Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Putting to high as the test is not skipped thanks to https://review.opendev.org/c/openstack/tempest/+/870974

Proposal to fix the rootcause (downloading the entire image in memory) is under review https://review.opendev.org/c/openstack/tempest/+/871000

Changed in nova:
importance: Critical → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/tempest/+/871000
Committed: https://opendev.org/openstack/tempest/commit/2c192f46db1b253fe312f6539bfda4735bd07e15
Submitter: "Zuul (22348)"
Branch: master

commit 2c192f46db1b253fe312f6539bfda4735bd07e15
Author: Dan Smith <email address hidden>
Date: Wed Jan 18 11:22:34 2023 -0800

    Chunked GET request support

    In one test, we are downloading the entire image (into memory) and
    re-uploading it. That works when the image is 16MiB but not when it
    is 1GiB. This adds support to the internal http client for chunked
    downloads (similar to upload), makes the image client able to take
    that flag, and finally makes the offending test do a chunked upload/
    download streaming operation.

    Note this un-skips the test, effectively reverting a6b7e334c
    because the test should no longer consume large amounts of memory.

    Related-Bug: #2002951
    Change-Id: I31e537538a1862e71091aa470da3b8e9c799bf15

Revision history for this message
Sylvain Bauza (sylvain-bauza) wrote :

Moving the Nova status of the bug to Fix Released as https://review.opendev.org/c/openstack/tempest/+/871000 fixed the root cause for the failing nova jobs.

Changed in nova:
status: Confirmed → Fix Released
Revision history for this message
Lukas Piwowarski (lukas-piwowarski) wrote :

I'm moving this to "Fix Released" as the bug was fixed in https://review.opendev.org/c/openstack/tempest/+/871000. I checked the test_attach_scsi_disk_with_config_drive test and in the last 7 days, the test failed 8 times. None failure out of these 8 failures was relevant to this bug [1].

[1] https://opensearch.logs.openstack.org/_dashboards/app/dashboards?security_tenant=global/view/a2282980-4327-11ed-8b28-cdde0c7cf02f?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:'test_name.keyword:%22tempest.api.compute.admin.test_volume.AttachSCSIVolumeTestJSON.test_attach_scsi_disk_with_config_drive%22%20and%20test_status.keyword:%22fail%22'),timeRestore:!f,title:'Subunit%20-%20Test%20Results',viewMode:view)

Changed in tempest:
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tempest (master)

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/870913
Reason: no activity for long time, feel free to restore it if needed

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by "Stephen Finucane <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/nova/+/870924

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.