neutron

Functional tests hits oom-killer which is causing job failures

Bug #1966394 reported by Slawek Kaplonski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Slawek Kaplonski

Bug Description

Failure example https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a75/833620/4/check/neutron-functional-with-uwsgi/a75e59b/testr_results.html

It seems that oom-killer is killing test runner and that is causing job failure:

mar 15 14:59:23 ubuntu-focal-ovh-gra1-0028840842 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-1.scope,task=/home/zuul/src/,pid=54377,uid=1000
mar 15 14:59:23 ubuntu-focal-ovh-gra1-0028840842 kernel: Out of memory: Killed process 54377 (/home/zuul/src/) total-vm:823632kB, anon-rss:676956kB, file-rss:4800kB, shmem-rss:0kB, UID:1000 pgtables:1508kB oom_score_adj:0
mar 15 14:59:23 ubuntu-focal-ovh-gra1-0028840842 kernel: oom_reaper: reaped process 54377 (/home/zuul/src/), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/835179

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/835179
Reason: I hope that https://review.opendev.org/c/openstack/neutron/+/836426 should be better way to fix that issue

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/836426
Committed: https://opendev.org/openstack/neutron/commit/b8dcb0b7afe06f92090d5d2c366b9aae3a532ebc
Submitter: "Zuul (22348)"
Branch: master

commit b8dcb0b7afe06f92090d5d2c366b9aae3a532ebc
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 5 09:11:11 2022 +0200

    Set bigger swap in the functional and fullstack jobs

    As we are hitting some memory limits (again) in functional tests
    job, lets configure 8GB of swap in all functional and fullstack jobs.

    Before that patch swap was configured to 4GB only in the FIPS jobs. Now
    it will be set to 8GB in all functional and fullstack jobs.

    Closes-Bug: #1966394

    Change-Id: I6b097d8409318a5dfe17e9673adb6c1431a59b0b

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/836120
Committed: https://opendev.org/openstack/neutron/commit/9bc447077bce642c25faed9306537b4f1bf57230
Submitter: "Zuul (22348)"
Branch: master

commit 9bc447077bce642c25faed9306537b4f1bf57230
Author: yatinkarel <email address hidden>
Date: Fri Apr 1 13:17:55 2022 +0530

    Enable dstat and memory_tracker in functional/fullstack jobs

    This Will help in troubleshooting failures related to high
    memory or cpu usage.

    Related-Bug: #1966394
    Change-Id: I74b0d53bfc54b71d3e8b2d46739a944e5f5a6b6f

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/837328

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/837328
Committed: https://opendev.org/openstack/neutron/commit/969a0db8b9acc468e994f437aede9952bb6acb78
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 969a0db8b9acc468e994f437aede9952bb6acb78
Author: yatinkarel <email address hidden>
Date: Fri Apr 1 13:17:55 2022 +0530

    Enable dstat and memory_tracker in functional/fullstack jobs

    This Will help in troubleshooting failures related to high
    memory or cpu usage.

    Related-Bug: #1966394
    Change-Id: I74b0d53bfc54b71d3e8b2d46739a944e5f5a6b6f
    (cherry picked from commit 9bc447077bce642c25faed9306537b4f1bf57230)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 21.0.0.0rc1

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/865993

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/865994

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/865995

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/866034

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/866036

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/865993
Committed: https://opendev.org/openstack/neutron/commit/aeed2f44a205d2b221d3aab16ddcd0d215925d87
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit aeed2f44a205d2b221d3aab16ddcd0d215925d87
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 5 09:11:11 2022 +0200

    Set bigger swap in the functional and fullstack jobs

    As we are hitting some memory limits (again) in functional tests
    job, lets configure 8GB of swap in all functional and fullstack jobs.

    Before that patch swap was configured to 4GB only in the FIPS jobs. Now
    it will be set to 8GB in all functional and fullstack jobs.

    Closes-Bug: #1966394

    Change-Id: I6b097d8409318a5dfe17e9673adb6c1431a59b0b
    (cherry picked from commit b8dcb0b7afe06f92090d5d2c366b9aae3a532ebc)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/865994
Committed: https://opendev.org/openstack/neutron/commit/4b374965dde02f6761d191737e5185bd4bb39374
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 4b374965dde02f6761d191737e5185bd4bb39374
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 5 09:11:11 2022 +0200

    Set bigger swap in the functional and fullstack jobs

    As we are hitting some memory limits (again) in functional tests
    job, lets configure 8GB of swap in all functional and fullstack jobs.

    Before that patch swap was configured to 4GB only in the FIPS jobs. Now
    it will be set to 8GB in all functional and fullstack jobs.

    Closes-Bug: #1966394

    Change-Id: I6b097d8409318a5dfe17e9673adb6c1431a59b0b
    (cherry picked from commit b8dcb0b7afe06f92090d5d2c366b9aae3a532ebc)

tags: added: in-stable-xena
tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/866034
Committed: https://opendev.org/openstack/neutron/commit/df3ea8c765648093343f9eaab005d57dfc84cd66
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit df3ea8c765648093343f9eaab005d57dfc84cd66
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 5 09:11:11 2022 +0200

    Set bigger swap in the functional and fullstack jobs

    As we are hitting some memory limits (again) in functional tests
    job, lets configure 8GB of swap in all functional and fullstack jobs.

    Before that patch swap was configured to 4GB only in the FIPS jobs. Now
    it will be set to 8GB in all functional and fullstack jobs.

    Closes-Bug: #1966394

    Conflicts:
        zuul.d/base.yaml

    Change-Id: I6b097d8409318a5dfe17e9673adb6c1431a59b0b
    (cherry picked from commit b8dcb0b7afe06f92090d5d2c366b9aae3a532ebc)
    (cherry picked from commit 8f76285bbb4c8343c784cdf593f0a36eb5b37863)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/866036
Committed: https://opendev.org/openstack/neutron/commit/6358c2fa56fcd84337cfe074dba96abb3b302261
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 6358c2fa56fcd84337cfe074dba96abb3b302261
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 5 09:11:11 2022 +0200

    Set bigger swap in the functional and fullstack jobs

    As we are hitting some memory limits (again) in functional tests
    job, lets configure 8GB of swap in all functional and fullstack jobs.

    Before that patch swap was configured to 4GB only in the FIPS jobs. Now
    it will be set to 8GB in all functional and fullstack jobs.

    Closes-Bug: #1966394

    Conflicts:
        zuul.d/base.yaml

    Change-Id: I6b097d8409318a5dfe17e9673adb6c1431a59b0b
    (cherry picked from commit b8dcb0b7afe06f92090d5d2c366b9aae3a532ebc)
    (cherry picked from commit 8f76285bbb4c8343c784cdf593f0a36eb5b37863)
    (cherry picked from commit df3ea8c765648093343f9eaab005d57dfc84cd66)

tags: added: in-stable-ussuri
tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/865995
Committed: https://opendev.org/openstack/neutron/commit/8f76285bbb4c8343c784cdf593f0a36eb5b37863
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 8f76285bbb4c8343c784cdf593f0a36eb5b37863
Author: Slawek Kaplonski <email address hidden>
Date: Tue Apr 5 09:11:11 2022 +0200

    Set bigger swap in the functional and fullstack jobs

    As we are hitting some memory limits (again) in functional tests
    job, lets configure 8GB of swap in all functional and fullstack jobs.

    Before that patch swap was configured to 4GB only in the FIPS jobs. Now
    it will be set to 8GB in all functional and fullstack jobs.

    Closes-Bug: #1966394

    Change-Id: I6b097d8409318a5dfe17e9673adb6c1431a59b0b
    (cherry picked from commit b8dcb0b7afe06f92090d5d2c366b9aae3a532ebc)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/866701

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/866702

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/866701
Committed: https://opendev.org/openstack/neutron/commit/4716449325916477d8ddf20fe6b25a88aef9e3dc
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 4716449325916477d8ddf20fe6b25a88aef9e3dc
Author: yatinkarel <email address hidden>
Date: Fri Apr 1 13:17:55 2022 +0530

    Enable dstat and memory_tracker in functional/fullstack jobs

    This Will help in troubleshooting failures related to high
    memory or cpu usage.

    Related-Bug: #1966394
    Change-Id: I74b0d53bfc54b71d3e8b2d46739a944e5f5a6b6f
    (cherry picked from commit 9bc447077bce642c25faed9306537b4f1bf57230)
    (cherry picked from commit 969a0db8b9acc468e994f437aede9952bb6acb78)
    Conflicts: roles/configure_functional_tests/tasks/main.yaml
            zuul.d/base.yaml

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/866702
Committed: https://opendev.org/openstack/neutron/commit/c23f17902350611fc60c2de5fb97d0b57190250b
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c23f17902350611fc60c2de5fb97d0b57190250b
Author: yatinkarel <email address hidden>
Date: Fri Apr 1 13:17:55 2022 +0530

    Enable dstat and memory_tracker in functional/fullstack jobs

    This Will help in troubleshooting failures related to high
    memory or cpu usage.

    Related-Bug: #1966394
    Change-Id: I74b0d53bfc54b71d3e8b2d46739a944e5f5a6b6f
    (cherry picked from commit 9bc447077bce642c25faed9306537b4f1bf57230)
    (cherry picked from commit 969a0db8b9acc468e994f437aede9952bb6acb78)
    Conflicts: roles/configure_functional_tests/tasks/main.yaml
            zuul.d/base.yaml
    (cherry picked from commit 4716449325916477d8ddf20fe6b25a88aef9e3dc)
    Conflicts: zuul.d/base.yaml

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.5.0

This issue was fixed in the openstack/neutron 19.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.3.0

This issue was fixed in the openstack/neutron 20.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron ussuri-eol

This issue was fixed in the openstack/neutron ussuri-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron victoria-eom

This issue was fixed in the openstack/neutron victoria-eom release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron wallaby-eom

This issue was fixed in the openstack/neutron wallaby-eom release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.