devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE

[Lee Yarwood]

https://e144061c9960115e0a97-1157fa9303f82335ed0bb07a311cffdc.ssl.cf1.rackcdn.com/788429/1/gate/tempest-full-py3/c39c269/job-output.txt

2021-04-28 12:22:36.944718 | controller | + functions-common:service_check:1600 : sudo systemctl is-enabled devstack@memory_tracker.service
2021-04-28 12:22:36.957103 | controller | enabled
2021-04-28 12:22:36.961376 | controller | + functions-common:service_check:1604 : sudo systemctl status devstack@memory_tracker.service --no-pager
2021-04-28 12:22:36.979435 | controller | ● devstack@memory_tracker.service - Devstack devstack@memory_tracker.service
2021-04-28 12:22:36.979509 | controller | Loaded: loaded (/etc/systemd/system/devstack@memory_tracker.service; enabled; vendor preset: enabled)
2021-04-28 12:22:36.979582 | controller | Active: failed (Result: exit-code) since Wed 2021-04-28 12:20:42 UTC; 1min 54s ago
2021-04-28 12:22:36.979599 | controller | Main PID: 63467 (code=exited, status=1/FAILURE)
2021-04-28 12:22:36.979611 | controller |
2021-04-28 12:22:36.979622 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 memory_tracker.sh[88642]: Traceback (most recent call last):
2021-04-28 12:22:36.979633 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 memory_tracker.sh[88642]: File "/opt/stack/devstack/tools/mlock_report.py", line 51, in <module>
2021-04-28 12:22:36.979658 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 memory_tracker.sh[88642]: main()
2021-04-28 12:22:36.979671 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 memory_tracker.sh[88642]: File "/opt/stack/devstack/tools/mlock_report.py", line 14, in main
2021-04-28 12:22:36.979683 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 memory_tracker.sh[88642]: print(_get_report())
2021-04-28 12:22:36.979698 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 memory_tracker.sh[88642]: File "/opt/stack/devstack/tools/mlock_report.py", line 30, in _get_report
2021-04-28 12:22:36.979713 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 memory_tracker.sh[88642]: for line in s:
2021-04-28 12:22:36.979750 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 memory_tracker.sh[88642]: ProcessLookupError: [Errno 3] No such process
2021-04-28 12:22:36.979802 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 systemd[1]: devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE
2021-04-28 12:22:36.979823 | controller | Apr 28 12:20:42 ubuntu-focal-rax-iad-0024388876 systemd[1]: devstack@memory_tracker.service: Failed with result 'exit-code'.

$ cat queries/memory_tracker.yaml
query: >
 message:"devstack@memory_tracker.service: Main process exited, code=exited, status=1/FAILURE" AND
 tags:"console"

$ elastic-recheck-query queries/memory_tracker.yaml
total hits: 13
build_branch
  76% master
  23% stable/victoria
build_change
  23% 788155
  15% 778885
  15% 787681
  15% 787956
  15% 788410
build_hostids
  23% b09be02c270969d708092b6d48260cc4042f25885e2925f45343ddae dbbacc63dd2aa1da3255c1e53a36fc3608704a67e737e21f9e8c5eed
  15% 12d9d23aa698345440ead27b40c9e95f962f698e1a603ec71a840a11
  15% 3e4b14e335580b9b5ac078e3f986fca4d090d98b894a86bd35f02580
  15% 89d4678620f4caae15dbdd5144c1c604f5379115d135e3d418aa678b
  15% d82f3544d10ec0199da2c9b972e1fd1b47e87e08dd75cd64c14c12f6
build_name
  23% neutron-grenade-dvr-multinode
  15% designate-grenade-pdns4
  15% neutron-rally-task
  15% neutron-tempest-ipv6-only
  15% nova-multi-cell
build_node
  76% ubuntu-focal
  23% ubuntu-bionic
build_queue
  69% check
  30% gate
build_status
  100% FAILURE
build_zuul_url
  100% N/A
filename
  53% job-output.txt
  30% controller/logs/devstacklog.txt
  15% controller/logs/old/devstacklog.txt
log_url
  15% https://a81f6680b0de419161e8-69746c322fe5e3cf8e2976f654b96d92.ssl.cf1.rackcdn.com/788155/1/check/neutron-grenade-dvr-multinode/9e68cfe/job-output.txt
  7% https://4be066bc934fad54ed0d-3addfaddf0077dd10f155c31a77f70c6.ssl.cf2.rackcdn.com/778885/10/gate/nova-multi-cell/457b669/controller/logs/devstacklog.txt
  7% https://4be066bc934fad54ed0d-3addfaddf0077dd10f155c31a77f70c6.ssl.cf2.rackcdn.com/778885/10/gate/nova-multi-cell/457b669/job-output.txt
  7% https://4fa76beded93c6a1a5ec-435c106d0f8e6ffd36591d2c40c1fe49.ssl.cf2.rackcdn.com/787956/1/check/neutron-tempest-ipv6-only/867c5a9/controller/logs/devstacklog.txt
  7% https://4fa76beded93c6a1a5ec-435c106d0f8e6ffd36591d2c40c1fe49.ssl.cf2.rackcdn.com/787956/1/check/neutron-tempest-ipv6-only/867c5a9/job-output.txt
node_provider
  53% rax-dfw
  15% inap-mtl01
  15% rax-iad
  15% rax-ord
port
  15% 41412
  15% 43948
  15% 49138
  7% 37426
  7% 41376
project
  53% openstack/neutron
  15% openstack/devstack
  15% openstack/nova
  15% openstack/python-designateclient
tags
  53% job-output.txt console console.html
  46% devstacklog.txt console console.html
voting
  100% 1
zuul_attempts
  100% 1
zuul_executor
  38% ze09.opendev.org
  15% ze01.opendev.org
  15% ze02.opendev.org
  15% ze04.opendev.org
  15% ze11.opendev.org

[Dr. Jens Harbott (j-harbott)]

Seems there has been a fix proposed for this some time ago already: https://review.opendev.org/c/openstack/devstack/+/620198

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/devstack/+/620198
Committed: https://opendev.org/openstack/devstack/commit/b02a43291cc5e2d5677ecbb80c0fd608d67a1374
Submitter: "Zuul (22348)"
Branch: master

commit b02a43291cc5e2d5677ecbb80c0fd608d67a1374
Author: Ian Wienand <email address hidden>
Date: Tue Nov 27 12:59:04 2018 +1100

    Handle disappearing pids in mlock_report.py

    If a pid disappears on us while we're reading, we should just continue
    on.

    EnvironmentError is just an alias for OSError since Python 3.3, so use
    the latter name. [0]

    [0] https://docs.python.org/3/library/exceptions.html#OSError

    Change-Id: I3a25cca328e1469f72c84a118a9691c1c0258bc4
    Closes-Bug: #1926434

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/devstack/+/825029

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Steve Baker <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/devstack/+/825029
Reason: no reviews