Ubuntu
linux package

test_maps in ubuntu_bpf failed with "Failed sockmap unexpected timeout" on D ARM64

Bug #1805806 reported by Po-Hsu Lin on 2018-11-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	ubuntu-kernel-tests	Triaged	Undecided	Unassigned
	linux (Ubuntu)	Invalid	Undecided	Unassigned

Bug Description

This issue can be found on 2 different ARM64 node, TunderX Cavium node "starmie" and Moonshot "ms10-34-mcdivittB0-kernel"

Running test_maps bpf test..
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  ....
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  ....
  Fork 100 tasks to 'test_hashmap_sizes'
  Fork 100 tasks to 'test_hashmap_walk'
  Fork 100 tasks to 'test_arraymap'
  Fork 100 tasks to 'test_arraymap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 100 tasks to 'test_hashmap_sizes'
  Fork 100 tasks to 'test_hashmap_walk'
  Fork 100 tasks to 'test_arraymap'
  Fork 100 tasks to 'test_arraymap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 100 tasks to 'test_hashmap_sizes'
  Fork 100 tasks to 'test_hashmap_walk'
  Fork 100 tasks to 'test_arraymap'
  Fork 100 tasks to 'test_arraymap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 100 tasks to 'test_hashmap_sizes'
  Fork 100 tasks to 'test_hashmap_walk'
  Fork 100 tasks to 'test_arraymap'
  Fork 100 tasks to 'test_arraymap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 100 tasks to 'test_hashmap_sizes'
  Fork 100 tasks to 'test_hashmap_walk'
  Fork 100 tasks to 'test_arraymap'
  Fork 100 tasks to 'test_arraymap_percpu'
  Fork 1024 tasks to 'test_update_delete'
  Fork 1024 tasks to 'test_update_delete'
  Fork 100 tasks to 'test_hashmap'
  Fork 100 tasks to 'test_hashmap_percpu'
  Fork 100 tasks to 'test_hashmap_sizes'
  Fork 100 tasks to 'test_hashmap_walk'
  Fork 100 tasks to 'test_arraymap'
  Fork 100 tasks to 'test_arraymap_percpu'
  Failed sockmap unexpected timeout

ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: linux-image-4.18.0-11-generic 4.18.0-11.12
ProcVersionSignature: User Name 4.18.0-11.12-generic 4.18.12
Uname: Linux 4.18.0-11-generic aarch64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Nov 28 12:22 seq
crw-rw---- 1 root audio 116, 33 Nov 28 12:22 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu13.1
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDmesg:
[67841.439762] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[67841.447419] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
Date: Thu Nov 29 10:04:28 2018
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Cavium ThunderX CRB
PciMultimedia:

ProcFB:
0 EFI VGA
1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.18.0-11-generic root=UUID=d4e5f461-24ef-47d8-a67f-42c82ff1efe8 ro
RelatedPackageVersions:
linux-restricted-modules-4.18.0-11-generic N/A
linux-backports-modules-4.18.0-11-generic N/A
linux-firmware 1.175.1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/12/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 5.11
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: To be filled by O.E.M.
dmi.board.vendor: To be filled by O.E.M.
dmi.board.version: To be filled by O.E.M.
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 0
dmi.chassis.vendor: Cavium
dmi.chassis.version: To be filled by O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr5.11:bd12/12/2012:svnCavium:pnThunderXCRB:pvrTobefilledbyO.E.M.:rvnTobefilledbyO.E.M.:rnTobefilledbyO.E.M.:rvrTobefilledbyO.E.M.:cvnCavium:ct0:cvrTobefilledbyO.E.M.:
dmi.product.family: Default string
dmi.product.name: ThunderX CRB
dmi.product.sku: Default string
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Cavium

See original description

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2018-11-29:

CRDA.txt Edit (468 bytes, text/plain; charset="utf-8")
Dependencies.txt Edit (2.4 KiB, text/plain; charset="utf-8")
Lspci.txt Edit (44.8 KiB, text/plain; charset="utf-8")
Lsusb.txt Edit (417 bytes, text/plain; charset="utf-8")
ProcCpuinfo.txt Edit (8.8 KiB, text/plain; charset="utf-8")
ProcCpuinfoMinimal.txt Edit (187 bytes, text/plain; charset="utf-8")
ProcEnviron.txt Edit (270 bytes, text/plain; charset="utf-8")
ProcInterrupts.txt Edit (42.1 KiB, text/plain; charset="utf-8")
ProcModules.txt Edit (3.7 KiB, text/plain; charset="utf-8")
UdevDb.txt Edit (138.4 KiB, text/plain; charset="utf-8")
WifiSyslog.txt Edit (105.9 KiB, text/plain; charset="utf-8")

Revision history for this message

Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote on 2018-11-29: Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1805806

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status:	New → Incomplete

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2018-11-30: Re: test_maps in ubuntu_bpf failed with "Failed sockmap unexpected timeout" on Cosmic ARM64

This issue could be found on B-4.18 as well, but with 2 ARM64 nodes, it's only failing with the TunderX Cavium node "starmie", passed on the Moonshot node "ms10-34-mcdivittB0-kernel" instead.

description:

updated

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2019-05-09:

For Disco,
failed on ThunderX (1 out of 2 runs)
failed on Moonshot

Changed in linux (Ubuntu Cosmic):
status:	New → Confirmed
Changed in linux (Ubuntu Disco):
status:	New → Confirmed
tags:	added: disco

Po-Hsu Lin (cypressyew) on 2019-07-22

summary:

test_maps in ubuntu_bpf failed with "Failed sockmap unexpected timeout"
- on Cosmic ARM64
+ on C/D ARM64

Brad Figg (brad-figg) on 2019-07-24

tags:

added: ubuntu-certified

Po-Hsu Lin (cypressyew) on 2019-08-30

Changed in linux (Ubuntu Cosmic):
status:	Confirmed → Won't Fix
summary:	test_maps in ubuntu_bpf failed with "Failed sockmap unexpected timeout" - on C/D ARM64 + on D ARM64

Po-Hsu Lin (cypressyew) on 2019-09-16

tags:

added: ubuntu-bpf

Sean Feole (sfeole) on 2019-12-16

tags:	added: sru-20191202
Changed in ubuntu-kernel-tests:
status:	New → Triaged

Sean Feole (sfeole) on 2020-01-24

tags:

added: sru-20200106

Steve Langasek (vorlon) on 2020-07-02

Changed in linux (Ubuntu Disco):
status:	Confirmed → Won't Fix

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2020-08-26:

Passed on Focal ARM64
5.4.0-44.48-generic

Changed in ubuntu-kernel-tests:
status:	Triaged → Fix Released
status:	Fix Released → Triaged

Revision history for this message

Po-Hsu Lin (cypressyew) wrote on 2020-08-26:

Need to check if we still have this issue on 5.3 / 5.0 Variants.

Revision history for this message

Kelsey Steele (kelsey-steele) wrote on 2020-10-01:

This error is populating on Focal 'aws : 5.4.0-1026.26 : amd64' on c5.large and ' aws : 5.4.0-1026.26 : arm64' on a1.2xlarge

Kelsey Steele (kelsey-steele) on 2020-10-02

tags:

added: sru-20200921

Kelsey Steele (kelsey-steele) on 2020-10-07

tags:	added: sru-2020010 removed: sru-20200106
tags:	added: aws focal

Revision history for this message

Sean Feole (sfeole) wrote on 2020-11-23:

affects aws cloud 5.X kernels.

Changed in ubuntu-kernel-tests:
status:	Triaged → Invalid
Changed in linux (Ubuntu):
status:	Incomplete → Invalid
Changed in ubuntu-kernel-tests:
status:	Invalid → Triaged
no longer affects:	linux (Ubuntu Disco)
no longer affects:	linux (Ubuntu Cosmic)
tags:	added: sru-20201109

Revision history for this message

Kelsey Steele (kelsey-steele) wrote on 2020-12-01:

#10

Found on Groovy 5.8.0-31.33 lowlatency & generic. Only failing on one node (lagalla) and passes on the other (rizzo)

12/01 05:19:36 DEBUG| utils:0116| Running './test_maps'
12/01 05:19:36 DEBUG| utils:0153| [stdout] Fork 1024 tasks to 'test_update_delete'
12/01 05:19:37 DEBUG| utils:0153| [stdout] Fork 1024 tasks to 'test_update_delete'
12/01 05:19:37 DEBUG| utils:0153| [stdout] Fork 100 tasks to 'test_hashmap'
12/01 05:19:37 DEBUG| utils:0153| [stdout] Fork 100 tasks to 'test_hashmap_percpu'
12/01 05:19:37 DEBUG| utils:0153| [stdout] Fork 100 tasks to 'test_hashmap_sizes'
12/01 05:19:37 DEBUG| utils:0153| [stdout] Fork 100 tasks to 'test_hashmap_walk'
12/01 05:19:37 DEBUG| utils:0153| [stdout] Fork 100 tasks to 'test_arraymap'
12/01 05:19:37 DEBUG| utils:0153| [stdout] Fork 100 tasks to 'test_arraymap_percpu'
12/01 05:19:38 DEBUG| utils:0153| [stdout] Failed sockmap unexpected timeout
12/01 05:19:38 ERROR| test:0414| Exception escaping from test:
Traceback (most recent call last):
  File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function
    return func(*args, **dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute
    postprocess_profiled_run, args, dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once
    self.run_once(*args, **dargs)
  File "/home/ubuntu/autotest/client/tests/ubuntu_bpf/ubuntu_bpf.py", line 99, in run_once
    self.results = utils.system_output(cmd, retain_output=True)
  File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output
    verbose=verbose, args=args).stdout
  File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run
    "Command returned non-zero exit status")
CmdError: Command <./test_maps> failed, rc=1, Command returned non-zero exit status
* Command:
    ./test_maps
Exit status: 1
Duration: 2.08336114883

Found on Groovy 5.8.0-31.33 lowlatency & generic. Only failing on one node (lagalla) and passes on the other (rizzo)

12/01 05:19:36 DEBUG|     utils:0116| Running './test_maps'
12/01 05:19:36 DEBUG|     utils:0153| [stdout] Fork 1024 tasks to 'test_update_delete'
12/01 05:19:37 DEBUG|     utils:0153| [stdout] Fork 1024 tasks to 'test_update_delete'
12/01 05:19:37 DEBUG|     utils:0153| [stdout] Fork 100 tasks to 'test_hashmap'
12/01 05:19:37 DEBUG|     utils:0153| [stdout] Fork 100 tasks to 'test_hashmap_percpu'
12/01 05:19:37 DEBUG|     utils:0153| [stdout] Fork 100 tasks to 'test_hashmap_sizes'
12/01 05:19:37 DEBUG|     utils:0153| [stdout] Fork 100 tasks to 'test_hashmap_walk'
12/01 05:19:37 DEBUG|     utils:0153| [stdout] Fork 100 tasks to 'test_arraymap'
12/01 05:19:37 DEBUG|     utils:0153| [stdout] Fork 100 tasks to 'test_arraymap_percpu'
12/01 05:19:38 DEBUG|     utils:0153| [stdout] Failed sockmap unexpected timeout
12/01 05:19:38 ERROR|      test:0414| Exception escaping from test:
Traceback (most recent call last):
  File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function
    return func(*args, **dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute
    postprocess_profiled_run, args, dargs)
  File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once
    self.run_once(*args, **dargs)
  File "/home/ubuntu/autotest/client/tests/ubuntu_bpf/ubuntu_bpf.py", line 99, in run_once
    self.results = utils.system_output(cmd, retain_output=True)
  File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output
    verbose=verbose, args=args).stdout
  File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run
    "Command returned non-zero exit status")
CmdError: Command <./test_maps> failed, rc=1, Command returned non-zero exit status
* Command: 
    ./test_maps
Exit status: 1
Duration: 2.08336114883

tags:

added: 5.8 groovy

Revision history for this message

Francis Ginther (fginther) wrote on 2021-02-19:

#11

Seen with linux-aws 5.4.0-1038.40~18.04.1.

tags:

added: sru-20210125

Revision history for this message

Ian May (ian-may) wrote on 2021-03-09:

#12

Found on focal/aws 5.4.0-1039.41

tags:

added: sru-20210222

Revision history for this message

Ian May (ian-may) wrote on 2021-04-12:

#13

focal/aws 5.4.0-1042.44

tags:

added: sru-20210315

Revision history for this message

Krzysztof Kozlowski (krzk) wrote on 2021-04-27:

#14

groovy/aws 5.8.0-1031.33

tags:	added: sru-20210412
tags:	added: amd64

Revision history for this message

Ian May (ian-may) wrote on 2021-05-24:

#15

groovy/linux-aws: 5.8.0-1034.36

tags:

added: sru-20210510

Revision history for this message

Ian May (ian-may) wrote on 2021-05-25:

#16

focal/linux-aws: 5.4.0-1049.51

Revision history for this message

Ian May (ian-may) wrote on 2021-05-27:

#17

Found on bionic/linux-aws-fips: 4.15.0-2045.47

Revision history for this message

Ian May (ian-may) wrote on 2021-05-27:

#18

Found on bionic/linux-aws-5.4: 5.4.0-1049.51~18.04.1

Revision history for this message

Ian May (ian-may) wrote on 2021-05-27:

#19

Found on bionic/linux-azure-4.15: 4.15.0-1115.128

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntulinux package

test_maps in ubuntu_bpf failed with "Failed sockmap unexpected timeout" on D ARM64

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
linux package