Illegal negative packet size -28

[Amol Kahat]

Description:

Patch[1] adds MTU size to ping6 command to catch network issues early.

[1] https://review.opendev.org/c/openstack/tripleo-ansible/+/814832

Actual Error:
2021-11-09 09:05:25 | 2021-11-09 09:05:25.571384 | fa163e5e-c3a9-25c4-d9eb-0000000015c9 | TIMING | tripleo_nodes_validation : Get nic for ping | overcloud-controller-0 | 0:01:48.827302 | 0.28s
2021-11-09 09:05:25 | 2021-11-09 09:05:25.586742 | fa163e5e-c3a9-25c4-d9eb-0000000015cb | TASK | Get MTU for lo table local proto kernel
2021-11-09 09:05:25 | 2021-11-09 09:05:25.840951 | fa163e5e-c3a9-25c4-d9eb-0000000015cb | CHANGED | Get MTU for lo table local proto kernel | overcloud-controller-0
2021-11-09 09:05:25 | 2021-11-09 09:05:25.843878 | fa163e5e-c3a9-25c4-d9eb-0000000015cb | TIMING | tripleo_nodes_validation : Get MTU for lo table local proto kernel | overcloud-controller-0 | 0:01:49.099788 | 0.26s
2021-11-09 09:05:25 | 2021-11-09 09:05:25.866105 | fa163e5e-c3a9-25c4-d9eb-0000000015cc | TASK | Check IP responsiveness
2021-11-09 09:05:26 | 2021-11-09 09:05:26.130634 | fa163e5e-c3a9-25c4-d9eb-0000000015cc | OK | Check IP responsiveness | overcloud-controller-0
2021-11-09 09:05:26 | 2021-11-09 09:05:26.133208 | fa163e5e-c3a9-25c4-d9eb-0000000015cc | TIMING | tripleo_nodes_validation : Check IP responsiveness | overcloud-controller-0 | 0:01:49.389116 | 0.27s
2021-11-09 09:05:26 | 2021-11-09 09:05:26.153808 | fa163e5e-c3a9-25c4-d9eb-0000000015cd | TASK | Validate packet with given MTU size can reach controller
2021-11-09 09:05:26 | 2021-11-09 09:05:26.508645 | fa163e5e-c3a9-25c4-d9eb-0000000015cd | FATAL | Validate packet with given MTU size can reach controller | overcloud-controller-0 | error={"changed": false, "cmd": ["ping6", "-w", "10", "-s", "-28", "-c", "1", "2001:db8:fd00:1000::289"], "delta": "0:00:00.004897", "end": "2021-11-09 09:05:26.486311", "msg": "non-zero return code", "rc": 2, "start": "2021-11-09 09:05:26.481414", "stderr": "ping: illegal negative packet size -28.", "stderr_lines": ["ping: illegal negative packet size -28."], "stdout": "", "stdout_lines": []}
2021-11-09 09:05:26 | 2021-11-09 09:05:26.511743 | fa163e5e-c3a9-25c4-d9eb-0000000015cd | TIMING | tripleo_nodes_validation : Validate packet with given MTU size can reach controller | overcloud-controller-0 | 0:01:49.767645 | 0.36s

Logs:
- https://logserver.rdoproject.org/16/36116/7/experimental/rdoinfo-tripleo-master-testing-centos-8-ovb-3ctlr_1comp-featureset035/e1c9837/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/9e6cc75/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[David Hill (david-hill-ubisoft)]

I don't think the issue is ipv6 here but most likely an issue in determining the MTU size for br-ex which is where it failed.

[David Hill (david-hill-ubisoft)]

https://review.opendev.org/c/openstack/tripleo-ansible/+/817220

[David Hill (david-hill-ubisoft)]

Zuul on the previous seems to fail here:

{'stdout': ' "format": "raw",', 'stderr': '+ type tripleo-mount-image\n+ tripleo-mount-image -a /home/zuul/overcloud-hardened-uefi-full.raw -m /tmp/tmp.zA12NVXL0T\n+ qemu-img info --output json /home/zuul/overcloud-hardened-uefi-full.raw\n+ grep \'"format": "raw"\'\n+ image_format=\'--format raw\'\n+ qemu-nbd --format raw --connect /dev/nbd0 /home/zuul/overcloud-hardened-uefi-full.raw\nqemu-nbd: Failed to set NBD socket\nqemu-nbd: Disconnect client, due to: Failed to send reply: Unable to write to socket: Broken pipe'}

... maybe that's why we didn't catch the ipv6 error in opendev ?

[Ronelle Landy (rlandy)]

Looks to be related to https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/816103

Testing https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/817260 (revert of above) with https://review.opendev.org/c/openstack/tripleo-ansible/+/817251 (revert related to this bug) in https://review.rdoproject.org/r/c/testproject/+/31165

[Ronelle Landy (rlandy)]

https://bugs.launchpad.net/tripleo/+bug/1950137 - error logged in this bug

[Ronelle Landy (rlandy)]

The revert is in - we are closing this