Third party rdo jobs are intermittently failing with ERROR: Failed to attach network adapter device to <UUID> (HTTP 500)

[Sandeep Yadav]

Description:

Third party rdo jobs are intermittently failing with ERROR: Failed to attach network adapter device to 8d06f19c-c430-453a-9139-5925344f7fe3 (HTTP 500) (Request-ID: req-912e40dd-1846-4aa0-bde5-b8f6e6178383

Logs:

https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-train/d811c4f/job-output.txt
~~~
2021-05-31 02:09:26.745315 | TASK [ovb-manage : Attach instance to public OVB network]
2021-05-31 02:09:48.523891 | primary | Failed to attach network adapter device to 8d06f19c-c430-453a-9139-5925344f7fe3 (HTTP 500) (Request-ID: req-912e40dd-1846-4aa0-bde5-b8f6e6178383)
2021-05-31 02:09:48.841589 | primary | ERROR
2021-05-31 02:09:48.842128 | primary | {
2021-05-31 02:09:48.842200 | primary | "delta": "0:00:21.085611",
2021-05-31 02:09:48.842243 | primary | "end": "2021-05-31 02:09:48.599335",
2021-05-31 02:09:48.842283 | primary | "msg": "non-zero return code",
2021-05-31 02:09:48.842321 | primary | "rc": 1,
2021-05-31 02:09:48.842358 | primary | "start": "2021-05-31 02:09:27.513724"
~~~

Issue is repeatedly happening at high frequency:

https://review.rdoproject.org/analytics/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15d,to:now))&_a=(columns:!(_source),filters:!(),index:logstash,interval:auto,query:(language:kuery,query:'build_status:%20%22FAILURE%22%20AND%20tags:%20%22job-output.txt%22%20AND%20message:%20%22Failed%20to%20attach%20network%20adapter%20device%22%20%20AND%20node_provider:%20%22vexxhost-nodepool-tripleo%22'),sort:!())

[Sandeep Yadav (sandeepyadav93)]

Infra team has opened a ticket with #vexx.

[wes hayutin (weshayutin)]

Still happening..
 Failed to attach network adapter device to 97ba9c66-cd26-40a4-80d1-a98c0b6b3d53 (HTTP 500) (Request-ID: req-340452bb-20bb-4895-b9a7-760d81201a47)

https://logserver.rdoproject.org/81/795181/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/94318c1/job-output.txt

[Marios Andreou (marios-b)]

619 hits
Jun 7, 2021 @ 14:38:45.444 - Jun 14, 2021 @ 14:38:45.444

https://review.rdoproject.org/analytics/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1w,to:now))&_a=(columns:!(_source),filters:!(),index:logstash,interval:auto,query:(language:kuery,query:'build_status:%20%22FAILURE%22%20AND%20tags:%20%22job-output.txt%22%20AND%20message:%20%22Failed%20to%20attach%20network%20adapter%20device%22%20%20AND%20node_provider:%20%22vexxhost-nodepool-tripleo%22'),sort:!())

[Marios Andreou (marios-b)]

(updating status for today CIX call)

we have an ongoing support case opened with vexx but it is still happening today so no updates yet

92 hits in 24 hours @ https://review.rdoproject.org/analytics/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now))&_a=(columns:!(_source),filters:!(),index:logstash,interval:auto,query:(language:kuery,query:'build_status:%20%22FAILURE%22%20AND%20tags:%20%22job-output.txt%22%20AND%20message:%20%22Failed%20to%20attach%20network%20adapter%20device%22%20%20AND%20node_provider:%20%22vexxhost-nodepool-tripleo%22'),sort:!())

(attached screenshot)

[wes hayutin (weshayutin)]

Thank you Thank you Thank you Thank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank youThank you Thank you Thank you

[Bhagyashri Shewale (bhagyashri-shewale)]

Hi All,

We are again facing this issue intermittently

2021-07-06 08:27:21.742472 | TASK [ovb-manage : Attach instance to public OVB network]
2021-07-06 08:27:49.310417 | primary | Failed to attach network adapter device to 7be923f0-812d-4aac-a230-150b1d9e91b4 (HTTP 500) (Request-ID: req-1f40f8e8-f30a-42b3-8499-6e3add589a99)
2021-07-06 08:27:49.904536 | primary | ERROR
2021-07-06 08:27:49.904911 | primary | {
2021-07-06 08:27:49.904962 | primary | "delta": "0:00:26.724349",
2021-07-06 08:27:49.904993 | primary | "end": "2021-07-06 08:27:49.412350",
2021-07-06 08:27:49.905022 | primary | "msg": "non-zero return code",
2021-07-06 08:27:49.905049 | primary | "rc": 1,
2021-07-06 08:27:49.905076 | primary | "start": "2021-07-06 08:27:22.688001"
2021-07-06 08:27:49.905102 | primary | }

[1]: https://logserver.rdoproject.org/73/34373/3/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/6e9fcaa/job-output.txt

[wes hayutin (weshayutin)]

Thanks all!!

no hits in 24 hours
https://review.rdoproject.org/analytics/goto/a7b1faa6f5e3f628b325cc601038079c

even the cleanup is running clean..
http://38.102.83.131/clean_stacks_vexx.log.txt

jobs are running well
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001

THANK YOU JPENA AND VEXXHOST! THANK YOU JPENA AND VEXXHOST!
THANK YOU JPENA AND VEXXHOST!
THANK YOU JPENA AND VEXXHOST!