Bulk Introspection finished with finished with error: Preprocessing hook validate_interfaces

[Sai Sindhur Malleni]

On a setup with 71 physical nodes, I tried openstack baremetal introspection bulk start. Some nodes succeeded, so nodes timed out, and in the case of some nodes(~5) the following was seen

 UUID d11d32b3-f220-4ac2-bb7d-0bbc761dfa2f finished with error: Preprocessing hook validate_interfaces: No suitable interfaces found
in {u'em4': {'ip': None, 'mac': u'b8:ca:3a:66:dd:d5'}, u'em1': {'ip': u'192.0.2.9', 'mac': u'b8:ca:3a:66:dd:d0'}, u'em3': {'ip': u'10.12.70.73', 'mac'
: u'b8:ca:3a:66:dd:d4'}, u'em2': {'ip': None, 'mac': u'b8:ca:3a:66:dd:d2'}}

Newton was the release being used.

[Seamanhanshan (yaojun)]

I'm working on baremetal introspection recently, i've found some problems during simultaneous introspection. I'll investigate this issue as well.

[Seamanhanshan (yaojun)]

pls paste your inspector.conf file, thx!

[Sai Sindhur Malleni (smalleni)]

https://gist.github.com/smalleni/d43304be84aa67cf3a17a1222ee0d74b

[Seamanhanshan (yaojun)]

i've gone through your config, within add_ports = pxe.

'em4': {'ip': None, 'mac': u'b8:ca:3a:66:dd:d5'},
'em1': {'ip': u'192.0.2.9', 'mac': u'b8:ca:3a:66:dd:d0'},
'em3': {'ip': u'10.12.70.73', 'mac': u'b8:ca:3a:66:dd:d4'},
'em2': {'ip': None, 'mac': u'b8:ca:3a:66:dd:d2'}}

Only one possibility is your PXE_MAC is not in these fours MACs, would you pls paste the full collected data? I want to know the value of boot_interface, thanks!

[Sai Sindhur Malleni (smalleni)]

https://gist.github.com/smalleni/d44367e8440eb5e5e75320d236ca3eca

[Sai Sindhur Malleni (smalleni)]

Now trying to introspect a 120 nodes with bulk introspection, see this frequently

Introspection for UUID 611c4b67-12d9-410a-bc27-97f81af24011 finished with error: Preprocessing hook validate_interfaces: No suitable interfaces found in {u'em4': {'ip': None, 'mac': u'b8:ca:3a:66:d7:1d'}, u'em1'
: {'ip': u'192.0.2.33', 'mac': u'b8:ca:3a:66:d7:18'}, u'em3': {'ip': u'10.12.70.90', 'mac': u'b8:ca:3a:66:d7:1c'}, u'em2': {'ip': None, 'mac': u'b8:ca:3a:66:d7:1a'}}
Introspection for UUID 34c5974d-b399-4901-894a-79d7ff4ffcd0 finished with error: Introspection timeout
Introspection for UUID 8903d550-391f-4345-a2d7-dad866f14cd7 finished successfully.
Introspection for UUID b4156c64-5341-45a5-9dc5-3672be579656 finished successfully.
Introspection for UUID c576a8bd-3213-4a67-8d28-e84517b13f43 finished successfully.
Introspection for UUID bdc1e7dd-cba3-4987-a87d-d81fdaeb751e finished successfully.
Introspection for UUID 00a205ef-4c18-41c1-8b29-c258210e50f9 finished successfully.
Introspection for UUID 69f4c50f-9833-4f6b-ad5c-b64057e037c9 finished successfully.
Introspection for UUID ccc8caa8-2c59-4a45-8dd7-c53ba11e5b2c finished successfully.
Introspection for UUID 85465cbe-d335-4e25-b569-f9f2674a03ea finished with error: Introspection timeout
Introspection for UUID 2a7decda-fea4-4c12-8d2c-d85a7896e78c finished with error: Introspection timeout
Introspection for UUID ad0e1ec9-552f-4a66-8060-bb857a8a4300 finished successfully.
Introspection for UUID f0c4c74c-0b64-4e21-b23d-5e28893450dd finished successfully.
Introspection for UUID 5c7f2e30-3970-4f5f-9ffe-6c31ae4411c2 finished successfully.
Introspection completed with errors:
d3ea7217-67cf-4e63-8db3-0e714064966c: Introspection timeout
3f87fe68-b9ae-4c38-bdc1-a11ab9e301ca: Introspection timeout
1a17c740-53a4-466c-b759-f308eb594e89: Introspection timeout
c3df42ac-2632-4c48-8683-f727b9307bd2: Introspection timeout
1f9e0a6d-4f15-4dc7-af76-cc4669dc4ab6: Introspection timeout
74dde734-377c-48c4-8fa6-5139e6538589: Introspection timeout
3060b66c-890e-465e-96a8-7d42e447e94d: Preprocessing hook validate_interfaces: No suitable interfaces found in {u'em4': {'ip': None, 'mac': u'b8:ca:3a:66:f5:55'}, u'em1': {'ip': u'192.0.2.40', 'mac': u'b8:ca:3a:6
6:f5:50'}, u'em3': {'ip': u'10.12.70.64', 'mac': u'b8:ca:3a:66:f5:54'}, u'em2': {'ip': None, 'mac': u'b8:ca:3a:66:f5:52'}}
d055e5f6-46fe-49e1-a753-5abe501f4db4: Introspection timeout

[Seamanhanshan (yaojun)]

thanks for providing so much info!

I've tested introspection data mentioned in #5, this data is valid, shall not cause the error of "No suitable interfaces found in", so this issue is strange!

[Seamanhanshan (yaojun)]

hello Sai, would you please add more debug code in plugins/standard.py:_validate_interfaces()

         if not result:
+ LOG.error("pxe_mac %s" % pxe_mac)
             raise utils.Error(_('No suitable interfaces found in %s') %
                               interfaces, data=data)

If some nodes fails, we can get pxe_mac and interfaces values, these two variables are very important for debugging.

[Dmitry Tantsur (divius)]

Hi Sai! Please provide the ramdisk logs from the failed nodes?