Import hangs on wrong IPMI credentials

Bug #1667776 reported by Joe Talerico
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Dmitry Tantsur

Bug Description

The issue was ironic by default uses the ADMINISTRATOR, this env we had a OPERATOR account. However, ironic baremetal import shouldn't just hang forever. There should be a timeout, or if we fail to get power status we break out with a error to the user.

[stack@undercloud ~]$ time openstack baremetal import --json instackenv.json [243/1996]
Started Mistral Workflow. Execution ID: 08be22c3-810d-47b9-81e0-b785a479f0fb
^CTraceback (most recent call last):
  File "/bin/openstack", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 209, in main
    return OpenStackShell().run(argv)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 267, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 387, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 190, in take_ac
tion
    instance_boot_option=parsed_args.instance_boot_option
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 38, in r
egister_or_update
    payload = ws.wait_for_message(execution.id)
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 137, in wait_for_mess
age
    body = self.recv()['body']
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 125, in recv
    return json.loads(self._ws.recv())
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 348, in recv
    opcode, data = self.recv_data()
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 365, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 378, in recv_data_frame
    frame = self.recv_frame()
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 410, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/usr/lib/python2.7/site-packages/websocket/_abnf.py", line 300, in recv_frame
    self.recv_header()
  File "/usr/lib/python2.7/site-packages/websocket/_abnf.py", line 249, in recv_header
    header = self.recv_strict(2)
  File "/usr/lib/python2.7/site-packages/websocket/_abnf.py", line 334, in recv_strict
    bytes = self.recv(min(16384, shortage))
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 476, in _recv
    return recv(self.sock, bufsize)
  File "/usr/lib/python2.7/site-packages/websocket/_socket.py", line 77, in recv
    bytes = sock.recv(bufsize)
KeyboardInterrupt

real 90m26.806s
user 0m0.466s
sys 0m0.090s

Revision history for this message
milan k (vetrisko) wrote :

Could you please elaborate on the description?
Is this really an Ironic issue? Shouldn't it be TripleO?

Joe Talerico (jtaleric)
affects: ironic → tripleo
Changed in tripleo:
status: New → Triaged
importance: Undecided → Medium
milestone: none → pike-1
Changed in tripleo:
milestone: pike-1 → pike-2
Changed in tripleo:
milestone: pike-2 → pike-3
Changed in tripleo:
milestone: pike-3 → pike-rc1
Changed in tripleo:
milestone: pike-rc1 → queens-1
Changed in tripleo:
milestone: queens-1 → queens-2
Changed in tripleo:
milestone: queens-2 → queens-3
Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Dmitry Tantsur (divius)
summary: - ironic inspector fails to move machines from enroll causing the command
- to hang
+ Import hangs on wrong IPMI credentials
Changed in tripleo:
assignee: nobody → Dmitry Tantsur (divius)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/552554

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/552554
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d97cd4a00556457914b783282c544b5233af0260
Submitter: Zuul
Branch: master

commit d97cd4a00556457914b783282c544b5233af0260
Author: Dmitry Tantsur <email address hidden>
Date: Wed Mar 14 14:20:15 2018 +0100

    Fix error handling in set_provision_state/set_power_state workflows

    Currently these workflows succeed in any case, since we don't have
    any condition to fail. This change makes them fail if the resulting
    state does not match the expected one.

    It also handles the case when a node goes into one of the failure
    states, so that we don't wait until timeout. Proper error message
    is returned to avoid confusing operators.

    Finally, it reduces the traffic between mistral and ironic by only
    requesting the required fields.

    Partial-Bug: #1755754
    Closes-Bug: #1667776
    Change-Id: Ice19306d4c4a2080b0337bc02a6ccee4a81411b5

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/559314

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 9.0.0

This issue was fixed in the openstack/tripleo-common 9.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/queens)

Reviewed: https://review.openstack.org/559314
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=27b2ef1d72365c105be7bc03eb1f67986688dfe7
Submitter: Zuul
Branch: stable/queens

commit 27b2ef1d72365c105be7bc03eb1f67986688dfe7
Author: Dmitry Tantsur <email address hidden>
Date: Wed Mar 14 14:20:15 2018 +0100

    Fix error handling in set_provision_state/set_power_state workflows

    Currently these workflows succeed in any case, since we don't have
    any condition to fail. This change makes them fail if the resulting
    state does not match the expected one.

    It also handles the case when a node goes into one of the failure
    states, so that we don't wait until timeout. Proper error message
    is returned to avoid confusing operators.

    Finally, it reduces the traffic between mistral and ironic by only
    requesting the required fields.

    Conflicts:
     workbooks/baremetal.yaml

    Partial-Bug: #1755754
    Closes-Bug: #1667776
    Change-Id: Ice19306d4c4a2080b0337bc02a6ccee4a81411b5
    (cherry picked from commit d97cd4a00556457914b783282c544b5233af0260)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 8.6.2

This issue was fixed in the openstack/tripleo-common 8.6.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.