On Wed, Aug 10, 2016 at 8:39 PM, Andres Rodriguez
<email address hidden> wrote:
> Hi Jeff,
>
> 1. What happens if you click on "Check Power" on the node details page.
> Does the power update correctly?

No... For example, after commissioning, MAAS says the node is in Ready
state, and says Power is ON, while UCSM says the power is OFF.
Clicking the Check Power link doesn't change this.  The only way I
could find to change it was to select "Power Off" from the Action
menu.  THEN MAAS correctly showed the power as being Off.

> 2. While you do "Check Power" can you attach /var/log/maas/*.log?

I can... I've attached the logs.  However, NOTHING appears in the logs
when I click "Check Power".  I just retried several times, and there
is no log activity when I do that.

>
> That said, looking at your statement
>
> "Note that UCSM shows the node as powered OFF, MAAS still shows the
> power ON. Once again, I have to manually select Power Off from the
> Action menu and then maas changes state to Power Off."
>
> MAAS does not check power every second. It checks it every 3 minutes or
> so. So the machine may finish commissioning, and the power may be showed
> as ON in MAAS, but the machine may be actually off. The reason for this
> is that after commissioning, a power query is requested, but what may be
> happening is that the power query happens while the node is still on
> (and powering off), and it doesn't update in the UI until it actually
> checks the power again.
>
> Based on the logs, it seems that the node has transitioned correctly
> between states and you are experiencing the above. I'll mark this bug as
> invalid for the time being. Please, re-open the bug is the following
> happens:
>
> 1. Commission the node.
> 2. The node is in Ready state, show's power is ON (but hardware itself is off).
> 3. You do one of the two:
>   3.1 You click on "Check Power" and the node remains as ON (even though the hardware is off) or.
>   3.2 After 5 minutes, the power hasn't changed automatically.

So I recommissioned the node.  After it was done, once again, state
became "Ready" but MAAS still said the power was on.  UCSM shows the
power to the blade as Off.
I clicked the "Check Power" button 5 times, and it never changed the
status in MAAS.
I checked the logs, and there was NO log activity at all to correspond
with my "Check Power" clicks.
I left the node to sit well over 5 minutes after it was marked Ready,
and the power state in MAAS still says ON.
I clicked the "Check Power" button 5 more times, and it still didn't
change the status in MAAS.
And again nothing in the logs from my clicks.

According to MAAS.log,  at 22:36, the node was changed to READY, and
the power was marked as ON until 22:56 when I manually used the "Power
Off" action in the Action menu in order to set the power status
correctly so I could deploy the node and kick off testing.

Aug 10 22:36:33 lucuma maas.node: [INFO] oil-pullman-02: Storage
layout was set to flat.
Aug 10 22:36:33 lucuma maas.node: [INFO] oil-pullman-02: Status
transition from COMMISSIONING to READY
Aug 10 22:40:31 lucuma maas.power: [ERROR] Power state could not be
queried: Failed to login to virsh console.
Aug 10 22:40:31 lucuma maas.power: [ERROR] binacle: Failed to refresh
power state: Failed to login to virsh console.
Aug 10 22:43:58 lucuma maas.import-images: [INFO] Started importing boot images.
Aug 10 22:43:58 lucuma maas.import-images: [INFO] Downloading image
descriptions from
http://10.1.10.1/MAAS/images-stream/streams/v1/index.json
Aug 10 22:43:59 lucuma maas.import-images: [INFO] Finished importing
boot images, the region does not have any new images.
Aug 10 22:44:00 lucuma maas.import-images: [INFO] Downloading image
descriptions from http://maas.ubuntu.com/images/ephemeral-v2/releases/
Aug 10 22:44:02 lucuma maas.bootsources: [INFO] Updated boot sources cache.
Aug 10 22:44:02 lucuma maas.bootresources: [INFO] Started importing of
boot images from 1 source(s).
Aug 10 22:44:02 lucuma maas.import-images: [INFO] Downloading image
descriptions from http://maas.ubuntu.com/images/ephemeral-v2/releases/
Aug 10 22:44:04 lucuma maas.bootresources: [INFO] Importing images
from source: http://maas.ubuntu.com/images/ephemeral-v2/releases/
Aug 10 22:44:07 lucuma maas.bootresources: [INFO] Finished importing
of boot images from 1 source(s).
Aug 10 22:44:07 lucuma maas.import-images: [INFO] Started importing boot images.
Aug 10 22:44:07 lucuma maas.import-images: [INFO] Downloading image
descriptions from
http://10.1.10.1/MAAS/images-stream/streams/v1/index.json
Aug 10 22:44:08 lucuma maas.import-images: [INFO] Finished importing
boot images, the region does not have any new images.
Aug 10 22:45:31 lucuma maas.power: [ERROR] Power state could not be
queried: Failed to login to virsh console.
Aug 10 22:45:31 lucuma maas.power: [ERROR] binacle: Failed to refresh
power state: Failed to login to virsh console.
Aug 10 22:50:31 lucuma maas.power: [ERROR] Power state could not be
queried: Failed to login to virsh console.
Aug 10 22:50:31 lucuma maas.power: [ERROR] binacle: Failed to refresh
power state: Failed to login to virsh console.
Aug 10 22:55:46 lucuma maas.power: [ERROR] Power state could not be
queried: Failed to login to virsh console.
Aug 10 22:55:46 lucuma maas.power: [ERROR] binacle: Failed to refresh
power state: Failed to login to virsh console.
Aug 10 22:56:44 lucuma maas.power: [INFO] Changing power state (off)
of node: oil-pullman-02 (4y3h8r)
Aug 10 22:56:46 lucuma maas.power: [INFO] Changed power state (off) of
node: oil-pullman-02 (4y3h8r)

I've attached the logs as well as more screen shots.

>
>
> ** Changed in: maas
>        Status: Incomplete => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1611999
>
> Title:
>   MAAS 2 does not properly probe power state on Cisco UCS systems
>
> Status in MAAS:
>   Invalid
>
> Bug description:
>   I'm trying to set up a blade in a Cisco chassis via UCSM.  I'm using
>   MAAS 2.0 and it looks like MAAS is failing to properly probe power.
>
>   tl;dr:
>   MAAS can successfully turn power on and off for nodes that are part of a UCSM controlled chassis.  MAAS can NOT correctly get power status from UCSM so user has to manually use Power Off from action menu in MAAS to put node in the correct state.
>
>
>   Here's what happens (and I'll attach screen shots showing each step).
>
>   1:  System not in MAAS.  UCSM shows blade power OFF.
>   2:  Power system on in UCSM.  System boots, enlists, shuts off.  UCSM shows power state OFF.  MAAS shows system in New state with no power type set.
>   3:  Configure power type with U/P, UUID and API URL and save changes
>   4:  MAAS checks power state and shows System in New state with Power ON, evne though UCSM shows that the blade power state is OFF.
>   5:  Clicking Check Now re-probes power and still shows as ON.
>   6:  Manually set Power OFF in MAAS by selecting Power Off from the Action menu.
>   7:  MAAS NOW shows power state as OFF.
>   8:  Check Now also says power state is OFF
>
>   At this point, maas.log only shows this for the UCS node:
>   bladernr@lucuma:/var/log/maas$ grep unfluffed-cherise maas.log
>   Aug 10 18:59:06 lucuma maas.api: [INFO] unfluffed-cherise: Enlisted new machine
>   Aug 10 20:11:12 lucuma maas.power: [INFO] Changing power state (off) of node: unfluffed-cherise (4y3h8r)
>   Aug 10 20:11:14 lucuma maas.power: [INFO] Changed power state (off) of node: unfluffed-cherise (4y3h8r)
>
>   The last two are where I manually set power OFF using the Action menu.
>
>   9:  I click Commission from the Action menu.
>   10:  MAAS shows power state as ON and node state as Commissioning.
>   11:  UCSM confirms that power has turned on, so MAAS CAN control power.
>   12:  Commissioning completes.  UCSM shows power state OFF, MAAS shows system as Ready, with power state ON.
>
>   In the MAAS long, we see where maas turned the power on:
>   Aug 10 20:14:42 lucuma maas.node: [INFO] unfluffed-cherise: Status transition from NEW to COMMISSIONING
>   Aug 10 20:14:42 lucuma maas.power: [INFO] Changing power state (on) of node: unfluffed-cherise (4y3h8r)
>   Aug 10 20:14:42 lucuma maas.node: [INFO] unfluffed-cherise: Commissioning started
>   Aug 10 20:14:44 lucuma maas.power: [INFO] Changed power state (on) of node: unfluffed-cherise (4y3h8r)
>
>   13: MAAS changes node state to Ready, Power remains ON
>
>   maas.log shows this:
>   Aug 10 20:17:52 lucuma maas.interface: [INFO] enp7s0 (physical) on unfluffed-cherise: Observed connected to fabric-1 via 10.1.10.0/23.
>   Aug 10 20:17:53 lucuma maas.node: [INFO] unfluffed-cherise: Storage layout was set to flat.
>   Aug 10 20:17:53 lucuma maas.node: [INFO] unfluffed-cherise: Status transition from COMMISSIONING to READY
>
>   Note that UCSM shows the node as powered OFF, MAAS still shows the
>   power ON.  Once again, I have to manually select Power Off from the
>   Action menu and then maas changes state to Power Off.
>
>   maas.log shows this when I manually tell it to turn the power off:
>   Aug 10 20:20:15 lucuma maas.power: [INFO] Changing power state (off) of node: unfluffed-cherise (4y3h8r)
>   Aug 10 20:20:17 lucuma maas.power: [INFO] Changed power state (off) of node: unfluffed-cherise (4y3h8r)
>
>   Now that MAAS thinks the power is off, I can begin a deployment.  MAAS
>   acquires the node correctly, changes state to Deploying, and sets the
>   Power state to ON... UCSM shows the node ON.  This appears in the MAAS
>   log when I do the deployment:
>
>   Aug 10 20:20:30 lucuma maas.node: [INFO] unfluffed-cherise: Status transition from READY to ALLOCATED
>   Aug 10 20:20:30 lucuma maas.node: [INFO] unfluffed-cherise: allocated to user bladernr
>   Aug 10 20:20:30 lucuma maas.interface: [INFO] Allocated automatic IP address 10.1.11.7 for enp6s0 (physical) on unfluffed-cherise.
>   Aug 10 20:20:30 lucuma maas.node: [INFO] unfluffed-cherise: Status transition from ALLOCATED to DEPLOYING
>   Aug 10 20:20:30 lucuma maas.power: [INFO] Changing power state (on) of node: unfluffed-cherise (4y3h8r)
>   Aug 10 20:20:33 lucuma maas.power: [INFO] Changed power state (on) of node: unfluffed-cherise (4y3h8r)
>   Aug 10 20:22:21 lucuma maas.preseed: [INFO] unfluffed-cherise: custom network and storage options are only supported on Ubuntu. Using flat storage layout.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1611999/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; status=Invalid; importance=Undecided; assignee=None;
> Launchpad-Bug-Tags: blocks-hwcert-server
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl bladernr
> Launchpad-Bug-Reporter: Jeff Lane (bladernr)
> Launchpad-Bug-Modifier: Andres Rodriguez (andreserl)
> Launchpad-Message-Rationale: Subscriber
> Launchpad-Message-For: bladernr



-- 
"Entropy isn't what it used to be."

Jeff Lane -
Server Certification Lead, Warrior Poet, Biker, Lover of Pie
Phone: 919-442-8649
Ubuntu Ham: W4KDH                          Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417  C466 4ABD 3635 3A14 B2DD