Instance creation fails with SSL, keystone v3

[Michael Skalka]

OS version is Ocata, with SSL enabled across the entire cloud.

Using the Keystone-LDAP charm to allow AD user authentication to the OS deployment. AD admin users can login, and have limited admin access.

If an AD user is added to a project on either the AD domain or the admin_default domain as an admin, they are able to request an instance but the instance creation errors out with: http://paste.ubuntu.com/24727101/

There is an associated error in nova-cloud-controller's apache2 nova-placement error log: http://paste.ubuntu.com/24727106/

Creating an instance with a local administrator on the admin_domain domain on a project in the admin domain works without issue. However it does not work while logged in as a local administrator (who has admin rights added) on a project created in the AD domain.

The root of the issue seems to be communication between the nova scheduler and the nova placement api, specifically where if a token originates from the AD domain it has insufficient privileges to perform administrative action between services.

[Antonio Rosales (arosales)]

Ryan,
Please let us know what information is needed as this bug is set to incomplete.

Michael,
Thanks for the bug report. Going forward please use https://paste.ubuntu.com/ and if necessary sanitize any sensitive data.

-thanks,
Antonio

[James Page (james-page)]

The error appears to be from the nova placement API, with an SSL error communicating with keystone:

SSLError: SSL exception connecting to https://keystone-ch2-g.xx.xx.yy:5000/v3/auth/tokens: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)

[Ryan Beisner (1chb1n)]

Agree @jamespage. I believe there is more than one modified charm in play in the model, related to other SSL and/or ksv3 issues.

What we're looking for is sanitized charm config options for nova-cloud-controller, nova-compute; sanitized juju status; what versions of the charms are deployed; and any custom modifications or workarounds made to the charms or the units.

[Michael Skalka (mskalka)]

Keystone:
    V: 11.0.1.dev17
    Config: http://paste.ubuntu.com/24727238/

Nova-CC:
    V: 15.0.2
    Config: http://paste.ubuntu.com/24727258/

Nova-compute-KVM:
    V: 15.0.2
    Config: http://paste.ubuntu.com/24727260/

Juju status pending retrieval and cleanup

[David Ames (thedac)]

The root cause for this is HAProxy timing out and terminating the TCP session. The hint in the error was the EOF:

SSLError: SSL exception connecting to https://keystone-<CUSTOMER_DOMAIN>.net:5000/v3/auth/tokens: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
                                                                                                                                                                                                                                                                                                       Bumping up the haproxy-*-timeout values for the API charms resolved the issue for CLI driven instance create commands.

There is still some question if instance creation via horizon has a remaining bug or timeout.

I am marking the nova-compute and nova-cloud-controller projects as invalid for this bug. If a horizon bug still remains we can add the openstack-dashboard project to this bug.