Coverage for pysaml2 is insufficient

[Thomas Goirand]

Releasing Kilo in Debian, I found out that Keystone just broke with pysaml2 2.0.0, and in fact needs 2.4.0. The unit tests just passed, but with pysaml2 2.0.0 Keystone just crashes with a stack dump.

Out of this, 2 remarks:
- requirements.txt is wrong and should ask for something higher than 2.0.0 (maybe 2.4.0, or something lower)
- unit tests should have detected the issue, meaning that coverage isn't good enough

[Morgan Fainberg (mdrnstm)]

Hi Thomas,

This is a fix that needs to be pushed to the Global Requirements to fix for the first issue (likely needs to be fixed in master *and* in stable/juno).

To the second remark, we do not currently test either the minimum installable (or variations thereof) in the gate, we currently only test the maximum(s) that can be installed. There is no way that we can detect with the unit (or integration tests) currently the matrix of every possible install version of every possible library.

If you can provide the traceback you're seeing we can look to see what assumption was made that was incorrect. I do encourage you to post the update to global requirements directly and provide me with a link to the review that sets the minimum that works with Keystone. If you do not wish to propose the change to global-requirements, I am more than happy to once I've looked at the traceback.

[Thomas Goirand (thomas-goirand)]

Hi Morgan,

Thanks for your reply.

I think you missed the point, because I didn't explain well enough. Of course I know that we aren't (yet) testing with the lower bound of our requirements. That's not the point. The point is, I'm running unit tests when building packages, and I should have been able to catch the issue with the pysaml2 version when building keystone. And I was not, because unit tests didn't cover well enough pysaml2. Instead, all unit tests passed, and what didn't work was running the keystone-all daemon, which crashed.

As for the global-requirements.txt, I'll try to find the time to make it happen. I've pushed it for master:
https://review.openstack.org/186996

Cheers,

Thomas

[Morgan Fainberg (mdrnstm)]

I also asked if you would post the traceback you were seeing since you had the active environment handy. I am not trying to be difficult here, I am trying to ease up the load on but triage so we can identify the scope of issues more quickly And what we are looking for with better but reports. This means we (the team doing this) can have a clear target for duplication.

If I don't know the error "keystone-all crashes" is a wide surface of things to be looking for and sometimes it's easy to see what the error is before even setting up the isolated test environment (or helps us to see what the tests are missing before we start duplicating the bug).

I'm not disagreeing this is an issue since I didn't close the bug. Thanks for the work on this. The traceback would still be useful but in its absence I'll still circle up on this in the near-ish term.

[Steve Martinelli (stevemar)]

is this still an issue? looks like there is now a minimum version of pysaml2

[Launchpad Janitor (janitor)]

[Expired for OpenStack Identity (keystone) because there has been no activity for 60 days.]