RefStack sign in sometimes slow

Bug #1499542 reported by Paul Van Eck
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
refstack
Fix Released
High
Paul Van Eck

Bug Description

Occasionally when signing in to refstack.net through openstackid, signing in takes a long time when openstackid sends a request to the RefStack /auth/signin_return API endpoint.

I've done some debugging on this issues, and the hang up seems to be coming from the portion of code where we validate a signature:

https://github.com/openstack/refstack/blob/master/refstack/api/utils.py#L287

Specifically:
verify_response = requests.post(
        CONF.osid.openstack_openid_endpoint, data=verify_params,
        verify=not CONF.api.app_dev_mode
)

This post request verifies with the openstackid server that an openid signature returned in openstackid's return redirect is valid.

There seems to be some delay security measure on the openstackid side, as this post request seems to sometimes take a long time. On times that it is slow, it always seems to take 128 seconds.

Maybe related to this: https://github.com/openstack-infra/openstackid/blob/master/app/services/security_policies/DelayCounterMeasure.php#L26, as 2^7= 128, but I am not sure.

It is odd that it doesn't happen on all requests. Could be fast one time, slow the next, then fast again. Will require more investigation.

This only seems to exist on the official refstack.net site, as I can't recreate the delay locally. Maybe not enough requests have been initiated over time locally?

Catherine Diep (cdiep)
Changed in refstack:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Catherine Diep (cdiep) wrote :

I checked again today and the sign in response is good .... so this is really an intermittent issue.

Paul Van Eck (pvaneck-z)
description: updated
Paul Van Eck (pvaneck-z)
description: updated
description: updated
Revision history for this message
Paul Van Eck (pvaneck-z) wrote :

I believe this has been resolved. The issue seemed to be that the RefStack server was using IPv6 when performing the TCP handshake with the openstackid server. This caused intermittent issues where it would sometimes take eight SYN transmissions (span of about 128 seconds) before a SYN-ACK was received back. Not sure why this was the case, but I made IPv4 take precedence over IPv6, so that the connections always used IPv4.

The reason I was unable to reproduce this locally was that my test environments always used IPv4.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Yep, further troubleshooting indicated some issue returning IPv6 responses from openstackid.org to refstack.net where packets were being dropped in Internap. As soon as we move the service to refstack.openstack.org this should cease to be a problem and we can have full IPv6 support.

Revision history for this message
Catherine Diep (cdiep) wrote :

refstack.net was moved to refstack.openstack.org on October 16, 2015. The slow sign-in is no longer noticed since then.

Changed in refstack:
assignee: nobody → Paul Van Eck (pvaneck-z)
status: Confirmed → Fix Committed
Revision history for this message
Catherine Diep (cdiep) wrote :

moved to refstack.openstack.org ( https://review.openstack.org/#/c/233309/)

Changed in refstack:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.