RefStack sign in sometimes slow

[Paul Van Eck]

Occasionally when signing in to refstack.net through openstackid, signing in takes a long time when openstackid sends a request to the RefStack /auth/signin_return API endpoint.

I've done some debugging on this issues, and the hang up seems to be coming from the portion of code where we validate a signature:

https://github.com/openstack/refstack/blob/master/refstack/api/utils.py#L287

Specifically:
verify_response = requests.post(
        CONF.osid.openstack_openid_endpoint, data=verify_params,
        verify=not CONF.api.app_dev_mode
)

This post request verifies with the openstackid server that an openid signature returned in openstackid's return redirect is valid.

There seems to be some delay security measure on the openstackid side, as this post request seems to sometimes take a long time. On times that it is slow, it always seems to take 128 seconds.

Maybe related to this: https://github.com/openstack-infra/openstackid/blob/master/app/services/security_policies/DelayCounterMeasure.php#L26, as 2^7= 128, but I am not sure.

It is odd that it doesn't happen on all requests. Could be fast one time, slow the next, then fast again. Will require more investigation.

This only seems to exist on the official refstack.net site, as I can't recreate the delay locally. Maybe not enough requests have been initiated over time locally?

[Catherine Diep (cdiep)]

I checked again today and the sign in response is good .... so this is really an intermittent issue.

[Paul Van Eck (pvaneck-z)]

I believe this has been resolved. The issue seemed to be that the RefStack server was using IPv6 when performing the TCP handshake with the openstackid server. This caused intermittent issues where it would sometimes take eight SYN transmissions (span of about 128 seconds) before a SYN-ACK was received back. Not sure why this was the case, but I made IPv4 take precedence over IPv6, so that the connections always used IPv4.

The reason I was unable to reproduce this locally was that my test environments always used IPv4.

[Jeremy Stanley (fungi)]

Yep, further troubleshooting indicated some issue returning IPv6 responses from openstackid.org to refstack.net where packets were being dropped in Internap. As soon as we move the service to refstack.openstack.org this should cease to be a problem and we can have full IPv6 support.

[Catherine Diep (cdiep)]

refstack.net was moved to refstack.openstack.org on October 16, 2015. The slow sign-in is no longer noticed since then.

[Catherine Diep (cdiep)]

moved to refstack.openstack.org ( https://review.openstack.org/#/c/233309/)