HPE 3PAR: SSH Connection rejected - max number of connections reached

[Raghavendra Tilay]

1. CUSTOMER IMPACT:
Limit on SSH connections to 3PAR array is preventing additional connections.

2. DETAILS:
From grepping the cinder logs – customer can see 128 times (over the last week or so) that;
the number of requests to create/delete a volume was greater than 24 requests
per minute with values as high as 110 requests per minute.
As per our understanding, 24 is the maximum number of SSH connections for this model of 3PAR.

3. ROOT CAUSE:
It seems that the hpe3parclient code is creating a new SSH/HTTP session for each request;
instead of creating and reusing sessions.

4. LOGS:
From cinder-volume log:

2021-03-22 10:20:45.703 34 ERROR oslo_messaging.rpc.server HTTPServiceUnavailable: Service Unavailable (HTTP 503) 31 - maximum number of server connections has been reached

Logs from backend i.e 3PAR event logs

rhaimi@cxo-datamine-01 /share/st909/prod/data/files/3PAR.INSERV/TierTwo/2021-03/CZ20060GC7/evtlog []$ cat evtlog.* |grep "Rejecting SSH Connection from IP sw_auth_helper General" |grep "Alert"
2021-03-01 14:08:05.83 GMT 3 20109349 Alert Major Rejecting SSH Connection from IP sw_auth_helper General SSH connection from 10.232.137.10 was rejected because SSH connections have been exhausted for this IP address.

5. SOLUTION:
[a] Instead of creating new session for each operation (eg. create/delete volume),
try to reuse existing session for as many operations as possible.
[b] Since WSAPI of 3PAR has evolved, use wsapi (http/https) requests and
get rid of SSH connections/commands.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/804698

[Sofia Enriquez (lsofia-enriquez)]

Summary: the 3PAR driver creates new SSH connection for each operation (eg. create/delete volume) because of that hits the max connections error.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/804698
Committed: https://opendev.org/openstack/cinder/commit/1daded795109b9cc8e3954befe513400fe1ef036
Submitter: "Zuul (22348)"
Branch: master

commit 1daded795109b9cc8e3954befe513400fe1ef036
Author: stack <email address hidden>
Date: Mon Aug 16 11:10:55 2021 +0000

    HPE 3PAR: Reuse existing session

    The 3PAR driver creates new SSH connection to the backend for each
    operation (eg. create/delete volume).

    There is a limit on maximum number of SSH connections from 3PAR
    backend i.e 24 connections.

    The problem is that when multiple concurrent operations are
    performed, this limit gets reached & further operations get
    errored out.

    This patch performs two things:
    [a] Instead of creating new session for each operation, an attempt is
    made to reuse existing session for as most operations.
    [b] Since WSAPI of 3PAR has evolved, wsapi (http/https) requests are
    used; thus SSH connections are no longer required.

    Closes-Bug: #1940069
    Change-Id: Ica59db93a0153be9ab2e5e9165651173d9b2afd0

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 20.0.0.0rc1 release candidate.