I finally found the cause of the issue. it is because the session limit was reached for rabbitmq. the default value was only 150 and it is controlled by haproxy.service_maxconn in heat template, which didn't exist in the template generated in my environment. so default 150 was used.
the number of sessions generated by my cluster of 14 nodes is a little over 200, exceeded the default 150, according to the statistics on http://haproxy_host:1993
I wonder how tripleo ci cloud in hp1 and hp2 regions set their session number. i didn't see service_maxconn specified anywhere in tripleo-cd element or tripleo-incubator.
I finally found the cause of the issue. it is because the session limit was reached for rabbitmq. the default value was only 150 and it is controlled by haproxy. service_ maxconn in heat template, which didn't exist in the template generated in my environment. so default 150 was used. haproxy_ host:1993
the number of sessions generated by my cluster of 14 nodes is a little over 200, exceeded the default 150, according to the statistics on http://
I wonder how tripleo ci cloud in hp1 and hp2 regions set their session number. i didn't see service_maxconn specified anywhere in tripleo-cd element or tripleo-incubator.