Comment 8 for bug 1917308

Revision history for this message
chen haochuan (martin1982) wrote :

For my reproduce system

./sysinv.log:9002:sysinv 2021-03-19 14:37:47.208 1359833 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.: HelmTillerFailure: Helm operation failure: Failed to obtain pending charts list: Helm operation failure: Error: write tcp 172.16.192.72:52144->192.188.204.2:5432: write: broken pipe

And 192.188.204.2:5432, postgres listen on this port
[sysadmin@controller-0 log(keystone_admin)]$ sudo netstat -ltnp | grep 5432
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 1357087/postgres
tcp6 0 0 :::5432 :::* LISTEN 1357087/postgres

[sysadmin@controller-0 log(keystone_admin)]$ ps -aux | grep postgres
nfsnobo+ 130497 0.1 0.4 213352 126384 ? Ssl 03:35 0:57 /tiller --storage=sql --sql-dialect=postgres --sql-connection-string=postgresql://admin-helmv2:X1=_Vx3F7T-GGu6L@192.188.204.2:5432/helmv2?sslmode=disable -listen :24134 -probe-listen :24135 -logtostderr -v 5
postgres 1357087 0.0 0.1 312924 34880 ? S< 14:17 0:00 /usr/bin/postgres -D /var/lib/postgresql/20.12 -c config_file=/etc/postgresql/postgresql.conf

And cluster ip "172.16.192.72", is pod armada-api address.
./pods/kube-system_calico-node-tq7k8_5e123081-7506-414c-a679-58a88c7e2795/calico-node/1.log:1731:2021-03-19T03:35:49.722858338Z stdout F 2021-03-19 03:35:49.720 [INFO][41] int_dataplane.go 825: Received *proto.WorkloadEndpointUpdate update from calculation graph msg=id:<orchestrator_id:"k8s" workload_id:"armada/armada-api-5fc6fb496c-qqdkm" endpoint_id:"eth0" > endpoint:<state:"active" name:"cali7815eef50b5" profile_ids:"kns.armada" profile_ids:"ksa.armada.armada-api" ipv4_nets:"172.16.192.72/32" >

[sysadmin@controller-0 log(keystone_admin)]$ kubectl get po -n armada -o wide | grep "172.16.192.72"
armada-api-5fc6fb496c-qqdkm 2/2 Running 2 11h 172.16.192.72 controller-0 <none> <none>

so after swact, tiller in aramda-api pod, could not access posgresl service