Brief Description
-----------------
System remove, reapply fails to start the Vault pods with "http: TLS handshake error from 172.16.154.5:40060: remote error: tls: bad certificate"
Severity
--------
Major
Steps to Reproduce
------------------
1)Upload,apply the app
2)perform the following steps to enable kubernetes,kv engine,inject the secret
Get the auth methods
********************
curl --insecure --header "X-Vault-Token:s.wi1Jab7k24PpNpieFmJwBfK1" https://10.102.120.240:8200/v1/sys/auth
Enable kubernetes auth
********************************
curl --insecure --header "X-Vault-Token:s.wi1Jab7k24PpNpieFmJwBfK1" \
--request POST \
--data '{"type":"kubernetes","description":"kubernetes auth"}' \
https://10.102.120.240:8200/v1/sys/auth/kubernetes
Configure kubernetes auth
***********************************
Get the ca cert and token from the vault pod
kubectl exec -n vault sva-vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
kubectl exec -n vault sva-vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
curl \
--insecure \
--header "X-Vault-Token:s.wi1Jab7k24PpNpieFmJwBfK1" \
--request POST \
--data '{"kubernetes_host": "https://10.96.0.1:443", "kubernetes_ca_cert":"-----BEGIN CERTIFICATE-----\nMIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJlcm5ldGVzMB4XDTIwMDcyNDEzMTY1M1oXDTMwMDcyMjEzMTY1M1owFTETMBEGA1UEAxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMUzG64PKNDGSrKREkG5WaLasWI/SDmJ5oyxhUYythuYAWgFfzbTUfkMaAYlWFjfqj5AcYzQuA8J6LkJbKBjZVNKL7pGSXBf1NkRdp90tqGH34FmuwPv7A7tpz3sfP25mRldb76DYk8bDP9qFzqaC5Z1Uqp30MP04wKFlHoxJ+zZ0xLsE5Yh8J/9pjlKr3LaQUrG3iXbxDYMPP3KWf6HtrUDqtZH/p1vpuUCP1QE4A2ZuA+Krt/AeGvXYUyr8Q1uYLV8WpoIZsCUVS88FGZRUF4pCE0v7Btvm1v2A8An6gGdIoJLMTUmDC5ZUrQIfsEsOVzy1I3nw9XGN2ktOATN2ECAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFEr9KBCWWNsS1vO+xV1bXYfvE9ajJMI3zG3sZ/55AnjBmQkJTIcRrhd6DxbZGor0400yvteRCGWTTQKvv1Dbpa0Q0xdm2pllGlSXSI4DRdv2V0Doa2l28AYZjemVoOXXfLYVSWY90CxTRsLvVYVEnm2ogkG7moTiTmZJh/cTmUZpyrR0xYniaOOLVsGSwgvzSPa7kt06DM4HFWqlfUVdDMt/9Resr4HjX5PIDtP5sBKEmuviUdiTxO5d6xmhhcJ1IDmZZ4meKo7fElKmpZTAkaBSv5V3ct2Hx31loO1xhfLOjAyp8cDHqMIWcNxt+Qlm2bGQZNA6Hs+xnLnk3stWw8=\n-----END CERTIFICATE-----", "token_reviewer_jwt":"eyJhbGciOiJSUzI1NiIsImtpZCI6IjBvMWpjTmc0Q0FmT2Z6LTh0UzN6bV9MVWVmQWZLVVJ1OU9ma1I4TzliNXcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJ2YXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJzdmEtdmF1bHQtdG9rZW4tbGZkdGoiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoic3ZhLXZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNGYwOTZmY2YtOWU5NC00NjM3LWI5YWYtOWZjZDViZGM2YjE2Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnZhdWx0OnN2YS12YXVsdCJ9.OxPAVADb2_5mXSJDZpbBLToQlph3e4b3QGut8d0pGS2w-qKSfbJ_ksBIsabtH4s7xOlN9lUdLGANixnebrS-9PmkKA6tcPFi36ghvHT7GAZvRy4gK02c2pb91BvsGAGwWMt_egzseIxsIXG5o6uPqLiH1vEDDRQvDRF_CKNg2S79CtHKl2gyfZAI97YET4NKL7kaGWjnOQui8n5KMhsQ-CENprMa7eH5BGFXR-VX4g5f3zURmo4tXG5KKBqFoqNN2mapUjONH2SnzA70wfYWtx5XZPd4TKBj-eGHLfk-xQV9wk51etGuE_pjzx1FKq8WY53LOfLEFtPPTig-bmIIGQ"}' \
https://10.102.120.240:8200/v1/auth/kubernetes/config
Read the config
****************
curl --insecure --header "X-Vault-Token:s.wi1Jab7k24PpNpieFmJwBfK1" https://10.102.120.240:8200/v1/auth/kubernetes/config
Create the policy
************
curl --insecure \
-H "X-Vault-Token: s.wi1Jab7k24PpNpieFmJwBfK1" \
-H "Content-Type: application/json" \
-X PUT \
-d '{"policy":"path \"secret/basic-secret/*\" {capabilities = [\"read\"]}"}' \
https://10.102.120.240:8200/v1/sys/policy/basic-secret-policy
Create the role with policy and namespace
***************************************
curl --insecure \
--header "X-Vault-Token:s.wi1Jab7k24PpNpieFmJwBfK1" \
--request POST \
--data '{ "bound_service_account_names": "basic-secret", "bound_service_account_namespaces": "pvtest", "policies": "basic-secret-policy", "max_ttl": "1800000"}' \
https://10.102.120.240:8200/v1/auth/kubernetes/role/basic-secret-role
Read the role
************
curl --insecure --header "X-Vault-Token:s.wi1Jab7k24PpNpieFmJwBfK1" https://10.102.120.240:8200/v1/auth/kubernetes/role/basic-secret-role
Enable the secret engine
*********************
curl --insecure --header "X-Vault-Token:s.wi1Jab7k24PpNpieFmJwBfK1" \
--request POST \
--data '{"type": "kv","version":"2"}' \
https://10.102.120.240:8200/v1/sys/mounts/secret
Create the secrets
*************************
curl --insecure \
-H "X-Vault-Token: s.wi1Jab7k24PpNpieFmJwBfK1" \
-H "Content-Type: application/json" \
-X POST -d '{"username":"pvtest","password":"Li69nux*"}' \
https://10.102.120.240:8200/v1/secret/basic-secret/helloworld
Check the secret
***********************
curl --insecure --header "X-Vault-Token:s.wi1Jab7k24PpNpieFmJwBfK1" https://10.102.120.240:8200/v1/secret/basic-secret/helloworld
cat helloworld.yaml
apiVersion: v1
kind: Namespace
metadata:
name: pvtest
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: basic-secret
namespace: pvtest
labels:
app: basic-secret
spec:
selector:
matchLabels:
app: basic-secret
replicas: 1
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/tls-skip-verify: "true"
vault.hashicorp.com/agent-inject-secret-helloworld: "secret/basic-secret/helloworld"
vault.hashicorp.com/agent-inject-template-helloworld: |
{{- with secret "secret/basic-secret/helloworld" -}}
{
"username" : "{{ .Data.username }}",
"password" : "{{ .Data.password }}"
}
{{- end }}
vault.hashicorp.com/role: "basic-secret-role"
labels:
app: basic-secret
spec:
serviceAccountName: basic-secret
containers:
- name: app
image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: basic-secret
namespace: pvtest
labels:
app: basic-secret
Apply the app and verify pod is running
********************************
kubectl create -f helloworld.yaml
Verify secrets injected into the pod
******************************************
kubectl exec -n pvtest basic-secret-55d6c9bb6f-4whbp -- cat /vault/secrets/helloworld
Defaulting container name to app.
Use 'kubectl describe pod/basic-secret-55d6c9bb6f-4whbp -n pvtest' to see all of the containers in this pod.
{
"username" : "pvtest",
"password" : "Li69nux*"
}
3)Remove the vault app without deleting the PVC
system application-remove vault
4)apply the app
system applciation-apply vault
5)Wait for vault pod init and they dont reach the ready state
[sysadmin@controller-1 ~(keystone_admin)]$ kubectl get pods -n vault
NAME READY STATUS RESTARTS AGE
sva-vault-0 0/1 Running 0 2m25s
sva-vault-1 0/1 Running 0 2m25s
sva-vault-2 0/1 Running 0 2m25s
sva-vault-agent-injector-db6878c69-z5rtg 1/1 Running 0 2m25s
sva-vault-manager-0 1/1 Running 0 2m25s
6)log show the following error
[sysadmin@controller-1 ~(keystone_admin)]$ tail -f /var/log/pods/vault_sva-vault-2_5747c4fe-446b-4ff7-8ce2-cd40df2a9342/vault/0.log
2020-07-24T20:24:35.666429779Z stderr F 2020-07-24T20:24:35.666Z [INFO] http: TLS handshake error from 172.16.154.5:40060: remote error: tls: bad certificate
2020-07-24T20:24:41.049837853Z stderr F 2020-07-24T20:24:41.049Z [INFO] http: TLS handshake error from 172.16.154.5:40088: remote error: tls: bad certificate
2020-07-24T20:24:46.484364442Z stderr F 2020-07-24T20:24:46.484Z [INFO] http: TLS handshake error from 172.16.154.5:40118: remote error: tls: bad certificate
2020-07-24T20:24:51.88965778Z stderr F 2020-07-24T20:24:51.889Z [INFO] http: TLS handshake error from 172.16.154.5:40150: remote error: tls: bad certificate
2020-07-24T20:24:57.315645712Z stderr F 2020-07-24T20:24:57.315Z [INFO] http: TLS handshake error from 172.16.154.5:40184: remote error: tls: bad certificate
2020-07-24T20:25:02.741276992Z stderr F 2020-07-24T20:25:02.741Z [INFO] http: TLS handshake error from 172.16.154.5:40212: remote error: tls: bad certificate
2020-07-24T20:25:08.161556419Z stderr F 2020-07-24T20:25:08.161Z [INFO] http: TLS handshake error from 172.16.154.5:40242: remote error: tls: bad certificate
2020-07-24T20:25:13.597918351Z stderr F 2020-07-24T20:25:13.597Z [INFO] http: TLS handshake error from 172.16.154.5:40270: remote error: tls: bad certificate
2020-07-24T20:25:19.011553862Z stderr F 2020-07-24T20:25:19.011Z [INFO] http: TLS handshake error from 172.16.154.5:40302: remote error: tls: bad certificate
2020-07-24T20:25:24.456478768Z stderr F 2020-07-24T20:25:24.456Z [INFO] http: TLS handshake error from 172.16.154.5:40338: remote error: tls: bad certificate
2020-07-24T20:25:29.862866032Z stderr F 2020-07-24T20:25:29.862Z [INFO] http: TLS handshake error from 172.16.154.5:40366: remote error: tls: bad certificate
2020-07-24T20:25:35.284795945Z stderr F 2020-07-24T20:25:35.284Z [INFO] http: TLS handshake error from 172.16.154.5:40398: remote error: tls: bad certificate
2020-07-24T20:25:40.731831023Z stderr F 2020-07-24T20:25:40.731Z [INFO] http: TLS handshake error from 172.16.154.5:40426: remote error: tls: bad certificate
2020-07-24T20:25:46.181830461Z stderr F 2020-07-24T20:25:46.181Z [INFO] http: TLS handshake error from 172.16.154.5:40458: remote error: tls: bad certificate
2020-07-24T20:25:51.617320245Z stderr F 2020-07-24T20:25:51.617Z [INFO] http: TLS handshake error from 172.16.154.5:40480: remote error: tls: bad certificate
2020-07-24T20:25:57.074625326Z stderr F 2020-07-24T20:25:57.074Z [INFO] http: TLS handshake error from 172.16.154.5:40522: remote error: tls: bad certificate
2020-07-24T20:26:02.515011819Z stderr F 2020-07-24T20:26:02.514Z [INFO] http: TLS handshake error from 172.16.154.5:40552: remote error: tls: bad certificate
Attached the server logs, please check
Expected Behavior
------------------
the pods should run successfully also the secrets should persist on the system
Actual Behavior
----------------
the pods doesn't reach to ready state
Reproducibility
---------------
100%
System Configuration
--------------------
standard wcp_3_6 ipv4
Branch/Pull Time/Commit
-----------------------
2020-07-24_00-00-00
Last Pass
---------
This is a new test scenario
Timestamp/Logs
--------------
2020-07-24T15:43:40.922783884Z
Test Activity
-------------
Feature Testing
Workaround
----------
Haven't found any
stx.5.0 / medium - failure scenario test-case; should be investigated