Activity log for bug #1846531

Date Who What changed Old value New value Message
2019-10-03 17:06:15 Radosław Piliszek bug added bug
2019-10-03 17:06:26 Radosław Piliszek nominated for series kolla-ansible/stein
2019-10-03 17:06:26 Radosław Piliszek bug task added kolla-ansible/stein
2019-10-03 17:06:26 Radosław Piliszek nominated for series kolla-ansible/train
2019-10-03 17:06:26 Radosław Piliszek bug task added kolla-ansible/train
2019-10-03 17:09:17 Radosław Piliszek attachment added docker-info.txt https://bugs.launchpad.net/kolla-ansible/+bug/1846531/+attachment/5294040/+files/docker-info.txt
2019-10-04 08:37:44 Mark Goddard description This is after enabling TLS which broke etcd. In Ansible: Read timed out. (read timeout=60) In Docker: Oct 02 18:23:08 primary dockerd[10089]: time="2019-10-02T18:23:08.586295237Z" level=debug msg="could not find network 3f560f255e07334fd1e21230d15cd44fb6d0f4029cf386884d1b096fc3b55107: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.0.2.1:2379: connect: connection refused\n; error #1: dial tcp 192.0.2.2:2379: connect: connection refused\n; error #2: dial tcp 192.0.2.3:2379: connect: connection refused\n" In etcd: 2019-10-02 18:26:05.742017 I | embed: rejected connection from "192.0.2.2:33296" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.742049 I | embed: rejected connection from "192.0.2.2:33298" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.805035 I | embed: rejected connection from "192.0.2.1:54146" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.806758 I | embed: rejected connection from "192.0.2.1:54144" (error "remote error: tls: bad certificate", ServerName "") Expected etcd to break only containers deployed by Zun, not all of deployment. This is after enabling internal TLS which broke etcd. In Ansible: Read timed out. (read timeout=60) In Docker: Oct 02 18:23:08 primary dockerd[10089]: time="2019-10-02T18:23:08.586295237Z" level=debug msg="could not find network 3f560f255e07334fd1e21230d15cd44fb6d0f4029cf386884d1b096fc3b55107: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.0.2.1:2379: connect: connection refused\n; error #1: dial tcp 192.0.2.2:2379: connect: connection refused\n; error #2: dial tcp 192.0.2.3:2379: connect: connection refused\n" In etcd: 2019-10-02 18:26:05.742017 I | embed: rejected connection from "192.0.2.2:33296" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.742049 I | embed: rejected connection from "192.0.2.2:33298" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.805035 I | embed: rejected connection from "192.0.2.1:54146" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.806758 I | embed: rejected connection from "192.0.2.1:54144" (error "remote error: tls: bad certificate", ServerName "") Expected etcd to break only containers deployed by Zun, not all of deployment.
2019-10-04 09:25:12 Radosław Piliszek description This is after enabling internal TLS which broke etcd. In Ansible: Read timed out. (read timeout=60) In Docker: Oct 02 18:23:08 primary dockerd[10089]: time="2019-10-02T18:23:08.586295237Z" level=debug msg="could not find network 3f560f255e07334fd1e21230d15cd44fb6d0f4029cf386884d1b096fc3b55107: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.0.2.1:2379: connect: connection refused\n; error #1: dial tcp 192.0.2.2:2379: connect: connection refused\n; error #2: dial tcp 192.0.2.3:2379: connect: connection refused\n" In etcd: 2019-10-02 18:26:05.742017 I | embed: rejected connection from "192.0.2.2:33296" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.742049 I | embed: rejected connection from "192.0.2.2:33298" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.805035 I | embed: rejected connection from "192.0.2.1:54146" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.806758 I | embed: rejected connection from "192.0.2.1:54144" (error "remote error: tls: bad certificate", ServerName "") Expected etcd to break only containers deployed by Zun, not all of deployment. The example failure given below is after enabling internal TLS which broke etcd. Any etcd failure is a valid trigger for this bug. In Ansible: Read timed out. (read timeout=60) In Docker: Oct 02 18:23:08 primary dockerd[10089]: time="2019-10-02T18:23:08.586295237Z" level=debug msg="could not find network 3f560f255e07334fd1e21230d15cd44fb6d0f4029cf386884d1b096fc3b55107: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 192.0.2.1:2379: connect: connection refused\n; error #1: dial tcp 192.0.2.2:2379: connect: connection refused\n; error #2: dial tcp 192.0.2.3:2379: connect: connection refused\n" In etcd: 2019-10-02 18:26:05.742017 I | embed: rejected connection from "192.0.2.2:33296" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.742049 I | embed: rejected connection from "192.0.2.2:33298" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.805035 I | embed: rejected connection from "192.0.2.1:54146" (error "remote error: tls: bad certificate", ServerName "") 2019-10-02 18:26:05.806758 I | embed: rejected connection from "192.0.2.1:54144" (error "remote error: tls: bad certificate", ServerName "") Expected etcd to break only containers deployed by Zun, not all of deployment.
2019-10-06 09:34:14 Radosław Piliszek kolla-ansible/train: importance Critical Medium
2019-10-06 09:35:22 Radosław Piliszek kolla-ansible/stein: assignee Radosław Piliszek (yoctozepto)
2019-11-14 14:05:48 Mark Goddard nominated for series kolla-ansible/ussuri
2019-11-14 14:05:48 Mark Goddard bug task added kolla-ansible/ussuri
2019-11-14 14:05:59 Mark Goddard kolla-ansible/ussuri: milestone 9.0.0 10.0.0
2019-11-18 17:58:53 Radosław Piliszek kolla-ansible/train: milestone 9.0.0
2019-11-18 17:58:55 Radosław Piliszek kolla-ansible/train: importance Medium Low
2019-11-18 17:58:58 Radosław Piliszek kolla-ansible/ussuri: milestone 10.0.0
2019-11-18 17:59:00 Radosław Piliszek kolla-ansible/ussuri: importance Medium Low
2019-11-18 17:59:06 Radosław Piliszek kolla-ansible/stein: importance Undecided Low
2019-11-18 17:59:08 Radosław Piliszek kolla-ansible/stein: status New Triaged
2020-05-19 07:34:57 Radosław Piliszek bug task deleted kolla-ansible/stein
2020-05-19 07:35:04 Radosław Piliszek bug task deleted kolla-ansible/train
2020-05-19 07:35:08 Radosław Piliszek bug task deleted kolla-ansible/ussuri
2020-05-19 07:35:15 Radosław Piliszek kolla-ansible: status Triaged Opinion
2020-05-19 07:35:19 Radosław Piliszek kolla-ansible: assignee Radosław Piliszek (yoctozepto)