etcd server fails to start after upgrade from 2.2 to 3.2

[Peter J. Holzer]

Ubuntu 16.04 includes etcd 2.2.5+dfsg-1ubuntu1
Ubuntu 18.04 includes etcd 3.2.17+dfsg-1

So an upgrade from 16.04 to 18.04 involves a direct upgrade from etcd 2.2 to 3.2.

According to the docs, this is not supported.

And indeed it doesn't work.

After the upgrade etcd server refuses to start:

May 27 13:21:06 hims systemd[1]: Starting etcd - highly-available key value store...
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://999.999.999.233:2379
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd/default
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://999.999.999.233:2380
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_INITIAL_CLUSTER=akran.XXX.YY.ZZ=http://999.999.999.232:2380,hims.XXX.YY.
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=XXXXXXXXXXXX
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
May 27 13:21:06 hims etcd[15185]: recognized and used environment variable ETCD_NAME=hims.XXX.YY.ZZ
May 27 13:21:06 hims etcd[15185]: etcd Version: 3.2.17
May 27 13:21:06 hims etcd[15185]: Git SHA: Not provided (use ./build instead of go build)
May 27 13:21:06 hims etcd[15185]: Go Version: go1.10
May 27 13:21:06 hims etcd[15185]: Go OS/Arch: linux/amd64
May 27 13:21:06 hims etcd[15185]: setting maximum number of CPUs to 24, total number of available CPUs is 24
May 27 13:21:06 hims etcd[15185]: the server is already initialized as member before, starting as etcd member...
May 27 13:21:06 hims etcd[15185]: listening for peers on http://0.0.0.0:2380
May 27 13:21:06 hims etcd[15185]: listening for client requests on 0.0.0.0:2379
May 27 13:21:06 hims etcd[15185]: recovered store from snapshot at index 147977273
May 27 13:21:06 hims etcd[15185]: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist
May 27 13:21:06 hims etcd[15185]: panic: recovering backend from snapshot error: database snapshot file path error: snap: snapshot file doesn't exist
May 27 13:21:06 hims etcd[15185]: panic: runtime error: invalid memory address or nil pointer dereference
May 27 13:21:06 hims etcd[15185]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xb3ad00]
May 27 13:21:06 hims etcd[15185]: goroutine 1 [running]:
May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdserver.NewServer.func1(0xc4202b6628, 0xc4202b6400)
May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdserver/server.go:284 +0x40
May 27 13:21:06 hims etcd[15185]: panic(0xd38d20, 0xc4207960f0)
May 27 13:21:06 hims etcd[15185]: /usr/lib/go-1.10/src/runtime/panic.go:505 +0x229
May 27 13:21:06 hims etcd[15185]: github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc42020d240, 0xf1f30b, 0x2a, 0xc4202b64a0, 0x1, 0x1)
May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x162
May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdserver.NewServer(0xc420272780, 0xc420272780, 0xfc6ca0, 0xc420796040)
May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdserver/server.go:379 +0x2589
May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/embed.StartEtcd(0xc420215c00, 0xc420340000, 0x0, 0x0)
May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/embed/etcd.go:157 +0x741
May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdmain.startEtcd(0xc420215c00, 0xeff649, 0x6, 0xc4202b7201, 0x2)
May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdmain/etcd.go:186 +0x73
May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdmain/etcd.go:103 +0x1369
May 27 13:21:06 hims etcd[15185]: github.com/coreos/etcd/etcdmain.Main()
May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/etcdmain/main.go:39 +0x12b
May 27 13:21:06 hims etcd[15185]: main.main()
May 27 13:21:06 hims etcd[15185]: /build/etcd-4uSRAK/etcd-3.2.17+dfsg/_build/src/github.com/coreos/etcd/main.go:28 +0x20

If I clear out /var/lib/etcd and attempt to re-join the cluster, I get the error message:

May 27 15:22:48 hims etcd[38731]: the running cluster version(2.2.0) is lower than the minimal cluster version(3.0.0) supported
May 27 15:22:48 hims etcd[38731]: incompatible with current running cluster

So it is necessary at least to upgrade the whole cluster from 2.2 to 3.0 (possibly via 2.3) before upgrading nodes to 3.2.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: etcd-server 3.2.17+dfsg-1
ProcVersionSignature: Ubuntu 4.4.0-148.174-generic 4.4.177
Uname: Linux 4.4.0-148-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.6
Architecture: amd64
Date: Mon May 27 16:09:30 2019
InstallationDate: Installed on 2018-04-04 (418 days ago)
InstallationMedia: Ubuntu-Server 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
SourcePackage: etcd
UpgradeStatus: Upgraded to bionic on 2019-05-27 (0 days ago)
modified.conffile..etc.default.etcd: [modified]
mtime.conffile..etc.default.etcd: 2019-05-27T15:20:47.342333

[Peter J. Holzer (ligneus)]

I just checked the non-LTS releases:

yakkety is also 2.2.5
zesty is 3.1

3.1 also insists of a minimum version of 3.0.

So there doesn't seem to be a possible upgrade path with ubuntu packages.