Failed client connect: Server connection error: tonic::transport::Error

[Björn Tillenius]

This is with MAAS 3.5.0~rc1-16267-g.e3f677f51

Whenever I try to do something (for example deploying a machine), I see this error in the UI:

  Failed client connect: Server connection error: tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })))

In the logs I can see this:

Apr 08 08:44:49 maas-dev maas-temporal[12545]: 2024/04/08 08:44:49 Loading config; env=production,zone=,configDir=/var/snap/maas/34705/temporal
Apr 08 08:44:49 maas-dev maas-temporal[12545]: 2024/04/08 08:44:49 Loading config files=[/var/snap/maas/34705/temporal/production.yaml]
Apr 08 08:44:49 maas-dev maas-temporal[12545]: {"level":"fatal","ts":"2024-04-08T08:44:49.606Z","msg":"Failed to get internode TLS client config","error":"failed to load client ca: unknown failure constructing cert pool for ca","logging-call-at":"f
actory.go:188","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Fatal\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/log/zap_logger.go:180\ngo.temporal.io/server/common/membership/ringpop.(*factory).getTLSChannel\n\t/build/temporal
-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/factory.go:188\ngo.temporal.io/server/common/membership/ringpop.(*factory).getTChannel.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/factory.go:158\nsync.(*Once).
doSlow\n\t/usr/lib/go-1.20/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/lib/go-1.20/src/sync/once.go:65\ngo.temporal.io/server/common/membership/ringpop.(*factory).getTChannel\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/
factory.go:151\ngo.temporal.io/server/common/membership/ringpop.(*factory).getMonitor.func1\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/factory.go:127\nsync.(*Once).doSlow\n\t/usr/lib/go-1.20/src/sync/once.go:74\nsync.(*
Once).Do\n\t/usr/lib/go-1.20/src/sync/once.go:65\ngo.temporal.io/server/common/membership/ringpop.(*factory).getMonitor\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/factory.go:114\ngo.temporal.io/server/common/membership/
ringpop.provideMembership\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/common/membership/ringpop/fx.go:50\nreflect.Value.call\n\t/usr/lib/go-1.20/src/reflect/value.go:586\nreflect.Value.Call\n\t/usr/lib/go-1.20/src/reflect/value.go:370\ngo.uber.or
g/dig.defaultInvoker\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/container.go:238\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:180\ngo.uber.org
/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*const
ructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList
.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSing
le.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Ca
ll\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramList.BuildList\n\t
/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/b
uild/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:287\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temp
oral-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8
dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t
/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/te
mporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/
temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/
vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).executeInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/ven
dor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.HistoryServiceProvider\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:
377\nreflect.Value.call\n\t/usr/lib/go-1.20/src/reflect/value.go:586\nreflect.Value.Call\n\t/usr/lib/go-1.20/src/reflect/value.go:370\ngo.uber.org/dig.defaultInvoker\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/container.go:
238\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:180\ngo.uber.org/dig.paramGroupedSlice.callGroupProviders\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.
org/dig/param.go:609\ngo.uber.org/dig.paramGroupedSlice.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:639\ngo.uber.org/dig.paramObjectField.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org
/dig/param.go:484\ngo.uber.org/dig.paramObject.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:412\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.g
o:150\ngo.uber.org/dig.(*constructorNode).Call\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/constructor.go:150\ngo.uber.org/dig.paramSingle.Build\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:
287\ngo.uber.org/dig.paramList.BuildList\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/param.go:150\ngo.uber.org/dig.(*Scope).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:122\ngo.uber.
org/dig.(*Container).Invoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/dig/invoke.go:82\ngo.uber.org/fx.runInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/invoke.go:108\ngo.uber.org/fx.(*module).exec
uteInvoke\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:279\ngo.uber.org/fx.(*module).executeInvokes\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/module.go:265\ngo.uber.org/fx.New\n\t/build/tem
poral-8dYnj9/temporal-1.22.5/src/vendor/go.uber.org/fx/app.go:502\ngo.temporal.io/server/temporal.NewServerFx\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/temporal/fx.go:161\ngo.temporal.io/server/temporal.NewServer\n\t/build/temporal-8dYnj9/tempo
ral-1.22.5/src/temporal/server.go:69\nmain.buildCLI.func2\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:189\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/co
mmand.go:163\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/urfave/cli/v2/app.go:313\ngithub.com/urfave/cli/v2.(*App).Run\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/vendor/github.com/u
rfave/cli/v2/app.go:224\nmain.main\n\t/build/temporal-8dYnj9/temporal-1.22.5/src/cmd/server/main.go:55\nruntime.main\n\t/usr/lib/go-1.20/src/runtime/proc.go:250"}

[Björn Tillenius (bjornt)]

This only happens if you were running 7f700c4d79d2a39bb527d4c1161d5e6e329555c7, before 2177bbe0fb2fea7d114e0099496380f3c41702fa. The former created CA certificates that couldn't be used by the Temporal SDK.

2177bbe0fb2fea7d114e0099496380f3c41702fa fixed the issue, so anyone using released version of MAAS wouldn't run into this.

For development instances, the following can be done to fix the issue:

sudo snap stop maas
sudo rm /var/snap/maas/current/certificates/*
sudo -u postgres psql -d maasdb -c "delete from maasserver_secret where path='global/maas-ca-certificate' OR path='global/cluster-certificate';"
sudo snap restart maas