Cluster of multiple snap store proxy instances not working

[Christoph Fiehe]

The documentation (https://docs.ubuntu.com/snap-store-proxy/en/install) says that it is possible to run multiple instances of the Snap Store Proxy. I am not able to get that working. I followed the installation guide and setup two virtual machines each running an instance of the Snap Store Proxy. The problem is that each instance creates different passwords for the databases roles, which cannot be shared between the Snap Store Proxies, so that there is only one instance that can access the shared database. On the other proxy some services are failing because of authentication failures (wrong passwords):

root@snap-store-proxy-srv1:/snap/snap-store-proxy# snap-proxy status
Store URL: http://snaps.eurodata.de
Store DB: ok
Store ID: *******
Status: approved
Connected Devices (updated daily): 0
Device Limit: 25
Internal Service Status:
  memcached: running
  nginx: running
  snapauth: running
  snapdevicegw: running
  snapdevicegw-local: running
  snapproxy: running
  snaprevs: running

root@snap-store-proxy-srv2:~# snap-proxy status
Store URL: http://snaps.eurodata.de
Store DB: ok
Store ID: not registered
Internal Service Status:
  memcached: running
  nginx: running
  snapauth: not running: 500 Server Error: INTERNAL SERVER ERROR for url: http://127.0.0.1:8005/_status/check
  snapdevicegw: running
  snapdevicegw-local: running
  snapproxy: running
  snaprevs: not running: 500 Server Error: INTERNAL SERVER ERROR for url: http://127.0.0.1:8002/_status/check

The documentation also says that it is enough to register one of the cluster members, but that does not work, too. Let's have a look at the store id. It is not accessible from the database for the other node or only hold locally on the system that was registered successfully.

It it currently possible to run that service as a cluster comprising two or more instances of the Snap Store Proxy? Is there a way to loadbalance it with the help of NGINX, Apache or HAProxy?

[Przemysław Suliga (suligap)]

Hi Christoph and thanks for this report!

We're aware of this and some other current limitations of this setup and are currently designing some related improvements. We will likely remove the current related documentation until this is addressed.

It should be possible to work around the db credentials issue. See:

    sudo snap get snap-store-proxy internal.snaprevs.db

and

    sudo snap get snap-store-proxy internal.snapauth.db

`sudo snap set ...` can be used to set the same connections string in all instances of the snap-store-proxy.

As for the registration. Only one proxy has to be registered. But others' config will have to be set based on the config of the registered one. `snap-proxy config` command can be used to both get and set all non-internal config options of the snap-store-proxy.

snap-store-proxy can be fronted with an external load balancer / reverse-proxy. However currently for it to function properly the scheme used between the end clients (devices) and the external LB has to be the same as the scheme exposed by the snap-store-proxy - meaning either http or https on both sides.

[Christoph Fiehe (cfiehe)]

Hi Przemysław,

thanks a lot for those helpful hints. I have reconfigured the second instance based on the configuration values (secret, keys, database credentials) of the first (registered) instance. It works like a charm:

root@snap-store-proxy-srv1:~# snap-proxy status
Store URL: http://snaps.mycompany.com
Store DB: ok
Store ID: *****
Status: approved
Connected Devices (updated daily): 2
Device Limit: 25
Internal Service Status:
  memcached: running
  nginx: running
  snapauth: running
  snapdevicegw: running
  snapdevicegw-local: running
  snapproxy: running
  snaprevs: running

root@snap-store-proxy-srv2:~# snap-proxy status
Store URL: http://snaps.mycompany.com
Store DB: ok
Store ID: *****
Status: approved
Connected Devices (updated daily): 2
Device Limit: 25
Internal Service Status:
  memcached: running
  nginx: running
  snapauth: running
  snapdevicegw: running
  snapdevicegw-local: running
  snapproxy: running
  snaprevs: running

Thank you very much!