sync-mirror-pocket fails after 30 minutes

[Jorge Merlino]

When a sync mirror command is issued and takes longer that 30 minutes to finish (which is not strange when landscape is installed for the first time) the activity shows as failed with result text "No transition: delivered=>delivered".

In the job-handler.log file we can see this error:

'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more'

so this seems to be by design. Furthermore, after this error is shown, reprepro continues running and eventually the sync process finishes successfully even if it is shown as failed in Landscape.

In conclusion I think that 30 minutes is not long enough for this timeout as sync activities can reasonably take several hours in case of large repositories and not super fast network connections.

[Steven LaCosse (motosteven)]

I am running into this issue as well with the initial sync with larger repos.
For example when syncing the release of bionic.

Issuing the command:

landscape-api sync-mirror-pocket release bionic ubuntu

fails after 30 min: error log same as Jorge.

(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', 0, 0) content = None

I do not see in the docs to configure this fail, is there a workaround to change the value?

[Rigoberto Sanchez (sanch1)]

I've hit the same issue, if you look at the activity you can see the progress still going, so it's definitely syncing? but I get those error messages too. Not sure where to change that setting either

[Andy Wu (qch2012)]

I've hit this issue in one of the deployment, the failure is caused by the default consumer_timeout v value(30min) in rabbitmq. The first sync pockets activity, depending on the network bandwidth, can take more than 30 min, while the sync is in progress, the rabbitmq close the session after the timeout, causing the acvitivty to fail in landscape.

The workaround is to increase or disable the consumer timeout before the first sync

juju ssh rabbitmq-server/leader

cd /etc/rabbitmq
echo "
[
  {rabbit, [
    {consumer_timeout, undefined}
  ]}
]. " | sudo tee /etc/rabbitmq/advanced.config

# restart rabbitmq
sudo systemctl restart rabbitmq-server

# ensure timeout is disabled
sudo rabbitmq-diagnostics environment | grep consumer_timeout