Agent is not working on lxd

[Johann David Krister Andersson]

After deploying jenkins and 2 jenkins agents on lxd using juju deploy and relate, the agent is not actually able to start running jobs

juju debug-log -n 20
unit-jenkins-2: 16:26:48 INFO unit.jenkins/2.juju-log Reactive main running for hook update-status
unit-jenkins-2: 16:26:48 INFO unit.jenkins/2.juju-log Initializing Apt Layer
unit-jenkins-2: 16:26:48 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:211:update_plugins
unit-jenkins-2: 16:26:48 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-2: 16:26:48 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:228:ready
unit-jenkins-2: 16:26:48 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:240:add_slaves
unit-jenkins-2: 16:26:49 INFO unit.jenkins/2.juju-log Slaves are unchanged - no need to do anything
unit-jenkins-2: 16:26:49 INFO unit.jenkins/2.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
unit-jenkins-2: 16:26:49 INFO unit.jenkins/2.juju-log status-set: active: Jenkins is running
unit-jenkins-2: 16:26:49 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-2: 16:28:50 INFO unit.jenkins-slave/2.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:28:50 ERROR unit.jenkins-slave/2.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-2: 16:28:50 INFO unit.jenkins-slave/2.juju-log Initializing Apt Layer
unit-jenkins-slave-2: 16:28:50 INFO unit.jenkins-slave/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:28:50 INFO unit.jenkins-slave/3.juju-log Reactive main running for hook update-status
unit-jenkins-slave-3: 16:28:50 ERROR unit.jenkins-slave/3.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-3: 16:28:50 INFO unit.jenkins-slave/3.juju-log Initializing Apt Layer
unit-jenkins-slave-2: 16:28:50 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-3: 16:28:50 INFO unit.jenkins-slave/3.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:28:51 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-2: 16:30:54 INFO unit.jenkins/2.juju-log Reactive main running for hook update-status
unit-jenkins-2: 16:30:54 INFO unit.jenkins/2.juju-log Initializing Apt Layer
unit-jenkins-2: 16:30:54 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:211:update_plugins
unit-jenkins-2: 16:30:54 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-2: 16:30:55 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:228:ready
unit-jenkins-2: 16:30:55 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:240:add_slaves
unit-jenkins-2: 16:30:55 INFO unit.jenkins/2.juju-log Slaves are unchanged - no need to do anything
unit-jenkins-2: 16:30:55 INFO unit.jenkins/2.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
unit-jenkins-2: 16:30:55 INFO unit.jenkins/2.juju-log status-set: active: Jenkins is running
unit-jenkins-2: 16:30:55 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-2: 16:33:32 INFO unit.jenkins-slave/2.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:33:32 ERROR unit.jenkins-slave/2.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-2: 16:33:32 INFO unit.jenkins-slave/2.juju-log Initializing Apt Layer
unit-jenkins-slave-2: 16:33:32 INFO unit.jenkins-slave/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-2: 16:33:32 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-3: 16:33:32 INFO unit.jenkins-slave/3.juju-log Reactive main running for hook update-status
unit-jenkins-slave-3: 16:33:33 ERROR unit.jenkins-slave/3.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-3: 16:33:33 INFO unit.jenkins-slave/3.juju-log Initializing Apt Layer
unit-jenkins-slave-3: 16:33:33 INFO unit.jenkins-slave/3.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:33:33 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log Reactive main running for hook update-status
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log Initializing Apt Layer
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:211:update_plugins
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:228:ready
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:240:add_slaves
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log Slaves are unchanged - no need to do anything
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
unit-jenkins-2: 16:35:45 INFO unit.jenkins/2.juju-log status-set: active: Jenkins is running
unit-jenkins-2: 16:35:46 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-2: 16:37:48 INFO unit.jenkins-slave/2.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:37:48 ERROR unit.jenkins-slave/2.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-2: 16:37:48 INFO unit.jenkins-slave/2.juju-log Initializing Apt Layer
unit-jenkins-slave-2: 16:37:48 INFO unit.jenkins-slave/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:37:48 INFO unit.jenkins-slave/3.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:37:48 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-3: 16:37:48 ERROR unit.jenkins-slave/3.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-3: 16:37:48 INFO unit.jenkins-slave/3.juju-log Initializing Apt Layer
unit-jenkins-slave-3: 16:37:48 INFO unit.jenkins-slave/3.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:37:48 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-2: 16:40:53 INFO unit.jenkins/2.juju-log Reactive main running for hook update-status
unit-jenkins-2: 16:40:53 INFO unit.jenkins/2.juju-log Initializing Apt Layer
unit-jenkins-2: 16:40:53 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:211:update_plugins
unit-jenkins-2: 16:40:54 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-2: 16:40:54 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:228:ready
unit-jenkins-2: 16:40:54 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:240:add_slaves
unit-jenkins-2: 16:40:54 INFO unit.jenkins/2.juju-log Slaves are unchanged - no need to do anything
unit-jenkins-2: 16:40:54 INFO unit.jenkins/2.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
unit-jenkins-2: 16:40:54 INFO unit.jenkins/2.juju-log status-set: active: Jenkins is running
unit-jenkins-2: 16:40:54 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-2: 16:42:21 INFO unit.jenkins-slave/2.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:42:21 ERROR unit.jenkins-slave/2.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-2: 16:42:21 INFO unit.jenkins-slave/2.juju-log Initializing Apt Layer
unit-jenkins-slave-2: 16:42:22 INFO unit.jenkins-slave/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-2: 16:42:22 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-3: 16:42:22 INFO unit.jenkins-slave/3.juju-log Reactive main running for hook update-status
unit-jenkins-slave-3: 16:42:22 ERROR unit.jenkins-slave/3.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-3: 16:42:22 INFO unit.jenkins-slave/3.juju-log Initializing Apt Layer
unit-jenkins-slave-3: 16:42:22 INFO unit.jenkins-slave/3.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:42:22 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-2: 16:46:07 INFO unit.jenkins/2.juju-log Reactive main running for hook update-status
unit-jenkins-2: 16:46:07 INFO unit.jenkins/2.juju-log Initializing Apt Layer
unit-jenkins-2: 16:46:07 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:211:update_plugins
unit-jenkins-2: 16:46:07 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-2: 16:46:07 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:228:ready
unit-jenkins-2: 16:46:07 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:240:add_slaves
unit-jenkins-2: 16:46:08 INFO unit.jenkins/2.juju-log Slaves are unchanged - no need to do anything
unit-jenkins-2: 16:46:08 INFO unit.jenkins/2.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
unit-jenkins-2: 16:46:08 INFO unit.jenkins/2.juju-log status-set: active: Jenkins is running
unit-jenkins-2: 16:46:08 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-2: 16:46:41 INFO unit.jenkins-slave/2.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:46:41 ERROR unit.jenkins-slave/2.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-2: 16:46:41 INFO unit.jenkins-slave/2.juju-log Initializing Apt Layer
unit-jenkins-slave-2: 16:46:42 INFO unit.jenkins-slave/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:46:42 INFO unit.jenkins-slave/3.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:46:42 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-3: 16:46:42 ERROR unit.jenkins-slave/3.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-3: 16:46:42 INFO unit.jenkins-slave/3.juju-log Initializing Apt Layer
unit-jenkins-slave-3: 16:46:42 INFO unit.jenkins-slave/3.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:46:42 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-2: 16:50:55 INFO unit.jenkins/2.juju-log Reactive main running for hook update-status
unit-jenkins-2: 16:50:55 INFO unit.jenkins/2.juju-log Initializing Apt Layer
unit-jenkins-2: 16:50:56 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:211:update_plugins
unit-jenkins-2: 16:50:56 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-2: 16:50:56 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:228:ready
unit-jenkins-2: 16:50:56 INFO unit.jenkins/2.juju-log Invoking reactive handler: reactive/jenkins.py:240:add_slaves
unit-jenkins-2: 16:50:56 INFO unit.jenkins/2.juju-log Slaves are unchanged - no need to do anything
unit-jenkins-2: 16:50:56 INFO unit.jenkins/2.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
unit-jenkins-2: 16:50:56 INFO unit.jenkins/2.juju-log status-set: active: Jenkins is running
unit-jenkins-2: 16:50:56 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-2: 16:51:08 INFO unit.jenkins-slave/2.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:51:08 ERROR unit.jenkins-slave/2.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-2: 16:51:08 INFO unit.jenkins-slave/2.juju-log Initializing Apt Layer
unit-jenkins-slave-2: 16:51:08 INFO unit.jenkins-slave/2.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:51:08 INFO unit.jenkins-slave/3.juju-log Reactive main running for hook update-status
unit-jenkins-slave-2: 16:51:08 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
unit-jenkins-slave-3: 16:51:08 ERROR unit.jenkins-slave/3.juju-log Unable to find implementation for relation: provides of jenkins-slave
unit-jenkins-slave-3: 16:51:08 INFO unit.jenkins-slave/3.juju-log Initializing Apt Layer
unit-jenkins-slave-3: 16:51:08 INFO unit.jenkins-slave/3.juju-log Invoking reactive handler: reactive/apt.py:50:ensure_package_status
unit-jenkins-slave-3: 16:51:09 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)

[Johann David Krister Andersson (jdkandersson)]

Adding a bit more detail, the problem is that the deployed jenkins agents actually never join Jenkins to be available for jobs. Jenkins reports them as disconnected.

The steps to reproduce are:
1. Run on Ubuntu 22.04 with an lxd controller
2. Create a model and deploy jenkins: juju deploy jenkins
3. Deploy jenkins agent: juju deploy -n 2 jenkins-slave
4. Relate jenkins and jenkins agent: juju add-relation jenkins jenkins-slave

Also check the web interface and make sure the port for jenkins agents is enabled.

[Johann David Krister Andersson (jdkandersson)]

juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
jenkins lxd localhost/localhost 2.9.33 unsupported 09:33:29+10:00

App Version Status Scale Charm Channel Rev Exposed Message
jenkins active 1 jenkins stable 39 no Jenkins is running
jenkins-slave active 2 jenkins-slave stable 14 no ready - using jenkins from relation

Unit Workload Agent Machine Public address Ports Message
jenkins-slave/2 active idle 5 10.173.84.221 ready - using jenkins from relation
jenkins-slave/3* active idle 6 10.173.84.209 ready - using jenkins from relation
jenkins/2* active idle 4 10.173.84.223 8080/tcp,48484/tcp Jenkins is running

Machine State Address Inst id Series AZ Message
4 started 10.173.84.223 juju-6b098c-4 focal Running
5 started 10.173.84.221 juju-6b098c-5 focal Running
6 started 10.173.84.209 juju-6b098c-6 focal Running

Relation provider Requirer Interface Type Message
jenkins-slave:slave jenkins:master jenkins-slave regular