controller-0: 20:43:58 INFO juju.worker.apicaller [f0e819] "controller-0" successfully connected to "localhost:17070"
controller-0: 20:43:58 INFO juju.worker.logforwarder config change - log forwarding not enabled
controller-0: 20:43:58 INFO juju.worker.logger logger worker started
controller-0: 20:43:58 DEBUG juju.worker.caas waiting for config and credential notifications
controller-0: 20:43:58 DEBUG juju.worker.caas reloading model config
controller-0: 20:43:58 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for kubeflow (f0e819dd-2fb4-4e25-8bdd-77edd9c85838)
controller-0: 20:43:58 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for kubeflow (f0e819dd-2fb4-4e25-8bdd-77edd9c85838)
controller-0: 20:43:58 DEBUG juju.worker.caas waiting for config and credential notifications
controller-0: 20:43:58 DEBUG juju.worker.caas waiting for config and credential notifications
controller-0: 20:43:59 DEBUG juju.worker.storageprovisioner volumes alive: [], dying: [], dead: []
controller-0: 20:43:59 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:43:59 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:43:59 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:02 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:45:02 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:02 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:02 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:02 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:02 DEBUG juju.worker.caasunitprovisioner service for argo-controller(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
controller-0: 20:45:02 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:45:03 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc00105efc0}
controller-0: 20:45:03 DEBUG juju.worker.caasunitprovisioner operator update for argo-controller
controller-0: 20:45:03 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:04 DEBUG juju.worker.caasunitprovisioner operator update for minio
controller-0: 20:45:04 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner volumes alive: [volume-0], dying: [], dead: []
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner not queuing volume for non-machine unit-minio-0
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner volume attachments alive: [{unit-minio-0 volume-0}], dying: [], dead: []
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner filesystems alive: [filesystem-0], dying: [], dead: []
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner not queuing volume attachment for non-machine unit-minio-0
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner not queuing filesystem for minio unit
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [{unit-minio-0 filesystem-0}], dying: [], dead: []
controller-0: 20:45:04 DEBUG juju.worker.storageprovisioner not queuing filesystem attachment for non-machine unit-minio-0
controller-0: 20:45:05 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:45:05 INFO juju.worker.caasprovisioner started operator for application "argo-controller"
controller-0: 20:45:05 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:45:05 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:05 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:05 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:05 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:06 DEBUG juju.worker.caasunitprovisioner operator update for argo-controller
controller-0: 20:45:07 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:45:08 DEBUG juju.worker.caasunitprovisioner operator update for argo-controller
controller-0: 20:45:08 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc00233c780}
application-argo-controller: 20:45:08 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-argo-controller: 20:45:08 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=argo-controller", "--debug"}
application-argo-controller: 20:45:08 DEBUG juju.agent read agent config, format "2.0"
application-argo-controller: 20:45:08 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-argo-controller: 20:45:08 INFO juju.cmd.jujud caas operator application-argo-controller start (2.8.6 [gc])
application-argo-controller: 20:45:08 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:45:08.962634614 +0000 UTC
application-argo-controller: 20:45:08 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:45:08.962854416 +0000 UTC
application-argo-controller: 20:45:08 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:45:08.963002317 +0000 UTC
application-argo-controller: 20:45:08 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:45:08 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:45:08.965941135 +0000 UTC
application-argo-controller: 20:45:08 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-argo-controller"
application-argo-controller: 20:45:08 DEBUG juju.worker.introspection stats worker now serving
application-argo-controller: 20:45:08 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:45:08.972608877 +0000 UTC
application-argo-controller: 20:45:08 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:45:08.982893942 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-argo-controller: 20:45:09 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-argo-controller: 20:45:09 INFO juju.worker.apicaller [f0e819] "application-argo-controller" successfully connected to "10.152.183.175:17070"
application-argo-controller: 20:45:09 DEBUG juju.api RPC connection died
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully
application-argo-controller: 20:45:09 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:45:09 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-argo-controller: 20:45:09 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-argo-controller: 20:45:09 INFO juju.worker.apicaller [f0e819] "application-argo-controller" successfully connected to "10.152.183.175:17070"
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:45:09.06196004 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:45:09.074322918 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:45:09.076931535 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:45:09.085158687 +0000 UTC
application-argo-controller: 20:45:09 INFO juju.worker.migrationminion migration phase is now: NONE
application-argo-controller: 20:45:09 INFO juju.worker.caasupgrader abort check blocked until version event received
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:45:09.102046393 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:45:09.106550521 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-argo-controller: 20:45:09 INFO juju.worker.caasupgrader unblocking abort check
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:45:09.125038638 +0000 UTC
controller-0: 20:45:09 DEBUG juju.worker.caasunitprovisioner service for pipelines-api(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
controller-0: 20:45:09 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:45:09.135176002 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:45:09.136303109 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:45:09.139373228 +0000 UTC
application-argo-controller: 20:45:09 INFO juju.worker.logger logger worker started
application-argo-controller: 20:45:09 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-argo-controller: 20:45:09 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:45:09.156919939 +0000 UTC
application-argo-controller: 20:45:09 DEBUG juju.network no lxc bridge addresses to filter for machine
application-argo-controller: 20:45:09 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-argo-controller: 20:45:09 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-argo-controller: 20:45:09 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-argo-controller: 20:45:09 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-argo-controller: 20:45:09 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-argo-controller: 20:45:09 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-argo-controller: 20:45:09 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-argo-controller: 20:45:09 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-argo-controller: 20:45:09 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:45:09.221772647 +0000 UTC
application-argo-controller: 20:45:09 INFO juju.worker.uniter.charm downloading local:kubernetes/argo-controller-0 from API server
application-argo-controller: 20:45:09 INFO juju.downloader downloading from local:kubernetes/argo-controller-0
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner volume attachments alive: [{unit-pipelines-db-0 volume-1}], dying: [], dead: []
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner not queuing volume attachment for non-machine unit-pipelines-db-0
application-argo-controller: 20:45:09 INFO juju.downloader download complete ("local:kubernetes/argo-controller-0")
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:09 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-db
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner volumes alive: [volume-1], dying: [], dead: []
controller-0: 20:45:09 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner not queuing volume for non-machine unit-pipelines-db-0
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [{unit-pipelines-db-0 filesystem-1}], dying: [], dead: []
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner not queuing filesystem attachment for non-machine unit-pipelines-db-0
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner filesystems alive: [filesystem-1], dying: [], dead: []
controller-0: 20:45:09 DEBUG juju.worker.storageprovisioner not queuing filesystem for pipelines-db unit
controller-0: 20:45:09 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-argo-controller: 20:45:09 INFO juju.downloader download verified ("local:kubernetes/argo-controller-0")
application-argo-controller: 20:45:09 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/argo-controller-0"
application-argo-controller: 20:45:09 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/argo-controller-0"
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/argo-controller-0"
application-argo-controller: 20:45:10 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000853080}
application-argo-controller: 20:45:10 INFO juju.worker.caasoperator operator "argo-controller" started
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter juju-run listener running
application-argo-controller: 20:45:10 DEBUG juju.worker.caasoperator.remotestate got application change
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter starting uniter for "argo-controller/0"
application-argo-controller: 20:45:10 DEBUG juju.worker.caasoperator start "argo-controller/0"
application-argo-controller: 20:45:10 INFO juju.worker.caasoperator start "argo-controller/0"
application-argo-controller: 20:45:10 DEBUG juju.worker.caasoperator "argo-controller/0" started
application-argo-controller: 20:45:10 DEBUG juju.worker.leadership argo-controller/0 making initial claim for argo-controller leadership
application-argo-controller: 20:45:10 INFO juju.worker.leadership argo-controller/0 promoted to leadership of argo-controller
application-argo-controller: 20:45:10 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-argo-controller-0
application-argo-controller: 20:45:10 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-argo-controller-0/jujuc using jujud path /var/lib/juju/tools/unit-argo-controller-0/jujud
application-argo-controller: 20:45:10 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-argo-controller-0/jujud
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-argo-controller-0/run.socket <nil>}
application-argo-controller: 20:45:10 INFO juju.worker.uniter unit "argo-controller/0" started
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter juju-run listener running
application-argo-controller: 20:45:10 INFO juju.worker.uniter resuming charm install
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter.operation running operation install local:kubernetes/argo-controller-0 for argo-controller/0
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter.operation preparing operation "install local:kubernetes/argo-controller-0" for argo-controller/0
application-argo-controller: 20:45:10 INFO juju.worker.uniter.charm downloading local:kubernetes/argo-controller-0 from API server
application-argo-controller: 20:45:10 INFO juju.downloader downloading from local:kubernetes/argo-controller-0
application-argo-controller: 20:45:10 INFO juju.downloader download complete ("local:kubernetes/argo-controller-0")
application-argo-controller: 20:45:10 INFO juju.downloader download verified ("local:kubernetes/argo-controller-0")
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter.operation executing operation "install local:kubernetes/argo-controller-0" for argo-controller/0
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/argo-controller-0"
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/argo-controller-0"
controller-0: 20:45:10 DEBUG juju.worker.caasunitprovisioner operator update for argo-controller
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/argo-controller-0"
application-argo-controller: 20:45:10 DEBUG juju.worker.uniter.operation committing operation "install local:kubernetes/argo-controller-0" for argo-controller/0
application-argo-controller: 20:45:10 INFO juju.worker.uniter hooks are retried true
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got storage change for argo-controller/0: [] ok=true
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got leader settings change for argo-controller/0: ok=true
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got trust config change for argo-controller/0: ok=true, hashes=[95123ce51dfbdc5c9461940c26671759329bef2ebc0a43a8a665b91e40475a3d]
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got update status interval change for argo-controller/0: ok=true
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got address change for argo-controller/0: ok=true, hashes=[]
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got relations change for argo-controller/0: ok=true
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got action change for argo-controller/0: [] ok=true
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got unit change for argo-controller/0
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.remotestate got config change for argo-controller/0: ok=true, hashes=[57ab9473d7bd45cfa3fa23fa51ded4db320507f1b514dbf7b60f3d9dd8c125a5]
application-argo-controller: 20:45:11 INFO juju.worker.uniter found queued "install" hook
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.operation running operation run install hook for argo-controller/0
application-argo-controller: 20:45:11 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run install hook)
application-argo-controller: 20:45:11 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run install hook)
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for argo-controller/0
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.operation executing operation "run install hook" for argo-controller/0
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-argo-controller: 20:45:11 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
application-argo-controller: 20:45:11 DEBUG jujuc running hook tool "config-get" for argo-controller/0-install-5881021002247560792
controller-0: 20:45:11 DEBUG juju.worker.caasunitprovisioner units changed: true
application-argo-controller: 20:45:11 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:11 INFO unit.argo-controller/0.juju-log Reactive main running for hook install
controller-0: 20:45:11 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:11 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:11 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:11 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-argo-controller: 20:45:11 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:11 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:11 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: starting handler dispatch, 12 flags set
tracer: set flag config.changed
tracer: set flag config.changed.bucket
tracer: set flag config.changed.executor
tracer: set flag config.changed.key-prefix
tracer: set flag config.changed.kubelet-insecure
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
application-argo-controller: 20:45:11 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:11 DEBUG unit.argo-controller/0.juju-log tracer: hooks phase, 0 handlers queued
controller-0: 20:45:11 DEBUG juju.worker.caasunitprovisioner units changed: true
application-argo-controller: 20:45:11 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:11 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: main dispatch loop, 3 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
tracer: ++   queue handler reactive/argo_controller.py:21:update_image
tracer: ++   queue handler reactive/docker_resource.py:7:auto_fetch
application-argo-controller: 20:45:11 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:11 INFO unit.argo-controller/0.juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch
controller-0: 20:45:11 DEBUG juju.worker.caasunitprovisioner service for minio(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 DEBUG unit.argo-controller/0.juju-log tracer: set flag layer.docker-resource.oci-image.fetched
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: set flag layer.docker-resource.pending
tracer: ++   queue handler reactive/docker_resource.py:18:fetch
controller-0: 20:45:12 INFO juju.worker.caasprovisioner started operator for application "minio"
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: set flag layer.docker-resource.auto-fetched
tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 INFO unit.argo-controller/0.juju-log Invoking reactive handler: reactive/argo_controller.py:21:update_image
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 INFO unit.argo-controller/0.juju-log Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 INFO unit.argo-controller/0.juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 INFO unit.argo-controller/0.juju-log status-set: maintenance: fetching resource: oci-image
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "status-set" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "resource-get" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 DEBUG unit.argo-controller/0.juju-log tracer: set flag layer.docker-resource.oci-image.available
application-argo-controller: 20:45:12 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-install-5881021002247560792
application-argo-controller: 20:45:12 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: cleared flag layer.docker-resource.pending
tracer: -- dequeue handler reactive/docker_resource.py:18:fetch
application-argo-controller: 20:45:12 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.operation committing operation "run install hook" for argo-controller/0
application-argo-controller: 20:45:12 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run install hook)
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.operation running operation accept leadership for argo-controller/0
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for argo-controller/0
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for argo-controller/0
application-argo-controller: 20:45:12 INFO juju.worker.uniter found queued "leader-elected" hook
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for argo-controller/0
application-argo-controller: 20:45:12 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run leader-elected hook)
application-argo-controller: 20:45:12 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run leader-elected hook)
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for argo-controller/0
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for argo-controller/0
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-argo-controller: 20:45:12 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
controller-0: 20:45:13 DEBUG juju.worker.caasunitprovisioner operator update for minio
application-argo-controller: 20:45:13 DEBUG jujuc running hook tool "config-get" for argo-controller/0-leader-elected-7980421435456175849
application-argo-controller: 20:45:13 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-leader-elected-7980421435456175849
application-argo-controller: 20:45:13 INFO unit.argo-controller/0.juju-log Reactive main running for hook leader-elected
application-argo-controller: 20:45:13 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-leader-elected-7980421435456175849
application-argo-controller: 20:45:13 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-leader-elected-7980421435456175849
application-argo-controller: 20:45:13 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: starting handler dispatch, 10 flags set
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
controller-0: 20:45:13 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-scheduledworkflow
controller-0: 20:45:13 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:13 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:13 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:13 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-argo-controller: 20:45:13 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-leader-elected-7980421435456175849
application-argo-controller: 20:45:13 DEBUG unit.argo-controller/0.juju-log tracer: hooks phase, 0 handlers queued
application-argo-controller: 20:45:13 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-leader-elected-7980421435456175849
application-argo-controller: 20:45:13 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-argo-controller: 20:45:13 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-leader-elected-7980421435456175849
application-argo-controller: 20:45:13 INFO unit.argo-controller/0.juju-log Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
controller-0: 20:45:13 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-api
application-argo-controller: 20:45:13 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:13 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for argo-controller/0
application-argo-controller: 20:45:13 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run leader-elected hook)
application-argo-controller: 20:45:13 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:13 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:13 DEBUG juju.worker.uniter.operation running operation run config-changed hook for argo-controller/0
application-argo-controller: 20:45:13 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run config-changed hook)
application-argo-controller: 20:45:13 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run config-changed hook)
application-argo-controller: 20:45:13 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for argo-controller/0
application-argo-controller: 20:45:13 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for argo-controller/0
application-argo-controller: 20:45:13 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-argo-controller: 20:45:14 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
application-argo-controller: 20:45:14 DEBUG jujuc running hook tool "config-get" for argo-controller/0-config-changed-4958695632458519698
application-argo-controller: 20:45:14 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-config-changed-4958695632458519698
application-argo-controller: 20:45:14 INFO unit.argo-controller/0.juju-log Reactive main running for hook config-changed
application-argo-controller: 20:45:14 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-config-changed-4958695632458519698
application-argo-controller: 20:45:14 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-config-changed-4958695632458519698
application-argo-controller: 20:45:14 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: starting handler dispatch, 10 flags set
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-argo-controller: 20:45:14 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-config-changed-4958695632458519698
application-argo-controller: 20:45:14 DEBUG unit.argo-controller/0.juju-log tracer: hooks phase, 0 handlers queued
application-argo-controller: 20:45:14 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-config-changed-4958695632458519698
application-argo-controller: 20:45:14 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-argo-controller: 20:45:14 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-config-changed-4958695632458519698
application-argo-controller: 20:45:14 INFO unit.argo-controller/0.juju-log Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-argo-controller: 20:45:14 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:14 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for argo-controller/0
application-argo-controller: 20:45:14 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run config-changed hook)
application-argo-controller: 20:45:14 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:14 INFO juju.worker.uniter found queued "start" hook
application-argo-controller: 20:45:14 DEBUG juju.worker.uniter.operation running operation run start hook for argo-controller/0
application-argo-controller: 20:45:14 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run start hook)
application-argo-controller: 20:45:14 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run start hook)
application-argo-controller: 20:45:14 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for argo-controller/0
application-argo-controller: 20:45:14 DEBUG juju.worker.uniter.operation executing operation "run start hook" for argo-controller/0
application-argo-controller: 20:45:14 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-argo-controller: 20:45:14 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
application-argo-controller: 20:45:15 DEBUG jujuc running hook tool "config-get" for argo-controller/0-start-2723204803782685733
application-argo-controller: 20:45:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-start-2723204803782685733
application-argo-controller: 20:45:15 INFO unit.argo-controller/0.juju-log Reactive main running for hook start
application-argo-controller: 20:45:15 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-start-2723204803782685733
controller-0: 20:45:15 DEBUG juju.worker.caasunitprovisioner operator update for pytorch-operator
controller-0: 20:45:15 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:15 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:15 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:15 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-argo-controller: 20:45:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-start-2723204803782685733
application-argo-controller: 20:45:15 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: starting handler dispatch, 10 flags set
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-argo-controller: 20:45:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-start-2723204803782685733
application-argo-controller: 20:45:15 DEBUG unit.argo-controller/0.juju-log tracer: hooks phase, 0 handlers queued
application-argo-controller: 20:45:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-start-2723204803782685733
application-argo-controller: 20:45:15 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-argo-controller: 20:45:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-start-2723204803782685733
application-argo-controller: 20:45:15 INFO unit.argo-controller/0.juju-log Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-argo-controller: 20:45:15 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:15 DEBUG juju.worker.uniter.operation committing operation "run start hook" for argo-controller/0
application-argo-controller: 20:45:15 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run start hook)
application-argo-controller: 20:45:15 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:15 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:15 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
controller-0: 20:45:16 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:45:17 DEBUG juju.worker.caasunitprovisioner operator update for minio
controller-0: 20:45:17 DEBUG juju.worker.caasunitprovisioner service for pipelines-persistence(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
controller-0: 20:45:17 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-persistence
controller-0: 20:45:18 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-minio: 20:45:18 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-minio: 20:45:18 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=minio", "--debug"}
application-minio: 20:45:18 DEBUG juju.agent read agent config, format "2.0"
application-minio: 20:45:18 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-minio: 20:45:18 INFO juju.cmd.jujud caas operator application-minio start (2.8.6 [gc])
application-minio: 20:45:18 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:45:18.507335118 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:45:18.507404718 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:45:18.50766342 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:45:18.507773121 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-minio"
application-minio: 20:45:18 DEBUG juju.worker.introspection stats worker now serving
application-minio: 20:45:18 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:45:18.519901502 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:45:18 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-minio: 20:45:18 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-minio: 20:45:18 INFO juju.worker.apicaller [f0e819] "application-minio" successfully connected to "10.152.183.175:17070"
application-minio: 20:45:18 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:45:18.534073797 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:45:18.534815402 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:45:18.544186864 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-minio: 20:45:18 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:45:18.544866669 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:45:18.555374839 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:45:18.562463287 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:45:18.563957597 +0000 UTC
application-minio: 20:45:18 INFO juju.worker.caasupgrader abort check blocked until version event received
application-minio: 20:45:18 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-minio: 20:45:18 INFO juju.worker.caasupgrader unblocking abort check
application-minio: 20:45:18 INFO juju.worker.migrationminion migration phase is now: NONE
application-minio: 20:45:18 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-minio: 20:45:18 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:45:18.575420573 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:45:18.575575474 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:45:18.575711075 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:45:18.575754175 +0000 UTC
application-minio: 20:45:18 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:45:18.579318499 +0000 UTC
application-minio: 20:45:18 INFO juju.worker.logger logger worker started
application-minio: 20:45:18 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-minio: 20:45:18 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-minio: 20:45:18 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-minio: 20:45:18 DEBUG juju.network no lxc bridge addresses to filter for machine
application-minio: 20:45:18 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-minio: 20:45:18 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-minio: 20:45:18 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-minio: 20:45:18 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-minio: 20:45:18 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-minio: 20:45:18 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-minio: 20:45:18 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-minio: 20:45:18 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:45:18.615949544 +0000 UTC
application-minio: 20:45:18 INFO juju.worker.uniter.charm downloading local:kubernetes/minio-0 from API server
application-minio: 20:45:18 INFO juju.downloader downloading from local:kubernetes/minio-0
application-minio: 20:45:18 INFO juju.downloader download complete ("local:kubernetes/minio-0")
application-minio: 20:45:18 INFO juju.downloader download verified ("local:kubernetes/minio-0")
application-minio: 20:45:18 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/minio-0"
application-minio: 20:45:18 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/minio-0"
controller-0: 20:45:18 DEBUG juju.worker.caasunitprovisioner service for pipelines-db(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
controller-0: 20:45:18 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-minio: 20:45:19 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/minio-0"
application-minio: 20:45:19 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc00085f800}
application-minio: 20:45:19 INFO juju.worker.caasoperator operator "minio" started
application-minio: 20:45:19 DEBUG juju.worker.uniter juju-run listener running
application-minio: 20:45:19 DEBUG juju.worker.caasoperator.remotestate got application change
application-minio: 20:45:19 DEBUG juju.worker.uniter starting uniter for "minio/0"
application-minio: 20:45:19 DEBUG juju.worker.caasoperator start "minio/0"
application-minio: 20:45:19 INFO juju.worker.caasoperator start "minio/0"
application-minio: 20:45:19 DEBUG juju.worker.caasoperator "minio/0" started
application-minio: 20:45:19 DEBUG juju.worker.leadership minio/0 making initial claim for minio leadership
application-minio: 20:45:19 INFO juju.worker.leadership minio/0 promoted to leadership of minio
application-minio: 20:45:19 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-minio-0
application-minio: 20:45:19 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-minio-0/jujuc using jujud path /var/lib/juju/tools/unit-minio-0/jujud
application-minio: 20:45:19 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-minio-0/jujud
application-minio: 20:45:19 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-minio-0/run.socket <nil>}
application-minio: 20:45:19 INFO juju.worker.uniter unit "minio/0" started
application-minio: 20:45:19 DEBUG juju.worker.uniter juju-run listener running
application-minio: 20:45:19 INFO juju.worker.uniter resuming charm install
application-minio: 20:45:19 DEBUG juju.worker.uniter.operation running operation install local:kubernetes/minio-0 for minio/0
application-minio: 20:45:19 DEBUG juju.worker.uniter.operation preparing operation "install local:kubernetes/minio-0" for minio/0
application-minio: 20:45:19 INFO juju.worker.uniter.charm downloading local:kubernetes/minio-0 from API server
application-minio: 20:45:19 INFO juju.downloader downloading from local:kubernetes/minio-0
application-minio: 20:45:19 INFO juju.downloader download complete ("local:kubernetes/minio-0")
application-minio: 20:45:19 INFO juju.downloader download verified ("local:kubernetes/minio-0")
application-minio: 20:45:19 DEBUG juju.worker.uniter.operation executing operation "install local:kubernetes/minio-0" for minio/0
application-minio: 20:45:19 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/minio-0"
application-minio: 20:45:19 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/minio-0"
controller-0: 20:45:19 DEBUG juju.worker.caasunitprovisioner operator update for seldon-core
controller-0: 20:45:19 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:19 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:19 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:19 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:19 DEBUG juju.worker.caasunitprovisioner operator update for minio
application-minio: 20:45:19 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/minio-0"
application-minio: 20:45:19 DEBUG juju.worker.uniter.operation committing operation "install local:kubernetes/minio-0" for minio/0
application-minio: 20:45:19 INFO juju.worker.uniter hooks are retried true
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got config change for minio/0: ok=true, hashes=[8b62a1d74d8d9f7018b381d8ba50d6738b7534976254aceb687ab2dc15194ff3]
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got address change for minio/0: ok=true, hashes=[]
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got trust config change for minio/0: ok=true, hashes=[256ee7351c49e1e2a2e489abc320b6dc0380684d3dc93935982f65d3c394c354]
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got relations change for minio/0: ok=true
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got storage change for minio/0: [minio-data/0] ok=true
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got action change for minio/0: [] ok=true
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got leader settings change for minio/0: ok=true
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got unit change for minio/0
application-minio: 20:45:20 DEBUG juju.worker.uniter.remotestate got update status interval change for minio/0: ok=true
application-minio: 20:45:20 INFO juju.worker.uniter found queued "install" hook
application-minio: 20:45:20 DEBUG juju.worker.uniter.operation running operation run install hook for minio/0
application-minio: 20:45:20 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run install hook)
application-minio: 20:45:20 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run install hook)
application-minio: 20:45:20 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for minio/0
application-minio: 20:45:20 DEBUG juju.worker.uniter.operation executing operation "run install hook" for minio/0
application-minio: 20:45:20 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-minio: 20:45:20 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
controller-0: 20:45:20 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:<nil>}
controller-0: 20:45:20 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-minio: 20:45:20 DEBUG jujuc running hook tool "config-get" for minio/0-install-6585021247932194050
application-minio: 20:45:20 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:20 INFO unit.minio/0.juju-log Reactive main running for hook install
application-minio: 20:45:20 DEBUG jujuc running hook tool "relation-ids" for minio/0-install-6585021247932194050
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 DEBUG unit.minio/0.juju-log tracer>
tracer: starting handler dispatch, 7 flags set
tracer: set flag config.changed
tracer: set flag config.changed.access-key
tracer: set flag config.changed.port
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 DEBUG unit.minio/0.juju-log tracer: hooks phase, 0 handlers queued
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 DEBUG unit.minio/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/docker_resource.py:7:auto_fetch
tracer: ++   queue handler reactive/minio.py:31:update_image
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:31:update_image
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 DEBUG unit.minio/0.juju-log tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 DEBUG unit.minio/0.juju-log tracer>
tracer: set flag layer.docker-resource.pending
tracer: ++   queue handler reactive/docker_resource.py:18:fetch
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 DEBUG unit.minio/0.juju-log tracer>
tracer: set flag layer.docker-resource.auto-fetched
tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch
application-argo-controller: 20:45:21 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:45:21 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:21 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-argo-controller: 20:45:21 DEBUG juju.worker.uniter.remotestate got relations change for argo-controller/0: ok=true
controller-0: 20:45:21 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:45:21 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:45:21 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:45:21 DEBUG juju.worker.caasunitprovisioner units changed: true
application-minio: 20:45:21 DEBUG juju.worker.caasoperator.remotestate got application change
application-minio: 20:45:21 DEBUG juju.worker.uniter.remotestate got relations change for minio/0: ok=true
controller-0: 20:45:21 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:21 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-argo-controller: 20:45:21 INFO juju.worker.uniter.relation joining relation "argo-controller:minio minio:minio"
application-minio: 20:45:21 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:21 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch
application-minio: 20:45:21 DEBUG juju.worker.caasoperator.remotestate got application change
application-minio: 20:45:21 DEBUG juju.worker.uniter.remotestate got relations change for minio/0: ok=true
application-argo-controller: 20:45:21 INFO juju.worker.uniter.relation joined relation "argo-controller:minio minio:minio"
application-argo-controller: 20:45:21 DEBUG juju.worker.uniter.operation running operation run relation-created (0; app: minio) hook for argo-controller/0
application-argo-controller: 20:45:21 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run relation-created (0; app: minio) hook)
application-argo-controller: 20:45:21 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run relation-created (0; app: minio) hook)
application-argo-controller: 20:45:21 DEBUG juju.worker.uniter.operation preparing operation "run relation-created (0; app: minio) hook" for argo-controller/0
application-minio: 20:45:21 DEBUG juju.worker.uniter.remotestate got a relation units change for minio/0 : {0 {map[argo-controller/0:{0}] map[] []}}
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 INFO unit.minio/0.juju-log status-set: maintenance: fetching resource: oci-image
application-argo-controller: 20:45:22 DEBUG juju.worker.uniter.operation executing operation "run relation-created (0; app: minio) hook" for argo-controller/0
application-argo-controller: 20:45:22 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-created hook
application-argo-controller: 20:45:22 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
application-argo-controller: 20:45:22 INFO juju.worker.uniter.operation skipped "minio-relation-created" hook (missing)
application-argo-controller: 20:45:22 DEBUG juju.worker.uniter.operation committing operation "run relation-created (0; app: minio) hook" for argo-controller/0
application-argo-controller: 20:45:22 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run relation-created (0; app: minio) hook)
application-argo-controller: 20:45:22 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:22 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:22 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-minio: 20:45:22 DEBUG jujuc running hook tool "status-set" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG jujuc running hook tool "resource-get" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG unit.minio/0.juju-log tracer: set flag layer.docker-resource.oci-image.available
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG unit.minio/0.juju-log tracer>
tracer: cleared flag layer.docker-resource.pending
tracer: -- dequeue handler reactive/docker_resource.py:18:fetch
controller-0: 20:45:22 DEBUG juju.worker.caasunitprovisioner units changed: true
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:36:start_charm
application-minio: 20:45:22 DEBUG jujuc running hook tool "is-leader" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 INFO unit.minio/0.juju-log status-set: maintenance: configuring container
application-minio: 20:45:22 DEBUG jujuc running hook tool "status-set" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG jujuc running hook tool "pod-spec-set" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG jujuc running hook tool "pod-spec-set" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 INFO unit.minio/0.juju-log status-set: maintenance: creating container
application-minio: 20:45:22 DEBUG jujuc running hook tool "status-set" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 DEBUG unit.minio/0.juju-log tracer>
tracer: set flag charm.started
tracer: -- dequeue handler reactive/minio.py:36:start_charm
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:13:charm_ready
controller-0: 20:45:22 DEBUG juju.worker.caasunitprovisioner units changed: true
application-minio: 20:45:22 DEBUG jujuc running hook tool "juju-log" for minio/0-install-6585021247932194050
application-minio: 20:45:22 INFO unit.minio/0.juju-log status-set: active: 
application-minio: 20:45:23 DEBUG jujuc running hook tool "status-set" for minio/0-install-6585021247932194050
application-minio: 20:45:23 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation committing operation "run install hook" for minio/0
application-minio: 20:45:23 DEBUG juju.machinelock machine lock released for minio/0 uniter (run install hook)
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:23 INFO juju.worker.uniter.relation joining relation "pipelines-api:minio minio:minio"
application-minio: 20:45:23 INFO juju.worker.uniter.relation joined relation "pipelines-api:minio minio:minio"
application-minio: 20:45:23 INFO juju.worker.uniter.relation joining relation "argo-controller:minio minio:minio"
application-minio: 20:45:23 INFO juju.worker.uniter.relation joined relation "argo-controller:minio minio:minio"
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation running operation run relation-created (3; app: pipelines-api) hook for minio/0
application-minio: 20:45:23 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run relation-created (3; app: pipelines-api) hook)
application-minio: 20:45:23 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run relation-created (3; app: pipelines-api) hook)
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation preparing operation "run relation-created (3; app: pipelines-api) hook" for minio/0
application-argo-controller: 20:45:23 DEBUG juju.worker.uniter.remotestate got a relation units change for argo-controller/0 : {0 {map[minio/0:{0}] map[] []}}
application-argo-controller: 20:45:23 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:23 DEBUG juju.worker.uniter.operation running operation run relation-joined (0; unit: minio/0) hook for argo-controller/0
application-argo-controller: 20:45:23 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run relation-joined (0; unit: minio/0) hook)
application-argo-controller: 20:45:23 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run relation-joined (0; unit: minio/0) hook)
application-argo-controller: 20:45:23 DEBUG juju.worker.uniter.operation preparing operation "run relation-joined (0; unit: minio/0) hook" for argo-controller/0
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation executing operation "run relation-created (3; app: pipelines-api) hook" for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-created hook
application-argo-controller: 20:45:23 DEBUG juju.worker.uniter.operation executing operation "run relation-joined (0; unit: minio/0) hook" for argo-controller/0
application-argo-controller: 20:45:23 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-joined hook
application-minio: 20:45:23 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-minio: 20:45:23 INFO juju.worker.uniter.operation skipped "minio-relation-created" hook (missing)
application-argo-controller: 20:45:23 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation committing operation "run relation-created (3; app: pipelines-api) hook" for minio/0
application-minio: 20:45:23 DEBUG juju.machinelock machine lock released for minio/0 uniter (run relation-created (3; app: pipelines-api) hook)
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation running operation run relation-created (0; app: argo-controller) hook for minio/0
application-minio: 20:45:23 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run relation-created (0; app: argo-controller) hook)
application-minio: 20:45:23 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run relation-created (0; app: argo-controller) hook)
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation preparing operation "run relation-created (0; app: argo-controller) hook" for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation executing operation "run relation-created (0; app: argo-controller) hook" for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-minio: 20:45:23 INFO juju.worker.uniter.operation skipped "minio-relation-created" hook (missing)
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation committing operation "run relation-created (0; app: argo-controller) hook" for minio/0
application-minio: 20:45:23 DEBUG juju.machinelock machine lock released for minio/0 uniter (run relation-created (0; app: argo-controller) hook)
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation running operation accept leadership for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for minio/0
application-minio: 20:45:23 INFO juju.worker.uniter found queued "leader-elected" hook
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for minio/0
application-minio: 20:45:23 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run leader-elected hook)
application-minio: 20:45:23 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run leader-elected hook)
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for minio/0
application-minio: 20:45:23 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-minio: 20:45:23 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-argo-controller: 20:45:23 DEBUG jujuc running hook tool "config-get" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:23 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:23 INFO unit.argo-controller/0.juju-log minio:0: Reactive main running for hook minio-relation-joined
application-argo-controller: 20:45:23 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-minio-relation-joined-6180520765098660594
application-minio: 20:45:23 DEBUG jujuc running hook tool "config-get" for minio/0-leader-elected-7627157263022500986
application-argo-controller: 20:45:23 DEBUG jujuc running hook tool "relation-list" for argo-controller/0-minio-relation-joined-6180520765098660594
application-minio: 20:45:23 DEBUG jujuc running hook tool "juju-log" for minio/0-leader-elected-7627157263022500986
application-minio: 20:45:23 INFO unit.minio/0.juju-log Reactive main running for hook leader-elected
application-argo-controller: 20:45:23 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:23 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: set flag endpoint.minio.joined
tracer: ++   queue handler reactive/argo_controller.py:26:start_charm
application-minio: 20:45:24 DEBUG jujuc running hook tool "relation-ids" for minio/0-leader-elected-7627157263022500986
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "relation-get" for argo-controller/0-minio-relation-joined-6180520765098660594
application-minio: 20:45:24 DEBUG jujuc running hook tool "relation-list" for minio/0-leader-elected-7627157263022500986
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:24 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: starting handler dispatch, 11 flags set
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:24 DEBUG jujuc running hook tool "relation-list" for minio/0-leader-elected-7627157263022500986
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:24 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: hooks phase, 0 handlers queued
application-minio: 20:45:24 DEBUG jujuc running hook tool "juju-log" for minio/0-leader-elected-7627157263022500986
application-minio: 20:45:24 DEBUG unit.minio/0.juju-log tracer>
tracer: starting handler dispatch, 8 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:24 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/argo_controller.py:26:start_charm
application-minio: 20:45:24 DEBUG jujuc running hook tool "juju-log" for minio/0-leader-elected-7627157263022500986
application-minio: 20:45:24 DEBUG unit.minio/0.juju-log tracer: hooks phase, 0 handlers queued
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:24 INFO unit.argo-controller/0.juju-log minio:0: Invoking reactive handler: reactive/argo_controller.py:26:start_charm
application-minio: 20:45:24 DEBUG jujuc running hook tool "juju-log" for minio/0-leader-elected-7627157263022500986
application-minio: 20:45:24 DEBUG unit.minio/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "is-leader" for argo-controller/0-minio-relation-joined-6180520765098660594
application-minio: 20:45:24 DEBUG jujuc running hook tool "juju-log" for minio/0-leader-elected-7627157263022500986
application-minio: 20:45:24 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:13:charm_ready
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:24 INFO unit.argo-controller/0.juju-log minio:0: status-set: maintenance: configuring container
application-minio: 20:45:24 DEBUG jujuc running hook tool "juju-log" for minio/0-leader-elected-7627157263022500986
application-minio: 20:45:24 INFO unit.minio/0.juju-log status-set: active: 
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "status-set" for argo-controller/0-minio-relation-joined-6180520765098660594
application-minio: 20:45:24 DEBUG jujuc running hook tool "status-set" for minio/0-leader-elected-7627157263022500986
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:24 INFO unit.argo-controller/0.juju-log minio:0: status-set: waiting: Waiting for full minio relation.
application-argo-controller: 20:45:24 DEBUG jujuc running hook tool "status-set" for argo-controller/0-minio-relation-joined-6180520765098660594
application-argo-controller: 20:45:24 DEBUG unit.argo-controller/0.minio-relation-joined Requires relation
application-argo-controller: 20:45:24 DEBUG unit.argo-controller/0.minio-relation-joined minio:0
application-argo-controller: 20:45:24 DEBUG unit.argo-controller/0.minio-relation-joined requires unit
application-argo-controller: 20:45:24 DEBUG unit.argo-controller/0.minio-relation-joined {}
controller-0: 20:45:24 DEBUG juju.worker.caasunitprovisioner units changed: true
application-minio: 20:45:24 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-minio: 20:45:24 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for minio/0
application-minio: 20:45:24 DEBUG juju.machinelock machine lock released for minio/0 uniter (run leader-elected hook)
application-minio: 20:45:24 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:24 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:0 Life:alive Attached:false Location:}
application-minio: 20:45:24 DEBUG juju.worker.uniter.storage still pending [storage-minio-data-0]
application-minio: 20:45:24 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:45:24 DEBUG juju.worker.uniter.operation running operation run config-changed hook for minio/0
application-minio: 20:45:24 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run config-changed hook)
application-minio: 20:45:24 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run config-changed hook)
application-minio: 20:45:24 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for minio/0
application-minio: 20:45:24 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for minio/0
application-minio: 20:45:24 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-minio: 20:45:24 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-argo-controller: 20:45:24 INFO juju.worker.uniter.operation ran "minio-relation-joined" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:24 DEBUG juju.worker.uniter.operation committing operation "run relation-joined (0; unit: minio/0) hook" for argo-controller/0
application-argo-controller: 20:45:24 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run relation-joined (0; unit: minio/0) hook)
application-argo-controller: 20:45:24 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:24 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:24 DEBUG juju.worker.uniter.operation running operation run relation-changed (0; unit: minio/0) hook for argo-controller/0
application-argo-controller: 20:45:24 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:24 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:24 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
controller-0: 20:45:24 INFO juju.worker.caasprovisioner started operator for application "pipelines-api"
application-argo-controller: 20:45:24 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
application-argo-controller: 20:45:24 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-changed hook
application-argo-controller: 20:45:24 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
application-minio: 20:45:24 DEBUG jujuc running hook tool "config-get" for minio/0-config-changed-2773076468762238644
application-minio: 20:45:25 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-2773076468762238644
application-minio: 20:45:25 INFO unit.minio/0.juju-log Reactive main running for hook config-changed
application-minio: 20:45:25 DEBUG jujuc running hook tool "relation-ids" for minio/0-config-changed-2773076468762238644
application-argo-controller: 20:45:25 DEBUG jujuc running hook tool "config-get" for argo-controller/0-minio-relation-changed-1882700875257187870
application-minio: 20:45:25 DEBUG jujuc running hook tool "relation-list" for minio/0-config-changed-2773076468762238644
application-argo-controller: 20:45:25 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:25 INFO unit.argo-controller/0.juju-log minio:0: Reactive main running for hook minio-relation-changed
application-minio: 20:45:25 DEBUG jujuc running hook tool "relation-list" for minio/0-config-changed-2773076468762238644
application-argo-controller: 20:45:25 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-minio-relation-changed-1882700875257187870
application-minio: 20:45:25 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-2773076468762238644
application-minio: 20:45:25 DEBUG unit.minio/0.juju-log tracer>
tracer: starting handler dispatch, 8 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-argo-controller: 20:45:25 DEBUG jujuc running hook tool "relation-list" for argo-controller/0-minio-relation-changed-1882700875257187870
application-minio: 20:45:25 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-2773076468762238644
application-minio: 20:45:25 DEBUG unit.minio/0.juju-log tracer: hooks phase, 0 handlers queued
application-argo-controller: 20:45:25 DEBUG jujuc running hook tool "relation-get" for argo-controller/0-minio-relation-changed-1882700875257187870
application-minio: 20:45:25 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-2773076468762238644
application-minio: 20:45:25 DEBUG unit.minio/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
application-argo-controller: 20:45:25 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:25 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: starting handler dispatch, 11 flags set
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:25 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-2773076468762238644
application-minio: 20:45:25 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:13:charm_ready
application-argo-controller: 20:45:25 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:25 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: hooks phase, 0 handlers queued
controller-0: 20:45:25 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-api
application-argo-controller: 20:45:25 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:25 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/argo_controller.py:26:start_charm
application-argo-controller: 20:45:26 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:26 INFO unit.argo-controller/0.juju-log minio:0: Invoking reactive handler: reactive/argo_controller.py:26:start_charm
application-argo-controller: 20:45:26 DEBUG jujuc running hook tool "is-leader" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:26 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:26 INFO unit.argo-controller/0.juju-log minio:0: status-set: maintenance: configuring container
application-argo-controller: 20:45:26 DEBUG jujuc running hook tool "status-set" for argo-controller/0-minio-relation-changed-1882700875257187870
application-minio: 20:45:26 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-2773076468762238644
application-minio: 20:45:26 INFO unit.minio/0.juju-log status-set: active: 
application-minio: 20:45:26 DEBUG jujuc running hook tool "status-set" for minio/0-config-changed-2773076468762238644
application-argo-controller: 20:45:27 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:27 INFO unit.argo-controller/0.juju-log minio:0: status-set: waiting: Waiting for full minio relation.
application-argo-controller: 20:45:27 DEBUG jujuc running hook tool "status-set" for argo-controller/0-minio-relation-changed-1882700875257187870
application-argo-controller: 20:45:27 DEBUG unit.argo-controller/0.minio-relation-changed Requires relation
application-argo-controller: 20:45:27 DEBUG unit.argo-controller/0.minio-relation-changed minio:0
application-argo-controller: 20:45:27 DEBUG unit.argo-controller/0.minio-relation-changed requires unit
application-argo-controller: 20:45:27 DEBUG unit.argo-controller/0.minio-relation-changed {}
application-minio: 20:45:27 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:28 INFO juju.worker.uniter.operation ran "minio-relation-changed" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:28 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
controller-0: 20:45:28 DEBUG juju.worker.caasunitprovisioner ensured deployment for minio for 1 units
application-minio: 20:45:28 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for minio/0
controller-0: 20:45:28 DEBUG juju.worker.caasunitprovisioner service for tf-job-operator(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:45:29 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pipelines-api: 20:45:29 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-api", "--debug"}
application-pipelines-api: 20:45:29 DEBUG juju.agent read agent config, format "2.0"
application-pipelines-api: 20:45:29 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pipelines-api: 20:45:29 INFO juju.cmd.jujud caas operator application-pipelines-api start (2.8.6 [gc])
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:45:29.157932809 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:45:29.15804321 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:45:29.158106411 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-api: 20:45:29 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pipelines-api"
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:45:29.160674928 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:45:29.176388131 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:45:29.176554632 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-api: 20:45:29 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-api: 20:45:29 INFO juju.worker.apicaller [f0e819] "application-pipelines-api" successfully connected to "10.152.183.175:17070"
application-pipelines-api: 20:45:29 DEBUG juju.api RPC connection died
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully
application-pipelines-api: 20:45:29 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-api: 20:45:29 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-api: 20:45:29 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-api: 20:45:29 INFO juju.worker.apicaller [f0e819] "application-pipelines-api" successfully connected to "10.152.183.175:17070"
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:45:29.238061639 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:45:29.2473367 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:45:29.247526601 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:45:29.248290306 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:45:29.249004411 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:45:29.254680848 +0000 UTC
application-pipelines-api: 20:45:29 INFO juju.worker.migrationminion migration phase is now: NONE
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:45:29.264936416 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:45:29.265085117 +0000 UTC
application-pipelines-api: 20:45:29 INFO juju.worker.logger logger worker started
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:45:29.265436319 +0000 UTC
application-pipelines-api: 20:45:29 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pipelines-api: 20:45:29 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pipelines-api: 20:45:29 INFO juju.worker.caasupgrader unblocking abort check
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:45:29.266136824 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:45:29.27012545 +0000 UTC
application-pipelines-api: 20:45:29 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-api: 20:45:29 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-api: 20:45:29 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pipelines-api: 20:45:29 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-api: 20:45:29 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-api: 20:45:29 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pipelines-api: 20:45:29 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pipelines-api: 20:45:29 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
controller-0: 20:45:29 DEBUG juju.worker.caasunitprovisioner service for pipelines-scheduledworkflow(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
controller-0: 20:45:29 DEBUG juju.worker.caasunitprovisioner operator update for tf-job-operator
application-pipelines-api: 20:45:29 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
controller-0: 20:45:29 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-api: 20:45:29 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-minio: 20:45:29 DEBUG juju.machinelock machine lock released for minio/0 uniter (run config-changed hook)
application-minio: 20:45:29 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:29 INFO juju.worker.uniter found queued "start" hook
application-minio: 20:45:29 DEBUG juju.worker.uniter.operation running operation run start hook for minio/0
application-minio: 20:45:29 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run start hook)
application-minio: 20:45:29 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run start hook)
application-minio: 20:45:29 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for minio/0
controller-0: 20:45:29 DEBUG juju.worker.caasunitprovisioner service for seldon-core(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:45:29 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:45:29.723335044 +0000 UTC
application-argo-controller: 20:45:29 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:29 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:29 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:45:29 INFO juju.worker.uniter.charm downloading local:kubernetes/pipelines-api-0 from API server
application-argo-controller: 20:45:29 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-api: 20:45:29 INFO juju.downloader downloading from local:kubernetes/pipelines-api-0
controller-0: 20:45:29 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-minio: 20:45:29 DEBUG juju.worker.uniter.operation executing operation "run start hook" for minio/0
application-minio: 20:45:29 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-pipelines-api: 20:45:29 INFO juju.downloader download complete ("local:kubernetes/pipelines-api-0")
application-minio: 20:45:29 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-pipelines-api: 20:45:29 INFO juju.downloader download verified ("local:kubernetes/pipelines-api-0")
application-pipelines-api: 20:45:29 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-api-0"
application-pipelines-api: 20:45:29 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-api-0"
application-minio: 20:45:30 DEBUG jujuc running hook tool "config-get" for minio/0-start-7870210387414426817
application-minio: 20:45:30 DEBUG jujuc running hook tool "juju-log" for minio/0-start-7870210387414426817
application-minio: 20:45:30 INFO unit.minio/0.juju-log Reactive main running for hook start
controller-0: 20:45:30 DEBUG juju.worker.caasunitprovisioner service for pytorch-operator(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-minio: 20:45:30 DEBUG jujuc running hook tool "relation-ids" for minio/0-start-7870210387414426817
application-minio: 20:45:30 DEBUG jujuc running hook tool "relation-list" for minio/0-start-7870210387414426817
application-minio: 20:45:30 DEBUG jujuc running hook tool "relation-list" for minio/0-start-7870210387414426817
application-minio: 20:45:30 DEBUG jujuc running hook tool "juju-log" for minio/0-start-7870210387414426817
application-minio: 20:45:30 DEBUG unit.minio/0.juju-log tracer>
tracer: starting handler dispatch, 8 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:30 DEBUG jujuc running hook tool "juju-log" for minio/0-start-7870210387414426817
application-minio: 20:45:30 DEBUG unit.minio/0.juju-log tracer: hooks phase, 0 handlers queued
application-minio: 20:45:30 DEBUG jujuc running hook tool "juju-log" for minio/0-start-7870210387414426817
application-minio: 20:45:30 DEBUG unit.minio/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-api-0"
application-pipelines-api: 20:45:30 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000823680}
application-pipelines-api: 20:45:30 INFO juju.worker.caasoperator operator "pipelines-api" started
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter juju-run listener running
application-minio: 20:45:30 DEBUG jujuc running hook tool "juju-log" for minio/0-start-7870210387414426817
application-minio: 20:45:30 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:13:charm_ready
application-pipelines-api: 20:45:30 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter starting uniter for "pipelines-api/0"
application-pipelines-api: 20:45:30 DEBUG juju.worker.caasoperator start "pipelines-api/0"
application-pipelines-api: 20:45:30 INFO juju.worker.caasoperator start "pipelines-api/0"
application-pipelines-api: 20:45:30 DEBUG juju.worker.caasoperator "pipelines-api/0" started
application-pipelines-api: 20:45:30 DEBUG juju.worker.leadership pipelines-api/0 making initial claim for pipelines-api leadership
application-pipelines-api: 20:45:30 INFO juju.worker.leadership pipelines-api/0 promoted to leadership of pipelines-api
application-pipelines-api: 20:45:30 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-api-0
application-pipelines-api: 20:45:30 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pipelines-api-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-api-0/jujud
application-pipelines-api: 20:45:30 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pipelines-api-0/jujud
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-api-0/run.socket <nil>}
application-pipelines-api: 20:45:30 INFO juju.worker.uniter unit "pipelines-api/0" started
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-api: 20:45:30 INFO juju.worker.uniter resuming charm install
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter.operation running operation install local:kubernetes/pipelines-api-0 for pipelines-api/0
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter.operation preparing operation "install local:kubernetes/pipelines-api-0" for pipelines-api/0
application-pipelines-api: 20:45:30 INFO juju.worker.uniter.charm downloading local:kubernetes/pipelines-api-0 from API server
application-pipelines-api: 20:45:30 INFO juju.downloader downloading from local:kubernetes/pipelines-api-0
application-minio: 20:45:30 DEBUG jujuc running hook tool "juju-log" for minio/0-start-7870210387414426817
application-minio: 20:45:30 INFO unit.minio/0.juju-log status-set: active: 
application-pipelines-api: 20:45:30 INFO juju.downloader download complete ("local:kubernetes/pipelines-api-0")
application-pipelines-api: 20:45:30 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-api: 20:45:30 INFO juju.downloader download verified ("local:kubernetes/pipelines-api-0")
application-minio: 20:45:30 DEBUG jujuc running hook tool "status-set" for minio/0-start-7870210387414426817
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter.operation executing operation "install local:kubernetes/pipelines-api-0" for pipelines-api/0
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-api-0"
application-pipelines-api: 20:45:30 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-api-0"
controller-0: 20:45:30 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-api
application-minio: 20:45:30 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-minio: 20:45:30 DEBUG juju.worker.uniter.operation committing operation "run start hook" for minio/0
application-minio: 20:45:30 DEBUG juju.machinelock machine lock released for minio/0 uniter (run start hook)
application-minio: 20:45:30 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:30 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:0 Life:alive Attached:false Location:}
application-minio: 20:45:30 DEBUG juju.worker.uniter.storage still pending [storage-minio-data-0]
application-minio: 20:45:30 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:45:30 DEBUG juju.worker.uniter.operation running operation run relation-joined (0; unit: argo-controller/0) hook for minio/0
application-minio: 20:45:30 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run relation-joined (0; unit: argo-controller/0) hook)
application-minio: 20:45:30 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run relation-joined (0; unit: argo-controller/0) hook)
application-minio: 20:45:30 DEBUG juju.worker.uniter.operation preparing operation "run relation-joined (0; unit: argo-controller/0) hook" for minio/0
application-minio: 20:45:30 DEBUG juju.worker.uniter.operation executing operation "run relation-joined (0; unit: argo-controller/0) hook" for minio/0
application-minio: 20:45:30 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-joined hook
application-minio: 20:45:30 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-minio: 20:45:31 DEBUG jujuc running hook tool "config-get" for minio/0-minio-relation-joined-4961600144343586317
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-api-0"
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.operation committing operation "install local:kubernetes/pipelines-api-0" for pipelines-api/0
application-pipelines-api: 20:45:31 INFO juju.worker.uniter hooks are retried true
application-minio: 20:45:31 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:31 INFO unit.minio/0.juju-log minio:0: Reactive main running for hook minio-relation-joined
application-minio: 20:45:31 DEBUG juju.worker.uniter.remotestate got address change for minio/0: ok=true, hashes=[5379216ebc32c12dc98a35b1910a26fc2cb86640750d873867c67287b42a70ee]
application-minio: 20:45:31 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got config change for pipelines-api/0: ok=true, hashes=[cdeba69efca8b193773f8bead59c96ed23411c32f7e971851e3922e54e563574]
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got unit change for pipelines-api/0
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got action change for pipelines-api/0: [] ok=true
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got relations change for pipelines-api/0: ok=true
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got address change for pipelines-api/0: ok=true, hashes=[]
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got trust config change for pipelines-api/0: ok=true, hashes=[7f9884bc998df3cb2761f63f838f8c09c8a7ce0ea62a095fae071ad6e1a23413]
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got update status interval change for pipelines-api/0: ok=true
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got leader settings change for pipelines-api/0: ok=true
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.remotestate got storage change for pipelines-api/0: [] ok=true
application-pipelines-api: 20:45:31 INFO juju.worker.uniter found queued "install" hook
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.operation running operation run install hook for pipelines-api/0
application-pipelines-api: 20:45:31 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run install hook)
application-pipelines-api: 20:45:31 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run install hook)
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for pipelines-api/0
application-minio: 20:45:31 DEBUG jujuc running hook tool "relation-ids" for minio/0-minio-relation-joined-4961600144343586317
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.operation executing operation "run install hook" for pipelines-api/0
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-pipelines-api: 20:45:31 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
controller-0: 20:45:31 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-minio: 20:45:31 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:31 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:31 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:31 DEBUG unit.minio/0.juju-log minio:0: tracer>
tracer: set flag endpoint.minio.joined
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-minio: 20:45:31 DEBUG juju.worker.caasoperator.remotestate got application change
application-minio: 20:45:31 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:31 DEBUG juju.worker.uniter.remotestate storage attachment change for minio/0: {storage-minio-data-0 {2 alive true /data}}
controller-0: 20:45:31 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-api: 20:45:31 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-install-8154239330009868646
controller-0: 20:45:31 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc002617740}
application-minio: 20:45:32 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:32 DEBUG unit.minio/0.juju-log minio:0: tracer>
tracer: starting handler dispatch, 9 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-api: 20:45:32 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:32 INFO unit.pipelines-api/0.juju-log Reactive main running for hook install
application-minio: 20:45:32 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:32 DEBUG unit.minio/0.juju-log minio:0: tracer: hooks phase, 0 handlers queued
application-minio: 20:45:32 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:32 DEBUG unit.minio/0.juju-log minio:0: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-minio: 20:45:32 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:32 INFO unit.minio/0.juju-log minio:0: Invoking reactive handler: reactive/minio.py:13:charm_ready
application-pipelines-api: 20:45:32 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-install-8154239330009868646
application-minio: 20:45:32 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:32 INFO unit.minio/0.juju-log minio:0: Invoking reactive handler: reactive/minio.py:18:configure_minio
application-pipelines-api: 20:45:32 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-install-8154239330009868646
application-minio: 20:45:32 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-joined-4961600144343586317
application-pipelines-api: 20:45:32 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-install-8154239330009868646
application-minio: 20:45:32 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-joined-4961600144343586317
application-pipelines-api: 20:45:32 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-install-8154239330009868646
application-minio: 20:45:32 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-joined-4961600144343586317
application-pipelines-api: 20:45:32 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-install-8154239330009868646
application-minio: 20:45:32 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-joined-4961600144343586317
application-pipelines-api: 20:45:32 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:32 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: starting handler dispatch, 13 flags set
tracer: set flag config.changed
tracer: set flag config.changed.grpc-port
tracer: set flag config.changed.http-port
tracer: set flag config.changed.minio-access-key
tracer: set flag config.changed.minio-bucket-name
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
application-minio: 20:45:33 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:33 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:33 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:33 INFO unit.minio/0.juju-log minio:0: status-set: active: 
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 DEBUG unit.pipelines-api/0.juju-log tracer: hooks phase, 0 handlers queued
application-minio: 20:45:33 DEBUG jujuc running hook tool "status-set" for minio/0-minio-relation-joined-4961600144343586317
application-minio: 20:45:33 DEBUG unit.minio/0.minio-relation-joined Provides
application-minio: 20:45:33 DEBUG unit.minio/0.minio-relation-joined YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:33 DEBUG unit.minio/0.minio-relation-joined minio:0
application-minio: 20:45:33 DEBUG unit.minio/0.minio-relation-joined Provides
application-minio: 20:45:33 DEBUG unit.minio/0.minio-relation-joined YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:33 DEBUG unit.minio/0.minio-relation-joined minio:3
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: main dispatch loop, 7 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql
tracer: ++   queue handler reactive/docker_resource.py:7:auto_fetch
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: cleared flag config.changed
tracer: -- dequeue handler reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 DEBUG unit.pipelines-api/0.juju-log tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:33 INFO juju.worker.uniter.operation ran "minio-relation-joined" hook (via explicit, bespoke hook script)
application-minio: 20:45:33 DEBUG juju.worker.uniter.operation committing operation "run relation-joined (0; unit: argo-controller/0) hook" for minio/0
application-pipelines-api: 20:45:33 DEBUG juju.worker.uniter.remotestate got a relation units change for pipelines-api/0 : {3 {map[minio/0:{1}] map[] []}}
application-argo-controller: 20:45:33 DEBUG juju.worker.uniter.remotestate got a relation units change for argo-controller/0 : {0 {map[minio/0:{1}] map[] []}}
application-argo-controller: 20:45:33 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:33 DEBUG juju.worker.uniter.operation running operation run relation-changed (0; unit: minio/0) hook for argo-controller/0
application-argo-controller: 20:45:33 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:33 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:33 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: set flag layer.docker-resource.pending
tracer: ++   queue handler reactive/docker_resource.py:18:fetch
application-argo-controller: 20:45:33 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
application-argo-controller: 20:45:33 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-changed hook
application-minio: 20:45:33 DEBUG juju.machinelock machine lock released for minio/0 uniter (run relation-joined (0; unit: argo-controller/0) hook)
application-minio: 20:45:33 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:33 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:45:33 DEBUG juju.worker.uniter.operation running operation run storage-attached (minio-data/0) hook for minio/0
application-minio: 20:45:33 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run storage-attached (minio-data/0) hook)
application-minio: 20:45:33 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run storage-attached (minio-data/0) hook)
application-minio: 20:45:33 DEBUG juju.worker.uniter.operation preparing operation "run storage-attached (minio-data/0) hook" for minio/0
application-argo-controller: 20:45:33 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
application-minio: 20:45:33 DEBUG juju.worker.uniter.operation executing operation "run storage-attached (minio-data/0) hook" for minio/0
application-minio: 20:45:33 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-data-storage-attached hook
application-minio: 20:45:33 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
controller-0: 20:45:33 DEBUG juju.worker.caasunitprovisioner service for minio(workload): &{Id:0f6fc8e3-48a5-427a-a263-04533dd18de2 Addresses:[local-cloud:10.152.183.102] Scale:0xc002864d00 Generation:0xc002864d08 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: set flag layer.docker-resource.auto-fetched
tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:45:33 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:33 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "config-get" for argo-controller/0-minio-relation-changed-6493246157928624429
application-minio: 20:45:34 DEBUG jujuc running hook tool "config-get" for minio/0-minio-data-storage-attached-3844453286847804765
application-pipelines-api: 20:45:34 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:34 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api
application-minio: 20:45:34 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:34 INFO unit.minio/0.juju-log Reactive main running for hook minio-data-storage-attached
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:34 INFO unit.argo-controller/0.juju-log minio:0: Reactive main running for hook minio-relation-changed
application-pipelines-api: 20:45:34 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:34 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-minio: 20:45:34 DEBUG jujuc running hook tool "relation-ids" for minio/0-minio-data-storage-attached-3844453286847804765
application-pipelines-api: 20:45:34 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:34 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-minio-relation-changed-6493246157928624429
application-minio: 20:45:34 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:34 DEBUG juju.worker.caasoperator trigger running status for caas unit minio/0
application-minio: 20:45:34 DEBUG juju.worker.uniter.remotestate got running status change for minio/0
application-minio: 20:45:34 DEBUG juju.worker.caasoperator request running status for "unit-minio-0" minio-0
application-pipelines-api: 20:45:34 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:34 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch
application-minio: 20:45:34 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-data-storage-attached-3844453286847804765
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "relation-list" for argo-controller/0-minio-relation-changed-6493246157928624429
application-pipelines-api: 20:45:34 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:34 INFO unit.pipelines-api/0.juju-log status-set: maintenance: fetching resource: oci-image
application-minio: 20:45:34 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:34 DEBUG unit.minio/0.juju-log tracer>
tracer: starting handler dispatch, 9 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "relation-get" for argo-controller/0-minio-relation-changed-6493246157928624429
application-pipelines-api: 20:45:34 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-install-8154239330009868646
application-minio: 20:45:34 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:34 DEBUG unit.minio/0.juju-log tracer: hooks phase, 0 handlers queued
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:34 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: set flag endpoint.minio.changed
tracer: ++   queue handler reactive/argo_controller.py:26:start_charm
application-minio: 20:45:34 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:34 DEBUG unit.minio/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:34 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: set flag endpoint.minio.changed.ip
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:8:mailman3_connected:minio
application-minio: 20:45:34 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:34 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:13:charm_ready
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:34 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: set flag endpoint.minio.changed.password
controller-0: 20:45:34 INFO juju.worker.caasprovisioner started operator for application "pipelines-db"
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:34 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: set flag endpoint.minio.changed.port
application-minio: 20:45:34 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:34 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:18:configure_minio
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:34 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: set flag endpoint.minio.changed.user
application-minio: 20:45:34 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-data-storage-attached-3844453286847804765
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:34 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: starting handler dispatch, 16 flags set
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.ip
tracer: set flag endpoint.minio.changed.password
tracer: set flag endpoint.minio.changed.port
tracer: set flag endpoint.minio.changed.user
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:34 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-data-storage-attached-3844453286847804765
application-argo-controller: 20:45:34 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:34 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: hooks phase, 0 handlers queued
application-minio: 20:45:35 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-data-storage-attached-3844453286847804765
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:8:mailman3_connected:minio
tracer: ++   queue handler reactive/argo_controller.py:26:start_charm
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 INFO unit.argo-controller/0.juju-log minio:0: Invoking reactive handler: reactive/argo_controller.py:26:start_charm
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "is-leader" for argo-controller/0-minio-relation-changed-6493246157928624429
application-pipelines-api: 20:45:35 DEBUG jujuc running hook tool "resource-get" for pipelines-api/0-install-8154239330009868646
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 INFO unit.argo-controller/0.juju-log minio:0: status-set: maintenance: configuring container
application-minio: 20:45:35 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-data-storage-attached-3844453286847804765
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "status-set" for argo-controller/0-minio-relation-changed-6493246157928624429
application-minio: 20:45:35 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:35 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-data-storage-attached-3844453286847804765
application-pipelines-api: 20:45:35 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:35 DEBUG unit.pipelines-api/0.juju-log tracer: set flag layer.docker-resource.oci-image.available
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "pod-spec-set" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "pod-spec-set" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 INFO unit.argo-controller/0.juju-log minio:0: status-set: maintenance: creating container
application-minio: 20:45:35 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:35 INFO unit.minio/0.juju-log status-set: active: 
application-pipelines-api: 20:45:35 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-install-8154239330009868646
application-pipelines-api: 20:45:35 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: cleared flag layer.docker-resource.pending
tracer: -- dequeue handler reactive/docker_resource.py:18:fetch
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "status-set" for argo-controller/0-minio-relation-changed-6493246157928624429
application-minio: 20:45:35 DEBUG juju.worker.uniter.remotestate storage attachment change for minio/0: {storage-minio-data-0 {2 alive true /data}}
application-minio: 20:45:35 DEBUG jujuc running hook tool "status-set" for minio/0-minio-data-storage-attached-3844453286847804765
application-minio: 20:45:35 DEBUG unit.minio/0.minio-data-storage-attached Provides
application-minio: 20:45:35 DEBUG unit.minio/0.minio-data-storage-attached YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:35 DEBUG unit.minio/0.minio-data-storage-attached minio:0
application-minio: 20:45:35 DEBUG unit.minio/0.minio-data-storage-attached Provides
application-minio: 20:45:35 DEBUG unit.minio/0.minio-data-storage-attached YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:35 DEBUG unit.minio/0.minio-data-storage-attached minio:3
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: set flag charm.started
tracer: ++   queue handler reactive/argo_controller.py:16:charm_ready
tracer: -- dequeue handler reactive/argo_controller.py:26:start_charm
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 INFO unit.argo-controller/0.juju-log minio:0: Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:8:mailman3_connected:minio
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: set flag endpoint.minio.connected
application-pipelines-api: 20:45:35 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:45:35 DEBUG juju.worker.uniter.operation committing operation "run install hook" for pipelines-api/0
application-pipelines-api: 20:45:35 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run install hook)
application-pipelines-api: 20:45:35 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:35 INFO juju.worker.uniter.relation joining relation "pipelines-api:mysql pipelines-db:mysql"
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: cleared flag endpoint.minio.changed.ip
application-pipelines-api: 20:45:35 INFO juju.worker.uniter.relation joined relation "pipelines-api:mysql pipelines-db:mysql"
application-argo-controller: 20:45:35 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:35 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: cleared flag endpoint.minio.changed.port
application-pipelines-api: 20:45:35 INFO juju.worker.uniter.relation joining relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api"
application-minio: 20:45:35 INFO juju.worker.uniter.operation ran "minio-data-storage-attached" hook (via explicit, bespoke hook script)
controller-0: 20:45:35 DEBUG juju.worker.caasunitprovisioner units changed: true
application-minio: 20:45:35 DEBUG juju.worker.uniter.operation committing operation "run storage-attached (minio-data/0) hook" for minio/0
application-minio: 20:45:36 DEBUG juju.machinelock machine lock released for minio/0 uniter (run storage-attached (minio-data/0) hook)
application-minio: 20:45:36 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:36 DEBUG juju.worker.uniter.operation running operation remote init for minio/0
application-minio: 20:45:36 DEBUG juju.worker.uniter.operation preparing operation "remote init" for minio/0
application-argo-controller: 20:45:36 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:36 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: cleared flag endpoint.minio.changed.user
application-pipelines-api: 20:45:36 INFO juju.worker.uniter.relation joined relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api"
application-minio: 20:45:36 DEBUG juju.worker.uniter.operation executing operation "remote init" for minio/0
application-minio: 20:45:36 DEBUG juju.worker.caasoperator request running status for "unit-minio-0" minio-0
application-minio: 20:45:36 DEBUG juju.worker.caasoperator remote init for "unit-minio-0" {PodName:minio-0 Initialising:true InitialisingTime:2020-10-21 20:45:32 +0000 UTC Running:false}
application-minio: 20:45:36 INFO juju.worker.caasoperator started pod init on "minio/0"
application-minio: 20:45:36 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/cxwsjpan.pid; exec sh -c 'mkdir -p /tmp/unit-minio-0345384077'; "]
application-pipelines-api: 20:45:36 INFO juju.worker.uniter.relation joining relation "pipelines-api:minio minio:minio"
application-pipelines-api: 20:45:36 INFO juju.worker.uniter.relation joined relation "pipelines-api:minio minio:minio"
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation running operation run relation-created (1; app: pipelines-db) hook for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-created (1; app: pipelines-db) hook)
application-pipelines-api: 20:45:36 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-created (1; app: pipelines-db) hook)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation preparing operation "run relation-created (1; app: pipelines-db) hook" for pipelines-api/0
application-minio: 20:45:36 DEBUG juju.worker.uniter.remotestate got a relation units change for minio/0 : {3 {map[pipelines-api/0:{0}] map[] []}}
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation executing operation "run relation-created (1; app: pipelines-db) hook" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running mysql-relation-created hook
application-argo-controller: 20:45:36 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:36 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: cleared flag endpoint.minio.changed.password
tracer: -- dequeue handler hooks/relations/generic-ip-port-user-pass/requires.py:8:mailman3_connected:minio
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-api: 20:45:36 INFO juju.worker.uniter.operation skipped "mysql-relation-created" hook (missing)
controller-0: 20:45:36 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-db
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation committing operation "run relation-created (1; app: pipelines-db) hook" for pipelines-api/0
application-argo-controller: 20:45:36 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:36 INFO unit.argo-controller/0.juju-log minio:0: Invoking reactive handler: reactive/argo_controller.py:16:charm_ready
application-pipelines-api: 20:45:36 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-created (1; app: pipelines-db) hook)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation running operation run relation-created (2; app: pipelines-persistence) hook for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-created (2; app: pipelines-persistence) hook)
application-pipelines-api: 20:45:36 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-created (2; app: pipelines-persistence) hook)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation preparing operation "run relation-created (2; app: pipelines-persistence) hook" for pipelines-api/0
application-minio: 20:45:36 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-minio/charm  } to {/tmp/unit-minio-0345384077 minio-0 juju-pod-init}
application-minio: 20:45:36 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/xmvwhkqb.pid; exec sh -c 'test -d /tmp/unit-minio-0345384077'; "]
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation executing operation "run relation-created (2; app: pipelines-persistence) hook" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running pipelines-api-relation-created hook
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-api: 20:45:36 INFO juju.worker.uniter.operation skipped "pipelines-api-relation-created" hook (missing)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation committing operation "run relation-created (2; app: pipelines-persistence) hook" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-created (2; app: pipelines-persistence) hook)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation running operation run relation-created (3; app: minio) hook for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-created (3; app: minio) hook)
application-pipelines-api: 20:45:36 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-created (3; app: minio) hook)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation preparing operation "run relation-created (3; app: minio) hook" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation executing operation "run relation-created (3; app: minio) hook" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-created hook
application-argo-controller: 20:45:36 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:36 INFO unit.argo-controller/0.juju-log minio:0: status-set: active: 
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-api: 20:45:36 INFO juju.worker.uniter.operation skipped "minio-relation-created" hook (missing)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation committing operation "run relation-created (3; app: minio) hook" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-created (3; app: minio) hook)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation running operation accept leadership for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for pipelines-api/0
application-pipelines-api: 20:45:36 INFO juju.worker.uniter found queued "leader-elected" hook
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run leader-elected hook)
application-pipelines-api: 20:45:36 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run leader-elected hook)
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for pipelines-api/0
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-pipelines-api: 20:45:36 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-argo-controller: 20:45:36 DEBUG jujuc running hook tool "status-set" for argo-controller/0-minio-relation-changed-6493246157928624429
application-argo-controller: 20:45:36 DEBUG unit.argo-controller/0.minio-relation-changed Requires relation
application-argo-controller: 20:45:36 DEBUG unit.argo-controller/0.minio-relation-changed minio:0
application-argo-controller: 20:45:36 DEBUG unit.argo-controller/0.minio-relation-changed requires unit
application-argo-controller: 20:45:36 DEBUG unit.argo-controller/0.minio-relation-changed {'ip': '"minio"', 'password': '"YR9E8ANXSPL30CQQJ3G17E82QV0BJD"', 'port': '9000', 'user': '"minio"'}
application-minio: 20:45:36 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/wztypcpf.pid; exec sh -c 'tar -xmf - -C /tmp/unit-minio-0345384077'; "]
application-pipelines-api: 20:45:36 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-leader-elected-4282254987853709245
application-argo-controller: 20:45:36 INFO juju.worker.uniter.operation ran "minio-relation-changed" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:36 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
application-argo-controller: 20:45:36 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:36 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:36 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:36 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-api: 20:45:36 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:36 INFO unit.pipelines-api/0.juju-log Reactive main running for hook leader-elected
controller-0: 20:45:36 DEBUG juju.worker.caasunitprovisioner service for minio(workload): &{Id:0f6fc8e3-48a5-427a-a263-04533dd18de2 Addresses:[local-cloud:10.152.183.102] Scale:0xc00279fc60 Generation:0xc00279fc68 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-leader-elected-4282254987853709245
application-minio: 20:45:37 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-minio-0345384077/ca.crt  } to {/tmp/unit-minio-0345384077/ca.crt minio-0 juju-pod-init}
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-leader-elected-4282254987853709245
application-minio: 20:45:37 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/hcglffgz.pid; exec sh -c 'test -d /tmp/unit-minio-0345384077/ca.crt'; "]
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: starting handler dispatch, 11 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG unit.pipelines-api/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: main dispatch loop, 5 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql
application-pipelines-api: 20:45:37 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:37 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
controller-0: 20:45:37 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc002c7d640}
application-minio: 20:45:38 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/zadfewut.pid; exec sh -c 'tar -xmf - -C /tmp/unit-minio-0345384077'; "]
controller-0: 20:45:38 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-db
application-pipelines-api: 20:45:38 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:38 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-api: 20:45:39 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:39 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api
application-minio: 20:45:39 DEBUG juju.worker.caasoperator operator service address: 10.152.183.110
application-minio: 20:45:39 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-minio-0/operator-client-cache.yaml  } to {/tmp/unit-minio-0345384077/operator-client-cache.yaml minio-0 juju-pod-init}
application-minio: 20:45:39 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/xxaxfeee.pid; exec sh -c 'test -d /tmp/unit-minio-0345384077/operator-client-cache.yaml'; "]
application-pipelines-db: 20:45:39 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pipelines-db: 20:45:39 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-db", "--debug"}
application-pipelines-db: 20:45:39 DEBUG juju.agent read agent config, format "2.0"
application-pipelines-db: 20:45:39 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pipelines-db: 20:45:39 INFO juju.cmd.jujud caas operator application-pipelines-db start (2.8.6 [gc])
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:45:39.663396272 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:45:39.663686274 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-db: 20:45:39 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pipelines-db"
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:45:39.669023509 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:45:39.673543339 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-db: 20:45:39 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-db: 20:45:39 INFO juju.worker.apicaller [f0e819] "application-pipelines-db" successfully connected to "10.152.183.175:17070"
application-pipelines-db: 20:45:39 DEBUG juju.api RPC connection died
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:45:39.689680747 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:45:39.69011205 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-db: 20:45:39 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-db: 20:45:39 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-db: 20:45:39 INFO juju.worker.apicaller [f0e819] "application-pipelines-db" successfully connected to "10.152.183.175:17070"
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:45:39.759936415 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:45:39.770195383 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:45:39.770374384 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:45:39.770391384 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:45:39.770413084 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:45:39.780962154 +0000 UTC
application-pipelines-db: 20:45:39 INFO juju.worker.migrationminion migration phase is now: NONE
application-pipelines-db: 20:45:39 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pipelines-db: 20:45:39 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pipelines-db: 20:45:39 INFO juju.worker.caasupgrader unblocking abort check
application-pipelines-db: 20:45:39 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:45:39.792539932 +0000 UTC
application-pipelines-db: 20:45:39 INFO juju.worker.logger logger worker started
application-pipelines-db: 20:45:39 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:45:39.794892247 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:45:39.804729513 +0000 UTC
application-pipelines-api: 20:45:39 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:39 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:45:39.827439264 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:45:39.827722966 +0000 UTC
controller-0: 20:45:39 DEBUG juju.worker.caasunitprovisioner ensured deployment for argo-controller for 1 units
application-minio: 20:45:39 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/phgswibg.pid; exec sh -c 'tar -xmf - -C /tmp/unit-minio-0345384077'; "]
application-pipelines-db: 20:45:39 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:45:39.994462076 +0000 UTC
application-pipelines-db: 20:45:39 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pipelines-db: 20:45:39 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-db: 20:45:39 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-db: 20:45:39 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pipelines-db: 20:45:39 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pipelines-db: 20:45:39 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pipelines-db: 20:45:39 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-pipelines-db: 20:45:40 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-db: 20:45:40 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-db: 20:45:40 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-db: 20:45:40 INFO juju.worker.uniter.charm downloading cs:~charmed-osm/mariadb-k8s-34 from API server
application-pipelines-db: 20:45:40 INFO juju.downloader downloading from cs:~charmed-osm/mariadb-k8s-34
application-pipelines-db: 20:45:40 INFO juju.downloader download complete ("cs:~charmed-osm/mariadb-k8s-34")
application-pipelines-db: 20:45:40 INFO juju.downloader download verified ("cs:~charmed-osm/mariadb-k8s-34")
application-minio: 20:45:40 DEBUG juju.worker.uniter.remotestate storage attachment change for minio/0: {storage-minio-data-0 {2 alive true /data}}
application-pipelines-db: 20:45:40 DEBUG juju.worker.uniter.charm preparing to deploy charm "cs:~charmed-osm/mariadb-k8s-34"
application-pipelines-db: 20:45:40 DEBUG juju.worker.uniter.charm deploying charm "cs:~charmed-osm/mariadb-k8s-34"
application-pipelines-api: 20:45:40 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-leader-elected-4282254987853709245
application-pipelines-api: 20:45:40 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql
application-minio: 20:45:40 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/ctdmhyne.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-minio-0 --charm-dir /tmp/unit-minio-0345384077/charm --send --operator-file /tmp/unit-minio-0345384077/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-minio-0345384077/ca.crt'; "]
application-pipelines-api: 20:45:40 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:45:40 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for pipelines-api/0
application-pipelines-api: 20:45:40 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run leader-elected hook)
application-pipelines-api: 20:45:40 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:40 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:45:40 DEBUG juju.worker.uniter.operation running operation run config-changed hook for pipelines-api/0
application-pipelines-api: 20:45:40 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run config-changed hook)
application-pipelines-api: 20:45:40 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run config-changed hook)
application-pipelines-api: 20:45:40 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for pipelines-api/0
application-pipelines-api: 20:45:40 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for pipelines-api/0
application-pipelines-api: 20:45:40 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-pipelines-api: 20:45:40 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
controller-0: 20:45:40 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-config-changed-4158734354908220558
application-minio: 20:45:41 DEBUG juju.worker.uniter.operation committing operation "remote init" for minio/0
application-minio: 20:45:41 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:45:41 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:45:41 DEBUG juju.worker.uniter.operation running operation run config-changed hook for minio/0
application-minio: 20:45:41 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run config-changed hook)
application-minio: 20:45:41 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run config-changed hook)
application-minio: 20:45:41 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for minio/0
application-minio: 20:45:41 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for minio/0
application-minio: 20:45:41 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 INFO unit.pipelines-api/0.juju-log Reactive main running for hook config-changed
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-4158734354908220558
controller-0: 20:45:41 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-db
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: starting handler dispatch, 11 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG unit.pipelines-api/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: main dispatch loop, 5 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-api: 20:45:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:41 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api
application-pipelines-api: 20:45:42 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:42 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-pipelines-api: 20:45:42 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-4158734354908220558
application-pipelines-api: 20:45:42 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql
application-minio: 20:45:42 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-pipelines-api: 20:45:42 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:45:43 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for pipelines-api/0
application-minio: 20:45:43 DEBUG jujuc running hook tool "config-get" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:43 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run config-changed hook)
application-pipelines-api: 20:45:43 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:43 INFO juju.worker.uniter found queued "start" hook
application-pipelines-api: 20:45:43 DEBUG juju.worker.uniter.operation running operation run start hook for pipelines-api/0
application-pipelines-api: 20:45:43 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run start hook)
application-pipelines-api: 20:45:43 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run start hook)
application-pipelines-api: 20:45:43 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for pipelines-api/0
application-pipelines-api: 20:45:43 DEBUG juju.worker.uniter.operation executing operation "run start hook" for pipelines-api/0
application-pipelines-api: 20:45:43 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-pipelines-api: 20:45:43 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-minio: 20:45:43 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-6151505624067528869
application-minio: 20:45:43 INFO unit.minio/0.juju-log Reactive main running for hook config-changed
application-pipelines-db: 20:45:43 DEBUG juju.worker.uniter.charm finishing deploy of charm "cs:~charmed-osm/mariadb-k8s-34"
application-pipelines-db: 20:45:43 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000948900}
application-pipelines-db: 20:45:43 INFO juju.worker.caasoperator operator "pipelines-db" started
application-pipelines-db: 20:45:43 DEBUG juju.worker.uniter juju-run listener running
application-minio: 20:45:43 DEBUG jujuc running hook tool "relation-ids" for minio/0-config-changed-6151505624067528869
application-pipelines-db: 20:45:43 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-api: 20:45:43 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-start-4789500969943010041
application-pipelines-db: 20:45:43 DEBUG juju.worker.uniter starting uniter for "pipelines-db/0"
application-pipelines-db: 20:45:43 DEBUG juju.worker.caasoperator start "pipelines-db/0"
application-pipelines-db: 20:45:43 INFO juju.worker.caasoperator start "pipelines-db/0"
application-pipelines-db: 20:45:43 DEBUG juju.worker.caasoperator "pipelines-db/0" started
application-pipelines-db: 20:45:43 DEBUG juju.worker.leadership pipelines-db/0 making initial claim for pipelines-db leadership
application-pipelines-db: 20:45:43 INFO juju.worker.leadership pipelines-db/0 promoted to leadership of pipelines-db
application-pipelines-db: 20:45:43 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-db-0
application-pipelines-db: 20:45:43 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pipelines-db-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-db-0/jujud
application-pipelines-db: 20:45:43 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pipelines-db-0/jujud
application-pipelines-db: 20:45:43 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-db-0/run.socket <nil>}
application-pipelines-db: 20:45:43 INFO juju.worker.uniter unit "pipelines-db/0" started
application-pipelines-db: 20:45:43 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-db: 20:45:43 INFO juju.worker.uniter resuming charm install
application-pipelines-db: 20:45:43 DEBUG juju.worker.uniter.operation running operation install cs:~charmed-osm/mariadb-k8s-34 for pipelines-db/0
application-pipelines-db: 20:45:43 DEBUG juju.worker.uniter.operation preparing operation "install cs:~charmed-osm/mariadb-k8s-34" for pipelines-db/0
application-pipelines-db: 20:45:43 INFO juju.worker.uniter.charm downloading cs:~charmed-osm/mariadb-k8s-34 from API server
application-pipelines-db: 20:45:43 INFO juju.downloader downloading from cs:~charmed-osm/mariadb-k8s-34
application-pipelines-db: 20:45:43 INFO juju.downloader download complete ("cs:~charmed-osm/mariadb-k8s-34")
application-pipelines-db: 20:45:43 INFO juju.downloader download verified ("cs:~charmed-osm/mariadb-k8s-34")
application-minio: 20:45:43 DEBUG jujuc running hook tool "relation-list" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:43 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:43 INFO unit.pipelines-api/0.juju-log Reactive main running for hook start
application-minio: 20:45:43 DEBUG jujuc running hook tool "relation-list" for minio/0-config-changed-6151505624067528869
application-minio: 20:45:43 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-6151505624067528869
application-minio: 20:45:43 DEBUG unit.minio/0.juju-log tracer>
tracer: starting handler dispatch, 9 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:43 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-6151505624067528869
application-minio: 20:45:43 DEBUG unit.minio/0.juju-log tracer: hooks phase, 0 handlers queued
application-minio: 20:45:43 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-6151505624067528869
application-minio: 20:45:43 DEBUG unit.minio/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-minio: 20:45:43 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-6151505624067528869
application-minio: 20:45:43 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:13:charm_ready
application-minio: 20:45:43 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-6151505624067528869
application-minio: 20:45:43 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:18:configure_minio
controller-0: 20:45:43 INFO juju.worker.caasprovisioner started operator for application "pipelines-persistence"
application-pipelines-db: 20:45:44 DEBUG juju.worker.uniter.operation executing operation "install cs:~charmed-osm/mariadb-k8s-34" for pipelines-db/0
application-pipelines-db: 20:45:44 DEBUG juju.worker.uniter.charm preparing to deploy charm "cs:~charmed-osm/mariadb-k8s-34"
application-pipelines-db: 20:45:44 DEBUG juju.worker.uniter.charm deploying charm "cs:~charmed-osm/mariadb-k8s-34"
application-minio: 20:45:44 DEBUG jujuc running hook tool "relation-get" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:44 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-start-4789500969943010041
application-minio: 20:45:44 DEBUG jujuc running hook tool "relation-get" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:44 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-start-4789500969943010041
application-minio: 20:45:44 DEBUG jujuc running hook tool "relation-set" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:44 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-start-4789500969943010041
application-minio: 20:45:44 DEBUG jujuc running hook tool "relation-set" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:44 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-start-4789500969943010041
application-minio: 20:45:44 DEBUG jujuc running hook tool "relation-set" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:44 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-start-4789500969943010041
application-minio: 20:45:44 DEBUG jujuc running hook tool "relation-set" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:44 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-start-4789500969943010041
application-minio: 20:45:44 DEBUG jujuc running hook tool "juju-log" for minio/0-config-changed-6151505624067528869
application-minio: 20:45:44 INFO unit.minio/0.juju-log status-set: active: 
application-pipelines-api: 20:45:44 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-start-4789500969943010041
application-minio: 20:45:44 DEBUG jujuc running hook tool "status-set" for minio/0-config-changed-6151505624067528869
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: starting handler dispatch, 11 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 DEBUG unit.pipelines-api/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: main dispatch loop, 5 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:19:broken:minio
application-pipelines-api: 20:45:45 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-start-4789500969943010041
application-pipelines-api: 20:45:45 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql
controller-0: 20:45:45 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc00333cf00}
application-minio: 20:45:46 DEBUG unit.minio/0.config-changed Provides
application-minio: 20:45:46 DEBUG unit.minio/0.config-changed YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:46 DEBUG unit.minio/0.config-changed minio:0
application-minio: 20:45:46 DEBUG unit.minio/0.config-changed Provides
application-minio: 20:45:46 DEBUG unit.minio/0.config-changed YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:46 DEBUG unit.minio/0.config-changed minio:3
application-pipelines-api: 20:45:46 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter.operation committing operation "run start hook" for pipelines-api/0
controller-0: 20:45:46 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc00315abc0}
application-pipelines-api: 20:45:46 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run start hook)
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter.operation running operation run relation-joined (3; unit: minio/0) hook for pipelines-api/0
application-pipelines-api: 20:45:46 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-joined (3; unit: minio/0) hook)
application-pipelines-api: 20:45:46 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-joined (3; unit: minio/0) hook)
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter.operation preparing operation "run relation-joined (3; unit: minio/0) hook" for pipelines-api/0
application-minio: 20:45:46 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-minio: 20:45:46 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for minio/0
application-minio: 20:45:46 DEBUG juju.machinelock machine lock released for minio/0 uniter (run config-changed hook)
application-minio: 20:45:46 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:46 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:45:46 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:45:46 DEBUG juju.worker.uniter.operation running operation run relation-joined (3; unit: pipelines-api/0) hook for minio/0
application-minio: 20:45:46 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run relation-joined (3; unit: pipelines-api/0) hook)
application-minio: 20:45:46 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run relation-joined (3; unit: pipelines-api/0) hook)
application-minio: 20:45:46 DEBUG juju.worker.uniter.operation preparing operation "run relation-joined (3; unit: pipelines-api/0) hook" for minio/0
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter.operation executing operation "run relation-joined (3; unit: minio/0) hook" for pipelines-api/0
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-joined hook
application-minio: 20:45:46 DEBUG juju.worker.uniter.operation executing operation "run relation-joined (3; unit: pipelines-api/0) hook" for minio/0
application-minio: 20:45:46 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-joined hook
application-minio: 20:45:46 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter.remotestate got a relation units change for pipelines-api/0 : {3 {map[minio/0:{2}] map[] []}}
application-argo-controller: 20:45:46 DEBUG juju.worker.uniter.remotestate got a relation units change for argo-controller/0 : {0 {map[minio/0:{2}] map[] []}}
application-argo-controller: 20:45:46 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:46 DEBUG juju.worker.uniter.operation running operation run relation-changed (0; unit: minio/0) hook for argo-controller/0
application-argo-controller: 20:45:46 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:46 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:46 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
application-pipelines-api: 20:45:46 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-argo-controller: 20:45:47 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
application-argo-controller: 20:45:47 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-changed hook
application-argo-controller: 20:45:47 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner operator update for argo-controller
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-persistence
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-db
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner operator update for tf-job-operator
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:45:47 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-minio: 20:45:47 DEBUG jujuc running hook tool "config-get" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-api: 20:45:47 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-argo-controller: 20:45:47 DEBUG jujuc running hook tool "config-get" for argo-controller/0-minio-relation-changed-1681941728775037402
application-pipelines-api: 20:45:48 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:48 INFO unit.pipelines-api/0.juju-log minio:3: Reactive main running for hook minio-relation-joined
application-argo-controller: 20:45:48 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:48 INFO unit.argo-controller/0.juju-log minio:0: Reactive main running for hook minio-relation-changed
application-minio: 20:45:48 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-1587248472434631741
application-minio: 20:45:48 INFO unit.minio/0.juju-log minio:3: Reactive main running for hook minio-relation-joined
application-argo-controller: 20:45:48 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-minio-relation-changed-1681941728775037402
application-pipelines-api: 20:45:48 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.charm finishing deploy of charm "cs:~charmed-osm/mariadb-k8s-34"
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.operation committing operation "install cs:~charmed-osm/mariadb-k8s-34" for pipelines-db/0
application-minio: 20:45:48 DEBUG jujuc running hook tool "relation-ids" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-db: 20:45:48 INFO juju.worker.uniter hooks are retried true
application-argo-controller: 20:45:48 DEBUG jujuc running hook tool "relation-list" for argo-controller/0-minio-relation-changed-1681941728775037402
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got update status interval change for pipelines-db/0: ok=true
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got config change for pipelines-db/0: ok=true, hashes=[7895a6fda895e3e388b0e3f0a3b9cce024035b1022161326cf4148bd140cb81b]
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got unit change for pipelines-db/0
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got storage change for pipelines-db/0: [database/1] ok=true
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got trust config change for pipelines-db/0: ok=true, hashes=[ed7ecb5d76214434b26162caae4471d6cfb6c5d9428b7decf3570080e60f57d9]
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got address change for pipelines-db/0: ok=true, hashes=[]
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got leader settings change for pipelines-db/0: ok=true
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got relations change for pipelines-db/0: ok=true
application-pipelines-api: 20:45:48 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.remotestate got action change for pipelines-db/0: [] ok=true
application-pipelines-db: 20:45:48 INFO juju.worker.uniter found queued "install" hook
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.operation running operation run install hook for pipelines-db/0
application-pipelines-db: 20:45:48 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run install hook)
application-pipelines-db: 20:45:48 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run install hook)
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for pipelines-db/0
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.operation executing operation "run install hook" for pipelines-db/0
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-pipelines-db: 20:45:48 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-argo-controller: 20:45:48 DEBUG jujuc running hook tool "relation-get" for argo-controller/0-minio-relation-changed-1681941728775037402
application-minio: 20:45:48 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-api: 20:45:48 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:49 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:49 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer>
tracer: set flag endpoint.minio.joined
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-argo-controller: 20:45:49 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:49 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: ++   queue handler reactive/argo_controller.py:16:charm_ready
application-minio: 20:45:49 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-joined-1587248472434631741
application-argo-controller: 20:45:49 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:49 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: set flag endpoint.minio.changed.ingress-address
application-pipelines-api: 20:45:49 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-minio: 20:45:49 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-joined-1587248472434631741
application-argo-controller: 20:45:49 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:49 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: set flag endpoint.minio.changed.private-address
application-pipelines-api: 20:45:49 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:49 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: set flag endpoint.minio.changed
application-minio: 20:45:49 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-joined-1587248472434631741
application-argo-controller: 20:45:49 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:49 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: starting handler dispatch, 17 flags set
tracer: set flag charm.started
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-api: 20:45:49 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:49 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.egress-subnets
application-argo-controller: 20:45:49 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:49 DEBUG unit.argo-controller/0.juju-log minio:0: tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:45:49 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:49 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.ingress-address
application-minio: 20:45:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-1587248472434631741
application-minio: 20:45:49 DEBUG unit.minio/0.juju-log minio:3: tracer>
tracer: starting handler dispatch, 9 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-db: 20:45:49 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-install-1272112431368911998
application-argo-controller: 20:45:49 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:49 DEBUG unit.argo-controller/0.juju-log minio:0: tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/argo_controller.py:16:charm_ready
application-pipelines-api: 20:45:49 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:49 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer>
tracer: set flag endpoint.minio.changed.ip
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:8:mailman3_connected:minio
application-minio: 20:45:49 DEBUG juju.worker.caasoperator trigger running status for caas unit minio/0
application-minio: 20:45:49 DEBUG juju.worker.uniter.remotestate got running status change for minio/0
application-minio: 20:45:49 DEBUG juju.worker.caasoperator request running status for "unit-minio-0" minio-0
application-argo-controller: 20:45:49 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:49 INFO unit.argo-controller/0.juju-log minio:0: Invoking reactive handler: reactive/argo_controller.py:16:charm_ready
application-minio: 20:45:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-1587248472434631741
application-minio: 20:45:49 DEBUG unit.minio/0.juju-log minio:3: tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:45:49 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:49 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.password
application-minio: 20:45:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-1587248472434631741
application-minio: 20:45:49 DEBUG unit.minio/0.juju-log minio:3: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-pipelines-api: 20:45:49 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:49 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.port
application-argo-controller: 20:45:49 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-minio-relation-changed-1681941728775037402
application-argo-controller: 20:45:49 INFO unit.argo-controller/0.juju-log minio:0: status-set: active: 
application-argo-controller: 20:45:50 DEBUG jujuc running hook tool "status-set" for argo-controller/0-minio-relation-changed-1681941728775037402
application-minio: 20:45:50 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-1587248472434631741
application-minio: 20:45:50 INFO unit.minio/0.juju-log minio:3: Invoking reactive handler: reactive/minio.py:13:charm_ready
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.private-address
application-minio: 20:45:50 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-1587248472434631741
application-minio: 20:45:50 INFO unit.minio/0.juju-log minio:3: Invoking reactive handler: reactive/minio.py:18:configure_minio
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.user
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-minio: 20:45:50 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-minio: 20:45:50 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-minio: 20:45:50 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-joined-1587248472434631741
application-minio: 20:45:50 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-minio: 20:45:50 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer>
tracer: starting handler dispatch, 20 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.ip
tracer: set flag endpoint.minio.changed.password
tracer: set flag endpoint.minio.changed.port
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.changed.user
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:50 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: hooks phase, 0 handlers queued
application-minio: 20:45:50 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-joined-1587248472434631741
application-minio: 20:45:50 INFO unit.minio/0.juju-log minio:3: status-set: active: 
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer>
tracer: main dispatch loop, 6 handlers queued
tracer: ++   queue handler hooks/relations/generic-ip-port-user-pass/requires.py:8:mailman3_connected:minio
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-minio: 20:45:50 DEBUG jujuc running hook tool "status-set" for minio/0-minio-relation-joined-1587248472434631741
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:45:50 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:50 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-api: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:51 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api
application-minio: 20:45:51 DEBUG unit.minio/0.minio-relation-joined Provides
application-minio: 20:45:51 DEBUG unit.minio/0.minio-relation-joined YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:51 DEBUG unit.minio/0.minio-relation-joined minio:0
application-minio: 20:45:51 DEBUG unit.minio/0.minio-relation-joined Provides
application-minio: 20:45:51 DEBUG unit.minio/0.minio-relation-joined YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:51 DEBUG unit.minio/0.minio-relation-joined minio:3
application-pipelines-api: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:51 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/generic-ip-port-user-pass/requires.py:8:mailman3_connected:minio
application-pipelines-persistence: 20:45:51 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pipelines-persistence: 20:45:51 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-persistence", "--debug"}
application-pipelines-persistence: 20:45:51 DEBUG juju.agent read agent config, format "2.0"
application-pipelines-persistence: 20:45:51 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pipelines-persistence: 20:45:51 INFO juju.cmd.jujud caas operator application-pipelines-persistence start (2.8.6 [gc])
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:45:51.209736059 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:45:51.20986576 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:45:51.211485471 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:45:51.211683073 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pipelines-persistence"
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:45:51.227360283 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-persistence: 20:45:51 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-persistence: 20:45:51 INFO juju.worker.apicaller [f0e819] "application-pipelines-persistence" successfully connected to "10.152.183.175:17070"
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:45:51.233039723 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:45:51.238987165 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:45:51.244144302 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:45:51.244183502 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:45:51.244216102 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:45:51.245611712 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:45:51.250331646 +0000 UTC
application-pipelines-persistence: 20:45:51 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pipelines-persistence: 20:45:51 INFO juju.worker.caasupgrader unblocking abort check
application-pipelines-persistence: 20:45:51 INFO juju.worker.migrationminion migration phase is now: NONE
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:45:51.262889834 +0000 UTC
application-pipelines-persistence: 20:45:51 INFO juju.worker.logger logger worker started
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:45:51.26506985 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:45:51.265216151 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:45:51.265333651 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:45:51.269509481 +0000 UTC
application-pipelines-api: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:51 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: set flag endpoint.minio.connected
application-pipelines-persistence: 20:45:51 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pipelines-persistence: 20:45:51 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-persistence: 20:45:51 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-persistence: 20:45:51 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pipelines-persistence: 20:45:51 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pipelines-persistence: 20:45:51 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:45:51.283036977 +0000 UTC
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-argo-controller: 20:45:51 INFO juju.worker.uniter.operation ran "minio-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:51 INFO unit.pipelines-db/0.juju-log Reactive main running for hook install
application-minio: 20:45:51 INFO juju.worker.uniter.operation ran "minio-relation-joined" hook (via explicit, bespoke hook script)
application-argo-controller: 20:45:51 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (0; unit: minio/0) hook" for argo-controller/0
application-pipelines-persistence: 20:45:51 INFO juju.worker.uniter.charm downloading local:kubernetes/pipelines-persistence-0 from API server
application-pipelines-persistence: 20:45:51 INFO juju.downloader downloading from local:kubernetes/pipelines-persistence-0
application-minio: 20:45:51 DEBUG juju.worker.uniter.operation committing operation "run relation-joined (3; unit: pipelines-api/0) hook" for minio/0
application-argo-controller: 20:45:51 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run relation-changed (0; unit: minio/0) hook)
application-argo-controller: 20:45:51 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:45:51 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:45:51 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-persistence: 20:45:51 INFO juju.downloader download complete ("local:kubernetes/pipelines-persistence-0")
application-minio: 20:45:51 DEBUG juju.machinelock machine lock released for minio/0 uniter (run relation-joined (3; unit: pipelines-api/0) hook)
application-minio: 20:45:51 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:51 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:45:51 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:45:51 DEBUG juju.worker.uniter.operation running operation run relation-changed (0; unit: argo-controller/0) hook for minio/0
application-minio: 20:45:51 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run relation-changed (0; unit: argo-controller/0) hook)
application-minio: 20:45:51 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run relation-changed (0; unit: argo-controller/0) hook)
application-minio: 20:45:51 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (0; unit: argo-controller/0) hook" for minio/0
application-pipelines-api: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:51 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: cleared flag endpoint.minio.changed.ip
application-pipelines-persistence: 20:45:51 INFO juju.downloader download verified ("local:kubernetes/pipelines-persistence-0")
application-minio: 20:45:51 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (0; unit: argo-controller/0) hook" for minio/0
application-minio: 20:45:51 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-changed hook
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-persistence-0"
application-pipelines-persistence: 20:45:51 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-persistence-0"
application-minio: 20:45:51 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-pipelines-db: 20:45:51 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-install-1272112431368911998
application-pipelines-api: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:51 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: cleared flag endpoint.minio.changed.port
application-pipelines-api: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:51 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: cleared flag endpoint.minio.changed.user
application-pipelines-db: 20:45:51 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-install-1272112431368911998
application-pipelines-api: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:51 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer>
tracer: cleared flag endpoint.minio.changed.password
tracer: -- dequeue handler hooks/relations/generic-ip-port-user-pass/requires.py:8:mailman3_connected:minio
application-pipelines-db: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:51 INFO unit.pipelines-db/0.juju-log Initializing Leadership Layer (is leader)
application-minio: 20:45:51 DEBUG jujuc running hook tool "config-get" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:51 DEBUG unit.pipelines-db/0.juju-log tracer: set flag leadership.is_leader
application-pipelines-api: 20:45:51 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-joined-2342698126156470502
application-pipelines-api: 20:45:51 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql
application-minio: 20:45:51 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-2486373400396982903
application-minio: 20:45:51 INFO unit.minio/0.juju-log minio:0: Reactive main running for hook minio-relation-changed
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-install-1272112431368911998
application-minio: 20:45:52 DEBUG jujuc running hook tool "relation-ids" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:52 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: starting handler dispatch, 30 flags set
tracer: set flag config.changed
tracer: set flag config.changed.database
tracer: set flag config.changed.ha-image
tracer: set flag config.changed.ha-mode
tracer: set flag config.changed.image
tracer: set flag config.changed.mysql_port
tracer: set flag config.changed.password
tracer: set flag config.changed.query-cache-size
tracer: set flag config.changed.query-cache-type
tracer: set flag config.changed.root_password
tracer: set flag config.changed.user
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag leadership.is_leader
application-minio: 20:45:52 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:52 DEBUG unit.pipelines-db/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:45:52 INFO juju.worker.uniter.operation ran "minio-relation-joined" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:45:52 DEBUG juju.worker.uniter.operation committing operation "run relation-joined (3; unit: minio/0) hook" for pipelines-api/0
controller-0: 20:45:52 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-api: 20:45:52 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-joined (3; unit: minio/0) hook)
application-pipelines-api: 20:45:52 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:52 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:45:52 DEBUG juju.worker.uniter.operation running operation run relation-changed (3; unit: minio/0) hook for pipelines-api/0
application-pipelines-api: 20:45:52 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-changed (3; unit: minio/0) hook)
application-pipelines-api: 20:45:52 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-changed (3; unit: minio/0) hook)
application-pipelines-api: 20:45:52 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (3; unit: minio/0) hook" for pipelines-api/0
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:52 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: main dispatch loop, 3 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:25:_handle_broken:mysql
tracer: ++   queue handler reactive/mariadb.py:22:restart
tracer: ++   queue handler reactive/mariadb.py:28:configure
application-pipelines-api: 20:45:52 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (3; unit: minio/0) hook" for pipelines-api/0
application-pipelines-api: 20:45:52 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-changed hook
application-pipelines-api: 20:45:52 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:52 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:22:restart
application-minio: 20:45:52 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:52 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:28:configure
application-minio: 20:45:52 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:52 INFO unit.pipelines-db/0.juju-log status-set: maintenance: Configuring mariadb-k8s container
application-minio: 20:45:52 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-api: 20:45:52 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-minio-relation-changed-3737828802717484300
controller-0: 20:45:52 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-install-1272112431368911998
application-minio: 20:45:52 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-2486373400396982903
application-minio: 20:45:52 DEBUG unit.minio/0.juju-log minio:0: tracer>
tracer: starting handler dispatch, 9 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-api: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:52 INFO unit.pipelines-api/0.juju-log minio:3: Reactive main running for hook minio-relation-changed
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-persistence-0"
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc0001d5680}
application-pipelines-persistence: 20:45:52 INFO juju.worker.caasoperator operator "pipelines-persistence" started
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:52 DEBUG unit.pipelines-db/0.juju-log tracer: set flag user
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.uniter starting uniter for "pipelines-persistence/0"
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.caasoperator start "pipelines-persistence/0"
application-pipelines-persistence: 20:45:52 INFO juju.worker.caasoperator start "pipelines-persistence/0"
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.leadership pipelines-persistence/0 making initial claim for pipelines-persistence leadership
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.caasoperator "pipelines-persistence/0" started
application-pipelines-persistence: 20:45:52 INFO juju.worker.leadership pipelines-persistence/0 promoted to leadership of pipelines-persistence
controller-0: 20:45:52 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-persistence: 20:45:52 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-persistence-0
application-pipelines-persistence: 20:45:52 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pipelines-persistence-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-persistence-0/jujud
application-pipelines-persistence: 20:45:52 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pipelines-persistence-0/jujud
application-minio: 20:45:52 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-2486373400396982903
application-minio: 20:45:52 DEBUG unit.minio/0.juju-log minio:0: tracer: hooks phase, 0 handlers queued
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-persistence-0/run.socket <nil>}
application-pipelines-persistence: 20:45:52 INFO juju.worker.uniter unit "pipelines-persistence/0" started
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-persistence: 20:45:52 INFO juju.worker.uniter resuming charm install
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.uniter.operation running operation install local:kubernetes/pipelines-persistence-0 for pipelines-persistence/0
application-pipelines-persistence: 20:45:52 DEBUG juju.worker.uniter.operation preparing operation "install local:kubernetes/pipelines-persistence-0" for pipelines-persistence/0
application-pipelines-persistence: 20:45:52 INFO juju.worker.uniter.charm downloading local:kubernetes/pipelines-persistence-0 from API server
application-pipelines-persistence: 20:45:52 INFO juju.downloader downloading from local:kubernetes/pipelines-persistence-0
application-pipelines-api: 20:45:52 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-persistence: 20:45:52 INFO juju.downloader download complete ("local:kubernetes/pipelines-persistence-0")
application-pipelines-persistence: 20:45:52 INFO juju.downloader download verified ("local:kubernetes/pipelines-persistence-0")
application-minio: 20:45:52 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-2486373400396982903
application-minio: 20:45:52 DEBUG unit.minio/0.juju-log minio:0: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-pipelines-db: 20:45:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:52 DEBUG unit.pipelines-db/0.juju-log tracer: set flag password
application-pipelines-api: 20:45:52 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-changed-3737828802717484300
controller-0: 20:45:52 DEBUG juju.worker.caasunitprovisioner service for pytorch-operator(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-minio: 20:45:53 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-2486373400396982903
application-minio: 20:45:53 INFO unit.minio/0.juju-log minio:0: Invoking reactive handler: reactive/minio.py:13:charm_ready
application-pipelines-persistence: 20:45:53 DEBUG juju.worker.uniter.operation executing operation "install local:kubernetes/pipelines-persistence-0" for pipelines-persistence/0
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-persistence: 20:45:53 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-persistence-0"
application-pipelines-db: 20:45:53 DEBUG unit.pipelines-db/0.juju-log tracer: set flag database
application-pipelines-persistence: 20:45:53 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-persistence-0"
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-minio: 20:45:53 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-2486373400396982903
application-minio: 20:45:53 INFO unit.minio/0.juju-log minio:0: Invoking reactive handler: reactive/minio.py:18:configure_minio
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:53 DEBUG unit.pipelines-db/0.juju-log tracer: set flag root_password
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-minio: 20:45:53 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:53 INFO unit.pipelines-db/0.juju-log set pod spec:
version: 2
containers:
  - name: mariadb-k8s
    image: rocks.canonical.com:443/mariadb/server:10.3
    ports:
      - containerPort: 3306
        protocol: TCP
        name: main
    config:
      MARIADB_ROOT_PASSWORD: root
      MARIADB_USER: mysql
      MARIADB_PASSWORD: password
      MARIADB_DATABASE: database
    kubernetes:
      readinessProbe:
        tcpSocket:
          port: 3306
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 3
      livenessProbe:
        tcpSocket:
          port: 3306
        initialDelaySeconds: 120
        periodSeconds: 10
        timeoutSeconds: 5
        successThreshold: 1
        failureThreshold: 3

application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-minio: 20:45:53 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:53 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-minio: 20:45:53 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:53 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: set flag mariadb-k8s.configured
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
tracer: -- dequeue handler reactive/mariadb.py:28:configure
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-minio: 20:45:53 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:53 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: hooks/relations/mysql/provides.py:25:_handle_broken:mysql
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:53 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-minio: 20:45:53 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:53 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer>
tracer: starting handler dispatch, 17 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:53 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-install-1272112431368911998
application-pipelines-db: 20:45:53 INFO unit.pipelines-db/0.juju-log status-set: active: ready
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:53 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:45:53 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-install-1272112431368911998
application-pipelines-api: 20:45:53 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:53 DEBUG unit.pipelines-api/0.juju-log minio:3: tracer>
tracer: main dispatch loop, 5 handlers queued
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-persistence: 20:45:53 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-persistence-0"
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.operation committing operation "install local:kubernetes/pipelines-persistence-0" for pipelines-persistence/0
application-minio: 20:45:54 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-2486373400396982903
application-minio: 20:45:54 INFO unit.minio/0.juju-log minio:0: status-set: active: 
application-pipelines-persistence: 20:45:54 INFO juju.worker.uniter hooks are retried true
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got storage change for pipelines-persistence/0: [] ok=true
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got unit change for pipelines-persistence/0
application-pipelines-api: 20:45:54 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:54 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got leader settings change for pipelines-persistence/0: ok=true
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got config change for pipelines-persistence/0: ok=true, hashes=[1eff9126f7d1b06a64c47ad558690f3b2344a64304e49235317bc50378f4fe21]
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got update status interval change for pipelines-persistence/0: ok=true
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got trust config change for pipelines-persistence/0: ok=true, hashes=[af533a75f93e76b4f389d29e2474d4285fc44bd818475ce07660626b5226319b]
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got relations change for pipelines-persistence/0: ok=true
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got action change for pipelines-persistence/0: [] ok=true
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.remotestate got address change for pipelines-persistence/0: ok=true, hashes=[]
application-pipelines-persistence: 20:45:54 INFO juju.worker.uniter found queued "install" hook
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.operation running operation run install hook for pipelines-persistence/0
application-pipelines-persistence: 20:45:54 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run install hook)
application-minio: 20:45:54 DEBUG jujuc running hook tool "status-set" for minio/0-minio-relation-changed-2486373400396982903
application-pipelines-persistence: 20:45:54 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run install hook)
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.operation executing operation "run install hook" for pipelines-persistence/0
application-minio: 20:45:54 DEBUG unit.minio/0.minio-relation-changed Provides
application-minio: 20:45:54 DEBUG unit.minio/0.minio-relation-changed YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:54 DEBUG unit.minio/0.minio-relation-changed minio:0
application-minio: 20:45:54 DEBUG unit.minio/0.minio-relation-changed Provides
application-minio: 20:45:54 DEBUG unit.minio/0.minio-relation-changed YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:54 DEBUG unit.minio/0.minio-relation-changed minio:3
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-pipelines-persistence: 20:45:54 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
application-pipelines-api: 20:45:54 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:54 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:45:54 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:54 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-db: 20:45:54 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:45:54 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:54 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation committing operation "run install hook" for pipelines-db/0
application-pipelines-db: 20:45:54 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run install hook)
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:45:54 INFO juju.worker.uniter.relation joining relation "pipelines-api:mysql pipelines-db:mysql"
application-pipelines-api: 20:45:54 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-minio-relation-changed-3737828802717484300
application-pipelines-api: 20:45:54 INFO unit.pipelines-api/0.juju-log minio:3: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql
application-minio: 20:45:54 INFO juju.worker.uniter.operation ran "minio-relation-changed" hook (via explicit, bespoke hook script)
controller-0: 20:45:54 DEBUG juju.worker.caasunitprovisioner units changed: true
application-minio: 20:45:54 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (0; unit: argo-controller/0) hook" for minio/0
application-pipelines-db: 20:45:54 INFO juju.worker.uniter.relation joined relation "pipelines-api:mysql pipelines-db:mysql"
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation running operation run relation-created (1; app: pipelines-api) hook for pipelines-db/0
application-pipelines-db: 20:45:54 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run relation-created (1; app: pipelines-api) hook)
application-pipelines-db: 20:45:54 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run relation-created (1; app: pipelines-api) hook)
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation preparing operation "run relation-created (1; app: pipelines-api) hook" for pipelines-db/0
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter.remotestate got a relation units change for pipelines-api/0 : {1 {map[pipelines-db/0:{0}] map[] []}}
application-minio: 20:45:54 DEBUG juju.machinelock machine lock released for minio/0 uniter (run relation-changed (0; unit: argo-controller/0) hook)
application-minio: 20:45:54 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:54 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:45:54 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:45:54 DEBUG juju.worker.uniter.operation running operation run relation-changed (3; unit: pipelines-api/0) hook for minio/0
application-minio: 20:45:54 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run relation-changed (3; unit: pipelines-api/0) hook)
application-minio: 20:45:54 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run relation-changed (3; unit: pipelines-api/0) hook)
application-minio: 20:45:54 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (3; unit: pipelines-api/0) hook" for minio/0
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation executing operation "run relation-created (1; app: pipelines-api) hook" for pipelines-db/0
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running mysql-relation-created hook
application-pipelines-persistence: 20:45:54 DEBUG jujuc running hook tool "config-get" for pipelines-persistence/0-install-598323711581072047
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-pipelines-db: 20:45:54 INFO juju.worker.uniter.operation skipped "mysql-relation-created" hook (missing)
application-minio: 20:45:54 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (3; unit: pipelines-api/0) hook" for minio/0
application-minio: 20:45:54 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation committing operation "run relation-created (1; app: pipelines-api) hook" for pipelines-db/0
application-pipelines-db: 20:45:54 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run relation-created (1; app: pipelines-api) hook)
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation running operation accept leadership for pipelines-db/0
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for pipelines-db/0
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for pipelines-db/0
application-pipelines-persistence: 20:45:54 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:54 INFO unit.pipelines-persistence/0.juju-log Reactive main running for hook install
application-pipelines-api: 20:45:54 INFO juju.worker.uniter.operation ran "minio-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:45:54 INFO juju.worker.uniter found queued "leader-elected" hook
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for pipelines-db/0
controller-0: 20:45:54 DEBUG juju.worker.caasunitprovisioner operator update for pytorch-operator
application-pipelines-db: 20:45:54 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run leader-elected hook)
application-pipelines-db: 20:45:54 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run leader-elected hook)
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for pipelines-db/0
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (3; unit: minio/0) hook" for pipelines-api/0
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for pipelines-db/0
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-pipelines-db: 20:45:54 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-pipelines-api: 20:45:54 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-changed (3; unit: minio/0) hook)
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter.operation running operation run relation-joined (1; unit: pipelines-db/0) hook for pipelines-api/0
application-pipelines-api: 20:45:54 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-joined (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:45:54 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-joined (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter.operation preparing operation "run relation-joined (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-persistence: 20:45:54 DEBUG jujuc running hook tool "relation-ids" for pipelines-persistence/0-install-598323711581072047
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter.operation executing operation "run relation-joined (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running mysql-relation-joined hook
controller-0: 20:45:54 DEBUG juju.worker.caasunitprovisioner service for minio(workload): &{Id:0f6fc8e3-48a5-427a-a263-04533dd18de2 Addresses:[local-cloud:10.152.183.102] Scale:0xc002b7cba0 Generation:0xc002b7cba8 Status:{Status:active Message: Data:map[] Since:<nil>}}
controller-0: 20:45:54 DEBUG juju.worker.caasunitprovisioner service for seldon-core(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:45:54 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-minio: 20:45:54 DEBUG jujuc running hook tool "config-get" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-db: 20:45:54 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-persistence: 20:45:54 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:54 DEBUG unit.pipelines-persistence/0.juju-log tracer: starting handler dispatch, 0 flags set
application-pipelines-db: 20:45:54 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-db: 20:45:54 INFO unit.pipelines-db/0.juju-log Reactive main running for hook leader-elected
application-minio: 20:45:55 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-1795589093372921465
application-minio: 20:45:55 INFO unit.minio/0.juju-log minio:3: Reactive main running for hook minio-relation-changed
application-pipelines-persistence: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:55 DEBUG unit.pipelines-persistence/0.juju-log tracer: hooks phase, 0 handlers queued
controller-0: 20:45:55 DEBUG juju.worker.caasunitprovisioner service for pipelines-api(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-pipelines-db: 20:45:55 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-api: 20:45:55 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-minio: 20:45:55 DEBUG jujuc running hook tool "relation-ids" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-persistence: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:55 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-api
tracer: ++   queue handler reactive/docker_resource.py:7:auto_fetch
application-pipelines-persistence: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:55 INFO unit.pipelines-persistence/0.juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch
application-minio: 20:45:55 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-db: 20:45:55 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-api: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:55 INFO unit.pipelines-api/0.juju-log mysql:1: Reactive main running for hook mysql-relation-joined
application-pipelines-persistence: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:55 DEBUG unit.pipelines-persistence/0.juju-log tracer: set flag layer.docker-resource.oci-image.fetched
controller-0: 20:45:55 DEBUG juju.worker.caasunitprovisioner service for pipelines-scheduledworkflow(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
controller-0: 20:45:55 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-minio: 20:45:55 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-db: 20:45:55 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-api: 20:45:55 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-persistence: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:55 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: set flag layer.docker-resource.pending
tracer: ++   queue handler reactive/docker_resource.py:18:fetch
application-pipelines-db: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-db: 20:45:55 INFO unit.pipelines-db/0.juju-log Initializing Leadership Layer (is leader)
application-minio: 20:45:55 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-persistence: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:55 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: set flag layer.docker-resource.auto-fetched
tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch
application-pipelines-api: 20:45:55 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-db: 20:45:55 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-api: 20:45:55 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-minio: 20:45:55 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-persistence: 20:45:55 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:55 INFO unit.pipelines-persistence/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-api
application-pipelines-db: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-db: 20:45:56 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: starting handler dispatch, 24 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-api: 20:45:56 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-minio: 20:45:56 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-1795589093372921465
application-minio: 20:45:56 DEBUG unit.minio/0.juju-log minio:3: tracer>
tracer: starting handler dispatch, 9 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-persistence: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:56 INFO unit.pipelines-persistence/0.juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch
application-pipelines-db: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-db: 20:45:56 DEBUG unit.pipelines-db/0.juju-log tracer: hooks phase, 0 handlers queued
application-minio: 20:45:56 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-1795589093372921465
application-minio: 20:45:56 DEBUG unit.minio/0.juju-log minio:3: tracer: hooks phase, 0 handlers queued
controller-0: 20:45:56 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-api: 20:45:56 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-persistence: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:56 INFO unit.pipelines-persistence/0.juju-log status-set: maintenance: fetching resource: oci-image
application-pipelines-db: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-db: 20:45:56 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:25:_handle_broken:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-minio: 20:45:56 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-1795589093372921465
application-minio: 20:45:56 DEBUG unit.minio/0.juju-log minio:3: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-pipelines-persistence: 20:45:56 DEBUG jujuc running hook tool "status-set" for pipelines-persistence/0-install-598323711581072047
application-pipelines-api: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:56 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: set flag endpoint.mysql.joined
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-db: 20:45:56 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-minio: 20:45:56 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-1795589093372921465
application-minio: 20:45:56 INFO unit.minio/0.juju-log minio:3: Invoking reactive handler: reactive/minio.py:13:charm_ready
application-pipelines-persistence: 20:45:56 DEBUG jujuc running hook tool "resource-get" for pipelines-persistence/0-install-598323711581072047
application-pipelines-api: 20:45:56 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-minio: 20:45:56 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-1795589093372921465
application-minio: 20:45:56 INFO unit.minio/0.juju-log minio:3: Invoking reactive handler: reactive/minio.py:18:configure_minio
application-pipelines-db: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-db: 20:45:56 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: hooks/relations/mysql/provides.py:25:_handle_broken:mysql
controller-0: 20:45:56 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-minio: 20:45:56 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-api: 20:45:56 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-minio: 20:45:56 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-db: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-leader-elected-7869355530064258228
application-pipelines-db: 20:45:56 INFO unit.pipelines-db/0.juju-log status-set: active: ready
application-pipelines-persistence: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:56 DEBUG unit.pipelines-persistence/0.juju-log tracer: set flag layer.docker-resource.oci-image.available
application-pipelines-api: 20:45:56 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-joined-4642336872551915597
controller-0: 20:45:56 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-db: 20:45:56 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-leader-elected-7869355530064258228
application-minio: 20:45:56 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-persistence: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-install-598323711581072047
application-pipelines-persistence: 20:45:56 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: cleared flag layer.docker-resource.pending
tracer: -- dequeue handler reactive/docker_resource.py:18:fetch
application-pipelines-api: 20:45:56 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-minio: 20:45:56 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-api: 20:45:56 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:56 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: starting handler dispatch, 18 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag endpoint.mysql.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:45:57 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-api: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:57 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: hooks phase, 0 handlers queued
application-minio: 20:45:57 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-api: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:57 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: main dispatch loop, 5 handlers queued
tracer: ++   queue handler hooks/relations/http/provides.py:15:broken:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:45:57 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for pipelines-db/0
application-pipelines-persistence: 20:45:57 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation committing operation "run install hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run install hook)
application-pipelines-db: 20:45:57 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run leader-elected hook)
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:0 Life:alive Attached:false Location:}
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter.storage still pending [storage-database-1]
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter.operation running operation run config-changed hook for pipelines-db/0
application-pipelines-db: 20:45:57 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run config-changed hook)
application-pipelines-db: 20:45:57 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run config-changed hook)
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for pipelines-db/0
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 INFO juju.worker.uniter.relation joining relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api"
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for pipelines-db/0
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-minio: 20:45:57 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-1795589093372921465
application-minio: 20:45:57 INFO unit.minio/0.juju-log minio:3: status-set: active: 
application-pipelines-db: 20:45:57 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
controller-0: 20:45:57 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:45:57 DEBUG juju.worker.caasunitprovisioner service for tf-job-operator(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-pipelines-persistence: 20:45:57 INFO juju.worker.uniter.relation joined relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api"
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation running operation run relation-created (2; app: pipelines-api) hook for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run relation-created (2; app: pipelines-api) hook)
application-pipelines-api: 20:45:57 DEBUG juju.worker.uniter.remotestate got a relation units change for pipelines-api/0 : {2 {map[pipelines-persistence/0:{0}] map[] []}}
application-pipelines-persistence: 20:45:57 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run relation-created (2; app: pipelines-api) hook)
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation preparing operation "run relation-created (2; app: pipelines-api) hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation executing operation "run relation-created (2; app: pipelines-api) hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running pipelines-api-relation-created hook
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
application-pipelines-persistence: 20:45:57 INFO juju.worker.uniter.operation skipped "pipelines-api-relation-created" hook (missing)
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation committing operation "run relation-created (2; app: pipelines-api) hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run relation-created (2; app: pipelines-api) hook)
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation running operation accept leadership for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 INFO juju.worker.uniter found queued "leader-elected" hook
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run leader-elected hook)
application-pipelines-persistence: 20:45:57 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run leader-elected hook)
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for pipelines-persistence/0
application-pipelines-api: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:57 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-minio: 20:45:57 DEBUG jujuc running hook tool "status-set" for minio/0-minio-relation-changed-1795589093372921465
application-pipelines-persistence: 20:45:57 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
application-minio: 20:45:57 DEBUG unit.minio/0.minio-relation-changed Provides
application-minio: 20:45:57 DEBUG unit.minio/0.minio-relation-changed YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:57 DEBUG unit.minio/0.minio-relation-changed minio:0
application-minio: 20:45:57 DEBUG unit.minio/0.minio-relation-changed Provides
application-minio: 20:45:57 DEBUG unit.minio/0.minio-relation-changed YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:45:57 DEBUG unit.minio/0.minio-relation-changed minio:3
application-pipelines-api: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:57 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:57 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-persistence: 20:45:57 DEBUG jujuc running hook tool "config-get" for pipelines-persistence/0-leader-elected-1058602104053275682
application-pipelines-db: 20:45:57 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-api: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:57 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api
application-minio: 20:45:57 INFO juju.worker.uniter.operation ran "minio-relation-changed" hook (via explicit, bespoke hook script)
application-minio: 20:45:57 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (3; unit: pipelines-api/0) hook" for minio/0
application-pipelines-persistence: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-leader-elected-1058602104053275682
application-pipelines-persistence: 20:45:57 INFO unit.pipelines-persistence/0.juju-log Reactive main running for hook leader-elected
application-minio: 20:45:57 DEBUG juju.machinelock machine lock released for minio/0 uniter (run relation-changed (3; unit: pipelines-api/0) hook)
application-minio: 20:45:57 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:45:57 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:45:57 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:45:57 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-minio: 20:45:57 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:45:57 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-db: 20:45:57 INFO unit.pipelines-db/0.juju-log Reactive main running for hook config-changed
application-pipelines-api: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:57 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql
application-pipelines-persistence: 20:45:57 DEBUG jujuc running hook tool "relation-ids" for pipelines-persistence/0-leader-elected-1058602104053275682
application-pipelines-db: 20:45:57 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-persistence: 20:45:57 DEBUG jujuc running hook tool "relation-list" for pipelines-persistence/0-leader-elected-1058602104053275682
application-pipelines-api: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-joined-4642336872551915597
application-pipelines-api: 20:45:57 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag mysql.connected
application-pipelines-persistence: 20:45:57 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-leader-elected-1058602104053275682
application-pipelines-persistence: 20:45:57 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: starting handler dispatch, 3 flags set
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-persistence: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-leader-elected-1058602104053275682
application-pipelines-persistence: 20:45:58 DEBUG unit.pipelines-persistence/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-persistence: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-leader-elected-1058602104053275682
application-pipelines-persistence: 20:45:58 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-api
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-db: 20:45:58 INFO unit.pipelines-db/0.juju-log Initializing Leadership Layer (is leader)
application-pipelines-persistence: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-leader-elected-1058602104053275682
application-pipelines-persistence: 20:45:58 INFO unit.pipelines-persistence/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-api
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-api: 20:45:58 INFO juju.worker.uniter.operation ran "mysql-relation-joined" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:45:58 DEBUG juju.worker.uniter.operation committing operation "run relation-joined (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-api: 20:45:58 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-joined (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:45:58 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:45:58 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:45:58 DEBUG juju.worker.uniter.operation running operation run relation-joined (2; unit: pipelines-persistence/0) hook for pipelines-api/0
application-pipelines-api: 20:45:58 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-joined (2; unit: pipelines-persistence/0) hook)
application-pipelines-api: 20:45:58 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-joined (2; unit: pipelines-persistence/0) hook)
application-pipelines-api: 20:45:58 DEBUG juju.worker.uniter.operation preparing operation "run relation-joined (2; unit: pipelines-persistence/0) hook" for pipelines-api/0
application-pipelines-api: 20:45:58 DEBUG juju.worker.uniter.operation executing operation "run relation-joined (2; unit: pipelines-persistence/0) hook" for pipelines-api/0
application-pipelines-api: 20:45:58 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running pipelines-api-relation-joined hook
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-db: 20:45:58 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: starting handler dispatch, 24 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-api: 20:45:58 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-db: 20:45:58 DEBUG unit.pipelines-db/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-db: 20:45:58 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:25:_handle_broken:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-pipelines-api: 20:45:58 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-persistence: 20:45:58 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-db: 20:45:58 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-pipelines-persistence: 20:45:58 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:58 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run leader-elected hook)
application-pipelines-persistence: 20:45:58 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:45:58 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:45:58 DEBUG juju.worker.uniter.operation running operation run config-changed hook for pipelines-persistence/0
application-pipelines-persistence: 20:45:58 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run config-changed hook)
application-pipelines-persistence: 20:45:58 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run config-changed hook)
application-pipelines-persistence: 20:45:58 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:58 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:58 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-pipelines-persistence: 20:45:58 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-db: 20:45:58 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: hooks/relations/mysql/provides.py:25:_handle_broken:mysql
application-pipelines-api: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:45:58 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Reactive main running for hook pipelines-api-relation-joined
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-db: 20:45:58 INFO unit.pipelines-db/0.juju-log status-set: active: ready
application-pipelines-db: 20:45:58 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-config-changed-2860321564806573748
application-pipelines-api: 20:45:58 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-persistence: 20:45:58 DEBUG jujuc running hook tool "config-get" for pipelines-persistence/0-config-changed-2419004674812017449
controller-0: 20:45:58 DEBUG juju.worker.caasunitprovisioner ensured deployment for minio for 1 units
application-pipelines-persistence: 20:45:58 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-config-changed-2419004674812017449
application-pipelines-persistence: 20:45:58 INFO unit.pipelines-persistence/0.juju-log Reactive main running for hook config-changed
application-pipelines-api: 20:45:58 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-persistence: 20:45:58 DEBUG jujuc running hook tool "relation-ids" for pipelines-persistence/0-config-changed-2419004674812017449
application-pipelines-api: 20:45:58 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-persistence: 20:45:58 DEBUG jujuc running hook tool "relation-list" for pipelines-persistence/0-config-changed-2419004674812017449
application-pipelines-api: 20:45:58 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
controller-0: 20:45:59 DEBUG juju.worker.caasunitprovisioner service for argo-controller(workload): &{Id: Addresses:[] Scale:0xc00387ea28 Generation:0xc00387ea40 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-db: 20:45:59 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-pipelines-persistence: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-config-changed-2419004674812017449
application-pipelines-persistence: 20:45:59 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: starting handler dispatch, 3 flags set
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-db: 20:45:59 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for pipelines-db/0
application-pipelines-db: 20:45:59 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run config-changed hook)
application-pipelines-db: 20:45:59 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:45:59 INFO juju.worker.uniter found queued "start" hook
application-pipelines-db: 20:45:59 DEBUG juju.worker.uniter.operation running operation run start hook for pipelines-db/0
application-pipelines-db: 20:45:59 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run start hook)
application-pipelines-db: 20:45:59 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run start hook)
application-pipelines-db: 20:45:59 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for pipelines-db/0
application-pipelines-api: 20:45:59 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-db: 20:45:59 DEBUG juju.worker.uniter.operation executing operation "run start hook" for pipelines-db/0
application-pipelines-db: 20:45:59 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-pipelines-persistence: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-config-changed-2419004674812017449
application-pipelines-persistence: 20:45:59 DEBUG unit.pipelines-persistence/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:45:59 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-pipelines-api: 20:45:59 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
controller-0: 20:45:59 DEBUG juju.worker.caasunitprovisioner service for pipelines-persistence(workload): &{Id: Addresses:[] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
application-pipelines-persistence: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-config-changed-2419004674812017449
application-pipelines-persistence: 20:45:59 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-api
application-pipelines-api: 20:45:59 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-db: 20:45:59 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-start-1924719056568943209
application-pipelines-persistence: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-config-changed-2419004674812017449
application-pipelines-persistence: 20:45:59 INFO unit.pipelines-persistence/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-api
controller-0: 20:45:59 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-api: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:45:59 DEBUG unit.pipelines-api/0.juju-log pipelines-api:2: tracer>
tracer: set flag endpoint.pipelines-api.joined
tracer: ++   queue handler hooks/relations/http/provides.py:11:joined:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:45:59 INFO unit.pipelines-db/0.juju-log Reactive main running for hook start
application-pipelines-api: 20:45:59 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-db: 20:45:59 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-start-1924719056568943209
application-pipelines-api: 20:45:59 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-db: 20:45:59 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-start-1924719056568943209
application-pipelines-api: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:45:59 DEBUG unit.pipelines-api/0.juju-log pipelines-api:2: tracer>
tracer: starting handler dispatch, 20 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag endpoint.mysql.joined
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag mysql.connected
application-pipelines-persistence: 20:45:59 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:45:59 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-start-1924719056568943209
application-pipelines-persistence: 20:45:59 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:59 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run config-changed hook)
application-pipelines-persistence: 20:45:59 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:45:59 INFO juju.worker.uniter found queued "start" hook
application-pipelines-persistence: 20:45:59 DEBUG juju.worker.uniter.operation running operation run start hook for pipelines-persistence/0
application-pipelines-persistence: 20:45:59 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run start hook)
application-pipelines-persistence: 20:45:59 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run start hook)
application-pipelines-persistence: 20:45:59 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:59 DEBUG juju.worker.uniter.operation executing operation "run start hook" for pipelines-persistence/0
application-pipelines-persistence: 20:45:59 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-pipelines-db: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:45:59 INFO unit.pipelines-db/0.juju-log Initializing Leadership Layer (is leader)
application-pipelines-persistence: 20:45:59 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
application-pipelines-api: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:45:59 DEBUG unit.pipelines-api/0.juju-log pipelines-api:2: tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:45:59 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:45:59 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:45:59 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: starting handler dispatch, 24 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-db: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:46:00 DEBUG unit.pipelines-db/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:46:00 DEBUG unit.pipelines-api/0.juju-log pipelines-api:2: tracer>
tracer: main dispatch loop, 5 handlers queued
tracer: ++   queue handler hooks/relations/http/provides.py:11:joined:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:46:00 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:25:_handle_broken:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-pipelines-persistence: 20:46:00 DEBUG jujuc running hook tool "config-get" for pipelines-persistence/0-start-8262521491245832847
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:46:00 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:46:00 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:46:00 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-persistence: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-start-8262521491245832847
application-pipelines-persistence: 20:46:00 INFO unit.pipelines-persistence/0.juju-log Reactive main running for hook start
application-pipelines-db: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:46:00 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: hooks/relations/mysql/provides.py:25:_handle_broken:mysql
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:46:00 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-db: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-start-1924719056568943209
application-pipelines-db: 20:46:00 INFO unit.pipelines-db/0.juju-log status-set: active: ready
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:46:00 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api
application-pipelines-persistence: 20:46:00 DEBUG jujuc running hook tool "relation-ids" for pipelines-persistence/0-start-8262521491245832847
application-pipelines-db: 20:46:00 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-start-1924719056568943209
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:46:00 DEBUG unit.pipelines-api/0.juju-log pipelines-api:2: tracer>
tracer: set flag pipelines-api.available
tracer: ++   queue handler reactive/pipelines_api.py:31:configure_http
application-pipelines-persistence: 20:46:00 DEBUG jujuc running hook tool "relation-list" for pipelines-persistence/0-start-8262521491245832847
controller-0: 20:46:00 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-persistence
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:46:00 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql
application-pipelines-persistence: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-start-8262521491245832847
application-pipelines-persistence: 20:46:00 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: starting handler dispatch, 3 flags set
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-api: 20:46:00 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http
application-pipelines-persistence: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-start-8262521491245832847
application-pipelines-persistence: 20:46:00 DEBUG unit.pipelines-persistence/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-persistence: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-start-8262521491245832847
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-persistence: 20:46:00 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-api
application-pipelines-db: 20:46:00 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter.operation committing operation "run start hook" for pipelines-db/0
application-pipelines-db: 20:46:00 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run start hook)
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:0 Life:alive Attached:false Location:}
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter.storage still pending [storage-database-1]
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter.operation running operation run relation-joined (1; unit: pipelines-api/0) hook for pipelines-db/0
application-pipelines-db: 20:46:00 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run relation-joined (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:00 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run relation-joined (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter.operation preparing operation "run relation-joined (1; unit: pipelines-api/0) hook" for pipelines-db/0
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter.operation executing operation "run relation-joined (1; unit: pipelines-api/0) hook" for pipelines-db/0
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running mysql-relation-joined hook
application-pipelines-db: 20:46:00 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-pipelines-persistence: 20:46:00 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-start-8262521491245832847
application-pipelines-persistence: 20:46:00 INFO unit.pipelines-persistence/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-api
application-pipelines-api: 20:46:00 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-pipelines-api-relation-joined-4039870654822968888
application-pipelines-db: 20:46:00 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:01 INFO unit.pipelines-db/0.juju-log mysql:1: Reactive main running for hook mysql-relation-joined
application-pipelines-persistence: 20:46:01 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter.operation committing operation "run start hook" for pipelines-persistence/0
application-pipelines-persistence: 20:46:01 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run start hook)
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter.operation running operation run relation-joined (2; unit: pipelines-api/0) hook for pipelines-persistence/0
application-pipelines-persistence: 20:46:01 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run relation-joined (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:01 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run relation-joined (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter.operation preparing operation "run relation-joined (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter.operation executing operation "run relation-joined (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running pipelines-api-relation-joined hook
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
application-pipelines-api: 20:46:01 INFO juju.worker.uniter.operation ran "pipelines-api-relation-joined" hook (via explicit, bespoke hook script)
application-pipelines-persistence: 20:46:01 DEBUG juju.worker.uniter.remotestate got a relation units change for pipelines-persistence/0 : {2 {map[pipelines-api/0:{1}] map[] []}}
application-pipelines-api: 20:46:01 DEBUG juju.worker.uniter.operation committing operation "run relation-joined (2; unit: pipelines-persistence/0) hook" for pipelines-api/0
application-pipelines-api: 20:46:01 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-joined (2; unit: pipelines-persistence/0) hook)
application-pipelines-api: 20:46:01 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:46:01 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:46:01 DEBUG juju.worker.uniter.operation running operation run relation-changed (1; unit: pipelines-db/0) hook for pipelines-api/0
application-pipelines-api: 20:46:01 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-changed (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:46:01 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-changed (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:46:01 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-api: 20:46:01 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-api: 20:46:01 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running mysql-relation-changed hook
application-pipelines-api: 20:46:01 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:01 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: set flag endpoint.mysql.joined
tracer: ++   queue handler hooks/relations/mysql/provides.py:20:_handle_joined:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-pipelines-persistence: 20:46:01 DEBUG jujuc running hook tool "config-get" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "relation-get" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-api: 20:46:01 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-persistence: 20:46:01 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:01 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Reactive main running for hook pipelines-api-relation-joined
application-pipelines-api: 20:46:01 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:01 INFO unit.pipelines-api/0.juju-log mysql:1: Reactive main running for hook mysql-relation-changed
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-persistence: 20:46:01 DEBUG jujuc running hook tool "relation-ids" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-api: 20:46:01 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:01 INFO unit.pipelines-db/0.juju-log mysql:1: Initializing Leadership Layer (is leader)
application-pipelines-persistence: 20:46:01 DEBUG jujuc running hook tool "relation-list" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-api: 20:46:01 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-persistence: 20:46:01 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:01 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer: set flag endpoint.pipelines-api.joined
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-api: 20:46:01 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-minio: 20:46:01 DEBUG juju.worker.uniter.remotestate storage attachment change for minio/0: {storage-minio-data-0 {2 alive true /data}}
application-minio: 20:46:01 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:46:01 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:46:01 DEBUG jujuc running hook tool "relation-get" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
controller-0: 20:46:01 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-db: 20:46:01 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:01 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: starting handler dispatch, 25 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag endpoint.mysql.joined
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: set flag endpoint.pipelines-api.changed
tracer: ++   queue handler hooks/relations/http/requires.py:10:changed:pipelines-api
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:02 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer: hooks phase, 0 handlers queued
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer: set flag endpoint.pipelines-api.changed.hostname
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:02 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:20:_handle_joined:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer: set flag endpoint.pipelines-api.changed.port
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:02 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: starting handler dispatch, 7 flags set
tracer: set flag endpoint.pipelines-api.changed
tracer: set flag endpoint.pipelines-api.changed.hostname
tracer: set flag endpoint.pipelines-api.changed.port
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:02 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/mysql/provides.py:20:_handle_joined:mysql
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:02 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: set flag mysql.database.requested
tracer: ++   queue handler reactive/mariadb.py:50:provide_database
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler hooks/relations/http/requires.py:10:changed:pipelines-api
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:02 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: starting handler dispatch, 21 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag endpoint.mysql.joined
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag mysql.connected
tracer: set flag pipelines-api.available
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:02 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: reactive/mariadb.py:50:provide_database
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-api
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:02 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:02 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: main dispatch loop, 6 handlers queued
tracer: ++   queue handler hooks/relations/http/provides.py:11:joined:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler reactive/pipelines_api.py:31:configure_http
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: set flag pipelines-api.available
tracer: ++   queue handler reactive/pipelines_persistence.py:23:start_charm
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:02 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-persistence: 20:46:02 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Invoking reactive handler: reactive/pipelines_persistence.py:23:start_charm
controller-0: 20:46:02 DEBUG juju.worker.caasunitprovisioner operator update for minio
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "is-leader" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:02 INFO unit.pipelines-db/0.juju-log mysql:1: The pod is not ready.
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:02 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-persistence: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:02 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: status-set: maintenance: configuring container
application-pipelines-db: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-db: 20:46:02 INFO unit.pipelines-db/0.juju-log mysql:1: status-set: active: ready
application-pipelines-api: 20:46:02 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:02 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "status-set" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-api: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:03 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-db: 20:46:03 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-mysql-relation-joined-4850416190347467311
application-pipelines-api: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:03 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:03 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
controller-0: 20:46:03 DEBUG juju.worker.caasunitprovisioner operator update for seldon-core
application-pipelines-api: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-api: 20:46:03 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:03 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: status-set: maintenance: creating container
application-pipelines-api: 20:46:03 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "status-set" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-api: 20:46:03 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-mysql-relation-changed-3080706905192581479
application-pipelines-db: 20:46:03 INFO juju.worker.uniter.operation ran "mysql-relation-joined" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter.operation committing operation "run relation-joined (1; unit: pipelines-api/0) hook" for pipelines-db/0
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:03 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: set flag charm.started
tracer: -- dequeue handler reactive/pipelines_persistence.py:23:start_charm
application-pipelines-db: 20:46:03 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run relation-joined (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:0 Life:alive Attached:false Location:}
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter.storage still pending [storage-database-1]
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter.operation running operation run relation-changed (1; unit: pipelines-api/0) hook for pipelines-db/0
application-pipelines-db: 20:46:03 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run relation-changed (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:03 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run relation-changed (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (1; unit: pipelines-api/0) hook" for pipelines-db/0
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (1; unit: pipelines-api/0) hook" for pipelines-db/0
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running mysql-relation-changed hook
application-pipelines-db: 20:46:03 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:03 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Invoking reactive handler: reactive/pipelines_persistence.py:13:charm_ready
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-persistence: 20:46:03 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: status-set: active: 
application-pipelines-api: 20:46:03 INFO juju.worker.uniter.operation ran "mysql-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:46:03 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-db: 20:46:03 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-api: 20:46:03 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-changed (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:46:03 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:46:03 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:46:03 DEBUG juju.worker.uniter.operation running operation run relation-changed (2; unit: pipelines-persistence/0) hook for pipelines-api/0
application-pipelines-api: 20:46:03 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-changed (2; unit: pipelines-persistence/0) hook)
application-pipelines-api: 20:46:03 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-changed (2; unit: pipelines-persistence/0) hook)
application-pipelines-api: 20:46:03 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (2; unit: pipelines-persistence/0) hook" for pipelines-api/0
application-pipelines-persistence: 20:46:03 DEBUG jujuc running hook tool "status-set" for pipelines-persistence/0-pipelines-api-relation-joined-5527532038358495253
application-pipelines-api: 20:46:03 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (2; unit: pipelines-persistence/0) hook" for pipelines-api/0
application-pipelines-api: 20:46:03 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running pipelines-api-relation-changed hook
application-pipelines-api: 20:46:03 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-db: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:03 INFO unit.pipelines-db/0.juju-log mysql:1: Reactive main running for hook mysql-relation-changed
application-pipelines-db: 20:46:03 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-mysql-relation-changed-7499748927614068682
controller-0: 20:46:03 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-api
application-pipelines-db: 20:46:03 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-api: 20:46:03 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-db: 20:46:03 DEBUG jujuc running hook tool "relation-get" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-persistence: 20:46:03 INFO juju.worker.uniter.operation ran "pipelines-api-relation-joined" hook (via explicit, bespoke hook script)
application-pipelines-persistence: 20:46:03 DEBUG juju.worker.uniter.operation committing operation "run relation-joined (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-pipelines-api: 20:46:03 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:03 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Reactive main running for hook pipelines-api-relation-changed
application-pipelines-persistence: 20:46:03 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run relation-joined (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:03 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:46:03 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:46:03 DEBUG juju.worker.uniter.operation running operation run relation-changed (2; unit: pipelines-api/0) hook for pipelines-persistence/0
application-pipelines-persistence: 20:46:03 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run relation-changed (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:03 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run relation-changed (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:03 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-pipelines-db: 20:46:03 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-persistence: 20:46:03 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-pipelines-persistence: 20:46:03 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running pipelines-api-relation-changed hook
application-pipelines-persistence: 20:46:03 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
controller-0: 20:46:04 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-minio: 20:46:04 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-api: 20:46:04 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-db: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:04 INFO unit.pipelines-db/0.juju-log mysql:1: Initializing Leadership Layer (is leader)
controller-0: 20:46:04 DEBUG juju.worker.caasunitprovisioner operator update for argo-controller
application-pipelines-db: 20:46:04 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-api: 20:46:04 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-persistence: 20:46:04 DEBUG jujuc running hook tool "config-get" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-db: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:04 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: starting handler dispatch, 26 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag endpoint.mysql.joined
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag mysql.database.requested
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-api: 20:46:04 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-persistence: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-persistence: 20:46:04 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Reactive main running for hook pipelines-api-relation-changed
application-pipelines-api: 20:46:04 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-db: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:04 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer: hooks phase, 0 handlers queued
application-pipelines-persistence: 20:46:04 DEBUG jujuc running hook tool "relation-ids" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-db: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:04 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: main dispatch loop, 3 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:20:_handle_joined:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
tracer: ++   queue handler reactive/mariadb.py:50:provide_database
application-pipelines-api: 20:46:04 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-persistence: 20:46:04 DEBUG jujuc running hook tool "relation-list" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-db: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:04 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-pipelines-persistence: 20:46:04 DEBUG jujuc running hook tool "relation-get" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-api: 20:46:04 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-persistence: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-persistence: 20:46:04 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: starting handler dispatch, 9 flags set
tracer: set flag charm.started
tracer: set flag endpoint.pipelines-api.changed
tracer: set flag endpoint.pipelines-api.changed.hostname
tracer: set flag endpoint.pipelines-api.changed.port
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag pipelines-api.available
application-pipelines-api: 20:46:04 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
controller-0: 20:46:04 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc003a6fa00}
application-pipelines-db: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:04 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: reactive/mariadb.py:50:provide_database
application-pipelines-persistence: 20:46:04 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-persistence: 20:46:04 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:46:04 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
controller-0: 20:46:04 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-argo-controller: 20:46:05 DEBUG juju.worker.caasoperator trigger running status for caas unit argo-controller/0
application-argo-controller: 20:46:05 DEBUG juju.worker.uniter.remotestate got running status change for argo-controller/0
application-argo-controller: 20:46:05 DEBUG juju.worker.caasoperator request running status for "unit-argo-controller-0" 98f9d8de-8d65-4cad-af65-85867d87b501
application-argo-controller: 20:46:05 DEBUG juju.kubernetes.provider.exec no pod named "98f9d8de-8d65-4cad-af65-85867d87b501" found
application-argo-controller: 20:46:05 DEBUG juju.kubernetes.provider.exec try get pod by UID for "98f9d8de-8d65-4cad-af65-85867d87b501"
controller-0: 20:46:05 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-scheduledworkflow
application-argo-controller: 20:46:05 DEBUG juju.worker.uniter.operation running operation remote init for argo-controller/0
application-argo-controller: 20:46:05 DEBUG juju.worker.uniter.operation preparing operation "remote init" for argo-controller/0
application-pipelines-persistence: 20:46:05 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-persistence: 20:46:05 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/http/requires.py:10:changed:pipelines-api
tracer: ++   queue handler reactive/pipelines_persistence.py:13:charm_ready
application-argo-controller: 20:46:05 DEBUG juju.worker.uniter.operation executing operation "remote init" for argo-controller/0
application-argo-controller: 20:46:05 DEBUG juju.worker.caasoperator request running status for "unit-argo-controller-0" argo-controller-5fdc64b6b4-t6r66
application-argo-controller: 20:46:05 DEBUG juju.worker.caasoperator remote init for "unit-argo-controller-0" {PodName:argo-controller-5fdc64b6b4-t6r66 Initialising:true InitialisingTime:2020-10-21 20:45:43 +0000 UTC Running:false}
application-argo-controller: 20:46:05 INFO juju.worker.caasoperator started pod init on "argo-controller/0"
application-argo-controller: 20:46:05 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/pezzogbk.pid; exec sh -c 'mkdir -p /tmp/unit-argo-controller-0971362255'; "]
application-pipelines-db: 20:46:05 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-persistence: 20:46:05 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-persistence: 20:46:05 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Invoking reactive handler: reactive/pipelines_persistence.py:13:charm_ready
application-pipelines-api: 20:46:05 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-argo-controller: 20:46:05 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-argo-controller/charm  } to {/tmp/unit-argo-controller-0971362255 argo-controller-5fdc64b6b4-t6r66 juju-pod-init}
application-argo-controller: 20:46:05 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/nlfvviqa.pid; exec sh -c 'test -d /tmp/unit-argo-controller-0971362255'; "]
application-pipelines-db: 20:46:05 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:05 INFO unit.pipelines-db/0.juju-log mysql:1: The pod is not ready.
application-pipelines-persistence: 20:46:05 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-persistence: 20:46:05 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-api
application-pipelines-api: 20:46:05 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:05 DEBUG unit.pipelines-api/0.juju-log pipelines-api:2: tracer>
tracer: starting handler dispatch, 21 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag endpoint.mysql.joined
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag mysql.connected
tracer: set flag pipelines-api.available
application-pipelines-db: 20:46:05 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:05 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/mysql/provides.py:20:_handle_joined:mysql
application-pipelines-persistence: 20:46:05 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-persistence: 20:46:05 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: status-set: active: 
controller-0: 20:46:06 DEBUG juju.worker.caasunitprovisioner ensured deployment for argo-controller for 1 units
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:06 DEBUG unit.pipelines-api/0.juju-log pipelines-api:2: tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-db: 20:46:06 INFO unit.pipelines-db/0.juju-log mysql:1: status-set: active: ready
application-argo-controller: 20:46:06 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ymbnyhhr.pid; exec sh -c 'tar -xmf - -C /tmp/unit-argo-controller-0971362255'; "]
application-pipelines-persistence: 20:46:06 DEBUG jujuc running hook tool "status-set" for pipelines-persistence/0-pipelines-api-relation-changed-1424071433667253186
application-pipelines-db: 20:46:06 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-mysql-relation-changed-7499748927614068682
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:06 DEBUG unit.pipelines-api/0.juju-log pipelines-api:2: tracer>
tracer: main dispatch loop, 6 handlers queued
tracer: ++   queue handler hooks/relations/http/provides.py:11:joined:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler reactive/pipelines_api.py:31:configure_http
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:06 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http
application-pipelines-persistence: 20:46:06 INFO juju.worker.uniter.operation ran "pipelines-api-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-persistence: 20:46:06 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-pipelines-persistence: 20:46:06 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run relation-changed (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:06 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:46:06 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:46:06 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-persistence: 20:46:06 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:46:06 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-argo-controller-0971362255/ca.crt  } to {/tmp/unit-argo-controller-0971362255/ca.crt argo-controller-5fdc64b6b4-t6r66 juju-pod-init}
application-argo-controller: 20:46:06 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/fvluespd.pid; exec sh -c 'test -d /tmp/unit-argo-controller-0971362255/ca.crt'; "]
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-db: 20:46:06 INFO juju.worker.uniter.operation ran "mysql-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:46:06 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (1; unit: pipelines-api/0) hook" for pipelines-db/0
application-pipelines-db: 20:46:06 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run relation-changed (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:06 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:46:06 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:0 Life:alive Attached:false Location:}
application-pipelines-db: 20:46:06 DEBUG juju.worker.uniter.storage still pending [storage-database-1]
application-pipelines-db: 20:46:06 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:06 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
controller-0: 20:46:06 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:06 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:06 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-argo-controller: 20:46:06 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ktnbkino.pid; exec sh -c 'tar -xmf - -C /tmp/unit-argo-controller-0971362255'; "]
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:06 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:06 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:06 INFO unit.pipelines-api/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql
application-pipelines-api: 20:46:06 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-pipelines-api: 20:46:07 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-pipelines-api-relation-changed-2709421062755110537
application-argo-controller: 20:46:07 DEBUG juju.worker.caasoperator operator service address: 10.152.183.147
application-argo-controller: 20:46:07 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-argo-controller-0/operator-client-cache.yaml  } to {/tmp/unit-argo-controller-0971362255/operator-client-cache.yaml argo-controller-5fdc64b6b4-t6r66 juju-pod-init}
application-argo-controller: 20:46:07 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/oyojthcb.pid; exec sh -c 'test -d /tmp/unit-argo-controller-0971362255/operator-client-cache.yaml'; "]
application-pipelines-api: 20:46:07 INFO juju.worker.uniter.operation ran "pipelines-api-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:46:07 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (2; unit: pipelines-persistence/0) hook" for pipelines-api/0
application-pipelines-api: 20:46:07 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-changed (2; unit: pipelines-persistence/0) hook)
application-pipelines-api: 20:46:07 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:46:07 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:46:07 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-api: 20:46:07 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:46:07 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/tytqmfme.pid; exec sh -c 'tar -xmf - -C /tmp/unit-argo-controller-0971362255'; "]
controller-0: 20:46:07 DEBUG juju.worker.caasunitprovisioner ensured deployment for pipelines-db for 1 units
application-argo-controller: 20:46:07 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/laoxqbcc.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-argo-controller-0 --charm-dir /tmp/unit-argo-controller-0971362255/charm --send --operator-file /tmp/unit-argo-controller-0971362255/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-argo-controller-0971362255/ca.crt'; "]
application-argo-controller: 20:46:07 DEBUG juju.worker.uniter.operation committing operation "remote init" for argo-controller/0
application-argo-controller: 20:46:07 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:46:08 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:46:10 DEBUG juju.worker.caasunitprovisioner service for minio(workload): &{Id:0f6fc8e3-48a5-427a-a263-04533dd18de2 Addresses:[local-cloud:10.152.183.102] Scale:0xc001aabe50 Generation:0xc001aabe58 Status:{Status:active Message: Data:map[] Since:<nil>}}
controller-0: 20:46:10 DEBUG juju.worker.caasunitprovisioner service for pipelines-db(workload): &{Id:90c9bd81-6ec7-4cbd-97c9-b6f113fa660c Addresses:[local-cloud:10.152.183.235] Scale:<nil> Generation:<nil> Status:{Status: Message: Data:map[] Since:<nil>}}
controller-0: 20:46:11 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-db: 20:46:12 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:46:12 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-pipelines-db: 20:46:12 DEBUG juju.worker.uniter.remotestate got address change for pipelines-db/0: ok=true, hashes=[1921fd5cd1ed12e39fddf184e724c8df1836a62b259b06ba827107f3f27402fe]
application-pipelines-db: 20:46:12 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:0 Life:alive Attached:false Location:}
application-pipelines-db: 20:46:12 DEBUG juju.worker.uniter.storage still pending [storage-database-1]
application-pipelines-db: 20:46:12 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:12 DEBUG juju.worker.uniter.operation running operation run config-changed hook for pipelines-db/0
application-pipelines-db: 20:46:12 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run config-changed hook)
application-pipelines-db: 20:46:12 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run config-changed hook)
application-pipelines-db: 20:46:12 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for pipelines-db/0
application-minio: 20:46:12 DEBUG juju.worker.uniter.remotestate storage attachment change for minio/0: {storage-minio-data-0 {2 alive true /data}}
application-minio: 20:46:12 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:46:12 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:46:12 DEBUG juju.worker.caasunitprovisioner ensured deployment for pipelines-persistence for 1 units
application-pipelines-db: 20:46:12 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for pipelines-db/0
application-pipelines-db: 20:46:12 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
controller-0: 20:46:13 DEBUG juju.worker.caasunitprovisioner service for argo-controller(workload): &{Id: Addresses:[] Scale:0xc0035dd648 Generation:0xc0035dd660 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-db: 20:46:13 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
controller-0: 20:46:13 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc002f8be00}
application-minio: 20:46:13 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:46:13 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-db: 20:46:13 DEBUG juju.worker.uniter.remotestate storage attachment change for pipelines-db/0: {storage-database-1 {2 alive true /var/lib/mysql}}
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-config-changed-6016950270519478086
controller-0: 20:46:14 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:46:14 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:46:14 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:46:14 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 INFO unit.pipelines-db/0.juju-log Reactive main running for hook config-changed
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 INFO unit.pipelines-db/0.juju-log Initializing Leadership Layer (is leader)
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: starting handler dispatch, 26 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag endpoint.mysql.joined
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag mysql.database.requested
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 DEBUG unit.pipelines-db/0.juju-log tracer: hooks phase, 0 handlers queued
controller-0: 20:46:14 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-db
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: main dispatch loop, 3 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:20:_handle_joined:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
tracer: ++   queue handler reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-pipelines-db: 20:46:14 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:14 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 INFO unit.pipelines-db/0.juju-log request -> mysql:1 for app -> pipelines-api
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 INFO unit.pipelines-db/0.juju-log db params: mysql:password@database
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "relation-get" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: cleared flag mysql.database.requested
tracer: -- dequeue handler reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: hooks/relations/mysql/provides.py:20:_handle_joined:mysql
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: set flag mysql.database.requested
tracer: ++   queue handler reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 INFO unit.pipelines-db/0.juju-log request -> mysql:1 for app -> pipelines-api
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 INFO unit.pipelines-db/0.juju-log db params: mysql:password@database
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: cleared flag mysql.database.requested
tracer: -- dequeue handler reactive/mariadb.py:50:provide_database
controller-0: 20:46:15 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "relation-set" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-db/0
application-pipelines-db: 20:46:15 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-db/0
application-pipelines-db: 20:46:15 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-db-0" pipelines-db-0
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "relation-set" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-config-changed-6016950270519478086
application-pipelines-db: 20:46:15 INFO unit.pipelines-db/0.juju-log status-set: active: ready
application-pipelines-db: 20:46:15 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-config-changed-6016950270519478086
application-argo-controller: 20:46:16 DEBUG juju.worker.caasoperator trigger running status for caas unit argo-controller/0
application-argo-controller: 20:46:16 DEBUG juju.worker.uniter.remotestate got running status change for argo-controller/0
application-argo-controller: 20:46:16 DEBUG juju.worker.caasoperator request running status for "unit-argo-controller-0" 98f9d8de-8d65-4cad-af65-85867d87b501
application-argo-controller: 20:46:16 DEBUG juju.kubernetes.provider.exec no pod named "98f9d8de-8d65-4cad-af65-85867d87b501" found
application-argo-controller: 20:46:16 DEBUG juju.kubernetes.provider.exec try get pod by UID for "98f9d8de-8d65-4cad-af65-85867d87b501"
application-argo-controller: 20:46:16 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:16 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:46:16 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for pipelines-db/0
application-pipelines-db: 20:46:16 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run config-changed hook)
application-pipelines-db: 20:46:16 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:46:16 DEBUG juju.worker.uniter.operation running operation remote init for pipelines-db/0
application-pipelines-db: 20:46:16 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pipelines-db/0
application-pipelines-db: 20:46:16 DEBUG juju.worker.uniter.operation executing operation "remote init" for pipelines-db/0
application-pipelines-db: 20:46:16 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-db-0" pipelines-db-0
application-pipelines-db: 20:46:16 DEBUG juju.worker.caasoperator remote init for "unit-pipelines-db-0" {PodName:pipelines-db-0 Initialising:true InitialisingTime:2020-10-21 20:46:10 +0000 UTC Running:false}
application-pipelines-db: 20:46:16 INFO juju.worker.caasoperator started pod init on "pipelines-db/0"
application-pipelines-db: 20:46:16 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/fnilrfyy.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-db-0813213318'; "]
application-pipelines-db: 20:46:16 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pipelines-db/charm  } to {/tmp/unit-pipelines-db-0813213318 pipelines-db-0 juju-pod-init}
application-pipelines-db: 20:46:16 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/cbbseogd.pid; exec sh -c 'test -d /tmp/unit-pipelines-db-0813213318'; "]
application-pipelines-db: 20:46:16 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ofiinnpj.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-db-0813213318'; "]
application-pipelines-api: 20:46:16 DEBUG juju.worker.uniter.remotestate got a relation units change for pipelines-api/0 : {1 {map[pipelines-db/0:{2}] map[] []}}
application-pipelines-api: 20:46:16 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:46:16 DEBUG juju.worker.uniter.operation running operation run relation-changed (1; unit: pipelines-db/0) hook for pipelines-api/0
application-pipelines-api: 20:46:16 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run relation-changed (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:46:16 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run relation-changed (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:46:16 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-api: 20:46:16 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-api: 20:46:16 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running mysql-relation-changed hook
application-pipelines-api: 20:46:16 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-db: 20:46:16 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pipelines-db-0813213318/ca.crt  } to {/tmp/unit-pipelines-db-0813213318/ca.crt pipelines-db-0 juju-pod-init}
application-pipelines-db: 20:46:16 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/bthvahzc.pid; exec sh -c 'test -d /tmp/unit-pipelines-db-0813213318/ca.crt'; "]
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 INFO unit.pipelines-api/0.juju-log mysql:1: Reactive main running for hook mysql-relation-changed
application-pipelines-db: 20:46:17 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/fkafjihm.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-db-0813213318'; "]
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: set flag endpoint.mysql.changed
tracer: ++   queue handler hooks/relations/http/provides.py:11:joined:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql
tracer: ++   queue handler reactive/pipelines_api.py:31:configure_http
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:46:17 DEBUG juju.worker.caasoperator operator service address: 10.152.183.107
application-pipelines-db: 20:46:17 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pipelines-db-0/operator-client-cache.yaml  } to {/tmp/unit-pipelines-db-0813213318/operator-client-cache.yaml pipelines-db-0 juju-pod-init}
application-pipelines-db: 20:46:17 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/frgsbppv.pid; exec sh -c 'test -d /tmp/unit-pipelines-db-0813213318/operator-client-cache.yaml'; "]
controller-0: 20:46:17 DEBUG juju.worker.caasunitprovisioner service for argo-controller(workload): &{Id: Addresses:[] Scale:0xc0036d7988 Generation:0xc0036d79a0 Status:{Status:active Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.database
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.egress-subnets
application-pipelines-api: 20:46:17 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:17 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.host
application-minio: 20:46:17 DEBUG juju.worker.uniter.remotestate storage attachment change for minio/0: {storage-minio-data-0 {2 alive true /data}}
application-minio: 20:46:17 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:46:17 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:18 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/bedcsyhq.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-db-0813213318'; "]
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.ingress-address
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.password
application-pipelines-db: 20:46:18 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/sdvstzin.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-db-0 --charm-dir /tmp/unit-pipelines-db-0813213318/charm --send --operator-file /tmp/unit-pipelines-db-0813213318/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-db-0813213318/ca.crt'; "]
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.port
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.private-address
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.root_password
controller-0: 20:46:18 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.user
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-db: 20:46:18 DEBUG juju.worker.uniter.operation committing operation "remote init" for pipelines-db/0
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-db: 20:46:18 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:18 DEBUG juju.worker.uniter.operation running operation run storage-attached (database/1) hook for pipelines-db/0
application-pipelines-db: 20:46:18 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run storage-attached (database/1) hook)
application-pipelines-db: 20:46:18 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run storage-attached (database/1) hook)
application-pipelines-db: 20:46:18 DEBUG juju.worker.uniter.operation preparing operation "run storage-attached (database/1) hook" for pipelines-db/0
application-pipelines-db: 20:46:18 DEBUG juju.worker.uniter.operation executing operation "run storage-attached (database/1) hook" for pipelines-db/0
application-pipelines-db: 20:46:18 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running database-storage-attached hook
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-db: 20:46:18 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: starting handler dispatch, 31 flags set
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag endpoint.mysql.changed
tracer: set flag endpoint.mysql.changed.database
tracer: set flag endpoint.mysql.changed.egress-subnets
tracer: set flag endpoint.mysql.changed.host
tracer: set flag endpoint.mysql.changed.ingress-address
tracer: set flag endpoint.mysql.changed.password
tracer: set flag endpoint.mysql.changed.port
tracer: set flag endpoint.mysql.changed.private-address
tracer: set flag endpoint.mysql.changed.root_password
tracer: set flag endpoint.mysql.changed.user
tracer: set flag endpoint.mysql.joined
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag mysql.connected
tracer: set flag pipelines-api.available
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:46:18 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: main dispatch loop, 7 handlers queued
tracer: ++   queue handler hooks/relations/http/provides.py:11:joined:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql
tracer: ++   queue handler reactive/pipelines_api.py:31:configure_http
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:18 INFO unit.pipelines-db/0.juju-log Reactive main running for hook database-storage-attached
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:18 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http
application-pipelines-api: 20:46:18 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-db: 20:46:18 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:18 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 INFO unit.pipelines-db/0.juju-log Initializing Leadership Layer (is leader)
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: starting handler dispatch, 25 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag endpoint.mysql.joined
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 DEBUG unit.pipelines-db/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:20:_handle_joined:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: set flag mysql.available
tracer: ++   queue handler reactive/pipelines_api.py:54:start_charm
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: hooks/relations/mysql/provides.py:20:_handle_joined:mysql
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: set flag mysql.changed
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: set flag mysql.database.requested
tracer: ++   queue handler reactive/mariadb.py:50:provide_database
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:50:provide_database
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer: cleared flag mysql.changed
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:54:start_charm
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "is-leader" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 INFO unit.pipelines-db/0.juju-log request -> mysql:1 for app -> pipelines-api
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:19 INFO unit.pipelines-api/0.juju-log mysql:1: status-set: maintenance: configuring container
application-pipelines-db: 20:46:19 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:19 INFO unit.pipelines-db/0.juju-log db params: mysql:password@database
application-pipelines-api: 20:46:19 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-db: 20:46:20 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:20 DEBUG jujuc running hook tool "relation-get" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:20 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:20 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: cleared flag mysql.database.requested
tracer: -- dequeue handler reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:46:20 DEBUG jujuc running hook tool "relation-set" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:20 DEBUG jujuc running hook tool "relation-set" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:20 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-database-storage-attached-2890574045096381775
application-pipelines-db: 20:46:20 INFO unit.pipelines-db/0.juju-log status-set: active: ready
application-pipelines-db: 20:46:20 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-database-storage-attached-2890574045096381775
controller-0: 20:46:20 DEBUG juju.worker.caasunitprovisioner service for pipelines-db(workload): &{Id:90c9bd81-6ec7-4cbd-97c9-b6f113fa660c Addresses:[local-cloud:10.152.183.235] Scale:0xc00240a420 Generation:0xc00240a428 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
controller-0: 20:46:20 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:46:21 INFO juju.worker.caasprovisioner started operator for application "pipelines-scheduledworkflow"
application-pipelines-api: 20:46:21 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-mysql-relation-changed-7504232914284791121
controller-0: 20:46:21 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-api: 20:46:21 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:21 INFO unit.pipelines-api/0.juju-log mysql:1: DEBUG
application-pipelines-api: 20:46:21 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:21 INFO unit.pipelines-api/0.juju-log mysql:1: {'ip': 'minio', 'port': 9000, 'user': 'minio', 'password': 'YR9E8ANXSPL30CQQJ3G17E82QV0BJD'}
application-pipelines-db: 20:46:21 INFO juju.worker.uniter.operation ran "database-storage-attached" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:46:21 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:21 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-db: 20:46:21 DEBUG juju.worker.uniter.operation committing operation "run storage-attached (database/1) hook" for pipelines-db/0
application-pipelines-api: 20:46:21 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-pipelines-db: 20:46:21 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run storage-attached (database/1) hook)
application-pipelines-db: 20:46:21 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:46:21 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:21 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:21 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-db: 20:46:21 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:21 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:46:21 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:21 INFO unit.pipelines-api/0.juju-log mysql:1: status-set: maintenance: creating container
controller-0: 20:46:21 DEBUG juju.worker.caasunitprovisioner service for pipelines-persistence(workload): &{Id: Addresses:[] Scale:0xc00143ebd8 Generation:0xc00143ebf0 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-persistence: 20:46:21 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-persistence/0
application-pipelines-persistence: 20:46:21 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-persistence/0
application-pipelines-persistence: 20:46:21 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-persistence-0" d67ee23b-6f10-4e86-b8a8-1b878f678466
application-pipelines-persistence: 20:46:21 DEBUG juju.kubernetes.provider.exec no pod named "d67ee23b-6f10-4e86-b8a8-1b878f678466" found
application-pipelines-persistence: 20:46:21 DEBUG juju.kubernetes.provider.exec try get pod by UID for "d67ee23b-6f10-4e86-b8a8-1b878f678466"
application-pipelines-persistence: 20:46:21 DEBUG juju.worker.uniter.operation running operation remote init for pipelines-persistence/0
application-pipelines-persistence: 20:46:21 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pipelines-persistence/0
application-pipelines-persistence: 20:46:21 DEBUG juju.worker.uniter.operation executing operation "remote init" for pipelines-persistence/0
application-pipelines-persistence: 20:46:21 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-persistence-0" pipelines-persistence-7986b9f7f5-8bdlx
application-pipelines-persistence: 20:46:21 DEBUG juju.worker.caasoperator remote init for "unit-pipelines-persistence-0" {PodName:pipelines-persistence-7986b9f7f5-8bdlx Initialising:true InitialisingTime:2020-10-21 20:46:14 +0000 UTC Running:false}
application-pipelines-persistence: 20:46:21 INFO juju.worker.caasoperator started pod init on "pipelines-persistence/0"
application-pipelines-api: 20:46:21 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-persistence: 20:46:21 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/unjybsyz.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-persistence-0514597734'; "]
application-pipelines-api: 20:46:21 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:21 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: set flag charm.started
tracer: -- dequeue handler reactive/pipelines_api.py:54:start_charm
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-persistence: 20:46:22 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pipelines-persistence/charm  } to {/tmp/unit-pipelines-persistence-0514597734 pipelines-persistence-7986b9f7f5-8bdlx juju-pod-init}
application-pipelines-persistence: 20:46:22 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/mkcajefx.pid; exec sh -c 'test -d /tmp/unit-pipelines-persistence-0514597734'; "]
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: cleared flag charm.started
tracer: ++   queue handler reactive/pipelines_api.py:54:start_charm
tracer: -- dequeue handler reactive/pipelines_api.py:26:charm_ready
controller-0: 20:46:22 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-scheduledworkflow
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:54:start_charm
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "is-leader" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-persistence: 20:46:22 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/srpwducd.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-persistence-0514597734'; "]
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 INFO unit.pipelines-api/0.juju-log mysql:1: status-set: maintenance: configuring container
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 INFO unit.pipelines-api/0.juju-log mysql:1: DEBUG
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 INFO unit.pipelines-api/0.juju-log mysql:1: {'ip': 'minio', 'port': 9000, 'user': 'minio', 'password': 'YR9E8ANXSPL30CQQJ3G17E82QV0BJD'}
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-pipelines-persistence: 20:46:22 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pipelines-persistence-0514597734/ca.crt  } to {/tmp/unit-pipelines-persistence-0514597734/ca.crt pipelines-persistence-7986b9f7f5-8bdlx juju-pod-init}
application-pipelines-persistence: 20:46:22 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ywqrvmyn.pid; exec sh -c 'test -d /tmp/unit-pipelines-persistence-0514597734/ca.crt'; "]
application-pipelines-api: 20:46:22 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:22 INFO unit.pipelines-api/0.juju-log mysql:1: status-set: maintenance: creating container
application-pipelines-api: 20:46:23 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:23 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.juju-log mysql:1: tracer>
tracer: set flag charm.started
tracer: ++   queue handler reactive/pipelines_api.py:26:charm_ready
tracer: -- dequeue handler reactive/pipelines_api.py:54:start_charm
application-pipelines-api: 20:46:23 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:23 INFO unit.pipelines-api/0.juju-log mysql:1: Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready
application-pipelines-api: 20:46:23 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-persistence: 20:46:23 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/grbezddh.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-persistence-0514597734'; "]
application-pipelines-api: 20:46:23 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:23 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:23 INFO unit.pipelines-api/0.juju-log mysql:1: status-set: active: 
application-pipelines-api: 20:46:23 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-mysql-relation-changed-7504232914284791121
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.mysql-relation-changed Requires relation
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.mysql-relation-changed minio:3
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.mysql-relation-changed requires unit
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.mysql-relation-changed {'egress-subnets': '10.152.183.102/32', 'ingress-address': '10.152.183.102', 'ip': '"minio"', 'password': '"YR9E8ANXSPL30CQQJ3G17E82QV0BJD"', 'port': '9000', 'private-address': '10.152.183.102', 'user': '"minio"'}
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.mysql-relation-changed Requires relation
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.mysql-relation-changed minio:3
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.mysql-relation-changed requires unit
application-pipelines-api: 20:46:23 DEBUG unit.pipelines-api/0.mysql-relation-changed {'egress-subnets': '10.152.183.102/32', 'ingress-address': '10.152.183.102', 'ip': '"minio"', 'password': '"YR9E8ANXSPL30CQQJ3G17E82QV0BJD"', 'port': '9000', 'private-address': '10.152.183.102', 'user': '"minio"'}
application-pipelines-persistence: 20:46:23 DEBUG juju.worker.caasoperator operator service address: 10.152.183.198
application-pipelines-persistence: 20:46:23 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pipelines-persistence-0/operator-client-cache.yaml  } to {/tmp/unit-pipelines-persistence-0514597734/operator-client-cache.yaml pipelines-persistence-7986b9f7f5-8bdlx juju-pod-init}
application-pipelines-persistence: 20:46:23 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/nntouctp.pid; exec sh -c 'test -d /tmp/unit-pipelines-persistence-0514597734/operator-client-cache.yaml'; "]
application-pipelines-persistence: 20:46:23 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/xojxnoel.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-persistence-0514597734'; "]
application-pipelines-api: 20:46:23 INFO juju.worker.uniter.operation ran "mysql-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:46:23 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (1; unit: pipelines-db/0) hook" for pipelines-api/0
application-pipelines-api: 20:46:23 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run relation-changed (1; unit: pipelines-db/0) hook)
application-pipelines-api: 20:46:23 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:46:23 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:46:23 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-persistence: 20:46:24 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/bltertjj.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-persistence-0 --charm-dir /tmp/unit-pipelines-persistence-0514597734/charm --send --operator-file /tmp/unit-pipelines-persistence-0514597734/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-persistence-0514597734/ca.crt'; "]
application-pipelines-db: 20:46:24 DEBUG juju.worker.uniter.remotestate storage attachment change for pipelines-db/0: {storage-database-1 {2 alive true /var/lib/mysql}}
application-pipelines-db: 20:46:24 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:24 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:46:24 DEBUG juju.worker.uniter.operation committing operation "remote init" for pipelines-persistence/0
application-pipelines-persistence: 20:46:24 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:46:25 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-scheduledworkflow
controller-0: 20:46:25 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-scheduledworkflow", "--debug"}
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.agent read agent config, format "2.0"
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.cmd.jujud caas operator application-pipelines-scheduledworkflow start (2.8.6 [gc])
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:46:26.886793046 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:46:26.886897747 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pipelines-scheduledworkflow"
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:46:26.887907254 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:46:26.896681517 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:46:26.896990619 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.worker.apicaller [f0e819] "application-pipelines-scheduledworkflow" successfully connected to "10.152.183.175:17070"
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.api RPC connection died
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:46:26.908122298 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.worker.apicaller [f0e819] "application-pipelines-scheduledworkflow" successfully connected to "10.152.183.175:17070"
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:46:26.92387271 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:46:26.93369578 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:46:26.933724181 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:46:26.934045283 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:46:26.934952589 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:46:26.942725245 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.worker.migrationminion migration phase is now: NONE
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.worker.caasupgrader unblocking abort check
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:46:26.950792602 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:46:26.950848203 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:46:26.951084004 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:46:26.951178705 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 INFO juju.worker.logger logger worker started
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:46:26.953440721 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:46:26 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:46:28.365642274 +0000 UTC
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
controller-0: 20:46:28 DEBUG juju.worker.caasunitprovisioner service for pipelines-db(workload): &{Id:90c9bd81-6ec7-4cbd-97c9-b6f113fa660c Addresses:[local-cloud:10.152.183.235] Scale:0xc0023d60c0 Generation:0xc0023d60c8 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-scheduledworkflow: 20:46:28 INFO juju.worker.uniter.charm downloading local:kubernetes/pipelines-scheduledworkflow-0 from API server
application-pipelines-scheduledworkflow: 20:46:28 INFO juju.downloader downloading from local:kubernetes/pipelines-scheduledworkflow-0
application-pipelines-scheduledworkflow: 20:46:28 INFO juju.downloader download complete ("local:kubernetes/pipelines-scheduledworkflow-0")
application-pipelines-scheduledworkflow: 20:46:28 INFO juju.downloader download verified ("local:kubernetes/pipelines-scheduledworkflow-0")
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-scheduledworkflow-0"
application-pipelines-scheduledworkflow: 20:46:28 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-scheduledworkflow-0"
controller-0: 20:46:28 INFO juju.worker.caasprovisioner started operator for application "pytorch-operator"
controller-0: 20:46:29 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-scheduledworkflow-0"
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc0008fb680}
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.worker.caasoperator operator "pipelines-scheduledworkflow" started
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter starting uniter for "pipelines-scheduledworkflow/0"
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.caasoperator start "pipelines-scheduledworkflow/0"
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.worker.caasoperator start "pipelines-scheduledworkflow/0"
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.caasoperator "pipelines-scheduledworkflow/0" started
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.leadership pipelines-scheduledworkflow/0 making initial claim for pipelines-scheduledworkflow leadership
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.worker.leadership pipelines-scheduledworkflow/0 promoted to leadership of pipelines-scheduledworkflow
controller-0: 20:46:29 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-scheduledworkflow
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-scheduledworkflow-0
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pipelines-scheduledworkflow-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-scheduledworkflow-0/jujud
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pipelines-scheduledworkflow-0/jujud
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/run.socket <nil>}
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.worker.uniter unit "pipelines-scheduledworkflow/0" started
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.worker.uniter resuming charm install
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter.operation running operation install local:kubernetes/pipelines-scheduledworkflow-0 for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter.operation preparing operation "install local:kubernetes/pipelines-scheduledworkflow-0" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.worker.uniter.charm downloading local:kubernetes/pipelines-scheduledworkflow-0 from API server
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.downloader downloading from local:kubernetes/pipelines-scheduledworkflow-0
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.downloader download complete ("local:kubernetes/pipelines-scheduledworkflow-0")
controller-0: 20:46:29 DEBUG juju.worker.caasunitprovisioner operator update for pytorch-operator
application-pipelines-scheduledworkflow: 20:46:29 INFO juju.downloader download verified ("local:kubernetes/pipelines-scheduledworkflow-0")
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter.operation executing operation "install local:kubernetes/pipelines-scheduledworkflow-0" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-scheduledworkflow-0"
application-pipelines-scheduledworkflow: 20:46:29 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-scheduledworkflow-0"
controller-0: 20:46:31 DEBUG juju.worker.caasunitprovisioner ensured deployment for pipelines-api for 1 units
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-scheduledworkflow-0"
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.operation committing operation "install local:kubernetes/pipelines-scheduledworkflow-0" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:31 INFO juju.worker.uniter hooks are retried true
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got storage change for pipelines-scheduledworkflow/0: [] ok=true
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got unit change for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got address change for pipelines-scheduledworkflow/0: ok=true, hashes=[]
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got leader settings change for pipelines-scheduledworkflow/0: ok=true
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got relations change for pipelines-scheduledworkflow/0: ok=true
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got config change for pipelines-scheduledworkflow/0: ok=true, hashes=[fa68127543db9337d9b4cad89b15248a8ea6ba33f8d78e404e35bc47069c2dfa]
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got action change for pipelines-scheduledworkflow/0: [] ok=true
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got trust config change for pipelines-scheduledworkflow/0: ok=true, hashes=[d5ebfb6b4ecb717d6de05891462437ece7c1b77c612433beeb5a25415e29eacb]
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.remotestate got update status interval change for pipelines-scheduledworkflow/0: ok=true
application-pipelines-scheduledworkflow: 20:46:31 INFO juju.worker.uniter found queued "install" hook
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.operation running operation run install hook for pipelines-scheduledworkflow/0
application-pipelines-db: 20:46:31 DEBUG juju.worker.uniter.remotestate storage attachment change for pipelines-db/0: {storage-database-1 {2 alive true /var/lib/mysql}}
application-pipelines-db: 20:46:31 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:31 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.machinelock acquire machine lock for pipelines-scheduledworkflow/0 uniter (run install hook)
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.machinelock machine lock acquired for pipelines-scheduledworkflow/0 uniter (run install hook)
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.operation executing operation "run install hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-pipelines-scheduledworkflow: 20:46:31 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/agent.socket <nil>}
controller-0: 20:46:31 DEBUG juju.worker.caasunitprovisioner service for pipelines-persistence(workload): &{Id: Addresses:[] Scale:0xc002469338 Generation:0xc002469350 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "config-get" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 INFO unit.pipelines-scheduledworkflow/0.juju-log Reactive main running for hook install
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer: starting handler dispatch, 0 flags set
controller-0: 20:46:32 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:46:32 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/docker_resource.py:7:auto_fetch
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 INFO unit.pipelines-scheduledworkflow/0.juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: set flag layer.docker-resource.pending
tracer: ++   queue handler reactive/docker_resource.py:18:fetch
application-pipelines-scheduledworkflow: 20:46:32 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:32 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: set flag layer.docker-resource.auto-fetched
tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch
application-pipelines-scheduledworkflow: 20:46:33 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:33 INFO unit.pipelines-scheduledworkflow/0.juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch
application-pytorch-operator: 20:46:33 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pytorch-operator: 20:46:33 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pytorch-operator", "--debug"}
application-pytorch-operator: 20:46:33 DEBUG juju.agent read agent config, format "2.0"
application-pytorch-operator: 20:46:33 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pytorch-operator: 20:46:33 INFO juju.cmd.jujud caas operator application-pytorch-operator start (2.8.6 [gc])
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:46:33.088075002 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:46:33.089717413 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pytorch-operator"
application-pytorch-operator: 20:46:33 DEBUG juju.worker.introspection stats worker now serving
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:46:33.090225016 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pytorch-operator: 20:46:33 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:46:33.098860474 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:46:33.101053389 +0000 UTC
application-pytorch-operator: 20:46:33 INFO juju.worker.apicaller [f0e819] "application-pytorch-operator" successfully connected to "10.152.183.175:17070"
application-pytorch-operator: 20:46:33 DEBUG juju.api RPC connection died
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:46:33.112326565 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:46:33 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pytorch-operator: 20:46:33 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pytorch-operator: 20:46:33 INFO juju.worker.apicaller [f0e819] "application-pytorch-operator" successfully connected to "10.152.183.175:17070"
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:46:33.138752243 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:46:33.148114206 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:46:33.148155106 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:46:33.148893311 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:46:33.153714944 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:46:33.156263961 +0000 UTC
application-pytorch-operator: 20:46:33 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pytorch-operator: 20:46:33 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pytorch-operator: 20:46:33 INFO juju.worker.caasupgrader unblocking abort check
application-pytorch-operator: 20:46:33 INFO juju.worker.migrationminion migration phase is now: NONE
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:46:33.16648663 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pytorch-operator: 20:46:33 INFO juju.worker.logger logger worker started
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:46:33.167097834 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:46:33.167443836 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:46:33.167510637 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:46:33.185092455 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pytorch-operator: 20:46:33 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pytorch-operator: 20:46:33 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pytorch-operator: 20:46:33 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pytorch-operator: 20:46:33 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pytorch-operator: 20:46:33 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pytorch-operator: 20:46:33 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pytorch-operator: 20:46:33 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pytorch-operator: 20:46:33 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-pytorch-operator: 20:46:33 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:46:33.201437965 +0000 UTC
application-pytorch-operator: 20:46:33 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:46:33 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:33 INFO unit.pipelines-scheduledworkflow/0.juju-log status-set: maintenance: fetching resource: oci-image
application-pytorch-operator: 20:46:33 INFO juju.worker.uniter.charm downloading local:kubernetes/pytorch-operator-0 from API server
application-pytorch-operator: 20:46:33 INFO juju.downloader downloading from local:kubernetes/pytorch-operator-0
application-pytorch-operator: 20:46:33 INFO juju.downloader download complete ("local:kubernetes/pytorch-operator-0")
application-pytorch-operator: 20:46:33 INFO juju.downloader download verified ("local:kubernetes/pytorch-operator-0")
application-pytorch-operator: 20:46:33 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pytorch-operator-0"
application-pytorch-operator: 20:46:33 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pytorch-operator-0"
application-pipelines-scheduledworkflow: 20:46:33 DEBUG jujuc running hook tool "status-set" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:33 DEBUG jujuc running hook tool "resource-get" for pipelines-scheduledworkflow/0-install-6982496711259604290
controller-0: 20:46:34 DEBUG juju.worker.caasunitprovisioner operator update for pytorch-operator
application-pipelines-scheduledworkflow: 20:46:35 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:35 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer: set flag layer.docker-resource.oci-image.available
application-pipelines-api: 20:46:35 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:46:35 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-pipelines-api: 20:46:35 DEBUG juju.worker.uniter.remotestate got address change for pipelines-api/0: ok=true, hashes=[0bd1590d712e8925317163b30f451871e30c2035bc3feda057e780ed4de2b601]
application-pipelines-api: 20:46:35 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:46:35 DEBUG juju.worker.uniter.operation running operation run config-changed hook for pipelines-api/0
application-pipelines-api: 20:46:35 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run config-changed hook)
application-pipelines-api: 20:46:35 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run config-changed hook)
application-pipelines-api: 20:46:35 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for pipelines-api/0
application-pipelines-api: 20:46:35 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for pipelines-api/0
application-pipelines-api: 20:46:35 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-pipelines-api: 20:46:35 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
controller-0: 20:46:35 INFO juju.worker.caasprovisioner started operator for application "seldon-core"
application-pipelines-scheduledworkflow: 20:46:35 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:35 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: cleared flag layer.docker-resource.pending
tracer: -- dequeue handler reactive/docker_resource.py:18:fetch
application-pipelines-db: 20:46:35 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-scheduledworkflow: 20:46:35 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:35 INFO unit.pipelines-scheduledworkflow/0.juju-log Invoking reactive handler: reactive/pipelines_scheduledworkflow.py:26:start_charm
application-pipelines-api: 20:46:35 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-scheduledworkflow: 20:46:35 DEBUG jujuc running hook tool "is-leader" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:35 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:35 INFO unit.pipelines-scheduledworkflow/0.juju-log status-set: maintenance: configuring container
application-pipelines-scheduledworkflow: 20:46:36 DEBUG jujuc running hook tool "status-set" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-api: 20:46:36 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:36 INFO unit.pipelines-api/0.juju-log Reactive main running for hook config-changed
application-pipelines-scheduledworkflow: 20:46:36 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:36 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:36 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-pipelines-api: 20:46:36 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-api: 20:46:36 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-5374791040633283155
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pytorch-operator-0"
application-pytorch-operator: 20:46:36 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000788300}
application-pytorch-operator: 20:46:36 INFO juju.worker.caasoperator operator "pytorch-operator" started
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter juju-run listener running
controller-0: 20:46:36 DEBUG juju.worker.caasunitprovisioner operator update for seldon-core
application-pipelines-scheduledworkflow: 20:46:36 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:36 INFO unit.pipelines-scheduledworkflow/0.juju-log status-set: maintenance: creating container
application-pytorch-operator: 20:46:36 DEBUG juju.worker.caasoperator.remotestate got application change
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter starting uniter for "pytorch-operator/0"
application-pytorch-operator: 20:46:36 DEBUG juju.worker.caasoperator start "pytorch-operator/0"
application-pytorch-operator: 20:46:36 INFO juju.worker.caasoperator start "pytorch-operator/0"
application-pytorch-operator: 20:46:36 DEBUG juju.worker.caasoperator "pytorch-operator/0" started
application-pytorch-operator: 20:46:36 DEBUG juju.worker.leadership pytorch-operator/0 making initial claim for pytorch-operator leadership
application-pytorch-operator: 20:46:36 INFO juju.worker.leadership pytorch-operator/0 promoted to leadership of pytorch-operator
application-pytorch-operator: 20:46:36 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pytorch-operator-0
application-pytorch-operator: 20:46:36 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pytorch-operator-0/jujuc using jujud path /var/lib/juju/tools/unit-pytorch-operator-0/jujud
application-pytorch-operator: 20:46:36 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pytorch-operator-0/jujud
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pytorch-operator-0/run.socket <nil>}
application-pytorch-operator: 20:46:36 INFO juju.worker.uniter unit "pytorch-operator/0" started
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter juju-run listener running
application-pytorch-operator: 20:46:36 INFO juju.worker.uniter resuming charm install
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter.operation running operation install local:kubernetes/pytorch-operator-0 for pytorch-operator/0
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter.operation preparing operation "install local:kubernetes/pytorch-operator-0" for pytorch-operator/0
application-pytorch-operator: 20:46:36 INFO juju.worker.uniter.charm downloading local:kubernetes/pytorch-operator-0 from API server
application-pytorch-operator: 20:46:36 INFO juju.downloader downloading from local:kubernetes/pytorch-operator-0
application-pytorch-operator: 20:46:36 INFO juju.downloader download complete ("local:kubernetes/pytorch-operator-0")
application-pipelines-api: 20:46:36 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-5374791040633283155
application-pytorch-operator: 20:46:36 INFO juju.downloader download verified ("local:kubernetes/pytorch-operator-0")
application-pipelines-scheduledworkflow: 20:46:36 DEBUG jujuc running hook tool "status-set" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter.operation executing operation "install local:kubernetes/pytorch-operator-0" for pytorch-operator/0
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pytorch-operator-0"
application-pytorch-operator: 20:46:36 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pytorch-operator-0"
application-pipelines-scheduledworkflow: 20:46:36 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:36 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: set flag charm.started
tracer: -- dequeue handler reactive/pipelines_scheduledworkflow.py:26:start_charm
application-pipelines-api: 20:46:36 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-scheduledworkflow: 20:46:36 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:36 INFO unit.pipelines-scheduledworkflow/0.juju-log Invoking reactive handler: reactive/pipelines_scheduledworkflow.py:16:charm_ready
application-pipelines-api: 20:46:36 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:37 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-scheduledworkflow: 20:46:37 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-scheduledworkflow: 20:46:37 INFO unit.pipelines-scheduledworkflow/0.juju-log status-set: active: 
application-pipelines-scheduledworkflow: 20:46:37 DEBUG jujuc running hook tool "status-set" for pipelines-scheduledworkflow/0-install-6982496711259604290
application-pipelines-api: 20:46:37 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-scheduledworkflow: 20:46:37 WARNING unit.pipelines-scheduledworkflow/0.install /var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/charm/reactive/pipelines_scheduledworkflow.py:37: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
application-pipelines-scheduledworkflow: 20:46:37 WARNING unit.pipelines-scheduledworkflow/0.install   crd = yaml.load(Path('files/crd-v1beta1.yaml').read_text())
application-pipelines-api: 20:46:37 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:37 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-scheduledworkflow: 20:46:37 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:46:37 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:37 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: starting handler dispatch, 33 flags set
tracer: set flag charm.started
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag endpoint.mysql.changed
tracer: set flag endpoint.mysql.changed.database
tracer: set flag endpoint.mysql.changed.egress-subnets
tracer: set flag endpoint.mysql.changed.host
tracer: set flag endpoint.mysql.changed.ingress-address
tracer: set flag endpoint.mysql.changed.password
tracer: set flag endpoint.mysql.changed.port
tracer: set flag endpoint.mysql.changed.private-address
tracer: set flag endpoint.mysql.changed.root_password
tracer: set flag endpoint.mysql.changed.user
tracer: set flag endpoint.mysql.joined
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag mysql.available
tracer: set flag mysql.connected
tracer: set flag pipelines-api.available
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.operation committing operation "run install hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.machinelock machine lock released for pipelines-scheduledworkflow/0 uniter (run install hook)
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.operation lock released for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.operation running operation accept leadership for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for pipelines-scheduledworkflow/0
application-pipelines-api: 20:46:37 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:37 DEBUG unit.pipelines-api/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-scheduledworkflow: 20:46:37 INFO juju.worker.uniter found queued "leader-elected" hook
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.machinelock acquire machine lock for pipelines-scheduledworkflow/0 uniter (run leader-elected hook)
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.machinelock machine lock acquired for pipelines-scheduledworkflow/0 uniter (run leader-elected hook)
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-pipelines-scheduledworkflow: 20:46:37 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/agent.socket <nil>}
application-pipelines-api: 20:46:37 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:37 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: main dispatch loop, 8 handlers queued
tracer: ++   queue handler hooks/relations/http/provides.py:11:joined:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql
tracer: ++   queue handler reactive/pipelines_api.py:26:charm_ready
tracer: ++   queue handler reactive/pipelines_api.py:31:configure_http
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:38 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready
application-pipelines-api: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:38 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http
application-pipelines-scheduledworkflow: 20:46:38 DEBUG jujuc running hook tool "config-get" for pipelines-scheduledworkflow/0-leader-elected-395805794588504871
application-pipelines-api: 20:46:38 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-scheduledworkflow: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-leader-elected-395805794588504871
application-pipelines-scheduledworkflow: 20:46:38 INFO unit.pipelines-scheduledworkflow/0.juju-log Reactive main running for hook leader-elected
application-pipelines-api: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:38 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pytorch-operator-0"
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.operation committing operation "install local:kubernetes/pytorch-operator-0" for pytorch-operator/0
application-pipelines-scheduledworkflow: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-leader-elected-395805794588504871
application-pipelines-scheduledworkflow: 20:46:38 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: starting handler dispatch, 4 flags set
tracer: set flag charm.started
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pytorch-operator: 20:46:38 INFO juju.worker.uniter hooks are retried true
controller-0: 20:46:38 DEBUG juju.worker.caasunitprovisioner ensured deployment for pipelines-scheduledworkflow for 1 units
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got address change for pytorch-operator/0: ok=true, hashes=[]
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got trust config change for pytorch-operator/0: ok=true, hashes=[f74eed7b2b9c5e960456b13a0b731976c48ead2ae64aa6d04614b540e50f943a]
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got relations change for pytorch-operator/0: ok=true
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got leader settings change for pytorch-operator/0: ok=true
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got action change for pytorch-operator/0: [] ok=true
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got update status interval change for pytorch-operator/0: ok=true
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got unit change for pytorch-operator/0
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got config change for pytorch-operator/0: ok=true, hashes=[b02a79b1c23294a0b182205fe125ab183a735dd863a1c0a6ed74d29339626749]
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.remotestate got storage change for pytorch-operator/0: [] ok=true
application-pytorch-operator: 20:46:38 INFO juju.worker.uniter found queued "install" hook
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.operation running operation run install hook for pytorch-operator/0
application-pytorch-operator: 20:46:38 DEBUG juju.machinelock acquire machine lock for pytorch-operator/0 uniter (run install hook)
application-pytorch-operator: 20:46:38 DEBUG juju.machinelock machine lock acquired for pytorch-operator/0 uniter (run install hook)
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for pytorch-operator/0
application-pipelines-api: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:38 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: cleared flag charm.started
tracer: ++   queue handler reactive/pipelines_api.py:54:start_charm
tracer: -- dequeue handler reactive/pipelines_api.py:26:charm_ready
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.operation executing operation "run install hook" for pytorch-operator/0
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-pipelines-scheduledworkflow: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-leader-elected-395805794588504871
application-pipelines-scheduledworkflow: 20:46:38 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer: hooks phase, 0 handlers queued
application-pytorch-operator: 20:46:38 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pytorch-operator-0/agent.socket <nil>}
application-pipelines-api: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:38 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:38 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-scheduledworkflow: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-leader-elected-395805794588504871
application-pipelines-scheduledworkflow: 20:46:38 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/pipelines_scheduledworkflow.py:16:charm_ready
application-pipelines-api: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:38 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api
application-pipelines-scheduledworkflow: 20:46:38 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-leader-elected-395805794588504871
application-pipelines-scheduledworkflow: 20:46:38 INFO unit.pipelines-scheduledworkflow/0.juju-log Invoking reactive handler: reactive/pipelines_scheduledworkflow.py:16:charm_ready
application-pipelines-api: 20:46:39 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:39 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql
application-pipelines-scheduledworkflow: 20:46:39 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-leader-elected-395805794588504871
application-pipelines-scheduledworkflow: 20:46:39 INFO unit.pipelines-scheduledworkflow/0.juju-log status-set: active: 
controller-0: 20:46:39 DEBUG juju.worker.caasunitprovisioner operator update for seldon-core
application-pytorch-operator: 20:46:39 DEBUG jujuc running hook tool "config-get" for pytorch-operator/0-install-5560343907498053945
application-pipelines-api: 20:46:39 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:39 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql
application-pipelines-scheduledworkflow: 20:46:39 DEBUG jujuc running hook tool "status-set" for pipelines-scheduledworkflow/0-leader-elected-395805794588504871
application-pytorch-operator: 20:46:39 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:39 INFO unit.pytorch-operator/0.juju-log Reactive main running for hook install
application-pipelines-api: 20:46:39 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-config-changed-5374791040633283155
application-pytorch-operator: 20:46:39 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:39 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: starting handler dispatch, 3 flags set
tracer: set flag config.changed
tracer: set flag config.changed.pytorch-default-image
tracer: set flag config.default.pytorch-default-image
controller-0: 20:46:39 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-api: 20:46:39 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:39 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:54:start_charm
application-pipelines-scheduledworkflow: 20:46:39 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.machinelock machine lock released for pipelines-scheduledworkflow/0 uniter (run leader-elected hook)
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.worker.uniter.operation lock released for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.worker.uniter.operation running operation run config-changed hook for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.machinelock acquire machine lock for pipelines-scheduledworkflow/0 uniter (run config-changed hook)
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.machinelock machine lock acquired for pipelines-scheduledworkflow/0 uniter (run config-changed hook)
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-pipelines-scheduledworkflow: 20:46:39 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/agent.socket <nil>}
application-pytorch-operator: 20:46:39 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:39 DEBUG unit.pytorch-operator/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:46:39 DEBUG jujuc running hook tool "is-leader" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-db: 20:46:39 DEBUG juju.worker.uniter.remotestate storage attachment change for pipelines-db/0: {storage-database-1 {2 alive true /var/lib/mysql}}
application-pipelines-db: 20:46:39 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:39 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pytorch-operator: 20:46:39 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:39 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/docker_resource.py:7:auto_fetch
tracer: ++   queue handler reactive/pytorch_operator.py:20:update_image
application-pipelines-api: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:40 INFO unit.pipelines-api/0.juju-log status-set: maintenance: configuring container
application-pipelines-scheduledworkflow: 20:46:40 DEBUG jujuc running hook tool "config-get" for pipelines-scheduledworkflow/0-config-changed-8457026680749724676
application-pytorch-operator: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:40 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch
application-pipelines-scheduledworkflow: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-config-changed-8457026680749724676
application-pipelines-scheduledworkflow: 20:46:40 INFO unit.pipelines-scheduledworkflow/0.juju-log Reactive main running for hook config-changed
application-pipelines-api: 20:46:40 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-config-changed-5374791040633283155
application-pytorch-operator: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:40 DEBUG unit.pytorch-operator/0.juju-log tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-scheduledworkflow: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-config-changed-8457026680749724676
application-pipelines-scheduledworkflow: 20:46:40 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: starting handler dispatch, 4 flags set
tracer: set flag charm.started
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-scheduledworkflow: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-config-changed-8457026680749724676
application-pipelines-scheduledworkflow: 20:46:40 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer: hooks phase, 0 handlers queued
application-pytorch-operator: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:40 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: set flag layer.docker-resource.pending
tracer: ++   queue handler reactive/docker_resource.py:18:fetch
application-pipelines-scheduledworkflow: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-config-changed-8457026680749724676
application-pipelines-scheduledworkflow: 20:46:40 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/pipelines_scheduledworkflow.py:16:charm_ready
application-pipelines-scheduledworkflow: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-config-changed-8457026680749724676
application-pipelines-scheduledworkflow: 20:46:40 INFO unit.pipelines-scheduledworkflow/0.juju-log Invoking reactive handler: reactive/pipelines_scheduledworkflow.py:16:charm_ready
application-pipelines-scheduledworkflow: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-config-changed-8457026680749724676
application-pipelines-scheduledworkflow: 20:46:40 INFO unit.pipelines-scheduledworkflow/0.juju-log status-set: active: 
application-pytorch-operator: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:40 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: set flag layer.docker-resource.auto-fetched
tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch
application-pipelines-scheduledworkflow: 20:46:40 DEBUG jujuc running hook tool "status-set" for pipelines-scheduledworkflow/0-config-changed-8457026680749724676
application-pytorch-operator: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:40 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/pytorch_operator.py:20:update_image
application-pytorch-operator: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:40 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch
application-pytorch-operator: 20:46:40 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:40 INFO unit.pytorch-operator/0.juju-log status-set: maintenance: fetching resource: oci-image
application-pytorch-operator: 20:46:41 DEBUG jujuc running hook tool "status-set" for pytorch-operator/0-install-5560343907498053945
controller-0: 20:46:41 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-api: 20:46:41 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:41 INFO unit.pipelines-api/0.juju-log DEBUG
application-pipelines-api: 20:46:41 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:41 INFO unit.pipelines-api/0.juju-log {'ip': 'minio', 'port': 9000, 'user': 'minio', 'password': 'YR9E8ANXSPL30CQQJ3G17E82QV0BJD'}
application-pipelines-api: 20:46:41 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:41 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:42 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-pipelines-api: 20:46:42 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:42 INFO unit.pipelines-api/0.juju-log status-set: maintenance: creating container
application-pipelines-api: 20:46:42 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:42 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-api/0
application-pipelines-api: 20:46:42 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-api/0
application-pipelines-api: 20:46:42 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-api-0" 060740d9-9aef-4765-81a1-73ba4a468772
application-pytorch-operator: 20:46:42 DEBUG jujuc running hook tool "resource-get" for pytorch-operator/0-install-5560343907498053945
controller-0: 20:46:42 INFO juju.worker.caasprovisioner started operator for application "tf-job-operator"
controller-0: 20:46:42 DEBUG juju.worker.caasunitprovisioner service for pipelines-api(workload): &{Id:41dd3516-000c-4bd2-8989-7194ac9df0e9 Addresses:[local-cloud:10.152.183.229] Scale:0xc003d21b68 Generation:0xc003d21b80 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:46:43 DEBUG juju.kubernetes.provider.exec no pod named "060740d9-9aef-4765-81a1-73ba4a468772" found
application-pipelines-api: 20:46:43 DEBUG juju.kubernetes.provider.exec try get pod by UID for "060740d9-9aef-4765-81a1-73ba4a468772"
application-pipelines-api: 20:46:43 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:43 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: set flag charm.started
tracer: ++   queue handler reactive/pipelines_api.py:26:charm_ready
tracer: -- dequeue handler reactive/pipelines_api.py:54:start_charm
application-pipelines-scheduledworkflow: 20:46:43 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:46:43 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:43 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready
application-pipelines-api: 20:46:43 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:43 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.machinelock machine lock released for pipelines-scheduledworkflow/0 uniter (run config-changed hook)
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.worker.uniter.operation lock released for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:43 INFO juju.worker.uniter found queued "start" hook
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.worker.uniter.operation running operation run start hook for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.machinelock acquire machine lock for pipelines-scheduledworkflow/0 uniter (run start hook)
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.machinelock machine lock acquired for pipelines-scheduledworkflow/0 uniter (run start hook)
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.worker.uniter.operation executing operation "run start hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-pipelines-scheduledworkflow: 20:46:43 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/agent.socket <nil>}
application-pipelines-api: 20:46:43 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-config-changed-5374791040633283155
application-pipelines-api: 20:46:43 INFO unit.pipelines-api/0.juju-log status-set: active: 
application-pytorch-operator: 20:46:43 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:43 DEBUG unit.pytorch-operator/0.juju-log tracer: set flag layer.docker-resource.oci-image.available
controller-0: 20:46:44 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-api: 20:46:44 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-config-changed-5374791040633283155
controller-0: 20:46:44 DEBUG juju.worker.caasunitprovisioner operator update for tf-job-operator
application-pytorch-operator: 20:46:44 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:44 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: cleared flag layer.docker-resource.pending
tracer: -- dequeue handler reactive/docker_resource.py:18:fetch
application-pipelines-scheduledworkflow: 20:46:44 DEBUG jujuc running hook tool "config-get" for pipelines-scheduledworkflow/0-start-4021287502151290713
application-pipelines-scheduledworkflow: 20:46:44 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-start-4021287502151290713
application-pipelines-scheduledworkflow: 20:46:44 INFO unit.pipelines-scheduledworkflow/0.juju-log Reactive main running for hook start
application-pytorch-operator: 20:46:44 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:44 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/pytorch_operator.py:25:start_charm
application-pipelines-scheduledworkflow: 20:46:44 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-start-4021287502151290713
application-pipelines-scheduledworkflow: 20:46:44 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: starting handler dispatch, 4 flags set
tracer: set flag charm.started
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-api: 20:46:44 DEBUG unit.pipelines-api/0.config-changed Requires relation
application-pipelines-api: 20:46:44 DEBUG unit.pipelines-api/0.config-changed minio:3
application-pipelines-api: 20:46:44 DEBUG unit.pipelines-api/0.config-changed requires unit
application-pipelines-api: 20:46:44 DEBUG unit.pipelines-api/0.config-changed {'egress-subnets': '10.152.183.102/32', 'ingress-address': '10.152.183.102', 'ip': '"minio"', 'password': '"YR9E8ANXSPL30CQQJ3G17E82QV0BJD"', 'port': '9000', 'private-address': '10.152.183.102', 'user': '"minio"'}
application-pipelines-scheduledworkflow: 20:46:44 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-start-4021287502151290713
application-pipelines-scheduledworkflow: 20:46:44 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer: hooks phase, 0 handlers queued
application-pytorch-operator: 20:46:44 DEBUG jujuc running hook tool "is-leader" for pytorch-operator/0-install-5560343907498053945
application-pipelines-scheduledworkflow: 20:46:44 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-start-4021287502151290713
application-pipelines-scheduledworkflow: 20:46:44 DEBUG unit.pipelines-scheduledworkflow/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/pipelines_scheduledworkflow.py:16:charm_ready
controller-0: 20:46:44 DEBUG juju.worker.caasunitprovisioner service for pipelines-scheduledworkflow(workload): &{Id: Addresses:[] Scale:0xc003fb2928 Generation:0xc003fb2940 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:46:44 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:46:44 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for pipelines-api/0
application-pipelines-persistence: 20:46:44 DEBUG juju.worker.uniter.remotestate got a relation units change for pipelines-persistence/0 : {2 {map[pipelines-api/0:{2}] map[] []}}
application-pipelines-persistence: 20:46:44 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:46:44 DEBUG juju.worker.uniter.operation running operation run relation-changed (2; unit: pipelines-api/0) hook for pipelines-persistence/0
application-pipelines-persistence: 20:46:44 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run relation-changed (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:44 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run relation-changed (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:44 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-minio: 20:46:44 DEBUG juju.worker.uniter.remotestate got a relation units change for minio/0 : {3 {map[pipelines-api/0:{1}] map[] []}}
application-pipelines-db: 20:46:44 DEBUG juju.worker.uniter.remotestate got a relation units change for pipelines-db/0 : {1 {map[pipelines-api/0:{1}] map[] []}}
application-pipelines-db: 20:46:44 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:44 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:44 DEBUG juju.worker.uniter.operation running operation run relation-changed (1; unit: pipelines-api/0) hook for pipelines-db/0
application-pipelines-db: 20:46:44 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run relation-changed (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:44 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run relation-changed (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:44 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (1; unit: pipelines-api/0) hook" for pipelines-db/0
application-minio: 20:46:44 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:46:44 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:46:44 DEBUG juju.worker.uniter.operation running operation run relation-changed (3; unit: pipelines-api/0) hook for minio/0
application-minio: 20:46:44 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run relation-changed (3; unit: pipelines-api/0) hook)
application-minio: 20:46:44 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run relation-changed (3; unit: pipelines-api/0) hook)
application-minio: 20:46:44 DEBUG juju.worker.uniter.operation preparing operation "run relation-changed (3; unit: pipelines-api/0) hook" for minio/0
application-pytorch-operator: 20:46:44 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:44 INFO unit.pytorch-operator/0.juju-log status-set: maintenance: configuring container
application-pipelines-api: 20:46:44 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run config-changed hook)
application-pipelines-api: 20:46:44 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:46:44 DEBUG juju.worker.uniter.operation running operation remote init for pipelines-api/0
application-pipelines-api: 20:46:44 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pipelines-api/0
application-pipelines-persistence: 20:46:44 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-pipelines-persistence: 20:46:44 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running pipelines-api-relation-changed hook
application-pipelines-api: 20:46:44 DEBUG juju.worker.uniter.operation executing operation "remote init" for pipelines-api/0
application-pipelines-api: 20:46:44 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-api-0" pipelines-api-58c5b9c64d-lrjjh
application-pipelines-db: 20:46:44 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (1; unit: pipelines-api/0) hook" for pipelines-db/0
application-pipelines-db: 20:46:44 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running mysql-relation-changed hook
application-pipelines-persistence: 20:46:44 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
application-pipelines-db: 20:46:44 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-pipelines-api: 20:46:44 DEBUG juju.worker.caasoperator remote init for "unit-pipelines-api-0" {PodName:pipelines-api-58c5b9c64d-lrjjh Initialising:true InitialisingTime:2020-10-21 20:46:35 +0000 UTC Running:false}
application-minio: 20:46:44 DEBUG juju.worker.uniter.operation executing operation "run relation-changed (3; unit: pipelines-api/0) hook" for minio/0
application-minio: 20:46:44 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running minio-relation-changed hook
application-pipelines-api: 20:46:44 INFO juju.worker.caasoperator started pod init on "pipelines-api/0"
application-pipelines-api: 20:46:44 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/spaolobp.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-api-0143007586'; "]
application-minio: 20:46:44 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-pipelines-scheduledworkflow: 20:46:45 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-start-4021287502151290713
application-pipelines-scheduledworkflow: 20:46:45 INFO unit.pipelines-scheduledworkflow/0.juju-log Invoking reactive handler: reactive/pipelines_scheduledworkflow.py:16:charm_ready
application-pytorch-operator: 20:46:45 DEBUG jujuc running hook tool "status-set" for pytorch-operator/0-install-5560343907498053945
application-pipelines-scheduledworkflow: 20:46:45 DEBUG jujuc running hook tool "juju-log" for pipelines-scheduledworkflow/0-start-4021287502151290713
application-pipelines-scheduledworkflow: 20:46:45 INFO unit.pipelines-scheduledworkflow/0.juju-log status-set: active: 
application-pytorch-operator: 20:46:45 DEBUG jujuc running hook tool "pod-spec-set" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:45 DEBUG jujuc running hook tool "pod-spec-set" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:45 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-minio: 20:46:45 DEBUG jujuc running hook tool "config-get" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-scheduledworkflow: 20:46:45 DEBUG jujuc running hook tool "status-set" for pipelines-scheduledworkflow/0-start-4021287502151290713
application-pipelines-db: 20:46:45 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-mysql-relation-changed-3594578898277895853
controller-0: 20:46:45 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-persistence: 20:46:45 DEBUG jujuc running hook tool "config-get" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:45 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:45 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Reactive main running for hook pipelines-api-relation-changed
application-pytorch-operator: 20:46:45 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:45 INFO unit.pytorch-operator/0.juju-log status-set: maintenance: creating container
application-pipelines-db: 20:46:45 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:45 INFO unit.pipelines-db/0.juju-log mysql:1: Reactive main running for hook mysql-relation-changed
application-minio: 20:46:45 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:45 INFO unit.minio/0.juju-log minio:3: Reactive main running for hook minio-relation-changed
application-pipelines-scheduledworkflow: 20:46:45 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-pipelines-scheduledworkflow: 20:46:45 DEBUG juju.worker.uniter.operation committing operation "run start hook" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:45 DEBUG juju.machinelock machine lock released for pipelines-scheduledworkflow/0 uniter (run start hook)
application-pipelines-scheduledworkflow: 20:46:45 DEBUG juju.worker.uniter.operation lock released for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:45 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-scheduledworkflow: 20:46:45 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
controller-0: 20:46:45 DEBUG juju.worker.caasunitprovisioner service for pipelines-scheduledworkflow(workload): &{Id: Addresses:[] Scale:0xc003b83cc8 Generation:0xc003b83ce0 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pytorch-operator: 20:46:46 DEBUG jujuc running hook tool "status-set" for pytorch-operator/0-install-5560343907498053945
application-pipelines-persistence: 20:46:46 DEBUG jujuc running hook tool "relation-ids" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
controller-0: 20:46:46 DEBUG juju.worker.caasunitprovisioner operator update for tf-job-operator
application-pipelines-api: 20:46:46 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pipelines-api/charm  } to {/tmp/unit-pipelines-api-0143007586 pipelines-api-58c5b9c64d-lrjjh juju-pod-init}
application-pipelines-api: 20:46:46 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/hbxxjpkq.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0143007586'; "]
application-minio: 20:46:46 DEBUG jujuc running hook tool "relation-ids" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-db: 20:46:46 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-persistence: 20:46:46 DEBUG jujuc running hook tool "relation-list" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pytorch-operator: 20:46:46 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:46 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: set flag charm.started
tracer: -- dequeue handler reactive/pytorch_operator.py:25:start_charm
application-pipelines-persistence: 20:46:46 DEBUG jujuc running hook tool "relation-get" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-scheduledworkflow-0" 8c345d1f-b75a-4dff-a4bb-67637ec9f804
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.kubernetes.provider.exec no pod named "8c345d1f-b75a-4dff-a4bb-67637ec9f804" found
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.kubernetes.provider.exec try get pod by UID for "8c345d1f-b75a-4dff-a4bb-67637ec9f804"
application-pipelines-db: 20:46:46 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-minio: 20:46:46 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.worker.uniter.operation running operation remote init for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.worker.uniter.operation executing operation "remote init" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-scheduledworkflow-0" pipelines-scheduledworkflow-744fdf4694-zq7xb
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.worker.caasoperator remote init for "unit-pipelines-scheduledworkflow-0" {PodName:pipelines-scheduledworkflow-744fdf4694-zq7xb Initialising:true InitialisingTime:2020-10-21 20:46:44 +0000 UTC Running:false}
application-pipelines-scheduledworkflow: 20:46:46 INFO juju.worker.caasoperator started pod init on "pipelines-scheduledworkflow/0"
application-pipelines-scheduledworkflow: 20:46:46 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ecfmmepr.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-scheduledworkflow-0124301682'; "]
application-pipelines-persistence: 20:46:46 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:46 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: set flag endpoint.pipelines-api.changed.egress-subnets
tracer: ++   queue handler hooks/relations/http/requires.py:10:changed:pipelines-api
tracer: ++   queue handler reactive/pipelines_persistence.py:13:charm_ready
application-pytorch-operator: 20:46:46 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:46 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/pytorch_operator.py:15:charm_ready
application-minio: 20:46:46 DEBUG jujuc running hook tool "relation-list" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-db: 20:46:46 DEBUG jujuc running hook tool "relation-get" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-persistence: 20:46:46 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:46 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer: set flag endpoint.pipelines-api.changed.ingress-address
application-pytorch-operator: 20:46:47 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-install-5560343907498053945
application-pytorch-operator: 20:46:47 INFO unit.pytorch-operator/0.juju-log status-set: active: 
application-minio: 20:46:47 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-persistence: 20:46:47 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:47 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer: set flag endpoint.pipelines-api.changed.private-address
application-pipelines-db: 20:46:47 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:47 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: set flag endpoint.mysql.changed
tracer: ++   queue handler hooks/relations/mysql/provides.py:20:_handle_joined:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-pytorch-operator: 20:46:47 DEBUG jujuc running hook tool "status-set" for pytorch-operator/0-install-5560343907498053945
application-pipelines-persistence: 20:46:47 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:47 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: starting handler dispatch, 12 flags set
tracer: set flag charm.started
tracer: set flag endpoint.pipelines-api.changed
tracer: set flag endpoint.pipelines-api.changed.egress-subnets
tracer: set flag endpoint.pipelines-api.changed.hostname
tracer: set flag endpoint.pipelines-api.changed.ingress-address
tracer: set flag endpoint.pipelines-api.changed.port
tracer: set flag endpoint.pipelines-api.changed.private-address
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag pipelines-api.available
application-pipelines-db: 20:46:47 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:47 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.egress-subnets
application-pipelines-db: 20:46:47 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-minio: 20:46:47 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-persistence: 20:46:47 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:47 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:46:47 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.ingress-address
application-pipelines-db: 20:46:47 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:47 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer: set flag endpoint.mysql.changed.private-address
application-pipelines-db: 20:46:48 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-persistence: 20:46:48 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:48 DEBUG unit.pipelines-persistence/0.juju-log pipelines-api:2: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/http/requires.py:10:changed:pipelines-api
tracer: ++   queue handler reactive/pipelines_persistence.py:13:charm_ready
application-minio: 20:46:48 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:48 DEBUG unit.minio/0.juju-log minio:3: tracer>
tracer: set flag endpoint.minio.changed
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-pipelines-db: 20:46:48 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:48 INFO unit.pipelines-db/0.juju-log mysql:1: Initializing Leadership Layer (is leader)
application-pipelines-persistence: 20:46:48 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:48 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Invoking reactive handler: reactive/pipelines_persistence.py:13:charm_ready
application-minio: 20:46:48 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:48 DEBUG unit.minio/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.egress-subnets
application-pipelines-db: 20:46:48 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-persistence: 20:46:48 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:48 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-api
application-minio: 20:46:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:49 DEBUG unit.minio/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.ingress-address
application-pipelines-db: 20:46:49 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:49 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: starting handler dispatch, 29 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag endpoint.mysql.changed
tracer: set flag endpoint.mysql.changed.egress-subnets
tracer: set flag endpoint.mysql.changed.ingress-address
tracer: set flag endpoint.mysql.changed.private-address
tracer: set flag endpoint.mysql.joined
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-persistence: 20:46:49 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-pipelines-persistence: 20:46:49 INFO unit.pipelines-persistence/0.juju-log pipelines-api:2: status-set: active: 
application-pipelines-persistence: 20:46:49 DEBUG jujuc running hook tool "status-set" for pipelines-persistence/0-pipelines-api-relation-changed-1855836913827302839
application-minio: 20:46:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:49 DEBUG unit.minio/0.juju-log minio:3: tracer: set flag endpoint.minio.changed.private-address
application-pipelines-db: 20:46:49 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:49 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer: hooks phase, 0 handlers queued
application-pipelines-scheduledworkflow: 20:46:49 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pipelines-scheduledworkflow/charm  } to {/tmp/unit-pipelines-scheduledworkflow-0124301682 pipelines-scheduledworkflow-744fdf4694-zq7xb juju-pod-init}
application-minio: 20:46:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:49 DEBUG unit.minio/0.juju-log minio:3: tracer>
tracer: starting handler dispatch, 13 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:46:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:49 DEBUG unit.minio/0.juju-log minio:3: tracer: hooks phase, 0 handlers queued
application-minio: 20:46:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:49 DEBUG unit.minio/0.juju-log minio:3: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-pipelines-api: 20:46:49 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/jtmcbmls.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0143007586'; "]
application-minio: 20:46:49 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:49 INFO unit.minio/0.juju-log minio:3: Invoking reactive handler: reactive/minio.py:13:charm_ready
application-pipelines-db: 20:46:49 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:49 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:20:_handle_joined:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-pipelines-scheduledworkflow: 20:46:49 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/rslbfzzf.pid; exec sh -c 'test -d /tmp/unit-pipelines-scheduledworkflow-0124301682'; "]
application-pipelines-db: 20:46:49 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:49 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-pipelines-db: 20:46:49 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:49 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: hooks/relations/mysql/provides.py:20:_handle_joined:mysql
application-pipelines-db: 20:46:49 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:49 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: set flag mysql.database.requested
tracer: ++   queue handler reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:46:50 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:50 INFO unit.pipelines-db/0.juju-log mysql:1: Invoking reactive handler: reactive/mariadb.py:50:provide_database
application-minio: 20:46:50 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:50 INFO unit.minio/0.juju-log minio:3: Invoking reactive handler: reactive/minio.py:18:configure_minio
application-minio: 20:46:50 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-db: 20:46:50 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:50 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:50 INFO unit.pipelines-db/0.juju-log mysql:1: request -> mysql:1 for app -> pipelines-api
application-minio: 20:46:50 DEBUG jujuc running hook tool "relation-get" for minio/0-minio-relation-changed-190281899271586485
application-pytorch-operator: 20:46:50 WARNING unit.pytorch-operator/0.install /var/lib/juju/agents/unit-pytorch-operator-0/charm/reactive/pytorch_operator.py:36: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
application-pytorch-operator: 20:46:50 WARNING unit.pytorch-operator/0.install   crd = yaml.load(Path('files/crds.yaml').read_text())
application-minio: 20:46:50 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-db: 20:46:50 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:50 INFO unit.pipelines-db/0.juju-log mysql:1: db params: mysql:password@database
application-pytorch-operator: 20:46:50 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-minio: 20:46:50 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-190281899271586485
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.operation committing operation "run install hook" for pytorch-operator/0
application-pytorch-operator: 20:46:50 DEBUG juju.machinelock machine lock released for pytorch-operator/0 uniter (run install hook)
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.operation lock released for pytorch-operator/0
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.operation running operation accept leadership for pytorch-operator/0
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for pytorch-operator/0
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for pytorch-operator/0
application-pipelines-db: 20:46:50 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pytorch-operator: 20:46:50 INFO juju.worker.uniter found queued "leader-elected" hook
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for pytorch-operator/0
application-pytorch-operator: 20:46:50 DEBUG juju.machinelock acquire machine lock for pytorch-operator/0 uniter (run leader-elected hook)
application-pytorch-operator: 20:46:50 DEBUG juju.machinelock machine lock acquired for pytorch-operator/0 uniter (run leader-elected hook)
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for pytorch-operator/0
application-pipelines-persistence: 20:46:50 INFO juju.worker.uniter.operation ran "pipelines-api-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-persistence: 20:46:50 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (2; unit: pipelines-api/0) hook" for pipelines-persistence/0
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for pytorch-operator/0
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-pipelines-persistence: 20:46:50 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run relation-changed (2; unit: pipelines-api/0) hook)
application-pipelines-persistence: 20:46:50 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:46:50 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:46:50 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pytorch-operator: 20:46:50 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pytorch-operator-0/agent.socket <nil>}
application-minio: 20:46:50 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-190281899271586485
application-pipelines-db: 20:46:50 DEBUG jujuc running hook tool "relation-get" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:51 DEBUG unit.pipelines-db/0.juju-log mysql:1: tracer>
tracer: cleared flag mysql.database.requested
tracer: -- dequeue handler reactive/mariadb.py:50:provide_database
controller-0: 20:46:51 DEBUG juju.worker.caasunitprovisioner ensured deployment for pytorch-operator for 1 units
application-minio: 20:46:51 DEBUG jujuc running hook tool "relation-set" for minio/0-minio-relation-changed-190281899271586485
application-pytorch-operator: 20:46:51 DEBUG jujuc running hook tool "config-get" for pytorch-operator/0-leader-elected-7502021865492068640
application-pipelines-db: 20:46:51 DEBUG jujuc running hook tool "relation-set" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-scheduledworkflow: 20:46:51 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/mivinjpg.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-scheduledworkflow-0124301682'; "]
application-minio: 20:46:51 DEBUG jujuc running hook tool "juju-log" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:51 INFO unit.minio/0.juju-log minio:3: status-set: active: 
application-pipelines-db: 20:46:51 DEBUG jujuc running hook tool "relation-set" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pytorch-operator: 20:46:51 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-leader-elected-7502021865492068640
application-pytorch-operator: 20:46:51 INFO unit.pytorch-operator/0.juju-log Reactive main running for hook leader-elected
application-minio: 20:46:51 DEBUG jujuc running hook tool "status-set" for minio/0-minio-relation-changed-190281899271586485
application-minio: 20:46:51 DEBUG unit.minio/0.minio-relation-changed Provides
application-minio: 20:46:51 DEBUG unit.minio/0.minio-relation-changed YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:46:51 DEBUG unit.minio/0.minio-relation-changed minio:0
application-minio: 20:46:51 DEBUG unit.minio/0.minio-relation-changed Provides
application-minio: 20:46:51 DEBUG unit.minio/0.minio-relation-changed YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:46:51 DEBUG unit.minio/0.minio-relation-changed minio:3
application-pytorch-operator: 20:46:51 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-leader-elected-7502021865492068640
application-pytorch-operator: 20:46:51 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: starting handler dispatch, 5 flags set
tracer: set flag charm.started
tracer: set flag config.default.pytorch-default-image
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-db: 20:46:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pipelines-db: 20:46:51 INFO unit.pipelines-db/0.juju-log mysql:1: status-set: active: ready
application-pipelines-db: 20:46:51 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-mysql-relation-changed-3594578898277895853
application-pytorch-operator: 20:46:51 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-leader-elected-7502021865492068640
application-pytorch-operator: 20:46:51 DEBUG unit.pytorch-operator/0.juju-log tracer: hooks phase, 0 handlers queued
application-minio: 20:46:51 INFO juju.worker.uniter.operation ran "minio-relation-changed" hook (via explicit, bespoke hook script)
application-minio: 20:46:51 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (3; unit: pipelines-api/0) hook" for minio/0
application-minio: 20:46:51 DEBUG juju.machinelock machine lock released for minio/0 uniter (run relation-changed (3; unit: pipelines-api/0) hook)
application-minio: 20:46:51 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:46:51 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:46:51 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:46:51 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pytorch-operator: 20:46:52 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-leader-elected-7502021865492068640
application-pytorch-operator: 20:46:52 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/pytorch_operator.py:15:charm_ready
application-pipelines-db: 20:46:52 INFO juju.worker.uniter.operation ran "mysql-relation-changed" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:46:52 DEBUG juju.worker.uniter.operation committing operation "run relation-changed (1; unit: pipelines-api/0) hook" for pipelines-db/0
controller-0: 20:46:52 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pytorch-operator: 20:46:52 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-leader-elected-7502021865492068640
application-pytorch-operator: 20:46:52 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/pytorch_operator.py:15:charm_ready
application-pytorch-operator: 20:46:52 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-leader-elected-7502021865492068640
application-pytorch-operator: 20:46:52 INFO unit.pytorch-operator/0.juju-log status-set: active: 
application-pipelines-api: 20:46:52 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pipelines-api-0143007586/ca.crt  } to {/tmp/unit-pipelines-api-0143007586/ca.crt pipelines-api-58c5b9c64d-lrjjh juju-pod-init}
application-pytorch-operator: 20:46:52 DEBUG jujuc running hook tool "status-set" for pytorch-operator/0-leader-elected-7502021865492068640
application-pipelines-api: 20:46:52 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/gohwxzmm.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0143007586/ca.crt'; "]
controller-0: 20:46:53 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-db: 20:46:53 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run relation-changed (1; unit: pipelines-api/0) hook)
application-pipelines-db: 20:46:53 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:46:53 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:53 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:46:53 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
controller-0: 20:46:53 DEBUG juju.worker.caasunitprovisioner service for pytorch-operator(workload): &{Id: Addresses:[] Scale:0xc0044f6378 Generation:0xc0044f6390 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-tf-job-operator: 20:46:54 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-tf-job-operator: 20:46:54 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=tf-job-operator", "--debug"}
application-tf-job-operator: 20:46:54 DEBUG juju.agent read agent config, format "2.0"
application-tf-job-operator: 20:46:54 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-tf-job-operator: 20:46:54 INFO juju.cmd.jujud caas operator application-tf-job-operator start (2.8.6 [gc])
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:46:54.161571494 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:46:54.161853197 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-tf-job-operator"
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:46:54.162054198 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.introspection stats worker now serving
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:46:54.162218799 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:46:54.172266075 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:46:54.17284208 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:46:54 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-tf-job-operator: 20:46:54 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-tf-job-operator: 20:46:54 INFO juju.worker.apicaller [f0e819] "application-tf-job-operator" successfully connected to "10.152.183.175:17070"
application-tf-job-operator: 20:46:54 DEBUG juju.api RPC connection died
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully
application-tf-job-operator: 20:46:54 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:46:54 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-tf-job-operator: 20:46:54 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-tf-job-operator: 20:46:54 INFO juju.worker.apicaller [f0e819] "application-tf-job-operator" successfully connected to "10.152.183.175:17070"
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:46:54.207503242 +0000 UTC
application-pytorch-operator: 20:46:54 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:46:54.216684012 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:46:54.216852813 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:46:54.218657727 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:46:54.218932629 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:46:54.218964329 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-tf-job-operator: 20:46:54 INFO juju.worker.caasupgrader abort check blocked until version event received
application-tf-job-operator: 20:46:54 INFO juju.worker.migrationminion migration phase is now: NONE
application-tf-job-operator: 20:46:54 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-tf-job-operator: 20:46:54 INFO juju.worker.caasupgrader unblocking abort check
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:46:54.281990207 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:46:54.287920752 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:46:54.288122953 +0000 UTC
application-tf-job-operator: 20:46:54 INFO juju.worker.logger logger worker started
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:46:54.288682358 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:46:54.296601018 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-tf-job-operator: 20:46:54 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-tf-job-operator: 20:46:54 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:46:54.311736632 +0000 UTC
application-tf-job-operator: 20:46:54 DEBUG juju.network no lxc bridge addresses to filter for machine
application-tf-job-operator: 20:46:54 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-tf-job-operator: 20:46:54 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-tf-job-operator: 20:46:54 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-tf-job-operator: 20:46:54 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-tf-job-operator: 20:46:54 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-tf-job-operator: 20:46:54 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-tf-job-operator: 20:46:54 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pytorch-operator: 20:46:54 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for pytorch-operator/0
application-pipelines-api: 20:46:54 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/punmjpgz.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0143007586'; "]
application-tf-job-operator: 20:46:54 INFO juju.worker.uniter.charm downloading local:kubernetes/tf-job-operator-0 from API server
application-tf-job-operator: 20:46:54 INFO juju.downloader downloading from local:kubernetes/tf-job-operator-0
application-pytorch-operator: 20:46:54 DEBUG juju.machinelock machine lock released for pytorch-operator/0 uniter (run leader-elected hook)
application-pytorch-operator: 20:46:54 DEBUG juju.worker.uniter.operation lock released for pytorch-operator/0
application-pytorch-operator: 20:46:54 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pytorch-operator: 20:46:54 DEBUG juju.worker.uniter.operation running operation run config-changed hook for pytorch-operator/0
application-pytorch-operator: 20:46:54 DEBUG juju.machinelock acquire machine lock for pytorch-operator/0 uniter (run config-changed hook)
application-pytorch-operator: 20:46:54 DEBUG juju.machinelock machine lock acquired for pytorch-operator/0 uniter (run config-changed hook)
application-pytorch-operator: 20:46:54 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for pytorch-operator/0
application-tf-job-operator: 20:46:54 INFO juju.downloader download complete ("local:kubernetes/tf-job-operator-0")
application-pytorch-operator: 20:46:54 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for pytorch-operator/0
application-pytorch-operator: 20:46:54 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-tf-job-operator: 20:46:54 INFO juju.downloader download verified ("local:kubernetes/tf-job-operator-0")
application-pytorch-operator: 20:46:54 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pytorch-operator-0/agent.socket <nil>}
application-tf-job-operator: 20:46:54 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/tf-job-operator-0"
application-tf-job-operator: 20:46:54 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/tf-job-operator-0"
controller-0: 20:46:54 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pytorch-operator: 20:46:54 DEBUG jujuc running hook tool "config-get" for pytorch-operator/0-config-changed-1265392462754281361
application-pytorch-operator: 20:46:54 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-config-changed-1265392462754281361
application-pytorch-operator: 20:46:54 INFO unit.pytorch-operator/0.juju-log Reactive main running for hook config-changed
application-pytorch-operator: 20:46:54 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-config-changed-1265392462754281361
application-pytorch-operator: 20:46:54 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: starting handler dispatch, 5 flags set
tracer: set flag charm.started
tracer: set flag config.default.pytorch-default-image
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pipelines-scheduledworkflow: 20:46:54 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pipelines-scheduledworkflow-0124301682/ca.crt  } to {/tmp/unit-pipelines-scheduledworkflow-0124301682/ca.crt pipelines-scheduledworkflow-744fdf4694-zq7xb juju-pod-init}
application-pipelines-scheduledworkflow: 20:46:54 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/zfkbdhxm.pid; exec sh -c 'test -d /tmp/unit-pipelines-scheduledworkflow-0124301682/ca.crt'; "]
application-pytorch-operator: 20:46:54 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-config-changed-1265392462754281361
application-pytorch-operator: 20:46:54 DEBUG unit.pytorch-operator/0.juju-log tracer: hooks phase, 0 handlers queued
controller-0: 20:46:54 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pytorch-operator: 20:46:55 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-config-changed-1265392462754281361
application-pytorch-operator: 20:46:55 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/pytorch_operator.py:15:charm_ready
controller-0: 20:46:55 DEBUG juju.worker.caasunitprovisioner service for pytorch-operator(workload): &{Id: Addresses:[] Scale:0xc00354c778 Generation:0xc00354c790 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pytorch-operator: 20:46:55 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-config-changed-1265392462754281361
application-pytorch-operator: 20:46:55 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/pytorch_operator.py:15:charm_ready
application-pytorch-operator: 20:46:55 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-config-changed-1265392462754281361
application-pytorch-operator: 20:46:55 INFO unit.pytorch-operator/0.juju-log status-set: active: 
application-pytorch-operator: 20:46:55 DEBUG jujuc running hook tool "status-set" for pytorch-operator/0-config-changed-1265392462754281361
controller-0: 20:46:55 DEBUG juju.worker.caasunitprovisioner operator update for tf-job-operator
application-pipelines-api: 20:46:55 DEBUG juju.worker.caasoperator operator service address: 10.152.183.108
application-pipelines-api: 20:46:55 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pipelines-api-0/operator-client-cache.yaml  } to {/tmp/unit-pipelines-api-0143007586/operator-client-cache.yaml pipelines-api-58c5b9c64d-lrjjh juju-pod-init}
application-pipelines-api: 20:46:55 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ntnciyxe.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0143007586/operator-client-cache.yaml'; "]
application-pytorch-operator: 20:46:55 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-pytorch-operator: 20:46:55 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for pytorch-operator/0
application-pytorch-operator: 20:46:55 DEBUG juju.machinelock machine lock released for pytorch-operator/0 uniter (run config-changed hook)
application-pytorch-operator: 20:46:55 DEBUG juju.worker.uniter.operation lock released for pytorch-operator/0
application-pytorch-operator: 20:46:55 INFO juju.worker.uniter found queued "start" hook
application-pytorch-operator: 20:46:55 DEBUG juju.worker.uniter.operation running operation run start hook for pytorch-operator/0
application-pytorch-operator: 20:46:55 DEBUG juju.machinelock acquire machine lock for pytorch-operator/0 uniter (run start hook)
application-pytorch-operator: 20:46:55 DEBUG juju.machinelock machine lock acquired for pytorch-operator/0 uniter (run start hook)
application-pytorch-operator: 20:46:55 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for pytorch-operator/0
application-pytorch-operator: 20:46:55 DEBUG juju.worker.uniter.operation executing operation "run start hook" for pytorch-operator/0
application-pytorch-operator: 20:46:55 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-pytorch-operator: 20:46:55 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pytorch-operator-0/agent.socket <nil>}
controller-0: 20:46:55 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-db: 20:46:55 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-db/0
application-pipelines-db: 20:46:55 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-db/0
application-pipelines-db: 20:46:55 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-db-0" pipelines-db-0
application-pipelines-db: 20:46:55 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:55 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-scheduledworkflow: 20:46:55 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/jsxeynos.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-scheduledworkflow-0124301682'; "]
application-pytorch-operator: 20:46:56 DEBUG jujuc running hook tool "config-get" for pytorch-operator/0-start-824440184068348819
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/tf-job-operator-0"
application-tf-job-operator: 20:46:56 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000948c00}
application-tf-job-operator: 20:46:56 INFO juju.worker.caasoperator operator "tf-job-operator" started
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter juju-run listener running
application-pytorch-operator: 20:46:56 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-start-824440184068348819
application-pytorch-operator: 20:46:56 INFO unit.pytorch-operator/0.juju-log Reactive main running for hook start
application-tf-job-operator: 20:46:56 DEBUG juju.worker.caasoperator.remotestate got application change
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter starting uniter for "tf-job-operator/0"
application-tf-job-operator: 20:46:56 DEBUG juju.worker.caasoperator start "tf-job-operator/0"
application-tf-job-operator: 20:46:56 INFO juju.worker.caasoperator start "tf-job-operator/0"
application-tf-job-operator: 20:46:56 DEBUG juju.worker.caasoperator "tf-job-operator/0" started
application-tf-job-operator: 20:46:56 DEBUG juju.worker.leadership tf-job-operator/0 making initial claim for tf-job-operator leadership
application-tf-job-operator: 20:46:56 INFO juju.worker.leadership tf-job-operator/0 promoted to leadership of tf-job-operator
application-tf-job-operator: 20:46:56 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-tf-job-operator-0
application-tf-job-operator: 20:46:56 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-tf-job-operator-0/jujuc using jujud path /var/lib/juju/tools/unit-tf-job-operator-0/jujud
application-tf-job-operator: 20:46:56 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-tf-job-operator-0/jujud
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-tf-job-operator-0/run.socket <nil>}
application-tf-job-operator: 20:46:56 INFO juju.worker.uniter unit "tf-job-operator/0" started
application-tf-job-operator: 20:46:56 DEBUG juju.worker.caasoperator.remotestate got application change
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter juju-run listener running
application-tf-job-operator: 20:46:56 INFO juju.worker.uniter resuming charm install
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter.operation running operation install local:kubernetes/tf-job-operator-0 for tf-job-operator/0
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter.operation preparing operation "install local:kubernetes/tf-job-operator-0" for tf-job-operator/0
application-tf-job-operator: 20:46:56 INFO juju.worker.uniter.charm downloading local:kubernetes/tf-job-operator-0 from API server
application-tf-job-operator: 20:46:56 INFO juju.downloader downloading from local:kubernetes/tf-job-operator-0
application-tf-job-operator: 20:46:56 INFO juju.downloader download complete ("local:kubernetes/tf-job-operator-0")
application-tf-job-operator: 20:46:56 INFO juju.downloader download verified ("local:kubernetes/tf-job-operator-0")
controller-0: 20:46:56 DEBUG juju.worker.caasunitprovisioner service for pipelines-db(workload): &{Id:90c9bd81-6ec7-4cbd-97c9-b6f113fa660c Addresses:[local-cloud:10.152.183.235] Scale:0xc00148a300 Generation:0xc00148a308 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pytorch-operator: 20:46:56 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-start-824440184068348819
application-pytorch-operator: 20:46:56 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: starting handler dispatch, 5 flags set
tracer: set flag charm.started
tracer: set flag config.default.pytorch-default-image
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter.operation executing operation "install local:kubernetes/tf-job-operator-0" for tf-job-operator/0
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/tf-job-operator-0"
application-tf-job-operator: 20:46:56 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/tf-job-operator-0"
application-pytorch-operator: 20:46:56 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-start-824440184068348819
application-pytorch-operator: 20:46:56 DEBUG unit.pytorch-operator/0.juju-log tracer: hooks phase, 0 handlers queued
application-pytorch-operator: 20:46:56 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-start-824440184068348819
application-pytorch-operator: 20:46:56 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/pytorch_operator.py:15:charm_ready
application-pytorch-operator: 20:46:56 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-start-824440184068348819
application-pytorch-operator: 20:46:56 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/pytorch_operator.py:15:charm_ready
application-pytorch-operator: 20:46:57 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-start-824440184068348819
application-pytorch-operator: 20:46:57 INFO unit.pytorch-operator/0.juju-log status-set: active: 
application-pytorch-operator: 20:46:57 DEBUG jujuc running hook tool "status-set" for pytorch-operator/0-start-824440184068348819
application-pytorch-operator: 20:46:57 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:46:57 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/eixmpcnm.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0143007586'; "]
application-pytorch-operator: 20:46:57 DEBUG juju.worker.uniter.operation committing operation "run start hook" for pytorch-operator/0
application-pytorch-operator: 20:46:57 DEBUG juju.machinelock machine lock released for pytorch-operator/0 uniter (run start hook)
application-pytorch-operator: 20:46:57 DEBUG juju.worker.uniter.operation lock released for pytorch-operator/0
application-pytorch-operator: 20:46:57 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pytorch-operator: 20:46:57 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-db: 20:46:57 DEBUG juju.worker.uniter.remotestate storage attachment change for pipelines-db/0: {storage-database-1 {2 alive true /var/lib/mysql}}
application-pipelines-db: 20:46:57 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:46:57 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:46:58 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pytorch-operator: 20:46:58 DEBUG juju.worker.caasoperator trigger running status for caas unit pytorch-operator/0
application-pytorch-operator: 20:46:58 DEBUG juju.worker.uniter.remotestate got running status change for pytorch-operator/0
application-pytorch-operator: 20:46:58 DEBUG juju.worker.caasoperator request running status for "unit-pytorch-operator-0" b95510d3-6d1e-4d72-9615-e6403906e412
application-seldon-core: 20:46:58 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-seldon-core: 20:46:58 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=seldon-core", "--debug"}
application-seldon-core: 20:46:58 DEBUG juju.agent read agent config, format "2.0"
application-seldon-core: 20:46:58 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-seldon-core: 20:46:58 INFO juju.cmd.jujud caas operator application-seldon-core start (2.8.6 [gc])
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:46:58.55881366 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:46:58.559012862 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:46:58.559171163 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-seldon-core"
application-seldon-core: 20:46:58 DEBUG juju.worker.introspection stats worker now serving
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:46:58.56982684 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.apicaller connecting with old password
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:46:58.570974248 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-seldon-core: 20:46:58 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:46:58.582141929 +0000 UTC
application-seldon-core: 20:46:58 INFO juju.worker.apicaller [f0e819] "application-seldon-core" successfully connected to "10.152.183.175:17070"
application-seldon-core: 20:46:58 DEBUG juju.api RPC connection died
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully
application-seldon-core: 20:46:58 DEBUG juju.worker.apicaller connecting with old password
application-seldon-core: 20:46:58 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-seldon-core: 20:46:58 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-seldon-core: 20:46:58 INFO juju.worker.apicaller [f0e819] "application-seldon-core" successfully connected to "10.152.183.175:17070"
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:46:58.607923716 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:46:58.618455292 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:46:58.618492292 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:46:58.618552893 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:46:58.620182704 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:46:58.623166926 +0000 UTC
application-seldon-core: 20:46:58 INFO juju.worker.caasupgrader abort check blocked until version event received
application-seldon-core: 20:46:58 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-seldon-core: 20:46:58 INFO juju.worker.caasupgrader unblocking abort check
application-seldon-core: 20:46:58 INFO juju.worker.migrationminion migration phase is now: NONE
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:46:58.632824596 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:46:58.633558401 +0000 UTC
application-seldon-core: 20:46:58 INFO juju.worker.logger logger worker started
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:46:58.634584009 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:46:58.63472471 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-seldon-core: 20:46:58 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:46:58.638665938 +0000 UTC
application-seldon-core: 20:46:58 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-seldon-core: 20:46:58 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-seldon-core: 20:46:58 DEBUG juju.network no lxc bridge addresses to filter for machine
application-seldon-core: 20:46:58 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-seldon-core: 20:46:58 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-seldon-core: 20:46:58 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-seldon-core: 20:46:58 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-seldon-core: 20:46:58 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-seldon-core: 20:46:58 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:46:58 DEBUG juju.worker.caasoperator operator service address: 10.152.183.6
application-pipelines-scheduledworkflow: 20:46:58 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/operator-client-cache.yaml  } to {/tmp/unit-pipelines-scheduledworkflow-0124301682/operator-client-cache.yaml pipelines-scheduledworkflow-744fdf4694-zq7xb juju-pod-init}
application-seldon-core: 20:46:59 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-seldon-core: 20:46:59 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:46:59.559967802 +0000 UTC
application-pipelines-scheduledworkflow: 20:47:00 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/wuvahwwk.pid; exec sh -c 'test -d /tmp/unit-pipelines-scheduledworkflow-0124301682/operator-client-cache.yaml'; "]
application-pytorch-operator: 20:47:00 DEBUG juju.kubernetes.provider.exec no pod named "b95510d3-6d1e-4d72-9615-e6403906e412" found
application-pytorch-operator: 20:47:00 DEBUG juju.kubernetes.provider.exec try get pod by UID for "b95510d3-6d1e-4d72-9615-e6403906e412"
application-seldon-core: 20:47:01 INFO juju.worker.uniter.charm downloading local:kubernetes/seldon-core-0 from API server
application-seldon-core: 20:47:01 INFO juju.downloader downloading from local:kubernetes/seldon-core-0
application-seldon-core: 20:47:01 INFO juju.downloader download complete ("local:kubernetes/seldon-core-0")
application-seldon-core: 20:47:01 INFO juju.downloader download verified ("local:kubernetes/seldon-core-0")
controller-0: 20:47:01 DEBUG juju.worker.caasunitprovisioner operator update for seldon-core
application-seldon-core: 20:47:02 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/seldon-core-0"
application-seldon-core: 20:47:02 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/seldon-core-0"
application-pipelines-api: 20:47:02 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/vnfybxcu.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-api-0 --charm-dir /tmp/unit-pipelines-api-0143007586/charm --send --operator-file /tmp/unit-pipelines-api-0143007586/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-api-0143007586/ca.crt'; "]
application-pytorch-operator: 20:47:02 DEBUG juju.worker.uniter.operation running operation remote init for pytorch-operator/0
application-pytorch-operator: 20:47:02 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pytorch-operator/0
controller-0: 20:47:02 DEBUG juju.worker.caasunitprovisioner service for pytorch-operator(workload): &{Id: Addresses:[] Scale:0xc003d20138 Generation:0xc003d20150 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pytorch-operator: 20:47:02 DEBUG juju.worker.uniter.operation executing operation "remote init" for pytorch-operator/0
application-pytorch-operator: 20:47:02 DEBUG juju.worker.caasoperator request running status for "unit-pytorch-operator-0" pytorch-operator-6974c76984-m47qt
application-pytorch-operator: 20:47:02 DEBUG juju.worker.caasoperator remote init for "unit-pytorch-operator-0" {PodName:pytorch-operator-6974c76984-m47qt Initialising:true InitialisingTime:2020-10-21 20:46:57 +0000 UTC Running:false}
application-pytorch-operator: 20:47:02 INFO juju.worker.caasoperator started pod init on "pytorch-operator/0"
application-pytorch-operator: 20:47:02 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/wmhvbuia.pid; exec sh -c 'mkdir -p /tmp/unit-pytorch-operator-0904576432'; "]
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/tf-job-operator-0"
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.operation committing operation "install local:kubernetes/tf-job-operator-0" for tf-job-operator/0
application-tf-job-operator: 20:47:02 INFO juju.worker.uniter hooks are retried true
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got leader settings change for tf-job-operator/0: ok=true
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got storage change for tf-job-operator/0: [] ok=true
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got relations change for tf-job-operator/0: ok=true
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got unit change for tf-job-operator/0
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got update status interval change for tf-job-operator/0: ok=true
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got address change for tf-job-operator/0: ok=true, hashes=[]
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got trust config change for tf-job-operator/0: ok=true, hashes=[7e4e2cd72662de0b4ee50f0b60a9f5007971d9e62f95a8c59271b4569b404ba2]
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got action change for tf-job-operator/0: [] ok=true
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.remotestate got config change for tf-job-operator/0: ok=true, hashes=[e8227833d18e064d11351ac4b81a8a5cd47f8d979eee256d8c7ace444d07359e]
application-tf-job-operator: 20:47:02 INFO juju.worker.uniter found queued "install" hook
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.operation running operation run install hook for tf-job-operator/0
application-tf-job-operator: 20:47:02 DEBUG juju.machinelock acquire machine lock for tf-job-operator/0 uniter (run install hook)
application-tf-job-operator: 20:47:02 DEBUG juju.machinelock machine lock acquired for tf-job-operator/0 uniter (run install hook)
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for tf-job-operator/0
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.operation executing operation "run install hook" for tf-job-operator/0
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-tf-job-operator: 20:47:02 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-tf-job-operator-0/agent.socket <nil>}
application-pytorch-operator: 20:47:03 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pytorch-operator/charm  } to {/tmp/unit-pytorch-operator-0904576432 pytorch-operator-6974c76984-m47qt juju-pod-init}
application-pytorch-operator: 20:47:03 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/kennxaup.pid; exec sh -c 'test -d /tmp/unit-pytorch-operator-0904576432'; "]
application-pipelines-scheduledworkflow: 20:47:03 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/sbmxdpfk.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-scheduledworkflow-0124301682'; "]
application-pipelines-api: 20:47:03 DEBUG juju.worker.uniter.operation committing operation "remote init" for pipelines-api/0
application-pipelines-api: 20:47:03 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:47:03 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-api: 20:47:03 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-tf-job-operator: 20:47:03 DEBUG jujuc running hook tool "config-get" for tf-job-operator/0-install-7476131894038043726
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/seldon-core-0"
application-seldon-core: 20:47:03 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc0008e7b00}
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter juju-run listener running
application-seldon-core: 20:47:03 INFO juju.worker.caasoperator operator "seldon-core" started
application-seldon-core: 20:47:03 DEBUG juju.worker.caasoperator.remotestate got application change
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter starting uniter for "seldon-core/0"
application-seldon-core: 20:47:03 DEBUG juju.worker.caasoperator start "seldon-core/0"
application-seldon-core: 20:47:03 INFO juju.worker.caasoperator start "seldon-core/0"
application-seldon-core: 20:47:03 DEBUG juju.worker.caasoperator "seldon-core/0" started
application-seldon-core: 20:47:03 DEBUG juju.worker.leadership seldon-core/0 making initial claim for seldon-core leadership
application-seldon-core: 20:47:03 INFO juju.worker.leadership seldon-core/0 promoted to leadership of seldon-core
application-seldon-core: 20:47:03 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-seldon-core-0
application-seldon-core: 20:47:03 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-seldon-core-0/jujuc using jujud path /var/lib/juju/tools/unit-seldon-core-0/jujud
application-seldon-core: 20:47:03 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-seldon-core-0/jujud
application-tf-job-operator: 20:47:03 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:03 INFO unit.tf-job-operator/0.juju-log Reactive main running for hook install
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-seldon-core-0/run.socket <nil>}
application-seldon-core: 20:47:03 INFO juju.worker.uniter unit "seldon-core/0" started
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter juju-run listener running
application-seldon-core: 20:47:03 INFO juju.worker.uniter resuming charm install
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter.operation running operation install local:kubernetes/seldon-core-0 for seldon-core/0
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter.operation preparing operation "install local:kubernetes/seldon-core-0" for seldon-core/0
application-seldon-core: 20:47:03 INFO juju.worker.uniter.charm downloading local:kubernetes/seldon-core-0 from API server
application-seldon-core: 20:47:03 INFO juju.downloader downloading from local:kubernetes/seldon-core-0
application-seldon-core: 20:47:03 INFO juju.downloader download complete ("local:kubernetes/seldon-core-0")
application-seldon-core: 20:47:03 INFO juju.downloader download verified ("local:kubernetes/seldon-core-0")
application-tf-job-operator: 20:47:03 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:03 DEBUG unit.tf-job-operator/0.juju-log tracer: starting handler dispatch, 0 flags set
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter.operation executing operation "install local:kubernetes/seldon-core-0" for seldon-core/0
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/seldon-core-0"
application-seldon-core: 20:47:03 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/seldon-core-0"
application-tf-job-operator: 20:47:03 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:03 DEBUG unit.tf-job-operator/0.juju-log tracer: hooks phase, 0 handlers queued
application-pytorch-operator: 20:47:03 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ruwewqpc.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pytorch-operator-0904576432'; "]
application-pipelines-scheduledworkflow: 20:47:03 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/wznhgegz.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-scheduledworkflow-0 --charm-dir /tmp/unit-pipelines-scheduledworkflow-0124301682/charm --send --operator-file /tmp/unit-pipelines-scheduledworkflow-0124301682/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-scheduledworkflow-0124301682/ca.crt'; "]
application-tf-job-operator: 20:47:03 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:03 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/docker_resource.py:7:auto_fetch
application-tf-job-operator: 20:47:04 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:04 INFO unit.tf-job-operator/0.juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch
application-tf-job-operator: 20:47:04 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:04 DEBUG unit.tf-job-operator/0.juju-log tracer: set flag layer.docker-resource.oci-image.fetched
application-tf-job-operator: 20:47:04 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:04 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: set flag layer.docker-resource.pending
tracer: ++   queue handler reactive/docker_resource.py:18:fetch
application-tf-job-operator: 20:47:04 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:04 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: set flag layer.docker-resource.auto-fetched
tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch
application-tf-job-operator: 20:47:04 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:04 INFO unit.tf-job-operator/0.juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch
controller-0: 20:47:04 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-persistence: 20:47:04 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-persistence/0
application-pipelines-persistence: 20:47:04 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-persistence/0
application-pipelines-persistence: 20:47:04 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-persistence-0" d67ee23b-6f10-4e86-b8a8-1b878f678466
application-pipelines-persistence: 20:47:04 DEBUG juju.kubernetes.provider.exec no pod named "d67ee23b-6f10-4e86-b8a8-1b878f678466" found
application-pipelines-persistence: 20:47:04 DEBUG juju.kubernetes.provider.exec try get pod by UID for "d67ee23b-6f10-4e86-b8a8-1b878f678466"
application-tf-job-operator: 20:47:04 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:04 INFO unit.tf-job-operator/0.juju-log status-set: maintenance: fetching resource: oci-image
application-pipelines-persistence: 20:47:04 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:47:04 DEBUG juju.worker.caasunitprovisioner service for pipelines-persistence(workload): &{Id: Addresses:[] Scale:0xc000c4d2b8 Generation:0xc000c4d2d0 Status:{Status:active Message: Data:map[] Since:<nil>}}
application-tf-job-operator: 20:47:04 DEBUG jujuc running hook tool "status-set" for tf-job-operator/0-install-7476131894038043726
application-pipelines-scheduledworkflow: 20:47:05 DEBUG juju.worker.uniter.operation committing operation "remote init" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:47:05 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:47:05 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-tf-job-operator: 20:47:05 DEBUG jujuc running hook tool "resource-get" for tf-job-operator/0-install-7476131894038043726
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/seldon-core-0"
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.operation committing operation "install local:kubernetes/seldon-core-0" for seldon-core/0
application-seldon-core: 20:47:05 INFO juju.worker.uniter hooks are retried true
application-tf-job-operator: 20:47:05 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:05 DEBUG unit.tf-job-operator/0.juju-log tracer: set flag layer.docker-resource.oci-image.available
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got storage change for seldon-core/0: [] ok=true
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got address change for seldon-core/0: ok=true, hashes=[]
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got leader settings change for seldon-core/0: ok=true
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got relations change for seldon-core/0: ok=true
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got trust config change for seldon-core/0: ok=true, hashes=[07de29e38c2f189a77dfc56df84fb9f04f4dd6d592fd31660e47cf9029f0a781]
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got config change for seldon-core/0: ok=true, hashes=[adf7847ed70e89eebfefdbcccc3a92ba05fa5f6f1becdab3ce09b854b36788aa]
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got unit change for seldon-core/0
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got action change for seldon-core/0: [] ok=true
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.remotestate got update status interval change for seldon-core/0: ok=true
application-seldon-core: 20:47:05 INFO juju.worker.uniter found queued "install" hook
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.operation running operation run install hook for seldon-core/0
application-seldon-core: 20:47:05 DEBUG juju.machinelock acquire machine lock for seldon-core/0 uniter (run install hook)
application-seldon-core: 20:47:05 DEBUG juju.machinelock machine lock acquired for seldon-core/0 uniter (run install hook)
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.operation preparing operation "run install hook" for seldon-core/0
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.operation executing operation "run install hook" for seldon-core/0
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running install hook
application-seldon-core: 20:47:05 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-seldon-core-0/agent.socket <nil>}
application-tf-job-operator: 20:47:05 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:05 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: cleared flag layer.docker-resource.pending
tracer: -- dequeue handler reactive/docker_resource.py:18:fetch
application-tf-job-operator: 20:47:05 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:05 INFO unit.tf-job-operator/0.juju-log Invoking reactive handler: reactive/tf_job_operator.py:25:start_charm
application-pytorch-operator: 20:47:05 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pytorch-operator-0904576432/ca.crt  } to {/tmp/unit-pytorch-operator-0904576432/ca.crt pytorch-operator-6974c76984-m47qt juju-pod-init}
application-pytorch-operator: 20:47:06 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/nebobmdw.pid; exec sh -c 'test -d /tmp/unit-pytorch-operator-0904576432/ca.crt'; "]
application-tf-job-operator: 20:47:06 DEBUG jujuc running hook tool "is-leader" for tf-job-operator/0-install-7476131894038043726
application-seldon-core: 20:47:06 DEBUG jujuc running hook tool "config-get" for seldon-core/0-install-6056959523845937220
application-tf-job-operator: 20:47:06 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:06 INFO unit.tf-job-operator/0.juju-log status-set: maintenance: configuring container
application-tf-job-operator: 20:47:06 DEBUG jujuc running hook tool "status-set" for tf-job-operator/0-install-7476131894038043726
application-pytorch-operator: 20:47:06 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/lkzbjdgf.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pytorch-operator-0904576432'; "]
application-seldon-core: 20:47:06 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:06 INFO unit.seldon-core/0.juju-log Reactive main running for hook install
application-seldon-core: 20:47:06 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:06 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: starting handler dispatch, 112 flags set
tracer: set flag config.changed
tracer: set flag config.changed.ambassador-enabled
tracer: set flag config.changed.ambassador-single-namespace
tracer: set flag config.changed.controller-id
tracer: set flag config.changed.default-user-id
tracer: set flag config.changed.engine-container-image-and-version
tracer: set flag config.changed.engine-container-image-pull-policy
tracer: set flag config.changed.engine-container-service-account-name
tracer: set flag config.changed.engine-container-user
tracer: set flag config.changed.engine-log-messages-externally
tracer: set flag config.changed.engine-prometheus-path
tracer: set flag config.changed.engine-server-grpc-port
tracer: set flag config.changed.engine-server-port
tracer: set flag config.changed.executor-container-image-and-version
tracer: set flag config.changed.executor-container-image-pull-policy
tracer: set flag config.changed.executor-container-service-account-name
tracer: set flag config.changed.executor-container-user
tracer: set flag config.changed.executor-prometheus-path
tracer: set flag config.changed.executor-request-logger-default-endpoint
tracer: set flag config.changed.executor-server-metrics-port-name
tracer: set flag config.changed.executor-server-port
tracer: set flag config.changed.istio-enabled
tracer: set flag config.changed.istio-gateway
tracer: set flag config.changed.istio-tls-mode
tracer: set flag config.changed.manager-create-resources
tracer: set flag config.changed.metrics-port
tracer: set flag config.changed.predictive-unit-default-env-secret-ref-name
tracer: set flag config.changed.predictive-unit-metrics-port-name
tracer: set flag config.changed.predictive-unit-service-port
tracer: set flag config.changed.related-image-engine
tracer: set flag config.changed.related-image-executor
tracer: set flag config.changed.related-image-explainer
tracer: set flag config.changed.related-image-mlflowserver-grpc
tracer: set flag config.changed.related-image-mlflowserver-rest
tracer: set flag config.changed.related-image-sklearnserver-grpc
tracer: set flag config.changed.related-image-sklearnserver-rest
tracer: set flag config.changed.related-image-storage-initializer
tracer: set flag config.changed.related-image-tensorflow
tracer: set flag config.changed.related-image-tfproxy-grpc
tracer: set flag config.changed.related-image-tfproxy-rest
tracer: set flag config.changed.related-image-xgboostserver-grpc
tracer: set flag config.changed.related-image-xgboostserver-rest
tracer: set flag config.changed.use-executor
tracer: set flag config.changed.watch-namespace
tracer: set flag config.changed.webhook-port
tracer: set flag config.default.ambassador-enabled
tracer: set flag config.default.ambassador-single-namespace
tracer: set flag config.default.controller-id
tracer: set flag config.default.default-user-id
tracer: set flag config.default.engine-container-image-and-version
tracer: set flag config.default.engine-container-image-pull-policy
tracer: set flag config.default.engine-container-service-account-name
tracer: set flag config.default.engine-container-user
tracer: set flag config.default.engine-log-messages-externally
tracer: set flag config.default.engine-prometheus-path
tracer: set flag config.default.engine-server-grpc-port
tracer: set flag config.default.engine-server-port
tracer: set flag config.default.executor-container-image-and-version
tracer: set flag config.default.executor-container-image-pull-policy
tracer: set flag config.default.executor-container-service-account-name
tracer: set flag config.default.executor-container-user
tracer: set flag config.default.executor-prometheus-path
tracer: set flag config.default.executor-request-logger-default-endpoint
tracer: set flag config.default.executor-server-metrics-port-name
tracer: set flag config.default.executor-server-port
tracer: set flag config.default.istio-enabled
tracer: set flag config.default.istio-gateway
tracer: set flag config.default.istio-tls-mode
tracer: set flag config.default.manager-create-resources
tracer: set flag config.default.metrics-port
tracer: set flag config.default.predictive-unit-default-env-secret-ref-name
tracer: set flag config.default.predictive-unit-metrics-port-name
tracer: set flag config.default.predictive-unit-service-port
tracer: set flag config.default.related-image-engine
tracer: set flag config.default.related-image-executor
tracer: set flag config.default.related-image-explainer
tracer: set flag config.default.related-image-mlflowserver-grpc
tracer: set flag config.default.related-image-mlflowserver-rest
tracer: set flag config.default.related-image-sklearnserver-grpc
tracer: set flag config.default.related-image-sklearnserver-rest
tracer: set flag config.default.related-image-storage-initializer
tracer: set flag config.default.related-image-tensorflow
tracer: set flag config.default.related-image-tfproxy-grpc
tracer: set flag config.default.related-image-tfproxy-rest
tracer: set flag config.default.related-image-xgboostserver-grpc
tracer: set flag config.default.related-image-xgboostserver-rest
tracer: set flag config.default.use-executor
tracer: set flag config.default.watch-namespace
tracer: set flag config.default.webhook-port
tracer: set flag config.set.ambassador-enabled
tracer: set flag config.set.default-user-id
tracer: set flag config.set.engine-container-image-and-version
tracer: set flag config.set.engine-container-image-pull-policy
tracer: set flag config.set.engine-container-service-account-name
tracer: set flag config.set.engine-container-user
tracer: set flag config.set.engine-prometheus-path
tracer: set flag config.set.engine-server-grpc-port
tracer: set flag config.set.engine-server-port
tracer: set flag config.set.executor-container-image-and-version
tracer: set flag config.set.executor-container-image-pull-policy
tracer: set flag config.set.executor-container-service-account-name
tracer: set flag config.set.executor-container-user
tracer: set flag config.set.executor-prometheus-path
tracer: set flag config.set.executor-request-logger-default-endpoint
tracer: set flag config.set.executor-server-metrics-port-name
tracer: set flag config.set.executor-server-port
tracer: set flag config.set.istio-gateway
tracer: set flag config.set.metrics-port
tracer: set flag config.set.predictive-unit-metrics-port-name
tracer: set flag config.set.predictive-unit-service-port
tracer: set flag config.set.use-executor
tracer: set flag config.set.webhook-port
application-tf-job-operator: 20:47:06 DEBUG jujuc running hook tool "pod-spec-set" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:06 DEBUG jujuc running hook tool "pod-spec-set" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:06 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
controller-0: 20:47:06 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:47:06 DEBUG juju.worker.caasunitprovisioner service for pipelines-api(workload): &{Id:41dd3516-000c-4bd2-8989-7194ac9df0e9 Addresses:[local-cloud:10.152.183.229] Scale:0xc0038103b8 Generation:0xc0038103d0 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-seldon-core: 20:47:06 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:06 DEBUG unit.seldon-core/0.juju-log tracer: hooks phase, 0 handlers queued
application-tf-job-operator: 20:47:06 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:06 INFO unit.tf-job-operator/0.juju-log status-set: maintenance: creating container
application-tf-job-operator: 20:47:06 DEBUG jujuc running hook tool "status-set" for tf-job-operator/0-install-7476131894038043726
application-seldon-core: 20:47:06 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:06 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/docker_resource.py:7:auto_fetch
tracer: ++   queue handler reactive/seldon_core.py:23:update_image
application-seldon-core: 20:47:07 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:07 INFO unit.seldon-core/0.juju-log Invoking reactive handler: reactive/seldon_core.py:23:update_image
application-tf-job-operator: 20:47:07 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:07 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: set flag charm.started
tracer: -- dequeue handler reactive/tf_job_operator.py:25:start_charm
application-pytorch-operator: 20:47:07 DEBUG juju.worker.caasoperator operator service address: 10.152.183.122
application-pytorch-operator: 20:47:07 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pytorch-operator-0/operator-client-cache.yaml  } to {/tmp/unit-pytorch-operator-0904576432/operator-client-cache.yaml pytorch-operator-6974c76984-m47qt juju-pod-init}
application-pytorch-operator: 20:47:07 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/flsptkoq.pid; exec sh -c 'test -d /tmp/unit-pytorch-operator-0904576432/operator-client-cache.yaml'; "]
application-seldon-core: 20:47:07 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:07 INFO unit.seldon-core/0.juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch
application-tf-job-operator: 20:47:07 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:07 INFO unit.tf-job-operator/0.juju-log Invoking reactive handler: reactive/tf_job_operator.py:15:charm_ready
application-seldon-core: 20:47:07 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:07 DEBUG unit.seldon-core/0.juju-log tracer: set flag layer.docker-resource.oci-image.fetched
application-tf-job-operator: 20:47:07 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-install-7476131894038043726
application-tf-job-operator: 20:47:07 INFO unit.tf-job-operator/0.juju-log status-set: active: 
application-seldon-core: 20:47:07 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:07 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: set flag layer.docker-resource.pending
tracer: ++   queue handler reactive/docker_resource.py:18:fetch
application-tf-job-operator: 20:47:07 DEBUG jujuc running hook tool "status-set" for tf-job-operator/0-install-7476131894038043726
application-seldon-core: 20:47:07 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:07 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: set flag layer.docker-resource.auto-fetched
tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch
application-seldon-core: 20:47:07 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:07 INFO unit.seldon-core/0.juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch
application-seldon-core: 20:47:07 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:07 INFO unit.seldon-core/0.juju-log status-set: maintenance: fetching resource: oci-image
application-seldon-core: 20:47:07 DEBUG jujuc running hook tool "status-set" for seldon-core/0-install-6056959523845937220
controller-0: 20:47:07 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:47:07 DEBUG juju.worker.caasunitprovisioner service for pipelines-scheduledworkflow(workload): &{Id: Addresses:[] Scale:0xc00451dea8 Generation:0xc00451dec0 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-seldon-core: 20:47:08 DEBUG jujuc running hook tool "resource-get" for seldon-core/0-install-6056959523845937220
application-tf-job-operator: 20:47:08 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.operation committing operation "run install hook" for tf-job-operator/0
application-tf-job-operator: 20:47:08 DEBUG juju.machinelock machine lock released for tf-job-operator/0 uniter (run install hook)
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.operation lock released for tf-job-operator/0
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.operation running operation accept leadership for tf-job-operator/0
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for tf-job-operator/0
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for tf-job-operator/0
application-tf-job-operator: 20:47:08 INFO juju.worker.uniter found queued "leader-elected" hook
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for tf-job-operator/0
application-tf-job-operator: 20:47:08 DEBUG juju.machinelock acquire machine lock for tf-job-operator/0 uniter (run leader-elected hook)
application-tf-job-operator: 20:47:08 DEBUG juju.machinelock machine lock acquired for tf-job-operator/0 uniter (run leader-elected hook)
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for tf-job-operator/0
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for tf-job-operator/0
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-seldon-core: 20:47:08 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:08 DEBUG unit.seldon-core/0.juju-log tracer: set flag layer.docker-resource.oci-image.available
controller-0: 20:47:08 DEBUG juju.worker.caasunitprovisioner ensured deployment for tf-job-operator for 1 units
application-tf-job-operator: 20:47:08 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-tf-job-operator-0/agent.socket <nil>}
application-seldon-core: 20:47:08 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:08 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: cleared flag layer.docker-resource.pending
tracer: -- dequeue handler reactive/docker_resource.py:18:fetch
application-seldon-core: 20:47:08 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:08 INFO unit.seldon-core/0.juju-log Invoking reactive handler: reactive/seldon_core.py:28:start_charm
application-pytorch-operator: 20:47:08 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ulkhbbmj.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pytorch-operator-0904576432'; "]
application-tf-job-operator: 20:47:08 DEBUG jujuc running hook tool "config-get" for tf-job-operator/0-leader-elected-6685637161273577226
application-seldon-core: 20:47:08 DEBUG jujuc running hook tool "is-leader" for seldon-core/0-install-6056959523845937220
application-tf-job-operator: 20:47:08 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-leader-elected-6685637161273577226
application-tf-job-operator: 20:47:08 INFO unit.tf-job-operator/0.juju-log Reactive main running for hook leader-elected
application-seldon-core: 20:47:08 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:08 INFO unit.seldon-core/0.juju-log status-set: maintenance: configuring container
controller-0: 20:47:08 DEBUG juju.worker.caasunitprovisioner units changed: true
application-seldon-core: 20:47:08 DEBUG jujuc running hook tool "status-set" for seldon-core/0-install-6056959523845937220
controller-0: 20:47:08 DEBUG juju.worker.caasunitprovisioner service for pipelines-db(workload): &{Id:90c9bd81-6ec7-4cbd-97c9-b6f113fa660c Addresses:[local-cloud:10.152.183.235] Scale:0xc0034b1430 Generation:0xc0034b1438 Status:{Status:active Message: Data:map[] Since:<nil>}}
application-seldon-core: 20:47:08 WARNING unit.seldon-core/0.install Generating a RSA private key
application-tf-job-operator: 20:47:09 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-leader-elected-6685637161273577226
application-tf-job-operator: 20:47:09 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: starting handler dispatch, 4 flags set
tracer: set flag charm.started
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-tf-job-operator: 20:47:09 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-leader-elected-6685637161273577226
application-tf-job-operator: 20:47:09 DEBUG unit.tf-job-operator/0.juju-log tracer: hooks phase, 0 handlers queued
application-seldon-core: 20:47:09 WARNING unit.seldon-core/0.install ..................++++
application-seldon-core: 20:47:09 WARNING unit.seldon-core/0.install ...++++
application-seldon-core: 20:47:09 WARNING unit.seldon-core/0.install writing new private key to 'key.pem'
application-seldon-core: 20:47:09 WARNING unit.seldon-core/0.install -----
controller-0: 20:47:09 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-tf-job-operator: 20:47:09 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-leader-elected-6685637161273577226
application-tf-job-operator: 20:47:09 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/tf_job_operator.py:15:charm_ready
application-tf-job-operator: 20:47:09 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-leader-elected-6685637161273577226
application-tf-job-operator: 20:47:09 INFO unit.tf-job-operator/0.juju-log Invoking reactive handler: reactive/tf_job_operator.py:15:charm_ready
application-tf-job-operator: 20:47:09 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-leader-elected-6685637161273577226
application-tf-job-operator: 20:47:09 INFO unit.tf-job-operator/0.juju-log status-set: active: 
application-tf-job-operator: 20:47:09 DEBUG jujuc running hook tool "status-set" for tf-job-operator/0-leader-elected-6685637161273577226
application-pipelines-db: 20:47:09 DEBUG juju.worker.uniter.remotestate storage attachment change for pipelines-db/0: {storage-database-1 {2 alive true /var/lib/mysql}}
application-pipelines-db: 20:47:09 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:47:09 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-tf-job-operator: 20:47:10 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-tf-job-operator: 20:47:10 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for tf-job-operator/0
controller-0: 20:47:10 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-tf-job-operator: 20:47:10 DEBUG juju.machinelock machine lock released for tf-job-operator/0 uniter (run leader-elected hook)
application-tf-job-operator: 20:47:10 DEBUG juju.worker.uniter.operation lock released for tf-job-operator/0
application-tf-job-operator: 20:47:10 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-tf-job-operator: 20:47:10 DEBUG juju.worker.uniter.operation running operation run config-changed hook for tf-job-operator/0
application-tf-job-operator: 20:47:10 DEBUG juju.machinelock acquire machine lock for tf-job-operator/0 uniter (run config-changed hook)
application-tf-job-operator: 20:47:10 DEBUG juju.machinelock machine lock acquired for tf-job-operator/0 uniter (run config-changed hook)
application-tf-job-operator: 20:47:10 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for tf-job-operator/0
application-tf-job-operator: 20:47:10 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for tf-job-operator/0
application-tf-job-operator: 20:47:10 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-tf-job-operator: 20:47:10 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-tf-job-operator-0/agent.socket <nil>}
application-pytorch-operator: 20:47:10 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/nihxmoqb.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pytorch-operator-0 --charm-dir /tmp/unit-pytorch-operator-0904576432/charm --send --operator-file /tmp/unit-pytorch-operator-0904576432/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pytorch-operator-0904576432/ca.crt'; "]
application-tf-job-operator: 20:47:10 DEBUG jujuc running hook tool "config-get" for tf-job-operator/0-config-changed-1439025330385828508
application-tf-job-operator: 20:47:10 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-config-changed-1439025330385828508
application-tf-job-operator: 20:47:10 INFO unit.tf-job-operator/0.juju-log Reactive main running for hook config-changed
application-tf-job-operator: 20:47:10 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-config-changed-1439025330385828508
application-tf-job-operator: 20:47:10 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: starting handler dispatch, 4 flags set
tracer: set flag charm.started
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-seldon-core: 20:47:10 DEBUG jujuc running hook tool "pod-spec-set" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:10 DEBUG jujuc running hook tool "pod-spec-set" for seldon-core/0-install-6056959523845937220
application-pipelines-db: 20:47:11 DEBUG juju.worker.caasoperator.remotestate got application change
application-seldon-core: 20:47:11 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-tf-job-operator: 20:47:11 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-config-changed-1439025330385828508
application-tf-job-operator: 20:47:11 DEBUG unit.tf-job-operator/0.juju-log tracer: hooks phase, 0 handlers queued
application-tf-job-operator: 20:47:11 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-config-changed-1439025330385828508
application-tf-job-operator: 20:47:11 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/tf_job_operator.py:15:charm_ready
application-seldon-core: 20:47:11 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:11 INFO unit.seldon-core/0.juju-log status-set: maintenance: creating container
application-seldon-core: 20:47:11 DEBUG jujuc running hook tool "status-set" for seldon-core/0-install-6056959523845937220
application-tf-job-operator: 20:47:11 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-config-changed-1439025330385828508
application-tf-job-operator: 20:47:11 INFO unit.tf-job-operator/0.juju-log Invoking reactive handler: reactive/tf_job_operator.py:15:charm_ready
application-tf-job-operator: 20:47:11 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-config-changed-1439025330385828508
application-tf-job-operator: 20:47:11 INFO unit.tf-job-operator/0.juju-log status-set: active: 
application-seldon-core: 20:47:11 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:11 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: set flag charm.started
tracer: -- dequeue handler reactive/seldon_core.py:28:start_charm
application-tf-job-operator: 20:47:11 DEBUG jujuc running hook tool "status-set" for tf-job-operator/0-config-changed-1439025330385828508
application-seldon-core: 20:47:12 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:12 INFO unit.seldon-core/0.juju-log Invoking reactive handler: reactive/seldon_core.py:18:charm_ready
application-seldon-core: 20:47:12 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-install-6056959523845937220
application-seldon-core: 20:47:12 INFO unit.seldon-core/0.juju-log status-set: active: 
application-seldon-core: 20:47:12 DEBUG jujuc running hook tool "status-set" for seldon-core/0-install-6056959523845937220
controller-0: 20:47:12 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:47:13 DEBUG juju.worker.caasunitprovisioner service for tf-job-operator(workload): &{Id: Addresses:[] Scale:0xc0022de1d8 Generation:0xc0022de1f0 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pytorch-operator: 20:47:13 DEBUG juju.worker.uniter.operation committing operation "remote init" for pytorch-operator/0
application-tf-job-operator: 20:47:13 DEBUG juju.worker.caasoperator trigger running status for caas unit tf-job-operator/0
application-tf-job-operator: 20:47:13 DEBUG juju.worker.uniter.remotestate got running status change for tf-job-operator/0
application-tf-job-operator: 20:47:13 DEBUG juju.worker.caasoperator request running status for "unit-tf-job-operator-0" 6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97
application-tf-job-operator: 20:47:14 DEBUG juju.kubernetes.provider.exec no pod named "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97" found
application-tf-job-operator: 20:47:14 DEBUG juju.kubernetes.provider.exec try get pod by UID for "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97"
application-pytorch-operator: 20:47:14 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:47:14 DEBUG juju.worker.uniter.remotestate storage attachment change for pipelines-db/0: {storage-database-1 {2 alive true /var/lib/mysql}}
application-pipelines-db: 20:47:14 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:47:14 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-tf-job-operator: 20:47:14 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-tf-job-operator: 20:47:14 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for tf-job-operator/0
application-tf-job-operator: 20:47:15 DEBUG juju.machinelock machine lock released for tf-job-operator/0 uniter (run config-changed hook)
application-tf-job-operator: 20:47:15 DEBUG juju.worker.uniter.operation lock released for tf-job-operator/0
application-tf-job-operator: 20:47:15 INFO juju.worker.uniter found queued "start" hook
application-tf-job-operator: 20:47:15 DEBUG juju.worker.uniter.operation running operation run start hook for tf-job-operator/0
application-tf-job-operator: 20:47:15 DEBUG juju.machinelock acquire machine lock for tf-job-operator/0 uniter (run start hook)
application-tf-job-operator: 20:47:15 DEBUG juju.machinelock machine lock acquired for tf-job-operator/0 uniter (run start hook)
application-tf-job-operator: 20:47:15 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for tf-job-operator/0
application-tf-job-operator: 20:47:15 DEBUG juju.worker.uniter.operation executing operation "run start hook" for tf-job-operator/0
application-tf-job-operator: 20:47:15 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-seldon-core: 20:47:15 INFO juju.worker.uniter.operation ran "install" hook (via explicit, bespoke hook script)
application-tf-job-operator: 20:47:15 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-tf-job-operator-0/agent.socket <nil>}
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.operation committing operation "run install hook" for seldon-core/0
application-seldon-core: 20:47:15 DEBUG juju.machinelock machine lock released for seldon-core/0 uniter (run install hook)
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.operation lock released for seldon-core/0
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.operation running operation accept leadership for seldon-core/0
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.operation preparing operation "accept leadership" for seldon-core/0
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.operation committing operation "accept leadership" for seldon-core/0
application-seldon-core: 20:47:15 INFO juju.worker.uniter found queued "leader-elected" hook
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.operation running operation run leader-elected hook for seldon-core/0
application-seldon-core: 20:47:15 DEBUG juju.machinelock acquire machine lock for seldon-core/0 uniter (run leader-elected hook)
application-seldon-core: 20:47:15 DEBUG juju.machinelock machine lock acquired for seldon-core/0 uniter (run leader-elected hook)
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.operation preparing operation "run leader-elected hook" for seldon-core/0
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.operation executing operation "run leader-elected hook" for seldon-core/0
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running leader-elected hook
application-seldon-core: 20:47:15 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-seldon-core-0/agent.socket <nil>}
application-tf-job-operator: 20:47:15 DEBUG jujuc running hook tool "config-get" for tf-job-operator/0-start-8570123322490755435
application-tf-job-operator: 20:47:15 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-start-8570123322490755435
application-tf-job-operator: 20:47:15 INFO unit.tf-job-operator/0.juju-log Reactive main running for hook start
application-seldon-core: 20:47:15 DEBUG jujuc running hook tool "config-get" for seldon-core/0-leader-elected-4604434254487899651
application-tf-job-operator: 20:47:15 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-start-8570123322490755435
application-tf-job-operator: 20:47:15 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: starting handler dispatch, 4 flags set
tracer: set flag charm.started
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
controller-0: 20:47:15 DEBUG juju.worker.caasunitprovisioner ensured deployment for seldon-core for 1 units
application-tf-job-operator: 20:47:15 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-start-8570123322490755435
application-tf-job-operator: 20:47:15 DEBUG unit.tf-job-operator/0.juju-log tracer: hooks phase, 0 handlers queued
application-seldon-core: 20:47:15 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-leader-elected-4604434254487899651
application-seldon-core: 20:47:15 INFO unit.seldon-core/0.juju-log Reactive main running for hook leader-elected
application-tf-job-operator: 20:47:15 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-start-8570123322490755435
application-tf-job-operator: 20:47:15 DEBUG unit.tf-job-operator/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/tf_job_operator.py:15:charm_ready
application-seldon-core: 20:47:16 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-leader-elected-4604434254487899651
application-seldon-core: 20:47:16 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: starting handler dispatch, 71 flags set
tracer: set flag charm.started
tracer: set flag config.default.ambassador-enabled
tracer: set flag config.default.ambassador-single-namespace
tracer: set flag config.default.controller-id
tracer: set flag config.default.default-user-id
tracer: set flag config.default.engine-container-image-and-version
tracer: set flag config.default.engine-container-image-pull-policy
tracer: set flag config.default.engine-container-service-account-name
tracer: set flag config.default.engine-container-user
tracer: set flag config.default.engine-log-messages-externally
tracer: set flag config.default.engine-prometheus-path
tracer: set flag config.default.engine-server-grpc-port
tracer: set flag config.default.engine-server-port
tracer: set flag config.default.executor-container-image-and-version
tracer: set flag config.default.executor-container-image-pull-policy
tracer: set flag config.default.executor-container-service-account-name
tracer: set flag config.default.executor-container-user
tracer: set flag config.default.executor-prometheus-path
tracer: set flag config.default.executor-request-logger-default-endpoint
tracer: set flag config.default.executor-server-metrics-port-name
tracer: set flag config.default.executor-server-port
tracer: set flag config.default.istio-enabled
tracer: set flag config.default.istio-gateway
tracer: set flag config.default.istio-tls-mode
tracer: set flag config.default.manager-create-resources
tracer: set flag config.default.metrics-port
tracer: set flag config.default.predictive-unit-default-env-secret-ref-name
tracer: set flag config.default.predictive-unit-metrics-port-name
tracer: set flag config.default.predictive-unit-service-port
tracer: set flag config.default.related-image-engine
tracer: set flag config.default.related-image-executor
tracer: set flag config.default.related-image-explainer
tracer: set flag config.default.related-image-mlflowserver-grpc
tracer: set flag config.default.related-image-mlflowserver-rest
tracer: set flag config.default.related-image-sklearnserver-grpc
tracer: set flag config.default.related-image-sklearnserver-rest
tracer: set flag config.default.related-image-storage-initializer
tracer: set flag config.default.related-image-tensorflow
tracer: set flag config.default.related-image-tfproxy-grpc
tracer: set flag config.default.related-image-tfproxy-rest
tracer: set flag config.default.related-image-xgboostserver-grpc
tracer: set flag config.default.related-image-xgboostserver-rest
tracer: set flag config.default.use-executor
tracer: set flag config.default.watch-namespace
tracer: set flag config.default.webhook-port
tracer: set flag config.set.ambassador-enabled
tracer: set flag config.set.default-user-id
tracer: set flag config.set.engine-container-image-and-version
tracer: set flag config.set.engine-container-image-pull-policy
tracer: set flag config.set.engine-container-service-account-name
tracer: set flag config.set.engine-container-user
tracer: set flag config.set.engine-prometheus-path
tracer: set flag config.set.engine-server-grpc-port
tracer: set flag config.set.engine-server-port
tracer: set flag config.set.executor-container-image-and-version
tracer: set flag config.set.executor-container-image-pull-policy
tracer: set flag config.set.executor-container-service-account-name
tracer: set flag config.set.executor-container-user
tracer: set flag config.set.executor-prometheus-path
tracer: set flag config.set.executor-request-logger-default-endpoint
tracer: set flag config.set.executor-server-metrics-port-name
tracer: set flag config.set.executor-server-port
tracer: set flag config.set.istio-gateway
tracer: set flag config.set.metrics-port
tracer: set flag config.set.predictive-unit-metrics-port-name
tracer: set flag config.set.predictive-unit-service-port
tracer: set flag config.set.use-executor
tracer: set flag config.set.webhook-port
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-tf-job-operator: 20:47:16 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-start-8570123322490755435
application-tf-job-operator: 20:47:16 INFO unit.tf-job-operator/0.juju-log Invoking reactive handler: reactive/tf_job_operator.py:15:charm_ready
application-seldon-core: 20:47:16 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-leader-elected-4604434254487899651
application-seldon-core: 20:47:16 DEBUG unit.seldon-core/0.juju-log tracer: hooks phase, 0 handlers queued
application-tf-job-operator: 20:47:16 DEBUG jujuc running hook tool "juju-log" for tf-job-operator/0-start-8570123322490755435
application-tf-job-operator: 20:47:16 INFO unit.tf-job-operator/0.juju-log status-set: active: 
application-seldon-core: 20:47:16 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-leader-elected-4604434254487899651
application-seldon-core: 20:47:16 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/seldon_core.py:18:charm_ready
application-tf-job-operator: 20:47:16 DEBUG jujuc running hook tool "status-set" for tf-job-operator/0-start-8570123322490755435
application-seldon-core: 20:47:16 DEBUG juju.worker.caasoperator.remotestate got application change
application-seldon-core: 20:47:16 DEBUG juju.worker.uniter.remotestate got address change for seldon-core/0: ok=true, hashes=[0a87cebe0fc04fc08b8ca2e318d2d5f3e69df283b7baaac895278a95fd759b3b]
application-seldon-core: 20:47:16 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-leader-elected-4604434254487899651
application-seldon-core: 20:47:16 INFO unit.seldon-core/0.juju-log Invoking reactive handler: reactive/seldon_core.py:18:charm_ready
controller-0: 20:47:16 DEBUG juju.worker.caasunitprovisioner units changed: true
application-seldon-core: 20:47:16 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-leader-elected-4604434254487899651
application-seldon-core: 20:47:16 INFO unit.seldon-core/0.juju-log status-set: active: 
controller-0: 20:47:16 DEBUG juju.worker.caasunitprovisioner service for pytorch-operator(workload): &{Id: Addresses:[] Scale:0xc000b23868 Generation:0xc000b238a0 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-tf-job-operator: 20:47:16 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-seldon-core: 20:47:16 DEBUG jujuc running hook tool "status-set" for seldon-core/0-leader-elected-4604434254487899651
application-tf-job-operator: 20:47:16 DEBUG juju.worker.uniter.operation committing operation "run start hook" for tf-job-operator/0
application-tf-job-operator: 20:47:16 DEBUG juju.machinelock machine lock released for tf-job-operator/0 uniter (run start hook)
application-tf-job-operator: 20:47:16 DEBUG juju.worker.uniter.operation lock released for tf-job-operator/0
application-tf-job-operator: 20:47:16 DEBUG juju.worker.uniter.operation running operation remote init for tf-job-operator/0
application-tf-job-operator: 20:47:16 DEBUG juju.worker.uniter.operation preparing operation "remote init" for tf-job-operator/0
application-tf-job-operator: 20:47:16 DEBUG juju.worker.uniter.operation executing operation "remote init" for tf-job-operator/0
application-tf-job-operator: 20:47:16 DEBUG juju.worker.caasoperator request running status for "unit-tf-job-operator-0" tf-job-operator-7c48d7ff49-zslkw
application-tf-job-operator: 20:47:16 DEBUG juju.worker.caasoperator remote init for "unit-tf-job-operator-0" {PodName:tf-job-operator-7c48d7ff49-zslkw Initialising:true InitialisingTime:2020-10-21 20:47:11 +0000 UTC Running:false}
application-tf-job-operator: 20:47:16 INFO juju.worker.caasoperator started pod init on "tf-job-operator/0"
application-tf-job-operator: 20:47:16 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/epypxhao.pid; exec sh -c 'mkdir -p /tmp/unit-tf-job-operator-0592792845'; "]
controller-0: 20:47:16 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-seldon-core: 20:47:16 INFO juju.worker.uniter.operation ran "leader-elected" hook (via explicit, bespoke hook script)
application-seldon-core: 20:47:17 DEBUG juju.worker.uniter.operation committing operation "run leader-elected hook" for seldon-core/0
application-seldon-core: 20:47:17 DEBUG juju.machinelock machine lock released for seldon-core/0 uniter (run leader-elected hook)
application-seldon-core: 20:47:17 DEBUG juju.worker.uniter.operation lock released for seldon-core/0
application-seldon-core: 20:47:17 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-seldon-core: 20:47:17 DEBUG juju.worker.uniter.operation running operation run config-changed hook for seldon-core/0
application-seldon-core: 20:47:17 DEBUG juju.machinelock acquire machine lock for seldon-core/0 uniter (run config-changed hook)
application-seldon-core: 20:47:17 DEBUG juju.machinelock machine lock acquired for seldon-core/0 uniter (run config-changed hook)
application-seldon-core: 20:47:17 DEBUG juju.worker.uniter.operation preparing operation "run config-changed hook" for seldon-core/0
application-seldon-core: 20:47:17 DEBUG juju.worker.uniter.operation executing operation "run config-changed hook" for seldon-core/0
application-seldon-core: 20:47:17 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running config-changed hook
application-seldon-core: 20:47:17 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-seldon-core-0/agent.socket <nil>}
application-seldon-core: 20:47:17 DEBUG juju.worker.caasoperator.remotestate got application change
application-seldon-core: 20:47:17 DEBUG jujuc running hook tool "config-get" for seldon-core/0-config-changed-4199091633599311502
application-seldon-core: 20:47:18 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-config-changed-4199091633599311502
application-seldon-core: 20:47:18 INFO unit.seldon-core/0.juju-log Reactive main running for hook config-changed
application-seldon-core: 20:47:18 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-config-changed-4199091633599311502
application-seldon-core: 20:47:18 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: starting handler dispatch, 71 flags set
tracer: set flag charm.started
tracer: set flag config.default.ambassador-enabled
tracer: set flag config.default.ambassador-single-namespace
tracer: set flag config.default.controller-id
tracer: set flag config.default.default-user-id
tracer: set flag config.default.engine-container-image-and-version
tracer: set flag config.default.engine-container-image-pull-policy
tracer: set flag config.default.engine-container-service-account-name
tracer: set flag config.default.engine-container-user
tracer: set flag config.default.engine-log-messages-externally
tracer: set flag config.default.engine-prometheus-path
tracer: set flag config.default.engine-server-grpc-port
tracer: set flag config.default.engine-server-port
tracer: set flag config.default.executor-container-image-and-version
tracer: set flag config.default.executor-container-image-pull-policy
tracer: set flag config.default.executor-container-service-account-name
tracer: set flag config.default.executor-container-user
tracer: set flag config.default.executor-prometheus-path
tracer: set flag config.default.executor-request-logger-default-endpoint
tracer: set flag config.default.executor-server-metrics-port-name
tracer: set flag config.default.executor-server-port
tracer: set flag config.default.istio-enabled
tracer: set flag config.default.istio-gateway
tracer: set flag config.default.istio-tls-mode
tracer: set flag config.default.manager-create-resources
tracer: set flag config.default.metrics-port
tracer: set flag config.default.predictive-unit-default-env-secret-ref-name
tracer: set flag config.default.predictive-unit-metrics-port-name
tracer: set flag config.default.predictive-unit-service-port
tracer: set flag config.default.related-image-engine
tracer: set flag config.default.related-image-executor
tracer: set flag config.default.related-image-explainer
tracer: set flag config.default.related-image-mlflowserver-grpc
tracer: set flag config.default.related-image-mlflowserver-rest
tracer: set flag config.default.related-image-sklearnserver-grpc
tracer: set flag config.default.related-image-sklearnserver-rest
tracer: set flag config.default.related-image-storage-initializer
tracer: set flag config.default.related-image-tensorflow
tracer: set flag config.default.related-image-tfproxy-grpc
tracer: set flag config.default.related-image-tfproxy-rest
tracer: set flag config.default.related-image-xgboostserver-grpc
tracer: set flag config.default.related-image-xgboostserver-rest
tracer: set flag config.default.use-executor
tracer: set flag config.default.watch-namespace
tracer: set flag config.default.webhook-port
tracer: set flag config.set.ambassador-enabled
tracer: set flag config.set.default-user-id
tracer: set flag config.set.engine-container-image-and-version
tracer: set flag config.set.engine-container-image-pull-policy
tracer: set flag config.set.engine-container-service-account-name
tracer: set flag config.set.engine-container-user
tracer: set flag config.set.engine-prometheus-path
tracer: set flag config.set.engine-server-grpc-port
tracer: set flag config.set.engine-server-port
tracer: set flag config.set.executor-container-image-and-version
tracer: set flag config.set.executor-container-image-pull-policy
tracer: set flag config.set.executor-container-service-account-name
tracer: set flag config.set.executor-container-user
tracer: set flag config.set.executor-prometheus-path
tracer: set flag config.set.executor-request-logger-default-endpoint
tracer: set flag config.set.executor-server-metrics-port-name
tracer: set flag config.set.executor-server-port
tracer: set flag config.set.istio-gateway
tracer: set flag config.set.metrics-port
tracer: set flag config.set.predictive-unit-metrics-port-name
tracer: set flag config.set.predictive-unit-service-port
tracer: set flag config.set.use-executor
tracer: set flag config.set.webhook-port
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
controller-0: 20:47:18 DEBUG juju.worker.caasunitprovisioner units changed: true
application-seldon-core: 20:47:18 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-config-changed-4199091633599311502
application-seldon-core: 20:47:18 DEBUG unit.seldon-core/0.juju-log tracer: hooks phase, 0 handlers queued
application-seldon-core: 20:47:18 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-config-changed-4199091633599311502
application-seldon-core: 20:47:18 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/seldon_core.py:18:charm_ready
application-seldon-core: 20:47:18 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-config-changed-4199091633599311502
application-seldon-core: 20:47:18 INFO unit.seldon-core/0.juju-log Invoking reactive handler: reactive/seldon_core.py:18:charm_ready
application-seldon-core: 20:47:18 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-config-changed-4199091633599311502
application-seldon-core: 20:47:18 INFO unit.seldon-core/0.juju-log status-set: active: 
application-seldon-core: 20:47:18 DEBUG jujuc running hook tool "status-set" for seldon-core/0-config-changed-4199091633599311502
controller-0: 20:47:18 DEBUG juju.worker.caasunitprovisioner service for seldon-core(workload): &{Id:e2d53e59-da83-47da-a3a1-ae25d9ef4d4c Addresses:[local-cloud:10.152.183.28] Scale:0xc0039d4a98 Generation:0xc0039d4ab0 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-tf-job-operator: 20:47:18 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-tf-job-operator/charm  } to {/tmp/unit-tf-job-operator-0592792845 tf-job-operator-7c48d7ff49-zslkw juju-pod-init}
application-tf-job-operator: 20:47:18 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/qmqlrpob.pid; exec sh -c 'test -d /tmp/unit-tf-job-operator-0592792845'; "]
application-seldon-core: 20:47:18 INFO juju.worker.uniter.operation ran "config-changed" hook (via explicit, bespoke hook script)
application-seldon-core: 20:47:21 DEBUG juju.worker.uniter.operation committing operation "run config-changed hook" for seldon-core/0
application-seldon-core: 20:47:21 DEBUG juju.machinelock machine lock released for seldon-core/0 uniter (run config-changed hook)
application-seldon-core: 20:47:21 DEBUG juju.worker.uniter.operation lock released for seldon-core/0
application-seldon-core: 20:47:21 INFO juju.worker.uniter found queued "start" hook
application-seldon-core: 20:47:21 DEBUG juju.worker.uniter.operation running operation run start hook for seldon-core/0
application-seldon-core: 20:47:21 DEBUG juju.machinelock acquire machine lock for seldon-core/0 uniter (run start hook)
application-seldon-core: 20:47:21 DEBUG juju.machinelock machine lock acquired for seldon-core/0 uniter (run start hook)
application-seldon-core: 20:47:21 DEBUG juju.worker.uniter.operation preparing operation "run start hook" for seldon-core/0
application-seldon-core: 20:47:21 DEBUG juju.worker.uniter.operation executing operation "run start hook" for seldon-core/0
application-seldon-core: 20:47:21 DEBUG juju.worker.uniter [AGENT-STATUS] executing: running start hook
application-seldon-core: 20:47:21 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-seldon-core-0/agent.socket <nil>}
application-tf-job-operator: 20:47:21 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/pibztsin.pid; exec sh -c 'tar -xmf - -C /tmp/unit-tf-job-operator-0592792845'; "]
application-seldon-core: 20:47:21 DEBUG jujuc running hook tool "config-get" for seldon-core/0-start-5091022574416672869
application-seldon-core: 20:47:21 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-start-5091022574416672869
application-seldon-core: 20:47:21 INFO unit.seldon-core/0.juju-log Reactive main running for hook start
application-seldon-core: 20:47:21 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-start-5091022574416672869
application-seldon-core: 20:47:21 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: starting handler dispatch, 71 flags set
tracer: set flag charm.started
tracer: set flag config.default.ambassador-enabled
tracer: set flag config.default.ambassador-single-namespace
tracer: set flag config.default.controller-id
tracer: set flag config.default.default-user-id
tracer: set flag config.default.engine-container-image-and-version
tracer: set flag config.default.engine-container-image-pull-policy
tracer: set flag config.default.engine-container-service-account-name
tracer: set flag config.default.engine-container-user
tracer: set flag config.default.engine-log-messages-externally
tracer: set flag config.default.engine-prometheus-path
tracer: set flag config.default.engine-server-grpc-port
tracer: set flag config.default.engine-server-port
tracer: set flag config.default.executor-container-image-and-version
tracer: set flag config.default.executor-container-image-pull-policy
tracer: set flag config.default.executor-container-service-account-name
tracer: set flag config.default.executor-container-user
tracer: set flag config.default.executor-prometheus-path
tracer: set flag config.default.executor-request-logger-default-endpoint
tracer: set flag config.default.executor-server-metrics-port-name
tracer: set flag config.default.executor-server-port
tracer: set flag config.default.istio-enabled
tracer: set flag config.default.istio-gateway
tracer: set flag config.default.istio-tls-mode
tracer: set flag config.default.manager-create-resources
tracer: set flag config.default.metrics-port
tracer: set flag config.default.predictive-unit-default-env-secret-ref-name
tracer: set flag config.default.predictive-unit-metrics-port-name
tracer: set flag config.default.predictive-unit-service-port
tracer: set flag config.default.related-image-engine
tracer: set flag config.default.related-image-executor
tracer: set flag config.default.related-image-explainer
tracer: set flag config.default.related-image-mlflowserver-grpc
tracer: set flag config.default.related-image-mlflowserver-rest
tracer: set flag config.default.related-image-sklearnserver-grpc
tracer: set flag config.default.related-image-sklearnserver-rest
tracer: set flag config.default.related-image-storage-initializer
tracer: set flag config.default.related-image-tensorflow
tracer: set flag config.default.related-image-tfproxy-grpc
tracer: set flag config.default.related-image-tfproxy-rest
tracer: set flag config.default.related-image-xgboostserver-grpc
tracer: set flag config.default.related-image-xgboostserver-rest
tracer: set flag config.default.use-executor
tracer: set flag config.default.watch-namespace
tracer: set flag config.default.webhook-port
tracer: set flag config.set.ambassador-enabled
tracer: set flag config.set.default-user-id
tracer: set flag config.set.engine-container-image-and-version
tracer: set flag config.set.engine-container-image-pull-policy
tracer: set flag config.set.engine-container-service-account-name
tracer: set flag config.set.engine-container-user
tracer: set flag config.set.engine-prometheus-path
tracer: set flag config.set.engine-server-grpc-port
tracer: set flag config.set.engine-server-port
tracer: set flag config.set.executor-container-image-and-version
tracer: set flag config.set.executor-container-image-pull-policy
tracer: set flag config.set.executor-container-service-account-name
tracer: set flag config.set.executor-container-user
tracer: set flag config.set.executor-prometheus-path
tracer: set flag config.set.executor-request-logger-default-endpoint
tracer: set flag config.set.executor-server-metrics-port-name
tracer: set flag config.set.executor-server-port
tracer: set flag config.set.istio-gateway
tracer: set flag config.set.metrics-port
tracer: set flag config.set.predictive-unit-metrics-port-name
tracer: set flag config.set.predictive-unit-service-port
tracer: set flag config.set.use-executor
tracer: set flag config.set.webhook-port
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-seldon-core: 20:47:21 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-start-5091022574416672869
application-seldon-core: 20:47:21 DEBUG unit.seldon-core/0.juju-log tracer: hooks phase, 0 handlers queued
application-seldon-core: 20:47:21 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-start-5091022574416672869
application-seldon-core: 20:47:21 DEBUG unit.seldon-core/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/seldon_core.py:18:charm_ready
application-seldon-core: 20:47:21 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-start-5091022574416672869
application-seldon-core: 20:47:21 INFO unit.seldon-core/0.juju-log Invoking reactive handler: reactive/seldon_core.py:18:charm_ready
application-seldon-core: 20:47:22 DEBUG jujuc running hook tool "juju-log" for seldon-core/0-start-5091022574416672869
application-seldon-core: 20:47:22 INFO unit.seldon-core/0.juju-log status-set: active: 
controller-0: 20:47:22 DEBUG juju.worker.caasunitprovisioner units changed: true
application-seldon-core: 20:47:22 DEBUG juju.worker.caasoperator trigger running status for caas unit seldon-core/0
application-seldon-core: 20:47:22 DEBUG juju.worker.uniter.remotestate got running status change for seldon-core/0
controller-0: 20:47:22 DEBUG juju.worker.caasunitprovisioner service for seldon-core(workload): &{Id:e2d53e59-da83-47da-a3a1-ae25d9ef4d4c Addresses:[local-cloud:10.152.183.28] Scale:0xc002e12a68 Generation:0xc002e12a80 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-seldon-core: 20:47:22 DEBUG juju.worker.caasoperator request running status for "unit-seldon-core-0" 99fe7a56-602c-405a-8b08-2ca156ab3f06
application-seldon-core: 20:47:22 DEBUG juju.kubernetes.provider.exec no pod named "99fe7a56-602c-405a-8b08-2ca156ab3f06" found
application-seldon-core: 20:47:22 DEBUG juju.kubernetes.provider.exec try get pod by UID for "99fe7a56-602c-405a-8b08-2ca156ab3f06"
application-seldon-core: 20:47:22 DEBUG jujuc running hook tool "status-set" for seldon-core/0-start-5091022574416672869
application-tf-job-operator: 20:47:22 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-tf-job-operator-0592792845/ca.crt  } to {/tmp/unit-tf-job-operator-0592792845/ca.crt tf-job-operator-7c48d7ff49-zslkw juju-pod-init}
application-tf-job-operator: 20:47:22 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/muwsnsms.pid; exec sh -c 'test -d /tmp/unit-tf-job-operator-0592792845/ca.crt'; "]
application-seldon-core: 20:47:22 INFO juju.worker.uniter.operation ran "start" hook (via explicit, bespoke hook script)
application-seldon-core: 20:47:22 DEBUG juju.worker.uniter.operation committing operation "run start hook" for seldon-core/0
application-seldon-core: 20:47:22 DEBUG juju.machinelock machine lock released for seldon-core/0 uniter (run start hook)
application-seldon-core: 20:47:22 DEBUG juju.worker.uniter.operation lock released for seldon-core/0
application-seldon-core: 20:47:22 DEBUG juju.worker.uniter.operation running operation remote init for seldon-core/0
application-seldon-core: 20:47:22 DEBUG juju.worker.uniter.operation preparing operation "remote init" for seldon-core/0
application-seldon-core: 20:47:22 DEBUG juju.worker.uniter.operation executing operation "remote init" for seldon-core/0
application-seldon-core: 20:47:22 DEBUG juju.worker.caasoperator request running status for "unit-seldon-core-0" seldon-core-6758ff7f64-mmc22
application-seldon-core: 20:47:22 DEBUG juju.worker.caasoperator remote init for "unit-seldon-core-0" {PodName:seldon-core-6758ff7f64-mmc22 Initialising:true InitialisingTime:2020-10-21 20:47:18 +0000 UTC Running:false}
application-seldon-core: 20:47:22 INFO juju.worker.caasoperator started pod init on "seldon-core/0"
application-seldon-core: 20:47:22 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/snzwxmbb.pid; exec sh -c 'mkdir -p /tmp/unit-seldon-core-0168883512'; "]
application-tf-job-operator: 20:47:22 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/cywdywbl.pid; exec sh -c 'tar -xmf - -C /tmp/unit-tf-job-operator-0592792845'; "]
application-seldon-core: 20:47:23 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-seldon-core/charm  } to {/tmp/unit-seldon-core-0168883512 seldon-core-6758ff7f64-mmc22 juju-pod-init}
application-seldon-core: 20:47:23 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/xpcgqmxo.pid; exec sh -c 'test -d /tmp/unit-seldon-core-0168883512'; "]
application-tf-job-operator: 20:47:23 DEBUG juju.worker.caasoperator operator service address: 10.152.183.150
application-tf-job-operator: 20:47:23 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-tf-job-operator-0/operator-client-cache.yaml  } to {/tmp/unit-tf-job-operator-0592792845/operator-client-cache.yaml tf-job-operator-7c48d7ff49-zslkw juju-pod-init}
application-tf-job-operator: 20:47:23 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/mpmnrvqq.pid; exec sh -c 'test -d /tmp/unit-tf-job-operator-0592792845/operator-client-cache.yaml'; "]
application-seldon-core: 20:47:23 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/xpeckymc.pid; exec sh -c 'tar -xmf - -C /tmp/unit-seldon-core-0168883512'; "]
application-tf-job-operator: 20:47:23 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/icyibdgg.pid; exec sh -c 'tar -xmf - -C /tmp/unit-tf-job-operator-0592792845'; "]
application-seldon-core: 20:47:24 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-seldon-core-0168883512/ca.crt  } to {/tmp/unit-seldon-core-0168883512/ca.crt seldon-core-6758ff7f64-mmc22 juju-pod-init}
application-seldon-core: 20:47:24 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/zrrgngqn.pid; exec sh -c 'test -d /tmp/unit-seldon-core-0168883512/ca.crt'; "]
application-tf-job-operator: 20:47:24 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/lisrzovm.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-tf-job-operator-0 --charm-dir /tmp/unit-tf-job-operator-0592792845/charm --send --operator-file /tmp/unit-tf-job-operator-0592792845/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-tf-job-operator-0592792845/ca.crt'; "]
controller-0: 20:47:25 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-api: 20:47:25 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-api/0
application-pipelines-api: 20:47:25 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-api/0
application-pipelines-api: 20:47:25 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-api-0" 060740d9-9aef-4765-81a1-73ba4a468772
application-pipelines-api: 20:47:25 DEBUG juju.kubernetes.provider.exec no pod named "060740d9-9aef-4765-81a1-73ba4a468772" found
application-pipelines-api: 20:47:25 DEBUG juju.kubernetes.provider.exec try get pod by UID for "060740d9-9aef-4765-81a1-73ba4a468772"
controller-0: 20:47:25 DEBUG juju.worker.caasunitprovisioner service for pipelines-api(workload): &{Id:41dd3516-000c-4bd2-8989-7194ac9df0e9 Addresses:[local-cloud:10.152.183.229] Scale:0xc004094c18 Generation:0xc004094c30 Status:{Status:active Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:47:25 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:47:25 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-api: 20:47:25 DEBUG juju.worker.caasoperator.remotestate got application change
application-tf-job-operator: 20:47:25 DEBUG juju.worker.uniter.operation committing operation "remote init" for tf-job-operator/0
application-tf-job-operator: 20:47:25 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-tf-job-operator: 20:47:25 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-seldon-core: 20:47:25 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/bcdsovpb.pid; exec sh -c 'tar -xmf - -C /tmp/unit-seldon-core-0168883512'; "]
application-tf-job-operator: 20:47:25 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-seldon-core: 20:47:26 DEBUG juju.worker.caasoperator operator service address: 10.152.183.94
application-seldon-core: 20:47:26 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-seldon-core-0/operator-client-cache.yaml  } to {/tmp/unit-seldon-core-0168883512/operator-client-cache.yaml seldon-core-6758ff7f64-mmc22 juju-pod-init}
application-seldon-core: 20:47:26 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ckwynbai.pid; exec sh -c 'test -d /tmp/unit-seldon-core-0168883512/operator-client-cache.yaml'; "]
application-seldon-core: 20:47:27 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/jxdhwyxs.pid; exec sh -c 'tar -xmf - -C /tmp/unit-seldon-core-0168883512'; "]
application-seldon-core: 20:47:27 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/rfhonwwi.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-seldon-core-0 --charm-dir /tmp/unit-seldon-core-0168883512/charm --send --operator-file /tmp/unit-seldon-core-0168883512/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-seldon-core-0168883512/ca.crt'; "]
application-seldon-core: 20:47:28 DEBUG juju.worker.uniter.operation committing operation "remote init" for seldon-core/0
application-seldon-core: 20:47:28 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-seldon-core: 20:47:28 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-seldon-core: 20:47:28 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:47:29 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:47:29 DEBUG juju.worker.caasunitprovisioner service for tf-job-operator(workload): &{Id: Addresses:[] Scale:0xc000c4c168 Generation:0xc000c4c180 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
controller-0: 20:47:30 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-scheduledworkflow: 20:47:30 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:47:30 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:47:30 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-scheduledworkflow-0" 8c345d1f-b75a-4dff-a4bb-67637ec9f804
application-pipelines-scheduledworkflow: 20:47:30 DEBUG juju.kubernetes.provider.exec no pod named "8c345d1f-b75a-4dff-a4bb-67637ec9f804" found
application-pipelines-scheduledworkflow: 20:47:30 DEBUG juju.kubernetes.provider.exec try get pod by UID for "8c345d1f-b75a-4dff-a4bb-67637ec9f804"
application-pipelines-scheduledworkflow: 20:47:30 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:47:30 DEBUG juju.worker.caasunitprovisioner service for pipelines-scheduledworkflow(workload): &{Id: Addresses:[] Scale:0xc0023e50c8 Generation:0xc0023e50e0 Status:{Status:active Message: Data:map[] Since:<nil>}}
controller-0: 20:47:30 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:47:32 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:47:32 DEBUG juju.worker.caasunitprovisioner service for seldon-core(workload): &{Id:e2d53e59-da83-47da-a3a1-ae25d9ef4d4c Addresses:[local-cloud:10.152.183.28] Scale:0xc001e4bfc8 Generation:0xc001e4bfe0 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
controller-0: 20:47:40 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pytorch-operator: 20:47:40 DEBUG juju.worker.caasoperator trigger running status for caas unit pytorch-operator/0
application-pytorch-operator: 20:47:40 DEBUG juju.worker.uniter.remotestate got running status change for pytorch-operator/0
application-pytorch-operator: 20:47:40 DEBUG juju.worker.caasoperator request running status for "unit-pytorch-operator-0" b95510d3-6d1e-4d72-9615-e6403906e412
application-pytorch-operator: 20:47:40 DEBUG juju.kubernetes.provider.exec no pod named "b95510d3-6d1e-4d72-9615-e6403906e412" found
application-pytorch-operator: 20:47:40 DEBUG juju.kubernetes.provider.exec try get pod by UID for "b95510d3-6d1e-4d72-9615-e6403906e412"
controller-0: 20:47:40 DEBUG juju.worker.caasunitprovisioner service for pytorch-operator(workload): &{Id: Addresses:[] Scale:0xc003724878 Generation:0xc0037248e0 Status:{Status:active Message: Data:map[] Since:<nil>}}
application-pytorch-operator: 20:47:40 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:47:40 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:47:44 DEBUG juju.worker.caasunitprovisioner units changed: true
application-tf-job-operator: 20:47:44 DEBUG juju.worker.caasoperator trigger running status for caas unit tf-job-operator/0
application-tf-job-operator: 20:47:44 DEBUG juju.worker.uniter.remotestate got running status change for tf-job-operator/0
application-tf-job-operator: 20:47:44 DEBUG juju.worker.caasoperator request running status for "unit-tf-job-operator-0" 6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97
application-tf-job-operator: 20:47:44 DEBUG juju.kubernetes.provider.exec no pod named "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97" found
application-tf-job-operator: 20:47:44 DEBUG juju.kubernetes.provider.exec try get pod by UID for "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97"
controller-0: 20:47:44 DEBUG juju.worker.caasunitprovisioner service for tf-job-operator(workload): &{Id: Addresses:[] Scale:0xc000d4a278 Generation:0xc000d4a2a0 Status:{Status:active Message: Data:map[] Since:<nil>}}
application-tf-job-operator: 20:47:44 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:47:44 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-seldon-core: 20:47:48 DEBUG juju.worker.caasoperator trigger running status for caas unit seldon-core/0
application-seldon-core: 20:47:48 DEBUG juju.worker.uniter.remotestate got running status change for seldon-core/0
application-seldon-core: 20:47:48 DEBUG juju.worker.caasoperator request running status for "unit-seldon-core-0" 99fe7a56-602c-405a-8b08-2ca156ab3f06
controller-0: 20:47:48 DEBUG juju.worker.caasunitprovisioner units changed: true
application-seldon-core: 20:47:48 DEBUG juju.kubernetes.provider.exec no pod named "99fe7a56-602c-405a-8b08-2ca156ab3f06" found
application-seldon-core: 20:47:48 DEBUG juju.kubernetes.provider.exec try get pod by UID for "99fe7a56-602c-405a-8b08-2ca156ab3f06"
application-seldon-core: 20:47:48 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:47:48 DEBUG juju.worker.caasunitprovisioner service for seldon-core(workload): &{Id:e2d53e59-da83-47da-a3a1-ae25d9ef4d4c Addresses:[local-cloud:10.152.183.28] Scale:0xc000c7ae98 Generation:0xc000c7aeb0 Status:{Status:active Message: Data:map[] Since:<nil>}}
controller-0: 20:47:48 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-seldon-core: 20:47:48 DEBUG juju.worker.caasoperator.remotestate got application change
application-argo-controller: 20:49:14 DEBUG juju.worker.uniter.remotestate update status timer triggered for argo-controller/0
application-argo-controller: 20:49:14 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:49:14 DEBUG juju.worker.uniter.operation running operation run update-status hook for argo-controller/0
application-argo-controller: 20:49:14 DEBUG juju.machinelock acquire machine lock for argo-controller/0 uniter (run update-status hook)
application-argo-controller: 20:49:14 DEBUG juju.machinelock machine lock acquired for argo-controller/0 uniter (run update-status hook)
application-argo-controller: 20:49:14 DEBUG juju.worker.uniter.operation preparing operation "run update-status hook" for argo-controller/0
application-argo-controller: 20:49:14 DEBUG juju.worker.uniter.operation executing operation "run update-status hook" for argo-controller/0
application-argo-controller: 20:49:14 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-argo-controller-0/agent.socket <nil>}
application-argo-controller: 20:49:14 DEBUG jujuc running hook tool "config-get" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:14 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:14 INFO unit.argo-controller/0.juju-log Reactive main running for hook update-status
application-argo-controller: 20:49:14 DEBUG jujuc running hook tool "relation-ids" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:14 DEBUG jujuc running hook tool "relation-list" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:15 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: starting handler dispatch, 17 flags set
tracer: set flag charm.started
tracer: set flag config.default.bucket
tracer: set flag config.default.executor
tracer: set flag config.default.key-prefix
tracer: set flag config.default.kubelet-insecure
tracer: set flag config.set.bucket
tracer: set flag config.set.executor
tracer: set flag config.set.key-prefix
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-argo-controller: 20:49:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:15 DEBUG unit.argo-controller/0.juju-log tracer: hooks phase, 0 handlers queued
application-argo-controller: 20:49:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:15 DEBUG unit.argo-controller/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/argo_controller.py:16:charm_ready
application-argo-controller: 20:49:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:15 INFO unit.argo-controller/0.juju-log Invoking reactive handler: reactive/argo_controller.py:16:charm_ready
application-argo-controller: 20:49:15 DEBUG jujuc running hook tool "juju-log" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:15 INFO unit.argo-controller/0.juju-log status-set: active: 
application-argo-controller: 20:49:15 DEBUG jujuc running hook tool "status-set" for argo-controller/0-update-status-4709462811780397663
application-argo-controller: 20:49:15 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
application-argo-controller: 20:49:15 DEBUG juju.worker.uniter.operation committing operation "run update-status hook" for argo-controller/0
application-argo-controller: 20:49:15 DEBUG juju.machinelock machine lock released for argo-controller/0 uniter (run update-status hook)
application-argo-controller: 20:49:15 DEBUG juju.worker.uniter.operation lock released for argo-controller/0
application-argo-controller: 20:49:15 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:49:32 DEBUG juju.worker.uniter.remotestate update status timer triggered for minio/0
application-minio: 20:49:32 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:49:32 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:49:32 DEBUG juju.worker.uniter.operation running operation run update-status hook for minio/0
application-minio: 20:49:32 DEBUG juju.machinelock acquire machine lock for minio/0 uniter (run update-status hook)
application-minio: 20:49:32 DEBUG juju.machinelock machine lock acquired for minio/0 uniter (run update-status hook)
application-minio: 20:49:32 DEBUG juju.worker.uniter.operation preparing operation "run update-status hook" for minio/0
application-minio: 20:49:33 DEBUG juju.worker.uniter.operation executing operation "run update-status hook" for minio/0
application-minio: 20:49:33 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-minio-0/agent.socket <nil>}
application-minio: 20:49:39 DEBUG jujuc running hook tool "config-get" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 DEBUG jujuc running hook tool "juju-log" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 INFO unit.minio/0.juju-log Reactive main running for hook update-status
application-minio: 20:49:39 DEBUG jujuc running hook tool "relation-ids" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 DEBUG jujuc running hook tool "relation-list" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 DEBUG jujuc running hook tool "relation-list" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 DEBUG jujuc running hook tool "juju-log" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 DEBUG unit.minio/0.juju-log tracer>
tracer: starting handler dispatch, 13 flags set
tracer: set flag charm.started
tracer: set flag config.default.access-key
tracer: set flag config.default.port
tracer: set flag config.set.access-key
tracer: set flag config.set.port
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-minio: 20:49:39 DEBUG jujuc running hook tool "juju-log" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 DEBUG unit.minio/0.juju-log tracer: hooks phase, 0 handlers queued
application-minio: 20:49:39 DEBUG jujuc running hook tool "juju-log" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 DEBUG unit.minio/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler reactive/minio.py:13:charm_ready
tracer: ++   queue handler reactive/minio.py:18:configure_minio
application-minio: 20:49:39 DEBUG jujuc running hook tool "juju-log" for minio/0-update-status-5176739796157689185
application-minio: 20:49:39 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:13:charm_ready
application-minio: 20:49:40 DEBUG jujuc running hook tool "juju-log" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 INFO unit.minio/0.juju-log Invoking reactive handler: reactive/minio.py:18:configure_minio
application-minio: 20:49:40 DEBUG jujuc running hook tool "relation-get" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 DEBUG jujuc running hook tool "relation-get" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 DEBUG jujuc running hook tool "relation-set" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 DEBUG jujuc running hook tool "relation-set" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 DEBUG jujuc running hook tool "relation-set" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 DEBUG jujuc running hook tool "relation-set" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 DEBUG jujuc running hook tool "juju-log" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 INFO unit.minio/0.juju-log status-set: active: 
application-minio: 20:49:40 DEBUG jujuc running hook tool "status-set" for minio/0-update-status-5176739796157689185
application-minio: 20:49:40 DEBUG unit.minio/0.update-status Provides
application-minio: 20:49:40 DEBUG unit.minio/0.update-status YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:49:40 DEBUG unit.minio/0.update-status minio:0
application-minio: 20:49:40 DEBUG unit.minio/0.update-status Provides
application-minio: 20:49:40 DEBUG unit.minio/0.update-status YR9E8ANXSPL30CQQJ3G17E82QV0BJD
application-minio: 20:49:40 DEBUG unit.minio/0.update-status minio:3
application-minio: 20:49:41 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
application-minio: 20:49:41 DEBUG juju.worker.uniter.operation committing operation "run update-status hook" for minio/0
application-minio: 20:49:41 DEBUG juju.machinelock machine lock released for minio/0 uniter (run update-status hook)
application-minio: 20:49:41 DEBUG juju.worker.uniter.operation lock released for minio/0
application-minio: 20:49:41 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:49:41 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:49:58 DEBUG juju.worker.uniter.remotestate update status timer triggered for pipelines-persistence/0
application-pipelines-persistence: 20:49:58 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:49:58 DEBUG juju.worker.uniter.operation running operation run update-status hook for pipelines-persistence/0
application-pipelines-persistence: 20:49:58 DEBUG juju.machinelock acquire machine lock for pipelines-persistence/0 uniter (run update-status hook)
application-pipelines-persistence: 20:49:58 DEBUG juju.machinelock machine lock acquired for pipelines-persistence/0 uniter (run update-status hook)
application-pipelines-persistence: 20:49:58 DEBUG juju.worker.uniter.operation preparing operation "run update-status hook" for pipelines-persistence/0
application-pipelines-persistence: 20:49:58 DEBUG juju.worker.uniter.operation executing operation "run update-status hook" for pipelines-persistence/0
application-pipelines-persistence: 20:49:58 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-persistence-0/agent.socket <nil>}
application-pipelines-persistence: 20:49:58 DEBUG jujuc running hook tool "config-get" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:58 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:58 INFO unit.pipelines-persistence/0.juju-log Reactive main running for hook update-status
application-pipelines-persistence: 20:49:58 DEBUG jujuc running hook tool "relation-ids" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:58 DEBUG jujuc running hook tool "relation-list" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: starting handler dispatch, 12 flags set
tracer: set flag charm.started
tracer: set flag endpoint.pipelines-api.changed
tracer: set flag endpoint.pipelines-api.changed.egress-subnets
tracer: set flag endpoint.pipelines-api.changed.hostname
tracer: set flag endpoint.pipelines-api.changed.ingress-address
tracer: set flag endpoint.pipelines-api.changed.port
tracer: set flag endpoint.pipelines-api.changed.private-address
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag pipelines-api.available
application-pipelines-persistence: 20:49:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 DEBUG unit.pipelines-persistence/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-persistence: 20:49:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 DEBUG unit.pipelines-persistence/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/http/requires.py:10:changed:pipelines-api
tracer: ++   queue handler reactive/pipelines_persistence.py:13:charm_ready
application-pipelines-persistence: 20:49:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 INFO unit.pipelines-persistence/0.juju-log Invoking reactive handler: reactive/pipelines_persistence.py:13:charm_ready
application-pipelines-persistence: 20:49:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 INFO unit.pipelines-persistence/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-api
application-pipelines-persistence: 20:49:59 DEBUG jujuc running hook tool "relation-get" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 DEBUG jujuc running hook tool "juju-log" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 INFO unit.pipelines-persistence/0.juju-log status-set: active: 
application-pipelines-persistence: 20:49:59 DEBUG jujuc running hook tool "status-set" for pipelines-persistence/0-update-status-3041774280452928578
application-pipelines-persistence: 20:49:59 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
application-pipelines-persistence: 20:49:59 DEBUG juju.worker.uniter.operation committing operation "run update-status hook" for pipelines-persistence/0
application-pipelines-persistence: 20:49:59 DEBUG juju.machinelock machine lock released for pipelines-persistence/0 uniter (run update-status hook)
application-pipelines-persistence: 20:49:59 DEBUG juju.worker.uniter.operation lock released for pipelines-persistence/0
application-pipelines-persistence: 20:49:59 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:50:24 DEBUG juju.worker.uniter.remotestate update status timer triggered for pipelines-api/0
application-pipelines-api: 20:50:24 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:50:25 DEBUG juju.worker.uniter.operation running operation run update-status hook for pipelines-api/0
application-pipelines-api: 20:50:25 DEBUG juju.machinelock acquire machine lock for pipelines-api/0 uniter (run update-status hook)
application-pipelines-api: 20:50:25 DEBUG juju.machinelock machine lock acquired for pipelines-api/0 uniter (run update-status hook)
application-pipelines-api: 20:50:25 DEBUG juju.worker.uniter.operation preparing operation "run update-status hook" for pipelines-api/0
application-pipelines-api: 20:50:25 DEBUG juju.worker.uniter.operation executing operation "run update-status hook" for pipelines-api/0
application-pipelines-api: 20:50:25 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket <nil>}
application-pipelines-api: 20:50:25 DEBUG jujuc running hook tool "config-get" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 INFO unit.pipelines-api/0.juju-log Reactive main running for hook update-status
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-list" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-ids" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: starting handler dispatch, 33 flags set
tracer: set flag charm.started
tracer: set flag config.default.grpc-port
tracer: set flag config.default.http-port
tracer: set flag config.default.minio-access-key
tracer: set flag config.default.minio-bucket-name
tracer: set flag config.set.grpc-port
tracer: set flag config.set.http-port
tracer: set flag config.set.minio-access-key
tracer: set flag config.set.minio-bucket-name
tracer: set flag endpoint.minio.changed
tracer: set flag endpoint.minio.changed.egress-subnets
tracer: set flag endpoint.minio.changed.ingress-address
tracer: set flag endpoint.minio.changed.private-address
tracer: set flag endpoint.minio.connected
tracer: set flag endpoint.minio.joined
tracer: set flag endpoint.mysql.changed
tracer: set flag endpoint.mysql.changed.database
tracer: set flag endpoint.mysql.changed.egress-subnets
tracer: set flag endpoint.mysql.changed.host
tracer: set flag endpoint.mysql.changed.ingress-address
tracer: set flag endpoint.mysql.changed.password
tracer: set flag endpoint.mysql.changed.port
tracer: set flag endpoint.mysql.changed.private-address
tracer: set flag endpoint.mysql.changed.root_password
tracer: set flag endpoint.mysql.changed.user
tracer: set flag endpoint.mysql.joined
tracer: set flag endpoint.pipelines-api.joined
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
tracer: set flag mysql.available
tracer: set flag mysql.connected
tracer: set flag pipelines-api.available
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG unit.pipelines-api/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: main dispatch loop, 8 handlers queued
tracer: ++   queue handler hooks/relations/http/provides.py:11:joined:pipelines-api
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles
tracer: ++   queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization
tracer: ++   queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql
tracer: ++   queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql
tracer: ++   queue handler reactive/pipelines_api.py:26:charm_ready
tracer: ++   queue handler reactive/pipelines_api.py:31:configure_http
tracer: ++   queue handler reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:36:update_image
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: cleared flag charm.started
tracer: ++   queue handler reactive/pipelines_api.py:54:start_charm
tracer: -- dequeue handler reactive/pipelines_api.py:26:charm_ready
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles
application-pipelines-api: 20:50:26 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:26 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:54:start_charm
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "is-leader" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 INFO unit.pipelines-api/0.juju-log status-set: maintenance: configuring container
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "relation-get" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 INFO unit.pipelines-api/0.juju-log DEBUG
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 INFO unit.pipelines-api/0.juju-log {'ip': 'minio', 'port': 9000, 'user': 'minio', 'password': 'YR9E8ANXSPL30CQQJ3G17E82QV0BJD'}
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "pod-spec-set" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 DEBUG juju.kubernetes.provider.specs decoding stream as JSON
application-pipelines-api: 20:50:27 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:27 INFO unit.pipelines-api/0.juju-log status-set: maintenance: creating container
application-pipelines-api: 20:50:28 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:28 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:28 DEBUG unit.pipelines-api/0.juju-log tracer>
tracer: set flag charm.started
tracer: ++   queue handler reactive/pipelines_api.py:26:charm_ready
tracer: -- dequeue handler reactive/pipelines_api.py:54:start_charm
application-pipelines-api: 20:50:28 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:28 INFO unit.pipelines-api/0.juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready
application-pipelines-api: 20:50:28 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:28 DEBUG jujuc running hook tool "relation-set" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:28 DEBUG jujuc running hook tool "juju-log" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:28 INFO unit.pipelines-api/0.juju-log status-set: active: 
application-pipelines-api: 20:50:28 DEBUG jujuc running hook tool "status-set" for pipelines-api/0-update-status-5501172351894512360
application-pipelines-api: 20:50:28 DEBUG unit.pipelines-api/0.update-status Requires relation
application-pipelines-api: 20:50:28 DEBUG unit.pipelines-api/0.update-status minio:3
application-pipelines-api: 20:50:28 DEBUG unit.pipelines-api/0.update-status requires unit
application-pipelines-api: 20:50:28 DEBUG unit.pipelines-api/0.update-status {'egress-subnets': '10.152.183.102/32', 'ingress-address': '10.152.183.102', 'ip': '"minio"', 'password': '"YR9E8ANXSPL30CQQJ3G17E82QV0BJD"', 'port': '9000', 'private-address': '10.152.183.102', 'user': '"minio"'}
application-pipelines-api: 20:50:28 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
application-pipelines-api: 20:50:28 DEBUG juju.worker.uniter.operation committing operation "run update-status hook" for pipelines-api/0
application-pipelines-api: 20:50:28 DEBUG juju.machinelock machine lock released for pipelines-api/0 uniter (run update-status hook)
application-pipelines-api: 20:50:28 DEBUG juju.worker.uniter.operation lock released for pipelines-api/0
application-pipelines-api: 20:50:28 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:50:50 DEBUG juju.worker.uniter.remotestate update status timer triggered for pipelines-db/0
application-pipelines-db: 20:50:50 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:50:50 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:50:50 DEBUG juju.worker.uniter.operation running operation run update-status hook for pipelines-db/0
application-pipelines-db: 20:50:50 DEBUG juju.machinelock acquire machine lock for pipelines-db/0 uniter (run update-status hook)
application-pipelines-db: 20:50:50 DEBUG juju.machinelock machine lock acquired for pipelines-db/0 uniter (run update-status hook)
application-pipelines-db: 20:50:50 DEBUG juju.worker.uniter.operation preparing operation "run update-status hook" for pipelines-db/0
application-pipelines-db: 20:50:50 DEBUG juju.worker.uniter.operation executing operation "run update-status hook" for pipelines-db/0
application-pipelines-db: 20:50:50 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pipelines-db-0/agent.socket <nil>}
application-pipelines-db: 20:50:50 DEBUG jujuc running hook tool "config-get" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 INFO unit.pipelines-db/0.juju-log Reactive main running for hook update-status
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "relation-ids" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "relation-list" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "is-leader" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 INFO unit.pipelines-db/0.juju-log Initializing Leadership Layer (is leader)
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "leader-get" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: starting handler dispatch, 29 flags set
tracer: set flag config.default.database
tracer: set flag config.default.ha-image
tracer: set flag config.default.ha-mode
tracer: set flag config.default.image
tracer: set flag config.default.mysql_port
tracer: set flag config.default.password
tracer: set flag config.default.query-cache-size
tracer: set flag config.default.query-cache-type
tracer: set flag config.default.root_password
tracer: set flag config.default.user
tracer: set flag config.set.database
tracer: set flag config.set.ha-image
tracer: set flag config.set.image
tracer: set flag config.set.mysql_port
tracer: set flag config.set.password
tracer: set flag config.set.query-cache-type
tracer: set flag config.set.root_password
tracer: set flag config.set.user
tracer: set flag database
tracer: set flag endpoint.mysql.changed
tracer: set flag endpoint.mysql.changed.egress-subnets
tracer: set flag endpoint.mysql.changed.ingress-address
tracer: set flag endpoint.mysql.changed.private-address
tracer: set flag endpoint.mysql.joined
tracer: set flag leadership.is_leader
tracer: set flag mariadb-k8s.configured
tracer: set flag password
tracer: set flag root_password
tracer: set flag user
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG unit.pipelines-db/0.juju-log tracer: hooks phase, 0 handlers queued
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: main dispatch loop, 2 handlers queued
tracer: ++   queue handler hooks/relations/mysql/provides.py:20:_handle_joined:mysql
tracer: ++   queue handler reactive/mariadb.py:40:set_mariadb_active
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:40:set_mariadb_active
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: hooks/relations/mysql/provides.py:20:_handle_joined:mysql
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: set flag mysql.database.requested
tracer: ++   queue handler reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 INFO unit.pipelines-db/0.juju-log Invoking reactive handler: reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 INFO unit.pipelines-db/0.juju-log request -> mysql:1 for app -> pipelines-api
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:51 INFO unit.pipelines-db/0.juju-log db params: mysql:password@database
application-pipelines-db: 20:50:51 DEBUG jujuc running hook tool "network-get" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:52 DEBUG jujuc running hook tool "relation-get" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:52 DEBUG unit.pipelines-db/0.juju-log tracer>
tracer: cleared flag mysql.database.requested
tracer: -- dequeue handler reactive/mariadb.py:50:provide_database
application-pipelines-db: 20:50:52 DEBUG jujuc running hook tool "relation-set" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:52 DEBUG jujuc running hook tool "relation-set" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:52 DEBUG jujuc running hook tool "juju-log" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:52 INFO unit.pipelines-db/0.juju-log status-set: active: ready
application-pipelines-db: 20:50:52 DEBUG jujuc running hook tool "status-set" for pipelines-db/0-update-status-2348269404390314067
application-pipelines-db: 20:50:52 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
application-pipelines-db: 20:50:52 DEBUG juju.worker.uniter.operation committing operation "run update-status hook" for pipelines-db/0
application-pipelines-db: 20:50:52 DEBUG juju.machinelock machine lock released for pipelines-db/0 uniter (run update-status hook)
application-pipelines-db: 20:50:52 DEBUG juju.worker.uniter.operation lock released for pipelines-db/0
application-pipelines-db: 20:50:52 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:50:52 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:51:59 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-minio: 20:51:59 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=minio", "--debug"}
application-minio: 20:51:59 DEBUG juju.agent read agent config, format "2.0"
application-minio: 20:51:59 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-minio: 20:51:59 INFO juju.cmd.jujud caas operator application-minio start (2.8.6 [gc])
application-minio: 20:51:59 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:51:59.546087494 +0000 UTC
application-minio: 20:51:59 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:51:59.546392897 +0000 UTC
application-minio: 20:51:59 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:51:59.546405297 +0000 UTC
application-minio: 20:51:59 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:51:59.546516198 +0000 UTC
application-minio: 20:51:59 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-minio"
application-minio: 20:51:59 DEBUG juju.worker.introspection stats worker now serving
application-minio: 20:51:59 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:51:59 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:51:59.557443006 +0000 UTC
application-minio: 20:51:59 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:51:59.567579205 +0000 UTC
application-pipelines-db: 20:51:59 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pipelines-db: 20:51:59 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-db", "--debug"}
application-pipelines-db: 20:51:59 DEBUG juju.agent read agent config, format "2.0"
application-pipelines-db: 20:51:59 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pipelines-db: 20:51:59 INFO juju.cmd.jujud caas operator application-pipelines-db start (2.8.6 [gc])
application-pipelines-db: 20:51:59 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:51:59.84261581 +0000 UTC
application-pipelines-db: 20:51:59 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:51:59.846002643 +0000 UTC
application-pipelines-db: 20:51:59 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:51:59.846083544 +0000 UTC
application-pipelines-db: 20:51:59 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pipelines-db"
application-pipelines-db: 20:51:59 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:51:59.846391447 +0000 UTC
application-pipelines-db: 20:51:59 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-db: 20:51:59 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-db: 20:51:59 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:51:59.857177953 +0000 UTC
application-pipelines-db: 20:51:59 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:51:59.866778547 +0000 UTC
application-tf-job-operator: 20:52:00 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-tf-job-operator: 20:52:00 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=tf-job-operator", "--debug"}
application-minio: 20:52:00 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-minio: 20:52:00 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:00 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:00 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-minio" cannot open api
application-pipelines-db: 20:52:00 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-db: 20:52:00 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:00 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:00 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-db" cannot open api
application-tf-job-operator: 20:52:01 DEBUG juju.agent read agent config, format "2.0"
application-tf-job-operator: 20:52:01 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-tf-job-operator: 20:52:01 INFO juju.cmd.jujud caas operator application-tf-job-operator start (2.8.6 [gc])
application-tf-job-operator: 20:52:01 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:52:01.769279523 +0000 UTC
application-tf-job-operator: 20:52:01 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:52:01.769383924 +0000 UTC
application-tf-job-operator: 20:52:01 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:52:01.769444525 +0000 UTC
application-tf-job-operator: 20:52:01 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:52:01.769669427 +0000 UTC
application-tf-job-operator: 20:52:01 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-tf-job-operator"
application-tf-job-operator: 20:52:01 DEBUG juju.worker.introspection stats worker now serving
application-tf-job-operator: 20:52:01 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:01 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:52:01.780443633 +0000 UTC
application-tf-job-operator: 20:52:01 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:52:01.780673935 +0000 UTC
application-tf-job-operator: 20:52:02 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-tf-job-operator: 20:52:02 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:02 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:03 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-tf-job-operator: 20:52:03 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:03 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:03 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-tf-job-operator" cannot open api
application-pipelines-db: 20:52:04 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:52:04 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-db: 20:52:05 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-db: 20:52:05 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:05 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:05 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-db" cannot open api
application-minio: 20:52:05 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-minio: 20:52:05 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:05 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:05 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-minio" cannot open api
application-tf-job-operator: 20:52:08 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:09 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-tf-job-operator: 20:52:09 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:09 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:09 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-tf-job-operator" cannot open api
application-pipelines-db: 20:52:09 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:52:09 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-db: 20:52:10 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-db: 20:52:10 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:10 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:10 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-db" cannot open api
application-minio: 20:52:10 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-minio: 20:52:10 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:10 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:10 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-minio" cannot open api
application-pipelines-scheduledworkflow: 20:52:14 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-scheduledworkflow", "--debug"}
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.agent read agent config, format "2.0"
application-pipelines-scheduledworkflow: 20:52:14 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pipelines-scheduledworkflow: 20:52:14 INFO juju.cmd.jujud caas operator application-pipelines-scheduledworkflow start (2.8.6 [gc])
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:52:14.414164945 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:52:14.414405947 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:52:14.414574749 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pipelines-scheduledworkflow"
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:52:14.424139543 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:52:14.424290844 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:52:14.433306932 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:14 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:14 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-db: 20:52:15 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-scheduledworkflow: 20:52:15 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-scheduledworkflow: 20:52:15 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:15 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:15 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-scheduledworkflow" cannot open api
application-minio: 20:52:15 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:15 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-tf-job-operator: 20:52:15 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:15 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:15 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-tf-job-operator" cannot open api
application-pipelines-db: 20:52:16 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-db: 20:52:16 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:16 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:16 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-db" cannot open api
application-minio: 20:52:16 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-minio: 20:52:16 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:16 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:16 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-minio" cannot open api
application-pipelines-scheduledworkflow: 20:52:18 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-scheduledworkflow: 20:52:19 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-scheduledworkflow: 20:52:19 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:19 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:19 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-scheduledworkflow" cannot open api
application-argo-controller: 20:52:20 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-argo-controller: 20:52:20 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=argo-controller", "--debug"}
application-pipelines-persistence: 20:52:20 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pipelines-persistence: 20:52:20 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-persistence", "--debug"}
application-argo-controller: 20:52:20 DEBUG juju.agent read agent config, format "2.0"
application-argo-controller: 20:52:20 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-argo-controller: 20:52:20 INFO juju.cmd.jujud caas operator application-argo-controller start (2.8.6 [gc])
application-argo-controller: 20:52:20 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:52:20.165831553 +0000 UTC
application-argo-controller: 20:52:20 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:20 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:52:20.166214957 +0000 UTC
application-argo-controller: 20:52:20 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-argo-controller"
application-argo-controller: 20:52:20 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:52:20.166783363 +0000 UTC
application-argo-controller: 20:52:20 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-persistence: 20:52:20 DEBUG juju.agent read agent config, format "2.0"
application-pipelines-persistence: 20:52:20 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pipelines-persistence: 20:52:20 INFO juju.cmd.jujud caas operator application-pipelines-persistence start (2.8.6 [gc])
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:52:20.168380978 +0000 UTC
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:52:20.16855368 +0000 UTC
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pipelines-persistence"
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:52:20.168921083 +0000 UTC
application-argo-controller: 20:52:20 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:52:20.176679559 +0000 UTC
application-argo-controller: 20:52:20 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:52:20.17786317 +0000 UTC
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:52:20.180101592 +0000 UTC
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:52:20.180146193 +0000 UTC
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:20 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:52:20.186842658 +0000 UTC
application-pipelines-persistence: 20:52:20 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:52:20.19021389 +0000 UTC
application-pipelines-persistence: 20:52:21 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-persistence: 20:52:21 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:21 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:52:21 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pytorch-operator: 20:52:21 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pytorch-operator", "--debug"}
application-pipelines-api: 20:52:21 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-pipelines-api: 20:52:21 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-api", "--debug"}
application-pipelines-api: 20:52:21 DEBUG juju.agent read agent config, format "2.0"
application-pipelines-api: 20:52:21 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pipelines-api: 20:52:21 INFO juju.cmd.jujud caas operator application-pipelines-api start (2.8.6 [gc])
application-pipelines-api: 20:52:21 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:52:21.31751274 +0000 UTC
application-pipelines-api: 20:52:21 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:52:21.318001845 +0000 UTC
application-pipelines-api: 20:52:21 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:52:21.318253647 +0000 UTC
application-pipelines-api: 20:52:21 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pipelines-api"
application-pipelines-api: 20:52:21 DEBUG juju.worker.introspection stats worker now serving
application-pipelines-api: 20:52:21 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-api: 20:52:21 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:52:21.329637557 +0000 UTC
application-pipelines-api: 20:52:21 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:52:21.330800669 +0000 UTC
application-pipelines-api: 20:52:21 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:52:21.341040968 +0000 UTC
application-pytorch-operator: 20:52:21 DEBUG juju.agent read agent config, format "2.0"
application-pytorch-operator: 20:52:21 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-pytorch-operator: 20:52:21 INFO juju.cmd.jujud caas operator application-pytorch-operator start (2.8.6 [gc])
application-pytorch-operator: 20:52:21 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:52:21.509701105 +0000 UTC
application-pytorch-operator: 20:52:21 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:52:21.509897107 +0000 UTC
application-pytorch-operator: 20:52:21 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:21 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-argo-controller: 20:52:21 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:21 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:52:21.511473722 +0000 UTC
application-pytorch-operator: 20:52:21 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-pytorch-operator"
application-pytorch-operator: 20:52:21 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:52:21.511842326 +0000 UTC
application-pytorch-operator: 20:52:21 DEBUG juju.worker.introspection stats worker now serving
application-argo-controller: 20:52:21 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:52:21 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:52:21.538515385 +0000 UTC
application-pytorch-operator: 20:52:21 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:52:21.548682383 +0000 UTC
application-tf-job-operator: 20:52:21 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-persistence: 20:52:22 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-persistence: 20:52:22 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:22 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:22 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-persistence" cannot open api
application-pipelines-api: 20:52:22 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-api: 20:52:22 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:22 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:22 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pytorch-operator: 20:52:22 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-argo-controller: 20:52:22 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:22 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:22 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:22 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-argo-controller" cannot open api
application-pytorch-operator: 20:52:22 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-db: 20:52:22 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:22 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-tf-job-operator: 20:52:22 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:22 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:22 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-tf-job-operator" cannot open api
application-minio: 20:52:23 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-api: 20:52:23 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-api: 20:52:23 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:23 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:23 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-api" cannot open api
application-pytorch-operator: 20:52:23 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pytorch-operator: 20:52:23 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:23 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:23 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pytorch-operator" cannot open api
application-pipelines-db: 20:52:23 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-db: 20:52:23 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:23 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:23 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-db" cannot open api
application-pipelines-scheduledworkflow: 20:52:23 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:52:24 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-minio: 20:52:24 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:24 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:24 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-minio" cannot open api
application-pipelines-scheduledworkflow: 20:52:24 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-scheduledworkflow: 20:52:24 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:24 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:24 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-scheduledworkflow" cannot open api
application-argo-controller: 20:52:25 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:52:26 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:26 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-argo-controller: 20:52:26 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:26 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:26 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-argo-controller" cannot open api
application-pipelines-persistence: 20:52:26 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:52:27 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pytorch-operator: 20:52:27 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:27 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:27 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pytorch-operator" cannot open api
application-pipelines-persistence: 20:52:27 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-persistence: 20:52:27 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:27 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:27 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-persistence" cannot open api
application-pipelines-api: 20:52:28 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-api: 20:52:29 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-api: 20:52:29 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:29 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:29 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-api" cannot open api
application-pipelines-scheduledworkflow: 20:52:29 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:30 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-scheduledworkflow: 20:52:30 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-scheduledworkflow: 20:52:30 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:30 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:30 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-scheduledworkflow" cannot open api
application-pipelines-db: 20:52:30 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:52:30 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:31 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-tf-job-operator: 20:52:31 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:31 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:31 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-tf-job-operator" cannot open api
application-argo-controller: 20:52:31 DEBUG juju.worker.apicaller connecting with old password
application-seldon-core: 20:52:31 INFO juju.cmd running jujud [2.8.6 372 78e7a5a60ef88bd281835a3dfb3522277474257f gc go1.14.10]
application-seldon-core: 20:52:31 DEBUG juju.cmd   args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=seldon-core", "--debug"}
application-pipelines-db: 20:52:31 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-db: 20:52:31 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:31 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:31 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-db" cannot open api
application-minio: 20:52:31 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-minio: 20:52:31 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:31 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:31 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-minio" cannot open api
application-seldon-core: 20:52:32 DEBUG juju.agent read agent config, format "2.0"
application-seldon-core: 20:52:32 INFO juju.worker.upgradesteps upgrade steps for 2.8.6 have already been run.
application-seldon-core: 20:52:32 INFO juju.cmd.jujud caas operator application-seldon-core start (2.8.6 [gc])
application-seldon-core: 20:52:32 DEBUG juju.worker.dependency "clock" manifold worker started at 2020-10-21 20:52:32.056760197 +0000 UTC
application-seldon-core: 20:52:32 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2020-10-21 20:52:32.0571125 +0000 UTC
application-seldon-core: 20:52:32 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-seldon-core"
application-seldon-core: 20:52:32 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2020-10-21 20:52:32.057705706 +0000 UTC
application-seldon-core: 20:52:32 DEBUG juju.worker.dependency "agent" manifold worker started at 2020-10-21 20:52:32.057732206 +0000 UTC
application-seldon-core: 20:52:32 DEBUG juju.worker.introspection stats worker now serving
application-seldon-core: 20:52:32 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2020-10-21 20:52:32.067056296 +0000 UTC
application-seldon-core: 20:52:32 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2020-10-21 20:52:32.068056105 +0000 UTC
application-seldon-core: 20:52:32 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:52:32 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:32 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-argo-controller: 20:52:32 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:32 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:32 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-argo-controller" cannot open api
application-seldon-core: 20:52:33 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-seldon-core: 20:52:33 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-seldon-core" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-seldon-core: 20:52:33 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-seldon-core" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-seldon-core: 20:52:33 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-seldon-core" cannot open api
application-pytorch-operator: 20:52:33 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pytorch-operator: 20:52:33 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:33 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:33 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pytorch-operator" cannot open api
application-pipelines-persistence: 20:52:33 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-api: 20:52:33 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-persistence: 20:52:34 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-persistence: 20:52:34 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:34 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:34 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-persistence" cannot open api
application-pipelines-api: 20:52:34 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-api: 20:52:34 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:34 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:34 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-api" cannot open api
application-seldon-core: 20:52:36 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-scheduledworkflow: 20:52:37 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:37 DEBUG juju.worker.apicaller connecting with old password
application-seldon-core: 20:52:37 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-seldon-core: 20:52:37 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-seldon-core" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-seldon-core: 20:52:37 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-seldon-core" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-seldon-core: 20:52:37 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-seldon-core" cannot open api
application-pytorch-operator: 20:52:38 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-scheduledworkflow: 20:52:38 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-scheduledworkflow: 20:52:38 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:38 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:38 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-scheduledworkflow" cannot open api
application-argo-controller: 20:52:38 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-argo-controller: 20:52:38 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:38 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:38 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-argo-controller" cannot open api
application-pytorch-operator: 20:52:39 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pytorch-operator: 20:52:39 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:39 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:39 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pytorch-operator" cannot open api
application-pipelines-persistence: 20:52:40 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:40 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:52:40 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-persistence: 20:52:41 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-persistence: 20:52:41 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:41 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:41 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-persistence" cannot open api
application-tf-job-operator: 20:52:41 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-tf-job-operator: 20:52:41 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:41 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-tf-job-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-tf-job-operator: 20:52:41 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-tf-job-operator" cannot open api
application-pipelines-api: 20:52:41 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:52:41 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-minio: 20:52:41 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:41 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-minio" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-minio: 20:52:41 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-minio" cannot open api
application-pipelines-db: 20:52:41 DEBUG juju.worker.apicaller connecting with old password
application-seldon-core: 20:52:41 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-api: 20:52:42 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-api: 20:52:42 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:42 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-api" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-api: 20:52:42 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-api" cannot open api
application-pipelines-db: 20:52:42 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-db: 20:52:42 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:42 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-db" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-db: 20:52:42 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-db" cannot open api
application-seldon-core: 20:52:42 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-seldon-core: 20:52:42 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-seldon-core" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-seldon-core: 20:52:42 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-seldon-core" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-seldon-core: 20:52:42 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-seldon-core" cannot open api
application-argo-controller: 20:52:44 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:52:44 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:45 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-argo-controller: 20:52:45 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:45 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-argo-controller" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-argo-controller: 20:52:45 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-argo-controller" cannot open api
application-pipelines-scheduledworkflow: 20:52:45 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:52:45 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pytorch-operator: 20:52:45 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:45 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pytorch-operator" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pytorch-operator: 20:52:45 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pytorch-operator" cannot open api
application-pipelines-scheduledworkflow: 20:52:46 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-scheduledworkflow: 20:52:46 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:46 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-scheduledworkflow" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-scheduledworkflow: 20:52:46 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-scheduledworkflow" cannot open api
application-pipelines-persistence: 20:52:48 DEBUG juju.worker.apicaller connecting with old password
application-seldon-core: 20:52:48 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-persistence: 20:52:49 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-pipelines-persistence: 20:52:49 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:49 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-pipelines-persistence" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-pipelines-persistence: 20:52:49 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-pipelines-persistence" cannot open api
application-seldon-core: 20:52:49 DEBUG juju.worker.apicaller [f0e819] failed to connect
application-seldon-core: 20:52:49 DEBUG juju.worker.dependency "api-caller" manifold worker stopped: [f0e819] "application-seldon-core" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-seldon-core: 20:52:49 ERROR juju.worker.dependency "api-caller" manifold worker returned unexpected error: [f0e819] "application-seldon-core" cannot open api: unable to connect to API: dial tcp 10.152.183.175:17070: connect: connection refused
application-seldon-core: 20:52:49 DEBUG juju.worker.dependency stack trace:
dial tcp 10.152.183.175:17070: connect: connection refused
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:689: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1109: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1084: unable to connect to API
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:980: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:635: 
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:213: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:156: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/connect.go:57: 
/workspace/_build/src/github.com/juju/juju/worker/apicaller/manifold.go:97: [f0e819] "application-seldon-core" cannot open api
application-pipelines-api: 20:52:49 DEBUG juju.worker.apicaller connecting with old password
controller-0: 20:52:50 INFO juju.worker.apicaller [f0e819] "controller-0" successfully connected to "localhost:17070"
application-pipelines-api: 20:52:50 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-api: 20:52:50 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-api: 20:52:50 INFO juju.worker.apicaller [f0e819] "application-pipelines-api" successfully connected to "10.152.183.175:17070"
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:50.819715796 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:50.829179484 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:50.829970792 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:50.8307886 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:50.831418105 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:50.832410915 +0000 UTC
application-pipelines-api: 20:52:50 INFO juju.worker.migrationminion migration phase is now: NONE
application-pipelines-api: 20:52:50 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:50.844366627 +0000 UTC
application-pipelines-api: 20:52:50 INFO juju.worker.logger logger worker started
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:50.844871131 +0000 UTC
application-pipelines-api: 20:52:50 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:50.845880541 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:50.846113043 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:50.846506047 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-api: 20:52:50 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pipelines-api: 20:52:50 INFO juju.worker.caasupgrader unblocking abort check
application-pipelines-api: 20:52:50 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-api: 20:52:50 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-api: 20:52:50 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:50.861122584 +0000 UTC
application-pipelines-api: 20:52:50 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pipelines-api: 20:52:50 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-api: 20:52:50 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-api: 20:52:50 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pipelines-api: 20:52:50 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pipelines-api: 20:52:50 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pipelines-api: 20:52:50 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-pipelines-api: 20:52:50 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-api: 20:52:50 INFO juju.worker.uniter.charm downloading local:kubernetes/pipelines-api-0 from API server
application-pipelines-api: 20:52:50 INFO juju.downloader downloading from local:kubernetes/pipelines-api-0
application-pipelines-api: 20:52:50 INFO juju.downloader download complete ("local:kubernetes/pipelines-api-0")
application-pipelines-api: 20:52:50 INFO juju.downloader download verified ("local:kubernetes/pipelines-api-0")
application-pipelines-api: 20:52:50 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-api-0"
application-pipelines-api: 20:52:50 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-api-0"
application-pipelines-api: 20:52:51 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-api-0"
application-pipelines-api: 20:52:51 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc0007fe180}
application-pipelines-api: 20:52:51 INFO juju.worker.caasoperator operator "pipelines-api" started
application-pipelines-api: 20:52:51 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-api: 20:52:51 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-api: 20:52:51 DEBUG juju.worker.uniter starting uniter for "pipelines-api/0"
application-pipelines-api: 20:52:51 DEBUG juju.worker.caasoperator start "pipelines-api/0"
application-pipelines-api: 20:52:51 INFO juju.worker.caasoperator start "pipelines-api/0"
application-pipelines-api: 20:52:51 DEBUG juju.worker.caasoperator "pipelines-api/0" started
application-pipelines-api: 20:52:51 DEBUG juju.worker.leadership pipelines-api/0 making initial claim for pipelines-api leadership
application-pipelines-api: 20:52:51 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-api-0
application-pipelines-api: 20:52:51 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pipelines-api-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-api-0/jujud
application-pipelines-api: 20:52:51 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pipelines-api-0/jujud
application-pipelines-api: 20:52:51 INFO juju.worker.uniter.relation joining relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api"
application-pipelines-api: 20:52:51 INFO juju.worker.uniter.relation joined relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api"
application-pipelines-api: 20:52:51 DEBUG juju.worker.caasoperator.remotestate got application change
application-argo-controller: 20:52:52 DEBUG juju.worker.apicaller connecting with old password
application-argo-controller: 20:52:52 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-argo-controller: 20:52:52 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-argo-controller: 20:52:52 INFO juju.worker.apicaller [f0e819] "application-argo-controller" successfully connected to "10.152.183.175:17070"
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:52.358386889 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:52.367731277 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:52.368673385 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:52.36917599 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:52.369572894 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:52.377632269 +0000 UTC
application-argo-controller: 20:52:52 INFO juju.worker.caasupgrader abort check blocked until version event received
application-argo-controller: 20:52:52 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-argo-controller: 20:52:52 INFO juju.worker.migrationminion migration phase is now: NONE
application-argo-controller: 20:52:52 INFO juju.worker.caasupgrader unblocking abort check
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:52.388211668 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:52.389554081 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:52.390827492 +0000 UTC
application-argo-controller: 20:52:52 INFO juju.worker.logger logger worker started
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:52.393132914 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-argo-controller: 20:52:52 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:52.395230634 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-argo-controller: 20:52:52 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:52.406242036 +0000 UTC
application-argo-controller: 20:52:52 DEBUG juju.network no lxc bridge addresses to filter for machine
application-argo-controller: 20:52:52 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-argo-controller: 20:52:52 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-argo-controller: 20:52:52 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-argo-controller: 20:52:52 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-argo-controller: 20:52:52 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-argo-controller: 20:52:52 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-argo-controller: 20:52:52 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-argo-controller: 20:52:52 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/argo-controller-0"
application-argo-controller: 20:52:52 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/argo-controller-0"
application-minio: 20:52:52 DEBUG juju.worker.apicaller connecting with old password
application-minio: 20:52:52 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-minio: 20:52:52 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-minio: 20:52:52 INFO juju.worker.apicaller [f0e819] "application-minio" successfully connected to "10.152.183.175:17070"
application-minio: 20:52:52 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:52.463927276 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:52.474307373 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:52.474608575 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-minio: 20:52:52 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:52.475215381 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:52.475603185 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:52.475967288 +0000 UTC
application-minio: 20:52:52 INFO juju.worker.migrationminion migration phase is now: NONE
application-minio: 20:52:52 INFO juju.worker.caasupgrader abort check blocked until version event received
application-minio: 20:52:52 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:52.488386104 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-minio: 20:52:52 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-minio: 20:52:52 INFO juju.worker.caasupgrader unblocking abort check
application-minio: 20:52:52 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:52.489409014 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:52.489701416 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:52.489893218 +0000 UTC
application-minio: 20:52:52 INFO juju.worker.logger logger worker started
application-minio: 20:52:52 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:52.494689963 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-minio: 20:52:52 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-minio: 20:52:52 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-minio: 20:52:52 DEBUG juju.network no lxc bridge addresses to filter for machine
application-minio: 20:52:52 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-minio: 20:52:52 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-minio: 20:52:52 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-minio: 20:52:52 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-minio: 20:52:52 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-minio: 20:52:52 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-minio: 20:52:52 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:52.508132889 +0000 UTC
application-minio: 20:52:52 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-minio: 20:52:52 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/minio-0"
application-minio: 20:52:52 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/minio-0"
application-tf-job-operator: 20:52:52 DEBUG juju.worker.apicaller connecting with old password
application-tf-job-operator: 20:52:52 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-tf-job-operator: 20:52:52 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-tf-job-operator: 20:52:52 INFO juju.worker.apicaller [f0e819] "application-tf-job-operator" successfully connected to "10.152.183.175:17070"
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:52.691796805 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:52.701189393 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:52.702091601 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:52.702169902 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:52.708593462 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:52.709200267 +0000 UTC
application-tf-job-operator: 20:52:52 INFO juju.worker.migrationminion migration phase is now: NONE
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:52.723535101 +0000 UTC
application-tf-job-operator: 20:52:52 INFO juju.worker.caasupgrader abort check blocked until version event received
application-tf-job-operator: 20:52:52 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-tf-job-operator: 20:52:52 INFO juju.worker.caasupgrader unblocking abort check
application-tf-job-operator: 20:52:52 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:52.723841504 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:52.725185317 +0000 UTC
application-tf-job-operator: 20:52:52 INFO juju.worker.logger logger worker started
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:52.725232317 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:52.740810163 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-tf-job-operator: 20:52:52 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-tf-job-operator: 20:52:52 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:52.752887276 +0000 UTC
application-tf-job-operator: 20:52:52 DEBUG juju.network no lxc bridge addresses to filter for machine
application-tf-job-operator: 20:52:52 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-tf-job-operator: 20:52:52 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-tf-job-operator: 20:52:52 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-tf-job-operator: 20:52:52 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-tf-job-operator: 20:52:52 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-tf-job-operator: 20:52:52 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-tf-job-operator: 20:52:52 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-tf-job-operator: 20:52:52 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/tf-job-operator-0"
application-tf-job-operator: 20:52:52 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/tf-job-operator-0"
application-pytorch-operator: 20:52:53 DEBUG juju.worker.apicaller connecting with old password
application-pytorch-operator: 20:52:53 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pytorch-operator: 20:52:53 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pytorch-operator: 20:52:53 INFO juju.worker.apicaller [f0e819] "application-pytorch-operator" successfully connected to "10.152.183.175:17070"
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:53.527526708 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:53.544258365 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:53.544312465 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:53.544555067 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:53.544648968 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:53.54480407 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pytorch-operator: 20:52:53 INFO juju.worker.migrationminion migration phase is now: NONE
application-pytorch-operator: 20:52:53 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:53.563201741 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pytorch-operator: 20:52:53 INFO juju.worker.caasupgrader unblocking abort check
application-pytorch-operator: 20:52:53 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:53.564492953 +0000 UTC
application-pytorch-operator: 20:52:53 INFO juju.worker.logger logger worker started
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:53.565103959 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:53.565119959 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pytorch-operator: 20:52:53 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pytorch-operator: 20:52:53 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:53.576383864 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pytorch-operator: 20:52:53 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pytorch-operator: 20:52:53 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pytorch-operator: 20:52:53 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pytorch-operator: 20:52:53 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pytorch-operator: 20:52:53 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pytorch-operator: 20:52:53 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pytorch-operator: 20:52:53 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-pytorch-operator: 20:52:53 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:53.615340428 +0000 UTC
application-pytorch-operator: 20:52:53 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pytorch-operator-0"
application-pytorch-operator: 20:52:53 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pytorch-operator-0"
application-pipelines-db: 20:52:53 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-db: 20:52:53 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-db: 20:52:53 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-db: 20:52:53 INFO juju.worker.apicaller [f0e819] "application-pipelines-db" successfully connected to "10.152.183.175:17070"
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:53.78271219 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:53.795792812 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:53.796227216 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:53.796340617 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:53.79876274 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:53.79980575 +0000 UTC
application-pipelines-db: 20:52:53 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pipelines-db: 20:52:53 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pipelines-db: 20:52:53 INFO juju.worker.caasupgrader unblocking abort check
application-pipelines-db: 20:52:53 INFO juju.worker.migrationminion migration phase is now: NONE
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:53.840620731 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:53.841434638 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:53.841469338 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:53.843091054 +0000 UTC
application-pipelines-db: 20:52:53 INFO juju.worker.logger logger worker started
application-pipelines-db: 20:52:53 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-db: 20:52:53 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:53.855060065 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-db: 20:52:53 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pipelines-db: 20:52:53 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-db: 20:52:53 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-db: 20:52:53 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pipelines-db: 20:52:53 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pipelines-db: 20:52:53 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pipelines-db: 20:52:53 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-pipelines-db: 20:52:53 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:53.869849003 +0000 UTC
application-pipelines-db: 20:52:53 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-db: 20:52:54 DEBUG juju.worker.uniter.charm preparing to deploy charm "cs:~charmed-osm/mariadb-k8s-34"
application-pipelines-db: 20:52:54 DEBUG juju.worker.uniter.charm deploying charm "cs:~charmed-osm/mariadb-k8s-34"
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/argo-controller-0"
application-argo-controller: 20:52:54 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc0007f4180}
application-argo-controller: 20:52:54 INFO juju.worker.caasoperator operator "argo-controller" started
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter juju-run listener running
application-argo-controller: 20:52:54 DEBUG juju.worker.caasoperator.remotestate got application change
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter starting uniter for "argo-controller/0"
application-argo-controller: 20:52:54 DEBUG juju.worker.caasoperator start "argo-controller/0"
application-argo-controller: 20:52:54 INFO juju.worker.caasoperator start "argo-controller/0"
application-argo-controller: 20:52:54 DEBUG juju.worker.caasoperator "argo-controller/0" started
application-argo-controller: 20:52:54 DEBUG juju.worker.leadership argo-controller/0 making initial claim for argo-controller leadership
application-argo-controller: 20:52:54 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-argo-controller-0
application-argo-controller: 20:52:54 INFO juju.worker.leadership argo-controller/0 promoted to leadership of argo-controller
application-argo-controller: 20:52:54 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-argo-controller-0/jujuc using jujud path /var/lib/juju/tools/unit-argo-controller-0/jujud
application-argo-controller: 20:52:54 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-argo-controller-0/jujud
application-argo-controller: 20:52:54 INFO juju.worker.uniter.relation joining relation "argo-controller:minio minio:minio"
application-argo-controller: 20:52:54 INFO juju.worker.uniter.relation joined relation "argo-controller:minio minio:minio"
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-argo-controller-0/run.socket <nil>}
application-argo-controller: 20:52:54 INFO juju.worker.uniter unit "argo-controller/0" started
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter juju-run listener running
application-argo-controller: 20:52:54 INFO juju.worker.uniter hooks are retried true
application-argo-controller: 20:52:54 DEBUG juju.worker.caasoperator request running status for "unit-argo-controller-0" 98f9d8de-8d65-4cad-af65-85867d87b501
application-minio: 20:52:54 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/minio-0"
application-minio: 20:52:54 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000825c80}
application-minio: 20:52:54 INFO juju.worker.caasoperator operator "minio" started
application-minio: 20:52:54 DEBUG juju.worker.uniter juju-run listener running
application-argo-controller: 20:52:54 DEBUG juju.kubernetes.provider.exec no pod named "98f9d8de-8d65-4cad-af65-85867d87b501" found
application-argo-controller: 20:52:54 DEBUG juju.kubernetes.provider.exec try get pod by UID for "98f9d8de-8d65-4cad-af65-85867d87b501"
application-minio: 20:52:54 DEBUG juju.worker.caasoperator.remotestate got application change
application-minio: 20:52:54 DEBUG juju.worker.uniter starting uniter for "minio/0"
application-minio: 20:52:54 DEBUG juju.worker.caasoperator start "minio/0"
application-minio: 20:52:54 INFO juju.worker.caasoperator start "minio/0"
application-minio: 20:52:54 DEBUG juju.worker.caasoperator "minio/0" started
application-minio: 20:52:54 DEBUG juju.worker.leadership minio/0 making initial claim for minio leadership
application-minio: 20:52:54 INFO juju.worker.leadership minio/0 promoted to leadership of minio
application-minio: 20:52:54 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-minio-0
application-minio: 20:52:54 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-minio-0/jujuc using jujud path /var/lib/juju/tools/unit-minio-0/jujud
application-minio: 20:52:54 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-minio-0/jujud
application-minio: 20:52:54 INFO juju.worker.uniter.relation joining relation "argo-controller:minio minio:minio"
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got config change for argo-controller/0: ok=true, hashes=[57ab9473d7bd45cfa3fa23fa51ded4db320507f1b514dbf7b60f3d9dd8c125a5]
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got relations change for argo-controller/0: ok=true
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got leader settings change for argo-controller/0: ok=true
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got unit change for argo-controller/0
application-minio: 20:52:54 INFO juju.worker.uniter.relation joined relation "argo-controller:minio minio:minio"
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got storage change for argo-controller/0: [] ok=true
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got address change for argo-controller/0: ok=true, hashes=[]
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got update status interval change for argo-controller/0: ok=true
application-minio: 20:52:54 INFO juju.worker.uniter.relation joining relation "pipelines-api:minio minio:minio"
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got trust config change for argo-controller/0: ok=true, hashes=[95123ce51dfbdc5c9461940c26671759329bef2ebc0a43a8a665b91e40475a3d]
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.remotestate got action change for argo-controller/0: [] ok=true
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.operation running operation remote init for argo-controller/0
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.operation preparing operation "remote init" for argo-controller/0
application-minio: 20:52:54 INFO juju.worker.uniter.relation joined relation "pipelines-api:minio minio:minio"
application-argo-controller: 20:52:54 DEBUG juju.worker.uniter.operation executing operation "remote init" for argo-controller/0
application-argo-controller: 20:52:54 DEBUG juju.worker.caasoperator request running status for "unit-argo-controller-0" argo-controller-5fdc64b6b4-t6r66
application-argo-controller: 20:52:54 DEBUG juju.worker.caasoperator remote init for "unit-argo-controller-0" {PodName:argo-controller-5fdc64b6b4-t6r66 Initialising:true InitialisingTime:2020-10-21 20:52:00 +0000 UTC Running:false}
application-argo-controller: 20:52:54 INFO juju.worker.caasoperator started pod init on "argo-controller/0"
application-argo-controller: 20:52:54 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/znoyrqyg.pid; exec sh -c 'mkdir -p /tmp/unit-argo-controller-0827932386'; "]
application-minio: 20:52:54 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-minio-0/run.socket <nil>}
application-minio: 20:52:54 INFO juju.worker.uniter unit "minio/0" started
application-minio: 20:52:54 DEBUG juju.worker.uniter juju-run listener running
application-minio: 20:52:54 INFO juju.worker.uniter hooks are retried true
application-minio: 20:52:54 DEBUG juju.worker.caasoperator request running status for "unit-minio-0" minio-0
application-tf-job-operator: 20:52:54 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/tf-job-operator-0"
application-tf-job-operator: 20:52:54 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000869800}
application-tf-job-operator: 20:52:54 INFO juju.worker.caasoperator operator "tf-job-operator" started
application-tf-job-operator: 20:52:54 DEBUG juju.worker.uniter juju-run listener running
application-tf-job-operator: 20:52:54 DEBUG juju.worker.caasoperator.remotestate got application change
application-tf-job-operator: 20:52:54 DEBUG juju.worker.uniter starting uniter for "tf-job-operator/0"
application-tf-job-operator: 20:52:54 DEBUG juju.worker.caasoperator start "tf-job-operator/0"
application-tf-job-operator: 20:52:54 DEBUG juju.worker.caasoperator trigger running status for caas unit tf-job-operator/0
application-tf-job-operator: 20:52:54 INFO juju.worker.caasoperator start "tf-job-operator/0"
application-tf-job-operator: 20:52:54 DEBUG juju.worker.caasoperator "tf-job-operator/0" started
application-tf-job-operator: 20:52:54 DEBUG juju.worker.leadership tf-job-operator/0 making initial claim for tf-job-operator leadership
application-tf-job-operator: 20:52:54 INFO juju.worker.leadership tf-job-operator/0 promoted to leadership of tf-job-operator
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got storage change for minio/0: [minio-data/0] ok=true
application-tf-job-operator: 20:52:54 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-tf-job-operator-0
application-tf-job-operator: 20:52:54 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-tf-job-operator-0/jujuc using jujud path /var/lib/juju/tools/unit-tf-job-operator-0/jujud
application-tf-job-operator: 20:52:54 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-tf-job-operator-0/jujud
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got action change for minio/0: [] ok=true
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got relations change for minio/0: ok=true
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got leader settings change for minio/0: ok=true
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got unit change for minio/0
application-tf-job-operator: 20:52:54 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-tf-job-operator-0/run.socket <nil>}
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got config change for minio/0: ok=true, hashes=[8b62a1d74d8d9f7018b381d8ba50d6738b7534976254aceb687ab2dc15194ff3]
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got address change for minio/0: ok=true, hashes=[5379216ebc32c12dc98a35b1910a26fc2cb86640750d873867c67287b42a70ee]
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got trust config change for minio/0: ok=true, hashes=[256ee7351c49e1e2a2e489abc320b6dc0380684d3dc93935982f65d3c394c354]
application-minio: 20:52:54 DEBUG juju.worker.uniter.remotestate got update status interval change for minio/0: ok=true
application-tf-job-operator: 20:52:54 INFO juju.worker.uniter unit "tf-job-operator/0" started
application-tf-job-operator: 20:52:54 DEBUG juju.worker.uniter juju-run listener running
application-minio: 20:52:54 DEBUG juju.worker.uniter.operation running operation remote init for minio/0
application-minio: 20:52:54 DEBUG juju.worker.uniter.operation preparing operation "remote init" for minio/0
application-tf-job-operator: 20:52:54 INFO juju.worker.uniter hooks are retried true
application-minio: 20:52:54 DEBUG juju.worker.uniter.operation executing operation "remote init" for minio/0
application-minio: 20:52:54 DEBUG juju.worker.caasoperator request running status for "unit-minio-0" minio-0
application-tf-job-operator: 20:52:54 DEBUG juju.worker.caasoperator request running status for "unit-tf-job-operator-0" 6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97
application-minio: 20:52:54 DEBUG juju.worker.caasoperator remote init for "unit-minio-0" {PodName:minio-0 Initialising:true InitialisingTime:2020-10-21 20:52:36 +0000 UTC Running:false}
application-minio: 20:52:54 INFO juju.worker.caasoperator started pod init on "minio/0"
application-minio: 20:52:54 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/qkduuzyr.pid; exec sh -c 'mkdir -p /tmp/unit-minio-0772539583'; "]
application-tf-job-operator: 20:52:55 DEBUG juju.kubernetes.provider.exec no pod named "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97" found
application-tf-job-operator: 20:52:55 DEBUG juju.kubernetes.provider.exec try get pod by UID for "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97"
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got update status interval change for tf-job-operator/0: ok=true
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got leader settings change for tf-job-operator/0: ok=true
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got address change for tf-job-operator/0: ok=true, hashes=[]
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got relations change for tf-job-operator/0: ok=true
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got config change for tf-job-operator/0: ok=true, hashes=[e8227833d18e064d11351ac4b81a8a5cd47f8d979eee256d8c7ace444d07359e]
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got storage change for tf-job-operator/0: [] ok=true
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got action change for tf-job-operator/0: [] ok=true
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got unit change for tf-job-operator/0
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got running status change for tf-job-operator/0
application-tf-job-operator: 20:52:55 DEBUG juju.worker.caasoperator request running status for "unit-tf-job-operator-0" 6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97
application-tf-job-operator: 20:52:55 DEBUG juju.kubernetes.provider.exec no pod named "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97" found
application-tf-job-operator: 20:52:55 DEBUG juju.kubernetes.provider.exec try get pod by UID for "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97"
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.remotestate got trust config change for tf-job-operator/0: ok=true, hashes=[7e4e2cd72662de0b4ee50f0b60a9f5007971d9e62f95a8c59271b4569b404ba2]
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.operation running operation remote init for tf-job-operator/0
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.operation preparing operation "remote init" for tf-job-operator/0
application-tf-job-operator: 20:52:55 DEBUG juju.worker.uniter.operation executing operation "remote init" for tf-job-operator/0
application-tf-job-operator: 20:52:55 DEBUG juju.worker.caasoperator request running status for "unit-tf-job-operator-0" tf-job-operator-7c48d7ff49-zslkw
application-tf-job-operator: 20:52:55 DEBUG juju.worker.caasoperator remote init for "unit-tf-job-operator-0" {PodName:tf-job-operator-7c48d7ff49-zslkw Initialising:true InitialisingTime:2020-10-21 20:51:55 +0000 UTC Running:false}
application-tf-job-operator: 20:52:55 INFO juju.worker.caasoperator started pod init on "tf-job-operator/0"
application-tf-job-operator: 20:52:55 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/yxysywpa.pid; exec sh -c 'mkdir -p /tmp/unit-tf-job-operator-0945146030'; "]
controller-0: 20:52:55 INFO juju.worker.logforwarder config change - log forwarding not enabled
controller-0: 20:52:55 INFO juju.worker.logger logger worker started
application-seldon-core: 20:52:55 DEBUG juju.worker.apicaller connecting with old password
controller-0: 20:52:55 DEBUG juju.worker.caas waiting for config and credential notifications
controller-0: 20:52:55 DEBUG juju.worker.caas reloading model config
application-argo-controller: 20:52:55 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-argo-controller/charm  } to {/tmp/unit-argo-controller-0827932386 argo-controller-5fdc64b6b4-t6r66 juju-pod-init}
application-argo-controller: 20:52:55 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/uibxbswl.pid; exec sh -c 'test -d /tmp/unit-argo-controller-0827932386'; "]
application-seldon-core: 20:52:55 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-seldon-core: 20:52:55 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
controller-0: 20:52:55 DEBUG juju.worker.caas waiting for config and credential notifications
controller-0: 20:52:55 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for kubeflow (f0e819dd-2fb4-4e25-8bdd-77edd9c85838)
controller-0: 20:52:55 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:52:55 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for kubeflow (f0e819dd-2fb4-4e25-8bdd-77edd9c85838)
controller-0: 20:52:55 DEBUG juju.worker.caas waiting for config and credential notifications
application-seldon-core: 20:52:55 INFO juju.worker.apicaller [f0e819] "application-seldon-core" successfully connected to "10.152.183.175:17070"
application-seldon-core: 20:52:55 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:55.684701915 +0000 UTC
application-seldon-core: 20:52:55 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:55.707436027 +0000 UTC
application-seldon-core: 20:52:55 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:55.707500527 +0000 UTC
application-seldon-core: 20:52:55 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:55.70781993 +0000 UTC
application-seldon-core: 20:52:55 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:55.707871231 +0000 UTC
application-seldon-core: 20:52:55 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
controller-0: 20:52:55 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:52:55 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volumes alive: [volume-0 volume-1], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-persistence
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volume "0" is already provisioned, nothing to do
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volume "1" is already provisioned, nothing to do
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
application-seldon-core: 20:52:55 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:55.961299491 +0000 UTC
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-scheduledworkflow
application-seldon-core: 20:52:55 INFO juju.worker.caasupgrader abort check blocked until version event received
application-seldon-core: 20:52:55 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-seldon-core: 20:52:55 INFO juju.worker.caasupgrader unblocking abort check
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner volume attachments alive: [{unit-pipelines-db-0 volume-1} {unit-minio-0 volume-0}], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:52:55 DEBUG juju.worker.storageprovisioner filesystems alive: [], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner volume-1 is already attached to unit-pipelines-db-0, will reattach
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner volume-0 is already attached to unit-minio-0, will reattach
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystems alive: [filesystem-0], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner operator update for minio
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystems alive: [filesystem-1], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner not queuing volume attachment for non-machine unit-pipelines-db-0
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner not queuing volume attachment for non-machine unit-minio-0
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystem "1" is already provisioned, nothing to do
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.apicaller connecting with old password
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [{unit-pipelines-db-0 filesystem-1}], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystem "0" is already provisioned, nothing to do
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystem-1 is already attached to unit-pipelines-db-0, will reattach
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystem attachment alive: [{unit-minio-0 filesystem-0}], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner filesystem-0 is already attached to unit-minio-0, will reattach
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-db
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner not queuing filesystem attachment for non-machine unit-minio-0
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner volume attachments alive: [], dying: [], dead: []
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:52:56 DEBUG juju.worker.storageprovisioner not queuing filesystem attachment for non-machine unit-pipelines-db-0
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner operator update for pytorch-operator
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-scheduledworkflow: 20:52:56 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner service for argo-controller(workload): &{Id: Addresses:[] Scale:0xc001a85f60 Generation:0xc001a85f68 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-seldon-core: 20:52:56 INFO juju.worker.migrationminion migration phase is now: NONE
application-seldon-core: 20:52:56 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:56.113445506 +0000 UTC
application-seldon-core: 20:52:56 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:56.113633208 +0000 UTC
application-seldon-core: 20:52:56 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:56.113677108 +0000 UTC
application-seldon-core: 20:52:56 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-seldon-core: 20:52:56 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:56.113802609 +0000 UTC
application-seldon-core: 20:52:56 INFO juju.worker.logger logger worker started
controller-0: 20:52:56 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-pipelines-scheduledworkflow: 20:52:56 INFO juju.worker.apicaller [f0e819] "application-pipelines-scheduledworkflow" successfully connected to "10.152.183.175:17070"
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:56.161114749 +0000 UTC
application-seldon-core: 20:52:56 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:56.180767632 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:56.180827532 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:56.180857233 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:56.181820742 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:56.186817188 +0000 UTC
application-seldon-core: 20:52:56 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-seldon-core: 20:52:56 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-seldon-core: 20:52:56 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:56.199339005 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pipelines-scheduledworkflow: 20:52:56 INFO juju.worker.caasupgrader unblocking abort check
application-pipelines-scheduledworkflow: 20:52:56 INFO juju.worker.migrationminion migration phase is now: NONE
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:56.253086904 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:56.253241306 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:56.253253406 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 INFO juju.worker.logger logger worker started
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:56.253375007 +0000 UTC
application-seldon-core: 20:52:56 DEBUG juju.network no lxc bridge addresses to filter for machine
application-seldon-core: 20:52:56 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-seldon-core: 20:52:56 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-seldon-core: 20:52:56 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-seldon-core: 20:52:56 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-seldon-core: 20:52:56 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-seldon-core: 20:52:56 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-seldon-core: 20:52:56 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:56.281034764 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:56.281276067 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-seldon-core: 20:52:56 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:56.303823076 +0000 UTC
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-scheduledworkflow-0"
application-seldon-core: 20:52:56 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/seldon-core-0"
application-seldon-core: 20:52:56 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/seldon-core-0"
application-pipelines-scheduledworkflow: 20:52:56 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-scheduledworkflow-0"
application-pipelines-api: 20:52:56 INFO juju.worker.leadership pipelines-api/0 promoted to leadership of pipelines-api
application-pipelines-api: 20:52:56 INFO juju.worker.uniter.relation joining relation "pipelines-api:minio minio:minio"
application-pipelines-api: 20:52:56 INFO juju.worker.uniter.relation joined relation "pipelines-api:minio minio:minio"
application-pipelines-api: 20:52:56 INFO juju.worker.uniter.relation joining relation "pipelines-api:mysql pipelines-db:mysql"
application-minio: 20:52:56 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-minio/charm  } to {/tmp/unit-minio-0772539583 minio-0 juju-pod-init}
application-pipelines-api: 20:52:56 INFO juju.worker.uniter.relation joined relation "pipelines-api:mysql pipelines-db:mysql"
application-minio: 20:52:56 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/wzqyclau.pid; exec sh -c 'test -d /tmp/unit-minio-0772539583'; "]
application-pipelines-api: 20:52:56 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-api-0/run.socket <nil>}
application-pipelines-api: 20:52:56 INFO juju.worker.uniter unit "pipelines-api/0" started
application-pipelines-api: 20:52:56 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-api: 20:52:56 INFO juju.worker.uniter hooks are retried true
application-pipelines-api: 20:52:56 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-api-0" 060740d9-9aef-4765-81a1-73ba4a468772
application-pipelines-api: 20:52:56 DEBUG juju.kubernetes.provider.exec no pod named "060740d9-9aef-4765-81a1-73ba4a468772" found
application-pipelines-api: 20:52:56 DEBUG juju.kubernetes.provider.exec try get pod by UID for "060740d9-9aef-4765-81a1-73ba4a468772"
application-pytorch-operator: 20:52:56 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pytorch-operator-0"
application-pytorch-operator: 20:52:56 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000639e00}
application-pytorch-operator: 20:52:56 INFO juju.worker.caasoperator operator "pytorch-operator" started
application-pytorch-operator: 20:52:56 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.charm finishing deploy of charm "cs:~charmed-osm/mariadb-k8s-34"
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000829500}
application-pipelines-db: 20:52:57 INFO juju.worker.caasoperator operator "pipelines-db" started
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter juju-run listener running
application-pytorch-operator: 20:52:57 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got leader settings change for pipelines-api/0: ok=true
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got trust config change for pipelines-api/0: ok=true, hashes=[7f9884bc998df3cb2761f63f838f8c09c8a7ce0ea62a095fae071ad6e1a23413]
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got update status interval change for pipelines-api/0: ok=true
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got storage change for pipelines-api/0: [] ok=true
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter starting uniter for "pytorch-operator/0"
application-pytorch-operator: 20:52:57 DEBUG juju.worker.caasoperator start "pytorch-operator/0"
application-pytorch-operator: 20:52:57 DEBUG juju.worker.caasoperator trigger running status for caas unit pytorch-operator/0
application-pytorch-operator: 20:52:57 INFO juju.worker.caasoperator start "pytorch-operator/0"
application-pytorch-operator: 20:52:57 DEBUG juju.worker.caasoperator "pytorch-operator/0" started
application-pytorch-operator: 20:52:57 DEBUG juju.worker.leadership pytorch-operator/0 making initial claim for pytorch-operator leadership
application-pytorch-operator: 20:52:57 INFO juju.worker.leadership pytorch-operator/0 promoted to leadership of pytorch-operator
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got relations change for pipelines-api/0: ok=true
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator.remotestate got application change
application-pytorch-operator: 20:52:57 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pytorch-operator-0
application-pytorch-operator: 20:52:57 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pytorch-operator-0/jujuc using jujud path /var/lib/juju/tools/unit-pytorch-operator-0/jujud
application-pytorch-operator: 20:52:57 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pytorch-operator-0/jujud
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter starting uniter for "pipelines-db/0"
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator start "pipelines-db/0"
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-db/0
application-pipelines-db: 20:52:57 INFO juju.worker.caasoperator start "pipelines-db/0"
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator "pipelines-db/0" started
application-pipelines-db: 20:52:57 DEBUG juju.worker.leadership pipelines-db/0 making initial claim for pipelines-db leadership
application-pipelines-db: 20:52:57 INFO juju.worker.leadership pipelines-db/0 promoted to leadership of pipelines-db
application-pipelines-db: 20:52:57 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-db-0
application-pipelines-db: 20:52:57 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pipelines-db-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-db-0/jujud
application-pipelines-db: 20:52:57 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pipelines-db-0/jujud
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got action change for pipelines-api/0: [] ok=true
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.apicaller connecting with old password
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got config change for pipelines-api/0: ok=true, hashes=[cdeba69efca8b193773f8bead59c96ed23411c32f7e971851e3922e54e563574]
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got address change for pipelines-api/0: ok=true, hashes=[0bd1590d712e8925317163b30f451871e30c2035bc3feda057e780ed4de2b601]
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.remotestate got unit change for pipelines-api/0
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.operation running operation remote init for pipelines-api/0
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pipelines-api/0
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pytorch-operator-0/run.socket <nil>}
application-pipelines-db: 20:52:57 INFO juju.worker.uniter.relation joining relation "pipelines-api:mysql pipelines-db:mysql"
application-pytorch-operator: 20:52:57 INFO juju.worker.uniter unit "pytorch-operator/0" started
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter juju-run listener running
application-pytorch-operator: 20:52:57 INFO juju.worker.uniter hooks are retried true
application-pipelines-api: 20:52:57 DEBUG juju.worker.uniter.operation executing operation "remote init" for pipelines-api/0
application-pipelines-api: 20:52:57 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-api-0" pipelines-api-58c5b9c64d-lrjjh
application-pytorch-operator: 20:52:57 DEBUG juju.worker.caasoperator request running status for "unit-pytorch-operator-0" b95510d3-6d1e-4d72-9615-e6403906e412
application-pipelines-persistence: 20:52:57 DEBUG juju.api successfully dialed "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-persistence: 20:52:57 INFO juju.api connection established to "wss://10.152.183.175:17070/model/f0e819dd-2fb4-4e25-8bdd-77edd9c85838/api"
application-pipelines-api: 20:52:57 DEBUG juju.worker.caasoperator remote init for "unit-pipelines-api-0" {PodName:pipelines-api-58c5b9c64d-lrjjh Initialising:true InitialisingTime:2020-10-21 20:52:18 +0000 UTC Running:false}
application-pipelines-api: 20:52:57 INFO juju.worker.caasoperator started pod init on "pipelines-api/0"
application-pytorch-operator: 20:52:57 DEBUG juju.kubernetes.provider.exec no pod named "b95510d3-6d1e-4d72-9615-e6403906e412" found
application-pytorch-operator: 20:52:57 DEBUG juju.kubernetes.provider.exec try get pod by UID for "b95510d3-6d1e-4d72-9615-e6403906e412"
application-pipelines-db: 20:52:57 INFO juju.worker.uniter.relation joined relation "pipelines-api:mysql pipelines-db:mysql"
application-pipelines-persistence: 20:52:57 INFO juju.worker.apicaller [f0e819] "application-pipelines-persistence" successfully connected to "10.152.183.175:17070"
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2020-10-21 20:52:57.424259791 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2020-10-21 20:52:57.434320485 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2020-10-21 20:52:57.434818589 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2020-10-21 20:52:57.43484589 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2020-10-21 20:52:57.435166292 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2020-10-21 20:52:57.445835792 +0000 UTC
application-pipelines-persistence: 20:52:57 INFO juju.worker.migrationminion migration phase is now: NONE
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2020-10-21 20:52:57.455295379 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.logger initial log config: "<root>=DEBUG"
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2020-10-21 20:52:57.46182374 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2020-10-21 20:52:57.462577747 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2020-10-21 20:52:57.462602147 +0000 UTC
application-pipelines-persistence: 20:52:57 INFO juju.worker.logger logger worker started
application-pipelines-persistence: 20:52:57 INFO juju.worker.caasupgrader abort check blocked until version event received
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.caasupgrader current agent binary version: 2.8.6
application-pipelines-persistence: 20:52:57 INFO juju.worker.caasupgrader unblocking abort check
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2020-10-21 20:52:57.47585157 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.logger reconfiguring logging from "<root>=DEBUG" to "<root>=DEBUG;unit=DEBUG"
application-pipelines-api: 20:52:57 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/eusyghps.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-api-0374730597'; "]
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.proxyupdater new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-persistence: 20:52:57 DEBUG juju.network no lxc bridge addresses to filter for machine
application-pipelines-persistence: 20:52:57 DEBUG juju.network cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-persistence: 20:52:57 DEBUG juju.network cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
application-pipelines-persistence: 20:52:57 DEBUG juju.network including address local-cloud:10.152.183.175 for machine
application-pipelines-persistence: 20:52:57 DEBUG juju.network addresses after filtering: [local-cloud:10.152.183.175]
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.apiaddressupdater updating API hostPorts to [[10.152.183.175:17070]]
application-pipelines-persistence: 20:52:57 DEBUG juju.agent API server address details [["10.152.183.175:17070"]] written to agent config as ["10.152.183.175:17070"]
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.dependency "operator" manifold worker started at 2020-10-21 20:52:57.50385113 +0000 UTC
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.proxyupdater applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.175,127.0.0.1,::1,localhost", AutoNoProxy:""}
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-db-0/run.socket <nil>}
application-pipelines-db: 20:52:57 INFO juju.worker.uniter unit "pipelines-db/0" started
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-db: 20:52:57 INFO juju.worker.uniter hooks are retried true
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-db-0" pipelines-db-0
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.uniter.charm preparing to deploy charm "local:kubernetes/pipelines-persistence-0"
application-pipelines-persistence: 20:52:57 DEBUG juju.worker.uniter.charm deploying charm "local:kubernetes/pipelines-persistence-0"
application-tf-job-operator: 20:52:57 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-tf-job-operator/charm  } to {/tmp/unit-tf-job-operator-0945146030 tf-job-operator-7c48d7ff49-zslkw juju-pod-init}
application-argo-controller: 20:52:57 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/aeilfxbj.pid; exec sh -c 'tar -xmf - -C /tmp/unit-argo-controller-0827932386'; "]
application-tf-job-operator: 20:52:57 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/sfnxdglg.pid; exec sh -c 'test -d /tmp/unit-tf-job-operator-0945146030'; "]
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got trust config change for pytorch-operator/0: ok=true, hashes=[f74eed7b2b9c5e960456b13a0b731976c48ead2ae64aa6d04614b540e50f943a]
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got relations change for pytorch-operator/0: ok=true
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got config change for pytorch-operator/0: ok=true, hashes=[b02a79b1c23294a0b182205fe125ab183a735dd863a1c0a6ed74d29339626749]
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got update status interval change for pytorch-operator/0: ok=true
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got unit change for pytorch-operator/0
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got storage change for pytorch-operator/0: [] ok=true
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got address change for pytorch-operator/0: ok=true, hashes=[]
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got running status change for pytorch-operator/0
application-pytorch-operator: 20:52:57 DEBUG juju.worker.caasoperator request running status for "unit-pytorch-operator-0" b95510d3-6d1e-4d72-9615-e6403906e412
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got update status interval change for pipelines-db/0: ok=true
application-pytorch-operator: 20:52:57 DEBUG juju.kubernetes.provider.exec no pod named "b95510d3-6d1e-4d72-9615-e6403906e412" found
application-pytorch-operator: 20:52:57 DEBUG juju.kubernetes.provider.exec try get pod by UID for "b95510d3-6d1e-4d72-9615-e6403906e412"
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got leader settings change for pipelines-db/0: ok=true
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got relations change for pipelines-db/0: ok=true
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got address change for pipelines-db/0: ok=true, hashes=[1921fd5cd1ed12e39fddf184e724c8df1836a62b259b06ba827107f3f27402fe]
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got config change for pipelines-db/0: ok=true, hashes=[7895a6fda895e3e388b0e3f0a3b9cce024035b1022161326cf4148bd140cb81b]
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got unit change for pipelines-db/0
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got storage change for pipelines-db/0: [database/1] ok=true
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got action change for pipelines-db/0: [] ok=true
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-db/0
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-db-0" pipelines-db-0
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.remotestate got trust config change for pipelines-db/0: ok=true, hashes=[ed7ecb5d76214434b26162caae4471d6cfb6c5d9428b7decf3570080e60f57d9]
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.operation running operation remote init for pipelines-db/0
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pipelines-db/0
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got action change for pytorch-operator/0: [] ok=true
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.remotestate got leader settings change for pytorch-operator/0: ok=true
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.operation running operation remote init for pytorch-operator/0
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pytorch-operator/0
application-pytorch-operator: 20:52:57 DEBUG juju.worker.uniter.operation executing operation "remote init" for pytorch-operator/0
application-pytorch-operator: 20:52:57 DEBUG juju.worker.caasoperator request running status for "unit-pytorch-operator-0" pytorch-operator-6974c76984-m47qt
application-pipelines-db: 20:52:57 DEBUG juju.worker.uniter.operation executing operation "remote init" for pipelines-db/0
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-db-0" pipelines-db-0
application-pipelines-db: 20:52:57 DEBUG juju.worker.caasoperator remote init for "unit-pipelines-db-0" {PodName:pipelines-db-0 Initialising:true InitialisingTime:2020-10-21 20:52:18 +0000 UTC Running:false}
application-pipelines-db: 20:52:57 INFO juju.worker.caasoperator started pod init on "pipelines-db/0"
application-pytorch-operator: 20:52:57 DEBUG juju.worker.caasoperator remote init for "unit-pytorch-operator-0" {PodName:pytorch-operator-6974c76984-m47qt Initialising:true InitialisingTime:2020-10-21 20:52:00 +0000 UTC Running:false}
application-pytorch-operator: 20:52:57 INFO juju.worker.caasoperator started pod init on "pytorch-operator/0"
application-pipelines-db: 20:52:57 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/juzncqor.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-db-0452256353'; "]
application-pytorch-operator: 20:52:57 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/fjjjwkxe.pid; exec sh -c 'mkdir -p /tmp/unit-pytorch-operator-0900121166'; "]
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-scheduledworkflow-0"
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc000842780}
application-pipelines-scheduledworkflow: 20:52:58 INFO juju.worker.caasoperator operator "pipelines-scheduledworkflow" started
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.uniter juju-run listener running
application-seldon-core: 20:52:58 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/seldon-core-0"
application-seldon-core: 20:52:58 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc00091c480}
application-seldon-core: 20:52:58 INFO juju.worker.caasoperator operator "seldon-core" started
application-seldon-core: 20:52:58 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.uniter starting uniter for "pipelines-scheduledworkflow/0"
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.caasoperator start "pipelines-scheduledworkflow/0"
application-pipelines-scheduledworkflow: 20:52:58 INFO juju.worker.caasoperator start "pipelines-scheduledworkflow/0"
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.caasoperator "pipelines-scheduledworkflow/0" started
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.leadership pipelines-scheduledworkflow/0 making initial claim for pipelines-scheduledworkflow leadership
application-pipelines-scheduledworkflow: 20:52:58 INFO juju.worker.leadership pipelines-scheduledworkflow/0 promoted to leadership of pipelines-scheduledworkflow
application-pipelines-scheduledworkflow: 20:52:58 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-scheduledworkflow-0
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pipelines-scheduledworkflow-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-scheduledworkflow-0/jujud
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pipelines-scheduledworkflow-0/jujud
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/run.socket <nil>}
application-pipelines-scheduledworkflow: 20:52:58 INFO juju.worker.uniter unit "pipelines-scheduledworkflow/0" started
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.uniter juju-run listener running
application-minio: 20:52:58 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/dadualaz.pid; exec sh -c 'tar -xmf - -C /tmp/unit-minio-0772539583'; "]
application-pipelines-scheduledworkflow: 20:52:58 INFO juju.worker.uniter hooks are retried true
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-scheduledworkflow-0" 8c345d1f-b75a-4dff-a4bb-67637ec9f804
application-seldon-core: 20:52:58 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.kubernetes.provider.exec no pod named "8c345d1f-b75a-4dff-a4bb-67637ec9f804" found
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.kubernetes.provider.exec try get pod by UID for "8c345d1f-b75a-4dff-a4bb-67637ec9f804"
application-seldon-core: 20:52:58 DEBUG juju.worker.uniter starting uniter for "seldon-core/0"
application-seldon-core: 20:52:58 DEBUG juju.worker.caasoperator start "seldon-core/0"
application-seldon-core: 20:52:58 INFO juju.worker.caasoperator start "seldon-core/0"
application-seldon-core: 20:52:58 DEBUG juju.worker.caasoperator "seldon-core/0" started
application-seldon-core: 20:52:58 DEBUG juju.worker.leadership seldon-core/0 making initial claim for seldon-core leadership
application-seldon-core: 20:52:58 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-seldon-core-0
application-seldon-core: 20:52:58 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-seldon-core-0/jujuc using jujud path /var/lib/juju/tools/unit-seldon-core-0/jujud
application-seldon-core: 20:52:58 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-seldon-core-0/jujud
application-pipelines-scheduledworkflow: 20:52:58 DEBUG juju.worker.caasoperator.remotestate got application change
application-seldon-core: 20:52:58 DEBUG juju.worker.caasoperator.remotestate got application change
application-seldon-core: 20:52:58 INFO juju.worker.leadership seldon-core/0 promoted to leadership of seldon-core
application-seldon-core: 20:52:58 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-seldon-core-0/run.socket <nil>}
application-seldon-core: 20:52:58 INFO juju.worker.uniter unit "seldon-core/0" started
application-seldon-core: 20:52:58 DEBUG juju.worker.uniter juju-run listener running
application-seldon-core: 20:52:58 INFO juju.worker.uniter hooks are retried true
application-seldon-core: 20:52:59 DEBUG juju.worker.caasoperator request running status for "unit-seldon-core-0" 99fe7a56-602c-405a-8b08-2ca156ab3f06
application-seldon-core: 20:52:59 DEBUG juju.kubernetes.provider.exec no pod named "99fe7a56-602c-405a-8b08-2ca156ab3f06" found
application-seldon-core: 20:52:59 DEBUG juju.kubernetes.provider.exec try get pod by UID for "99fe7a56-602c-405a-8b08-2ca156ab3f06"
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got action change for pipelines-scheduledworkflow/0: [] ok=true
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got update status interval change for pipelines-scheduledworkflow/0: ok=true
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got relations change for pipelines-scheduledworkflow/0: ok=true
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got leader settings change for pipelines-scheduledworkflow/0: ok=true
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got storage change for pipelines-scheduledworkflow/0: [] ok=true
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got unit change for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got trust config change for pipelines-scheduledworkflow/0: ok=true, hashes=[d5ebfb6b4ecb717d6de05891462437ece7c1b77c612433beeb5a25415e29eacb]
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got address change for pipelines-scheduledworkflow/0: ok=true, hashes=[]
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.remotestate got config change for pipelines-scheduledworkflow/0: ok=true, hashes=[fa68127543db9337d9b4cad89b15248a8ea6ba33f8d78e404e35bc47069c2dfa]
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.operation running operation remote init for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pipelines-scheduledworkflow/0
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got action change for seldon-core/0: [] ok=true
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got trust config change for seldon-core/0: ok=true, hashes=[07de29e38c2f189a77dfc56df84fb9f04f4dd6d592fd31660e47cf9029f0a781]
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got update status interval change for seldon-core/0: ok=true
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got config change for seldon-core/0: ok=true, hashes=[adf7847ed70e89eebfefdbcccc3a92ba05fa5f6f1becdab3ce09b854b36788aa]
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got relations change for seldon-core/0: ok=true
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got unit change for seldon-core/0
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got storage change for seldon-core/0: [] ok=true
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.uniter.operation executing operation "remote init" for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-scheduledworkflow-0" pipelines-scheduledworkflow-744fdf4694-zq7xb
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got leader settings change for seldon-core/0: ok=true
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.remotestate got address change for seldon-core/0: ok=true, hashes=[0a87cebe0fc04fc08b8ca2e318d2d5f3e69df283b7baaac895278a95fd759b3b]
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.operation running operation remote init for seldon-core/0
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.operation preparing operation "remote init" for seldon-core/0
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.worker.caasoperator remote init for "unit-pipelines-scheduledworkflow-0" {PodName:pipelines-scheduledworkflow-744fdf4694-zq7xb Initialising:true InitialisingTime:2020-10-21 20:52:18 +0000 UTC Running:false}
application-pipelines-scheduledworkflow: 20:52:59 INFO juju.worker.caasoperator started pod init on "pipelines-scheduledworkflow/0"
application-seldon-core: 20:52:59 DEBUG juju.worker.uniter.operation executing operation "remote init" for seldon-core/0
application-seldon-core: 20:52:59 DEBUG juju.worker.caasoperator request running status for "unit-seldon-core-0" seldon-core-6758ff7f64-mmc22
application-pipelines-scheduledworkflow: 20:52:59 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/psaejnsi.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-scheduledworkflow-0518329923'; "]
application-seldon-core: 20:52:59 DEBUG juju.worker.caasoperator remote init for "unit-seldon-core-0" {PodName:seldon-core-6758ff7f64-mmc22 Initialising:true InitialisingTime:2020-10-21 20:52:36 +0000 UTC Running:false}
application-seldon-core: 20:52:59 INFO juju.worker.caasoperator started pod init on "seldon-core/0"
application-pipelines-api: 20:52:59 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pipelines-api/charm  } to {/tmp/unit-pipelines-api-0374730597 pipelines-api-58c5b9c64d-lrjjh juju-pod-init}
application-seldon-core: 20:52:59 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/oohbwyro.pid; exec sh -c 'mkdir -p /tmp/unit-seldon-core-0673265913'; "]
application-pipelines-api: 20:52:59 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/pqgzassh.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0374730597'; "]
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.uniter.charm finishing deploy of charm "local:kubernetes/pipelines-persistence-0"
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.caasoperator starting caas operator juju-run listener on &{tcp :30666 0xc0007ab680}
application-pipelines-persistence: 20:52:59 INFO juju.worker.caasoperator operator "pipelines-persistence" started
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.uniter starting uniter for "pipelines-persistence/0"
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.caasoperator start "pipelines-persistence/0"
application-pipelines-persistence: 20:52:59 INFO juju.worker.caasoperator start "pipelines-persistence/0"
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.caasoperator "pipelines-persistence/0" started
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.leadership pipelines-persistence/0 making initial claim for pipelines-persistence leadership
application-pipelines-persistence: 20:52:59 INFO juju.worker.leadership pipelines-persistence/0 promoted to leadership of pipelines-persistence
application-pipelines-persistence: 20:52:59 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-persistence-0
application-pipelines-persistence: 20:52:59 DEBUG juju.agent.tools jujuc not found at /var/lib/juju/tools/unit-pipelines-persistence-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-persistence-0/jujud
application-pipelines-persistence: 20:52:59 DEBUG juju.agent.tools target tools path /var/lib/juju/tools/unit-pipelines-persistence-0/jujud
application-pipelines-persistence: 20:52:59 INFO juju.worker.uniter.relation joining relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api"
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.caasoperator.remotestate got application change
application-pipelines-persistence: 20:52:59 INFO juju.worker.uniter.relation joined relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api"
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.uniter starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-persistence-0/run.socket <nil>}
application-pipelines-persistence: 20:52:59 INFO juju.worker.uniter unit "pipelines-persistence/0" started
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.uniter juju-run listener running
application-pipelines-persistence: 20:52:59 INFO juju.worker.uniter hooks are retried true
application-pipelines-persistence: 20:52:59 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-persistence-0" d67ee23b-6f10-4e86-b8a8-1b878f678466
application-pipelines-persistence: 20:52:59 DEBUG juju.kubernetes.provider.exec no pod named "d67ee23b-6f10-4e86-b8a8-1b878f678466" found
application-pipelines-persistence: 20:52:59 DEBUG juju.kubernetes.provider.exec try get pod by UID for "d67ee23b-6f10-4e86-b8a8-1b878f678466"
controller-0: 20:52:59 DEBUG juju.worker.caasunitprovisioner service for pipelines-api(workload): &{Id:41dd3516-000c-4bd2-8989-7194ac9df0e9 Addresses:[local-cloud:10.152.183.229] Scale:0xc001b06bf0 Generation:0xc001b06bf8 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got trust config change for pipelines-persistence/0: ok=true, hashes=[af533a75f93e76b4f389d29e2474d4285fc44bd818475ce07660626b5226319b]
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got unit change for pipelines-persistence/0
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got storage change for pipelines-persistence/0: [] ok=true
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got relations change for pipelines-persistence/0: ok=true
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got action change for pipelines-persistence/0: [] ok=true
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got leader settings change for pipelines-persistence/0: ok=true
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got address change for pipelines-persistence/0: ok=true, hashes=[]
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got config change for pipelines-persistence/0: ok=true, hashes=[1eff9126f7d1b06a64c47ad558690f3b2344a64304e49235317bc50378f4fe21]
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.remotestate got update status interval change for pipelines-persistence/0: ok=true
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.operation running operation remote init for pipelines-persistence/0
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.operation preparing operation "remote init" for pipelines-persistence/0
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.uniter.operation executing operation "remote init" for pipelines-persistence/0
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-persistence-0" pipelines-persistence-7986b9f7f5-8bdlx
application-pipelines-persistence: 20:53:00 DEBUG juju.worker.caasoperator remote init for "unit-pipelines-persistence-0" {PodName:pipelines-persistence-7986b9f7f5-8bdlx Initialising:true InitialisingTime:2020-10-21 20:52:10 +0000 UTC Running:false}
application-pipelines-persistence: 20:53:00 INFO juju.worker.caasoperator started pod init on "pipelines-persistence/0"
application-pipelines-persistence: 20:53:00 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ekahmiho.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-persistence-0759730194'; "]
application-tf-job-operator: 20:53:00 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/yehoxrrb.pid; exec sh -c 'tar -xmf - -C /tmp/unit-tf-job-operator-0945146030'; "]
application-pipelines-db: 20:53:00 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pipelines-db/charm  } to {/tmp/unit-pipelines-db-0452256353 pipelines-db-0 juju-pod-init}
application-pipelines-db: 20:53:00 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/xkzkoymv.pid; exec sh -c 'test -d /tmp/unit-pipelines-db-0452256353'; "]
application-pytorch-operator: 20:53:00 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pytorch-operator/charm  } to {/tmp/unit-pytorch-operator-0900121166 pytorch-operator-6974c76984-m47qt juju-pod-init}
application-pytorch-operator: 20:53:00 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/kbfxifzj.pid; exec sh -c 'test -d /tmp/unit-pytorch-operator-0900121166'; "]
controller-0: 20:53:00 DEBUG juju.worker.caasunitprovisioner operator update for argo-controller
application-argo-controller: 20:53:00 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-argo-controller-0827932386/ca.crt  } to {/tmp/unit-argo-controller-0827932386/ca.crt argo-controller-5fdc64b6b4-t6r66 juju-pod-init}
application-seldon-core: 20:53:01 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-seldon-core/charm  } to {/tmp/unit-seldon-core-0673265913 seldon-core-6758ff7f64-mmc22 juju-pod-init}
application-pipelines-scheduledworkflow: 20:53:01 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pipelines-scheduledworkflow/charm  } to {/tmp/unit-pipelines-scheduledworkflow-0518329923 pipelines-scheduledworkflow-744fdf4694-zq7xb juju-pod-init}
application-minio: 20:53:01 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-minio-0772539583/ca.crt  } to {/tmp/unit-minio-0772539583/ca.crt minio-0 juju-pod-init}
application-seldon-core: 20:53:01 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/adilmjph.pid; exec sh -c 'test -d /tmp/unit-seldon-core-0673265913'; "]
application-pipelines-scheduledworkflow: 20:53:01 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/dsguvmnz.pid; exec sh -c 'test -d /tmp/unit-pipelines-scheduledworkflow-0518329923'; "]
application-argo-controller: 20:53:01 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/pyvqprae.pid; exec sh -c 'test -d /tmp/unit-argo-controller-0827932386/ca.crt'; "]
application-minio: 20:53:01 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ewtrqksr.pid; exec sh -c 'test -d /tmp/unit-minio-0772539583/ca.crt'; "]
controller-0: 20:53:01 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pipelines-persistence: 20:53:01 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/application-pipelines-persistence/charm  } to {/tmp/unit-pipelines-persistence-0759730194 pipelines-persistence-7986b9f7f5-8bdlx juju-pod-init}
application-pipelines-persistence: 20:53:01 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/xqrwhszi.pid; exec sh -c 'test -d /tmp/unit-pipelines-persistence-0759730194'; "]
application-pipelines-db: 20:53:01 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/wqttnhhk.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-db-0452256353'; "]
application-pytorch-operator: 20:53:01 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ftqxozlc.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pytorch-operator-0900121166'; "]
application-pipelines-api: 20:53:01 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/wrsohryo.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0374730597'; "]
application-tf-job-operator: 20:53:02 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-tf-job-operator-0945146030/ca.crt  } to {/tmp/unit-tf-job-operator-0945146030/ca.crt tf-job-operator-7c48d7ff49-zslkw juju-pod-init}
application-tf-job-operator: 20:53:02 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/arkhglzo.pid; exec sh -c 'test -d /tmp/unit-tf-job-operator-0945146030/ca.crt'; "]
application-argo-controller: 20:53:02 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/nrthwvmi.pid; exec sh -c 'tar -xmf - -C /tmp/unit-argo-controller-0827932386'; "]
application-seldon-core: 20:53:02 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/grvpyken.pid; exec sh -c 'tar -xmf - -C /tmp/unit-seldon-core-0673265913'; "]
application-pipelines-scheduledworkflow: 20:53:03 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/bbrzrsjr.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-scheduledworkflow-0518329923'; "]
application-minio: 20:53:03 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/joqreipj.pid; exec sh -c 'tar -xmf - -C /tmp/unit-minio-0772539583'; "]
application-pipelines-persistence: 20:53:04 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/twvhntoe.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-persistence-0759730194'; "]
application-pipelines-db: 20:53:04 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pipelines-db-0452256353/ca.crt  } to {/tmp/unit-pipelines-db-0452256353/ca.crt pipelines-db-0 juju-pod-init}
application-pipelines-db: 20:53:04 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/yiuqbisp.pid; exec sh -c 'test -d /tmp/unit-pipelines-db-0452256353/ca.crt'; "]
application-pytorch-operator: 20:53:04 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pytorch-operator-0900121166/ca.crt  } to {/tmp/unit-pytorch-operator-0900121166/ca.crt pytorch-operator-6974c76984-m47qt juju-pod-init}
application-pytorch-operator: 20:53:04 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/cyoqjkfm.pid; exec sh -c 'test -d /tmp/unit-pytorch-operator-0900121166/ca.crt'; "]
application-tf-job-operator: 20:53:04 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ihhogocn.pid; exec sh -c 'tar -xmf - -C /tmp/unit-tf-job-operator-0945146030'; "]
application-pipelines-api: 20:53:05 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pipelines-api-0374730597/ca.crt  } to {/tmp/unit-pipelines-api-0374730597/ca.crt pipelines-api-58c5b9c64d-lrjjh juju-pod-init}
application-pipelines-api: 20:53:05 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/jwuakenr.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0374730597/ca.crt'; "]
application-argo-controller: 20:53:05 DEBUG juju.worker.caasoperator operator service address: 10.152.183.147
application-argo-controller: 20:53:05 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-argo-controller-0/operator-client-cache.yaml  } to {/tmp/unit-argo-controller-0827932386/operator-client-cache.yaml argo-controller-5fdc64b6b4-t6r66 juju-pod-init}
application-argo-controller: 20:53:05 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ouhwldcv.pid; exec sh -c 'test -d /tmp/unit-argo-controller-0827932386/operator-client-cache.yaml'; "]
application-seldon-core: 20:53:06 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-seldon-core-0673265913/ca.crt  } to {/tmp/unit-seldon-core-0673265913/ca.crt seldon-core-6758ff7f64-mmc22 juju-pod-init}
application-seldon-core: 20:53:06 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/gzlvcbsm.pid; exec sh -c 'test -d /tmp/unit-seldon-core-0673265913/ca.crt'; "]
application-pipelines-scheduledworkflow: 20:53:06 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pipelines-scheduledworkflow-0518329923/ca.crt  } to {/tmp/unit-pipelines-scheduledworkflow-0518329923/ca.crt pipelines-scheduledworkflow-744fdf4694-zq7xb juju-pod-init}
application-pipelines-scheduledworkflow: 20:53:06 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/srsjtfez.pid; exec sh -c 'test -d /tmp/unit-pipelines-scheduledworkflow-0518329923/ca.crt'; "]
application-minio: 20:53:06 DEBUG juju.worker.caasoperator operator service address: 10.152.183.110
application-minio: 20:53:06 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-minio-0/operator-client-cache.yaml  } to {/tmp/unit-minio-0772539583/operator-client-cache.yaml minio-0 juju-pod-init}
application-minio: 20:53:06 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/unjqsnyc.pid; exec sh -c 'test -d /tmp/unit-minio-0772539583/operator-client-cache.yaml'; "]
application-pipelines-db: 20:53:06 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/cdxwftgu.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-db-0452256353'; "]
application-pytorch-operator: 20:53:06 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/orizvxhs.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pytorch-operator-0900121166'; "]
application-pipelines-persistence: 20:53:06 DEBUG juju.kubernetes.provider.exec copying from {/tmp/unit-pipelines-persistence-0759730194/ca.crt  } to {/tmp/unit-pipelines-persistence-0759730194/ca.crt pipelines-persistence-7986b9f7f5-8bdlx juju-pod-init}
application-pipelines-persistence: 20:53:06 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/jnahmqfh.pid; exec sh -c 'test -d /tmp/unit-pipelines-persistence-0759730194/ca.crt'; "]
application-tf-job-operator: 20:53:06 DEBUG juju.worker.caasoperator operator service address: 10.152.183.150
application-tf-job-operator: 20:53:06 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-tf-job-operator-0/operator-client-cache.yaml  } to {/tmp/unit-tf-job-operator-0945146030/operator-client-cache.yaml tf-job-operator-7c48d7ff49-zslkw juju-pod-init}
application-tf-job-operator: 20:53:06 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/zmatruej.pid; exec sh -c 'test -d /tmp/unit-tf-job-operator-0945146030/operator-client-cache.yaml'; "]
application-pipelines-api: 20:53:06 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/qsmpvswq.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0374730597'; "]
application-seldon-core: 20:53:07 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/blarwtfc.pid; exec sh -c 'tar -xmf - -C /tmp/unit-seldon-core-0673265913'; "]
application-argo-controller: 20:53:07 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/safsdbdg.pid; exec sh -c 'tar -xmf - -C /tmp/unit-argo-controller-0827932386'; "]
application-minio: 20:53:07 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ahouymmy.pid; exec sh -c 'tar -xmf - -C /tmp/unit-minio-0772539583'; "]
application-pipelines-scheduledworkflow: 20:53:07 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/yqhqqruc.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-scheduledworkflow-0518329923'; "]
application-pipelines-persistence: 20:53:07 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/nydgmewd.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-persistence-0759730194'; "]
application-tf-job-operator: 20:53:07 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/qimuzuwn.pid; exec sh -c 'tar -xmf - -C /tmp/unit-tf-job-operator-0945146030'; "]
application-pipelines-db: 20:53:07 DEBUG juju.worker.caasoperator operator service address: 10.152.183.107
application-pipelines-db: 20:53:07 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pipelines-db-0/operator-client-cache.yaml  } to {/tmp/unit-pipelines-db-0452256353/operator-client-cache.yaml pipelines-db-0 juju-pod-init}
application-pipelines-db: 20:53:07 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/yhoycsqa.pid; exec sh -c 'test -d /tmp/unit-pipelines-db-0452256353/operator-client-cache.yaml'; "]
application-pytorch-operator: 20:53:08 DEBUG juju.worker.caasoperator operator service address: 10.152.183.122
application-pytorch-operator: 20:53:08 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pytorch-operator-0/operator-client-cache.yaml  } to {/tmp/unit-pytorch-operator-0900121166/operator-client-cache.yaml pytorch-operator-6974c76984-m47qt juju-pod-init}
application-pytorch-operator: 20:53:08 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/iegezfaq.pid; exec sh -c 'test -d /tmp/unit-pytorch-operator-0900121166/operator-client-cache.yaml'; "]
application-pipelines-api: 20:53:08 DEBUG juju.worker.caasoperator operator service address: 10.152.183.108
application-pipelines-api: 20:53:08 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pipelines-api-0/operator-client-cache.yaml  } to {/tmp/unit-pipelines-api-0374730597/operator-client-cache.yaml pipelines-api-58c5b9c64d-lrjjh juju-pod-init}
application-pipelines-api: 20:53:08 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/lfzxziye.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0374730597/operator-client-cache.yaml'; "]
controller-0: 20:53:08 DEBUG juju.worker.caasunitprovisioner ensured deployment for pipelines-persistence for 1 units
application-seldon-core: 20:53:08 DEBUG juju.worker.caasoperator operator service address: 10.152.183.94
application-seldon-core: 20:53:08 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-seldon-core-0/operator-client-cache.yaml  } to {/tmp/unit-seldon-core-0673265913/operator-client-cache.yaml seldon-core-6758ff7f64-mmc22 juju-pod-init}
application-seldon-core: 20:53:08 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/cmpxohss.pid; exec sh -c 'test -d /tmp/unit-seldon-core-0673265913/operator-client-cache.yaml'; "]
application-argo-controller: 20:53:08 DEBUG juju.kubernetes.provider.exec exec on pod "argo-controller-5fdc64b6b4-t6r66" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/sykzolbh.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-argo-controller-0 --charm-dir /tmp/unit-argo-controller-0827932386/charm --send --operator-file /tmp/unit-argo-controller-0827932386/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-argo-controller-0827932386/ca.crt'; "]
application-pipelines-scheduledworkflow: 20:53:08 DEBUG juju.worker.caasoperator operator service address: 10.152.183.6
application-pipelines-scheduledworkflow: 20:53:08 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pipelines-scheduledworkflow-0/operator-client-cache.yaml  } to {/tmp/unit-pipelines-scheduledworkflow-0518329923/operator-client-cache.yaml pipelines-scheduledworkflow-744fdf4694-zq7xb juju-pod-init}
application-pipelines-scheduledworkflow: 20:53:08 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/zcbvtjha.pid; exec sh -c 'test -d /tmp/unit-pipelines-scheduledworkflow-0518329923/operator-client-cache.yaml'; "]
controller-0: 20:53:08 DEBUG juju.worker.caasunitprovisioner ensured deployment for pipelines-scheduledworkflow for 1 units
application-minio: 20:53:08 DEBUG juju.kubernetes.provider.exec exec on pod "minio-0" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/hxxnoptb.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-minio-0 --charm-dir /tmp/unit-minio-0772539583/charm --send --operator-file /tmp/unit-minio-0772539583/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-minio-0772539583/ca.crt'; "]
application-pipelines-persistence: 20:53:09 DEBUG juju.worker.caasoperator operator service address: 10.152.183.198
application-pipelines-persistence: 20:53:09 DEBUG juju.kubernetes.provider.exec copying from {/var/lib/juju/agents/unit-pipelines-persistence-0/operator-client-cache.yaml  } to {/tmp/unit-pipelines-persistence-0759730194/operator-client-cache.yaml pipelines-persistence-7986b9f7f5-8bdlx juju-pod-init}
application-pipelines-persistence: 20:53:09 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/eedjgqwp.pid; exec sh -c 'test -d /tmp/unit-pipelines-persistence-0759730194/operator-client-cache.yaml'; "]
application-tf-job-operator: 20:53:09 DEBUG juju.kubernetes.provider.exec exec on pod "tf-job-operator-7c48d7ff49-zslkw" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/zwiqzree.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-tf-job-operator-0 --charm-dir /tmp/unit-tf-job-operator-0945146030/charm --send --operator-file /tmp/unit-tf-job-operator-0945146030/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-tf-job-operator-0945146030/ca.crt'; "]
application-pipelines-db: 20:53:09 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/arervypw.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-db-0452256353'; "]
application-pipelines-api: 20:53:09 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/jqjqenuu.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0374730597'; "]
application-pytorch-operator: 20:53:09 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/seeujdkd.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pytorch-operator-0900121166'; "]
controller-0: 20:53:09 DEBUG juju.worker.caasunitprovisioner ensured deployment for minio for 1 units
controller-0: 20:53:09 DEBUG juju.worker.caasunitprovisioner service for seldon-core(workload): &{Id:e2d53e59-da83-47da-a3a1-ae25d9ef4d4c Addresses:[local-cloud:10.152.183.28] Scale:0xc001871150 Generation:0xc001871158 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-seldon-core: 20:53:09 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/issodvvg.pid; exec sh -c 'tar -xmf - -C /tmp/unit-seldon-core-0673265913'; "]
application-pipelines-scheduledworkflow: 20:53:09 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/nvxygidg.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-scheduledworkflow-0518329923'; "]
application-minio: 20:53:10 DEBUG juju.worker.uniter.operation committing operation "remote init" for minio/0
application-minio: 20:53:10 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:53:10 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-minio: 20:53:10 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-argo-controller: 20:53:10 DEBUG juju.worker.uniter.operation committing operation "remote init" for argo-controller/0
application-argo-controller: 20:53:10 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-argo-controller: 20:53:10 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-persistence: 20:53:10 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ydbusxin.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-persistence-0759730194'; "]
controller-0: 20:53:10 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pytorch-operator: 20:53:10 DEBUG juju.kubernetes.provider.exec exec on pod "pytorch-operator-6974c76984-m47qt" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/abhodxdu.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pytorch-operator-0 --charm-dir /tmp/unit-pytorch-operator-0900121166/charm --send --operator-file /tmp/unit-pytorch-operator-0900121166/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pytorch-operator-0900121166/ca.crt'; "]
application-pipelines-db: 20:53:10 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-db-0" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/cvnqhjzp.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-db-0 --charm-dir /tmp/unit-pipelines-db-0452256353/charm --send --operator-file /tmp/unit-pipelines-db-0452256353/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-db-0452256353/ca.crt'; "]
application-tf-job-operator: 20:53:10 DEBUG juju.worker.uniter.operation committing operation "remote init" for tf-job-operator/0
controller-0: 20:53:10 DEBUG juju.worker.caasunitprovisioner service for tf-job-operator(workload): &{Id: Addresses:[] Scale:0xc0017fa0c0 Generation:0xc0017fa0c8 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-tf-job-operator: 20:53:10 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-tf-job-operator: 20:53:10 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-api: 20:53:10 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-api-58c5b9c64d-lrjjh" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/mrpkrdpw.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-api-0 --charm-dir /tmp/unit-pipelines-api-0374730597/charm --send --operator-file /tmp/unit-pipelines-api-0374730597/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-api-0374730597/ca.crt'; "]
application-pipelines-scheduledworkflow: 20:53:10 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-scheduledworkflow-744fdf4694-zq7xb" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/nmnccell.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-scheduledworkflow-0 --charm-dir /tmp/unit-pipelines-scheduledworkflow-0518329923/charm --send --operator-file /tmp/unit-pipelines-scheduledworkflow-0518329923/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-scheduledworkflow-0518329923/ca.crt'; "]
controller-0: 20:53:11 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-scheduledworkflow
controller-0: 20:53:11 DEBUG juju.worker.caasunitprovisioner ensured deployment for pipelines-db for 1 units
controller-0: 20:53:11 DEBUG juju.worker.caasunitprovisioner service for pipelines-api(workload): &{Id:41dd3516-000c-4bd2-8989-7194ac9df0e9 Addresses:[local-cloud:10.152.183.229] Scale:0xc0017c29d0 Generation:0xc0017c29d8 Status:{Status:waiting Message: Data:map[] Since:<nil>}}
application-seldon-core: 20:53:11 DEBUG juju.kubernetes.provider.exec exec on pod "seldon-core-6758ff7f64-mmc22" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/rrixbrlf.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-seldon-core-0 --charm-dir /tmp/unit-seldon-core-0673265913/charm --send --operator-file /tmp/unit-seldon-core-0673265913/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-seldon-core-0673265913/ca.crt'; "]
controller-0: 20:53:11 DEBUG juju.worker.caasunitprovisioner ensured deployment for pytorch-operator for 1 units
application-pipelines-persistence: 20:53:12 DEBUG juju.kubernetes.provider.exec exec on pod "pipelines-persistence-7986b9f7f5-8bdlx" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/tsewfucj.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-persistence-0 --charm-dir /tmp/unit-pipelines-persistence-0759730194/charm --send --operator-file /tmp/unit-pipelines-persistence-0759730194/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-persistence-0759730194/ca.crt'; "]
controller-0: 20:53:12 DEBUG juju.worker.caasunitprovisioner ensured deployment for tf-job-operator for 1 units
application-pytorch-operator: 20:53:12 DEBUG juju.worker.uniter.operation committing operation "remote init" for pytorch-operator/0
controller-0: 20:53:12 DEBUG juju.worker.caasunitprovisioner operator update for minio
application-pytorch-operator: 20:53:13 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
application-pytorch-operator: 20:53:13 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "update-status"
controller-0: 20:53:13 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-db: 20:53:13 DEBUG juju.worker.uniter.operation committing operation "remote init" for pipelines-db/0
application-pipelines-api: 20:53:13 DEBUG juju.worker.uniter.operation committing operation "remote init" for pipelines-api/0
application-pipelines-scheduledworkflow: 20:53:13 DEBUG juju.worker.uniter.operation committing operation "remote init" for pipelines-scheduledworkflow/0
application-seldon-core: 20:53:13 DEBUG juju.worker.uniter.operation committing operation "remote init" for seldon-core/0
application-pipelines-scheduledworkflow: 20:53:13 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-scheduledworkflow: 20:53:13 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-db: 20:53:13 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:53:13 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:53:13 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-seldon-core: 20:53:13 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-seldon-core: 20:53:13 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pipelines-api: 20:53:14 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:53:14 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
controller-0: 20:53:14 DEBUG juju.worker.caasunitprovisioner operator update for pytorch-operator
application-pipelines-persistence: 20:53:14 DEBUG juju.worker.uniter.operation committing operation "remote init" for pipelines-persistence/0
application-pipelines-persistence: 20:53:14 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:53:14 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
controller-0: 20:53:14 DEBUG juju.worker.caasunitprovisioner ensured deployment for argo-controller for 1 units
controller-0: 20:53:15 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:53:15 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:53:16 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc003c8b540}
controller-0: 20:53:16 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
controller-0: 20:53:16 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-minio: 20:53:16 DEBUG juju.worker.caasoperator.remotestate got application change
application-tf-job-operator: 20:53:18 DEBUG juju.worker.caasoperator trigger running status for caas unit tf-job-operator/0
application-tf-job-operator: 20:53:18 DEBUG juju.worker.uniter.remotestate got running status change for tf-job-operator/0
application-tf-job-operator: 20:53:18 DEBUG juju.worker.caasoperator request running status for "unit-tf-job-operator-0" 6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97
application-tf-job-operator: 20:53:18 DEBUG juju.kubernetes.provider.exec no pod named "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97" found
application-tf-job-operator: 20:53:18 DEBUG juju.kubernetes.provider.exec try get pod by UID for "6d3302fe-c93d-4cf7-8868-7d4ba4ec0b97"
application-tf-job-operator: 20:53:18 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:53:18 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-minio: 20:53:18 DEBUG juju.worker.caasoperator trigger running status for caas unit minio/0
application-minio: 20:53:18 DEBUG juju.worker.uniter.remotestate got running status change for minio/0
application-minio: 20:53:18 DEBUG juju.worker.caasoperator request running status for "unit-minio-0" minio-0
application-minio: 20:53:18 DEBUG juju.worker.uniter.storage next hook op for storage-minio-data-0: {Kind:2 Life:alive Attached:true Location:/data}
application-minio: 20:53:18 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:53:18 DEBUG juju.worker.caasoperator.remotestate got application change
application-pytorch-operator: 20:53:18 DEBUG juju.worker.uniter.remotestate retry hook timer triggered for pytorch-operator/0
application-pytorch-operator: 20:53:18 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
application-pytorch-operator: 20:53:18 DEBUG juju.worker.uniter.operation running operation run update-status hook for pytorch-operator/0
application-pytorch-operator: 20:53:18 DEBUG juju.machinelock acquire machine lock for pytorch-operator/0 uniter (run update-status hook)
application-pytorch-operator: 20:53:18 DEBUG juju.machinelock machine lock acquired for pytorch-operator/0 uniter (run update-status hook)
application-pytorch-operator: 20:53:18 DEBUG juju.worker.uniter.operation preparing operation "run update-status hook" for pytorch-operator/0
application-pytorch-operator: 20:53:18 DEBUG juju.worker.uniter.operation executing operation "run update-status hook" for pytorch-operator/0
application-pytorch-operator: 20:53:18 DEBUG juju.worker.uniter.runner starting jujuc server  {unix @/var/lib/juju/agents/unit-pytorch-operator-0/agent.socket <nil>}
application-pytorch-operator: 20:53:19 DEBUG jujuc running hook tool "config-get" for pytorch-operator/0-update-status-753277784026843633
application-pytorch-operator: 20:53:19 DEBUG juju.worker.caasoperator trigger running status for caas unit pytorch-operator/0
application-pytorch-operator: 20:53:19 DEBUG juju.worker.uniter.remotestate got running status change for pytorch-operator/0
application-pytorch-operator: 20:53:19 DEBUG juju.worker.caasoperator request running status for "unit-pytorch-operator-0" b95510d3-6d1e-4d72-9615-e6403906e412
application-pytorch-operator: 20:53:19 DEBUG juju.kubernetes.provider.exec no pod named "b95510d3-6d1e-4d72-9615-e6403906e412" found
application-pytorch-operator: 20:53:19 DEBUG juju.kubernetes.provider.exec try get pod by UID for "b95510d3-6d1e-4d72-9615-e6403906e412"
application-pytorch-operator: 20:53:19 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-update-status-753277784026843633
application-pytorch-operator: 20:53:19 INFO unit.pytorch-operator/0.juju-log Reactive main running for hook update-status
controller-0: 20:53:19 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-pytorch-operator: 20:53:19 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-update-status-753277784026843633
application-pytorch-operator: 20:53:19 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: starting handler dispatch, 5 flags set
tracer: set flag charm.started
tracer: set flag config.default.pytorch-default-image
tracer: set flag layer.docker-resource.auto-fetched
tracer: set flag layer.docker-resource.oci-image.available
tracer: set flag layer.docker-resource.oci-image.fetched
application-pytorch-operator: 20:53:19 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-update-status-753277784026843633
application-pytorch-operator: 20:53:19 DEBUG unit.pytorch-operator/0.juju-log tracer: hooks phase, 0 handlers queued
application-pytorch-operator: 20:53:19 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-update-status-753277784026843633
application-pytorch-operator: 20:53:19 DEBUG unit.pytorch-operator/0.juju-log tracer>
tracer: main dispatch loop, 1 handlers queued
tracer: ++   queue handler reactive/pytorch_operator.py:15:charm_ready
application-pytorch-operator: 20:53:19 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-update-status-753277784026843633
application-pytorch-operator: 20:53:19 INFO unit.pytorch-operator/0.juju-log Invoking reactive handler: reactive/pytorch_operator.py:15:charm_ready
controller-0: 20:53:20 DEBUG juju.worker.caasunitprovisioner units changed: true
application-pytorch-operator: 20:53:20 DEBUG jujuc running hook tool "juju-log" for pytorch-operator/0-update-status-753277784026843633
application-pytorch-operator: 20:53:20 INFO unit.pytorch-operator/0.juju-log status-set: active: 
application-pytorch-operator: 20:53:20 DEBUG jujuc running hook tool "status-set" for pytorch-operator/0-update-status-753277784026843633
application-pipelines-db: 20:53:20 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:53:20 DEBUG juju.worker.caasunitprovisioner operator update for seldon-core
application-argo-controller: 20:53:20 DEBUG juju.worker.caasoperator trigger running status for caas unit argo-controller/0
application-argo-controller: 20:53:20 DEBUG juju.worker.uniter.remotestate got running status change for argo-controller/0
application-argo-controller: 20:53:20 DEBUG juju.worker.caasoperator request running status for "unit-argo-controller-0" 98f9d8de-8d65-4cad-af65-85867d87b501
application-argo-controller: 20:53:20 DEBUG juju.kubernetes.provider.exec no pod named "98f9d8de-8d65-4cad-af65-85867d87b501" found
application-argo-controller: 20:53:20 DEBUG juju.kubernetes.provider.exec try get pod by UID for "98f9d8de-8d65-4cad-af65-85867d87b501"
application-argo-controller: 20:53:20 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pytorch-operator: 20:53:20 INFO juju.worker.uniter.operation ran "update-status" hook (via explicit, bespoke hook script)
application-pytorch-operator: 20:53:20 DEBUG juju.worker.uniter.operation committing operation "run update-status hook" for pytorch-operator/0
application-pytorch-operator: 20:53:20 DEBUG juju.machinelock machine lock released for pytorch-operator/0 uniter (run update-status hook)
application-pytorch-operator: 20:53:20 DEBUG juju.worker.uniter.operation lock released for pytorch-operator/0
application-pytorch-operator: 20:53:20 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pytorch-operator: 20:53:20 DEBUG juju.worker.uniter [AGENT-STATUS] idle: 
application-pytorch-operator: 20:53:20 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-db: 20:53:21 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-db/0
application-pipelines-db: 20:53:21 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-db/0
application-pipelines-db: 20:53:21 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-db-0" pipelines-db-0
application-pipelines-db: 20:53:21 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:53:21 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:53:21 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:53:21 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:53:21 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:53:22 DEBUG juju.worker.caasunitprovisioner service for argo-controller(workload): &{Id: Addresses:[] Scale:0xc0048932b0 Generation:0xc0048932b8 Status:{Status:active Message: Data:map[] Since:<nil>}}
application-seldon-core: 20:53:22 DEBUG juju.worker.caasoperator trigger running status for caas unit seldon-core/0
application-seldon-core: 20:53:22 DEBUG juju.worker.uniter.remotestate got running status change for seldon-core/0
application-seldon-core: 20:53:22 DEBUG juju.worker.caasoperator request running status for "unit-seldon-core-0" 99fe7a56-602c-405a-8b08-2ca156ab3f06
application-seldon-core: 20:53:22 DEBUG juju.kubernetes.provider.exec no pod named "99fe7a56-602c-405a-8b08-2ca156ab3f06" found
application-seldon-core: 20:53:22 DEBUG juju.kubernetes.provider.exec try get pod by UID for "99fe7a56-602c-405a-8b08-2ca156ab3f06"
application-seldon-core: 20:53:22 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-persistence: 20:53:22 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-persistence/0
application-pipelines-persistence: 20:53:22 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-persistence/0
application-pipelines-persistence: 20:53:22 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-persistence-0" d67ee23b-6f10-4e86-b8a8-1b878f678466
application-pipelines-persistence: 20:53:22 DEBUG juju.kubernetes.provider.exec no pod named "d67ee23b-6f10-4e86-b8a8-1b878f678466" found
application-pipelines-persistence: 20:53:22 DEBUG juju.kubernetes.provider.exec try get pod by UID for "d67ee23b-6f10-4e86-b8a8-1b878f678466"
application-pipelines-persistence: 20:53:22 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:53:23 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-scheduledworkflow: 20:53:23 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:53:23 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-scheduledworkflow/0
application-pipelines-scheduledworkflow: 20:53:23 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-scheduledworkflow-0" 8c345d1f-b75a-4dff-a4bb-67637ec9f804
application-pipelines-scheduledworkflow: 20:53:23 DEBUG juju.kubernetes.provider.exec no pod named "8c345d1f-b75a-4dff-a4bb-67637ec9f804" found
application-pipelines-scheduledworkflow: 20:53:23 DEBUG juju.kubernetes.provider.exec try get pod by UID for "8c345d1f-b75a-4dff-a4bb-67637ec9f804"
application-pipelines-scheduledworkflow: 20:53:23 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-pipelines-api: 20:53:24 DEBUG juju.worker.caasoperator trigger running status for caas unit pipelines-api/0
application-pipelines-api: 20:53:24 DEBUG juju.worker.uniter.remotestate got running status change for pipelines-api/0
application-pipelines-api: 20:53:24 DEBUG juju.worker.caasoperator request running status for "unit-pipelines-api-0" 060740d9-9aef-4765-81a1-73ba4a468772
application-pipelines-api: 20:53:24 DEBUG juju.kubernetes.provider.exec no pod named "060740d9-9aef-4765-81a1-73ba4a468772" found
application-pipelines-api: 20:53:24 DEBUG juju.kubernetes.provider.exec try get pod by UID for "060740d9-9aef-4765-81a1-73ba4a468772"
application-pipelines-api: 20:53:24 DEBUG juju.worker.uniter no operations in progress; waiting for changes
controller-0: 20:53:24 DEBUG juju.worker.caasunitprovisioner ensured deployment for seldon-core for 1 units
controller-0: 20:53:25 DEBUG juju.worker.caasunitprovisioner service for pipelines-scheduledworkflow(workload): &{Id: Addresses:[] Scale:0xc0038a1ce0 Generation:0xc0038a1ce8 Status:{Status:active Message: Data:map[] Since:<nil>}}
controller-0: 20:53:26 DEBUG juju.worker.caasunitprovisioner service for tf-job-operator(workload): &{Id: Addresses:[] Scale:0xc003d03400 Generation:0xc003d03408 Status:{Status:active Message: Data:map[] Since:<nil>}}
application-pipelines-api: 20:53:27 DEBUG juju.worker.caasoperator.remotestate got application change
application-minio: 20:53:27 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:53:27 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc004270100}
controller-0: 20:53:27 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:53:28 DEBUG juju.worker.caasunitprovisioner operator update for pipelines-persistence
controller-0: 20:53:28 DEBUG juju.worker.caasunitprovisioner ensured deployment for pipelines-api for 1 units
controller-0: 20:53:29 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-seldon-core: 20:53:29 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:53:31 DEBUG juju.worker.caasunitprovisioner deployment changed: true
application-pipelines-db: 20:53:32 DEBUG juju.worker.uniter.remotestate storage attachment change for pipelines-db/0: {storage-database-1 {2 alive true /var/lib/mysql}}
application-pipelines-db: 20:53:32 DEBUG juju.worker.uniter.storage next hook op for storage-database-1: {Kind:2 Life:alive Attached:true Location:/var/lib/mysql}
application-pipelines-db: 20:53:32 DEBUG juju.worker.uniter no operations in progress; waiting for changes
application-seldon-core: 20:53:32 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:53:33 DEBUG juju.worker.caasunitprovisioner desiredScale changed to 1
application-pipelines-api: 20:53:33 DEBUG juju.worker.caasoperator.remotestate got application change
controller-0: 20:53:33 DEBUG juju.worker.caasprovisioner using caas operator info {ImagePath:jujusolutions/jujud-operator:2.8.6.372 Version:2.8.6 APIAddresses:[10.152.183.175:17070] Tags:map[juju-controller-uuid:c6472f8a-fc51-4dca-87e7-cbc1d86db7d7 juju-model-uuid:f0e819dd-2fb4-4e25-8bdd-77edd9c85838] CharmStorage:0xc00197ee00}
controller-0: 20:53:34 DEBUG juju.worker.caasunitprovisioner deployment changed: true
controller-0: 20:53:34 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:53:35 DEBUG juju.worker.caasunitprovisioner units changed: true
controller-0: 20:53:35 DEBUG juju.worker.caasunitprovisioner deployment changed: true