2020-09-28 19:14:06 INFO juju.cmd supercommand.go:54 running jujud [2.8.4 212 602dbc79f0fe84685ae5fa84d90cdbe318706c9c gc go1.14.9] 2020-09-28 19:14:06 DEBUG juju.cmd supercommand.go:55 args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=pipelines-api", "--debug"} 2020-09-28 19:14:06 DEBUG juju.agent agent.go:583 read agent config, format "2.0" 2020-09-28 19:14:06 INFO juju.worker.upgradesteps worker.go:70 upgrade steps for 2.8.4 have already been run. 2020-09-28 19:14:06 INFO juju.cmd.jujud caasoperator.go:200 caas operator application-pipelines-api start (2.8.4 [gc]) 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "clock" manifold worker started at 2020-09-28 19:14:06.566878067 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "upgrade-steps-gate" manifold worker started at 2020-09-28 19:14:06.567153983 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "agent" manifold worker started at 2020-09-28 19:14:06.568456251 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "upgrade-steps-flag" manifold worker started at 2020-09-28 19:14:06.568567175 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.introspection socket.go:97 introspection worker listening on "@jujud-application-pipelines-api" 2020-09-28 19:14:06 DEBUG juju.worker.introspection socket.go:127 stats worker now serving 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "api-config-watcher" manifold worker started at 2020-09-28 19:14:06.580579001 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "migration-fortress" manifold worker started at 2020-09-28 19:14:06.580902794 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.apicaller connect.go:128 connecting with old password 2020-09-28 19:14:06 DEBUG juju.api apiclient.go:1105 successfully dialed "wss://10.152.183.208:17070/model/9cdef33f-cf9a-4631-8797-10cb2e1e98b6/api" 2020-09-28 19:14:06 INFO juju.api apiclient.go:637 connection established to "wss://10.152.183.208:17070/model/9cdef33f-cf9a-4631-8797-10cb2e1e98b6/api" 2020-09-28 19:14:06 INFO juju.worker.apicaller connect.go:158 [9cdef3] "application-pipelines-api" successfully connected to "10.152.183.208:17070" 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "api-caller" manifold worker started at 2020-09-28 19:14:06.604109356 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "upgrade-steps-runner" manifold worker started at 2020-09-28 19:14:06.613402064 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:584 "upgrade-steps-runner" manifold worker completed successfully 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "log-sender" manifold worker started at 2020-09-28 19:14:06.614334897 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "migration-minion" manifold worker started at 2020-09-28 19:14:06.614386322 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "upgrader" manifold worker started at 2020-09-28 19:14:06.614508193 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "migration-inactive-flag" manifold worker started at 2020-09-28 19:14:06.61627474 +0000 UTC 2020-09-28 19:14:06 INFO juju.worker.caasupgrader upgrader.go:112 abort check blocked until version event received 2020-09-28 19:14:06 DEBUG juju.worker.caasupgrader upgrader.go:127 current agent binary version: 2.8.4 2020-09-28 19:14:06 INFO juju.worker.caasupgrader upgrader.go:118 unblocking abort check 2020-09-28 19:14:06 INFO juju.worker.migrationminion worker.go:140 migration phase is now: NONE 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "proxy-config-updater" manifold worker started at 2020-09-28 19:14:06.626482541 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.logger logger.go:64 initial log config: "=DEBUG" 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "logging-config-updater" manifold worker started at 2020-09-28 19:14:06.626683114 +0000 UTC 2020-09-28 19:14:06 INFO juju.worker.logger logger.go:118 logger worker started 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "api-address-updater" manifold worker started at 2020-09-28 19:14:06.627432191 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "charm-dir" manifold worker started at 2020-09-28 19:14:06.627664644 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.logger logger.go:92 reconfiguring logging from "=DEBUG" to "=DEBUG;unit=DEBUG" 2020-09-28 19:14:06 DEBUG juju.worker.proxyupdater proxyupdater.go:171 applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.208,127.0.0.1,::1,localhost", AutoNoProxy:""} 2020-09-28 19:14:06 DEBUG juju.worker.proxyupdater proxyupdater.go:288 new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"127.0.0.1,::1,localhost", AutoNoProxy:""} 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "hook-retry-strategy" manifold worker started at 2020-09-28 19:14:06.63462076 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.network network.go:200 no lxc bridge addresses to filter for machine 2020-09-28 19:14:06 DEBUG juju.network network.go:236 cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring) 2020-09-28 19:14:06 DEBUG juju.network network.go:236 cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring) 2020-09-28 19:14:06 DEBUG juju.network network.go:185 including address local-cloud:10.152.183.208 for machine 2020-09-28 19:14:06 DEBUG juju.network network.go:254 addresses after filtering: [local-cloud:10.152.183.208] 2020-09-28 19:14:06 DEBUG juju.worker.apiaddressupdater apiaddressupdater.go:98 updating API hostPorts to [[10.152.183.208:17070]] 2020-09-28 19:14:06 DEBUG juju.agent agent.go:619 API server address details [["10.152.183.208:17070"]] written to agent config as ["10.152.183.208:17070"] 2020-09-28 19:14:06 DEBUG juju.worker.dependency engine.go:564 "operator" manifold worker started at 2020-09-28 19:14:06.645878782 +0000 UTC 2020-09-28 19:14:06 DEBUG juju.worker.proxyupdater proxyupdater.go:171 applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.152.183.208,127.0.0.1,::1,localhost", AutoNoProxy:""} 2020-09-28 19:14:06 INFO juju.worker.uniter.charm bundles.go:79 downloading local:kubernetes/pipelines-api-0 from API server 2020-09-28 19:14:06 INFO juju.downloader download.go:111 downloading from local:kubernetes/pipelines-api-0 2020-09-28 19:14:06 INFO juju.downloader download.go:94 download complete ("local:kubernetes/pipelines-api-0") 2020-09-28 19:14:06 INFO juju.downloader download.go:174 download verified ("local:kubernetes/pipelines-api-0") 2020-09-28 19:14:06 DEBUG juju.worker.uniter.charm manifest_deployer.go:135 preparing to deploy charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:14:06 DEBUG juju.worker.uniter.charm manifest_deployer.go:124 deploying charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:14:08 DEBUG juju.worker.uniter.charm manifest_deployer.go:156 finishing deploy of charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:14:08 DEBUG juju.worker.caasoperator caasoperator.go:373 starting caas operator juju-run listener on &{tcp :30666 0xc0005eb680} 2020-09-28 19:14:08 INFO juju.worker.caasoperator caasoperator.go:402 operator "pipelines-api" started 2020-09-28 19:14:08 DEBUG juju.worker.uniter runlistener.go:116 juju-run listener running 2020-09-28 19:14:08 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:14:08 DEBUG juju.worker.uniter uniter.go:212 starting uniter for "pipelines-api/0" 2020-09-28 19:14:08 DEBUG juju.worker.caasoperator runner.go:355 start "pipelines-api/0" 2020-09-28 19:14:08 INFO juju.worker.caasoperator runner.go:527 start "pipelines-api/0" 2020-09-28 19:14:08 DEBUG juju.worker.caasoperator runner.go:364 "pipelines-api/0" started 2020-09-28 19:14:08 DEBUG juju.worker.leadership tracker.go:125 pipelines-api/0 making initial claim for pipelines-api leadership 2020-09-28 19:14:08 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-api-0 2020-09-28 19:14:08 DEBUG juju.agent.tools symlinks.go:47 jujuc not found at /var/lib/juju/tools/unit-pipelines-api-0/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-api-0/jujud 2020-09-28 19:14:08 DEBUG juju.agent.tools symlinks.go:50 target tools path /var/lib/juju/tools/unit-pipelines-api-0/jujud 2020-09-28 19:14:08 INFO juju.worker.leadership tracker.go:194 pipelines-api/0 promoted to leadership of pipelines-api 2020-09-28 19:14:08 DEBUG juju.worker.uniter uniter.go:775 starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-api-0/run.socket } 2020-09-28 19:14:08 INFO juju.worker.uniter uniter.go:298 unit "pipelines-api/0" started 2020-09-28 19:14:08 INFO juju.worker.uniter uniter.go:308 resuming charm install 2020-09-28 19:14:08 DEBUG juju.worker.uniter runlistener.go:116 juju-run listener running 2020-09-28 19:14:08 DEBUG juju.worker.uniter.operation executor.go:83 running operation install local:kubernetes/pipelines-api-0 2020-09-28 19:14:08 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "install local:kubernetes/pipelines-api-0" 2020-09-28 19:14:08 INFO juju.worker.uniter.charm bundles.go:79 downloading local:kubernetes/pipelines-api-0 from API server 2020-09-28 19:14:08 INFO juju.downloader download.go:111 downloading from local:kubernetes/pipelines-api-0 2020-09-28 19:14:08 INFO juju.downloader download.go:94 download complete ("local:kubernetes/pipelines-api-0") 2020-09-28 19:14:08 INFO juju.downloader download.go:174 download verified ("local:kubernetes/pipelines-api-0") 2020-09-28 19:14:08 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "install local:kubernetes/pipelines-api-0" 2020-09-28 19:14:08 DEBUG juju.worker.uniter.charm manifest_deployer.go:135 preparing to deploy charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:14:08 DEBUG juju.worker.uniter.charm manifest_deployer.go:124 deploying charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:14:09 DEBUG juju.worker.uniter.charm manifest_deployer.go:156 finishing deploy of charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:14:09 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "install local:kubernetes/pipelines-api-0" 2020-09-28 19:14:09 INFO juju.worker.uniter uniter.go:349 hooks are retried true 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:523 got trust config change: ok=true, hashes=[7f9884bc998df3cb2761f63f838f8c09c8a7ce0ea62a095fae071ad6e1a23413] 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:544 got address change: ok=true, hashes=[] 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:512 got config change: ok=true, hashes=[3da95c4ad53ef52c4e6d84e4744a140197f8ee389904ab3f4c9eaf6f3ce8e7c0] 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:593 got update status interval change: ok=true 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:573 got relations change: ok=true 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:555 got leader settings change: ok=true 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:583 got storage change: [] ok=true 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:461 got unit change 2020-09-28 19:14:09 DEBUG juju.worker.uniter.remotestate watcher.go:565 got action change: [] ok=true 2020-09-28 19:14:09 INFO juju.worker.uniter resolver.go:147 found queued "install" hook 2020-09-28 19:14:09 DEBUG juju.worker.uniter.operation executor.go:83 running operation run install hook 2020-09-28 19:14:09 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run install hook) 2020-09-28 19:14:09 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run install hook) 2020-09-28 19:14:09 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run install hook" 2020-09-28 19:14:09 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run install hook" 2020-09-28 19:14:09 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running install hook 2020-09-28 19:14:09 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:10 INFO juju-log Reactive main running for hook install 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:10 DEBUG juju-log tracer> tracer: starting handler dispatch, 15 flags set tracer: set flag config.changed tracer: set flag config.changed.grpc-port tracer: set flag config.changed.http-port tracer: set flag config.changed.minio-access-key tracer: set flag config.changed.minio-bucket-name tracer: set flag config.changed.minio-secret-key tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:10 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:10 DEBUG juju-log tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/docker_resource.py:7:auto_fetch tracer: ++ queue handler reactive/pipelines_api.py:36:update_image 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:10 INFO juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch 2020-09-28 19:14:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:10 DEBUG juju-log tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 DEBUG juju-log tracer> tracer: set flag layer.docker-resource.pending tracer: ++ queue handler reactive/docker_resource.py:18:fetch 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 DEBUG juju-log tracer> tracer: set flag layer.docker-resource.auto-fetched tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:36:update_image 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 DEBUG juju-log tracer> tracer: cleared flag config.changed tracer: -- dequeue handler reactive/pipelines_api.py:36:update_image 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 INFO juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:11 INFO juju-log status-set: maintenance: fetching resource: oci-image 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:14:11 DEBUG jujuc server.go:211 running hook tool "resource-get" 2020-09-28 19:14:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:12 DEBUG juju-log tracer: set flag layer.docker-resource.oci-image.available 2020-09-28 19:14:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:12 DEBUG juju-log tracer> tracer: cleared flag layer.docker-resource.pending tracer: -- dequeue handler reactive/docker_resource.py:18:fetch 2020-09-28 19:14:12 INFO juju.worker.uniter.operation runhook.go:142 ran "install" hook (via explicit, bespoke hook script) 2020-09-28 19:14:12 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run install hook" 2020-09-28 19:14:12 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run install hook) 2020-09-28 19:14:12 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:pipelines-visualization pipelines-visualization:pipelines-visualization" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:pipelines-visualization pipelines-visualization:pipelines-visualization" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:mysql pipelines-db:mysql" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:mysql pipelines-db:mysql" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:14:12 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {10 {map[minio/0:{1}] map[] []}} 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:kubeflow-profiles kubeflow-profiles:kubeflow-profiles" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:kubeflow-profiles kubeflow-profiles:kubeflow-profiles" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:minio minio:minio" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:minio minio:minio" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-ui:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:14:12 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-ui:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:14:12 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (7; app: pipelines-db) hook 2020-09-28 19:14:12 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (7; app: pipelines-db) hook) 2020-09-28 19:14:12 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (7; app: pipelines-db) hook) 2020-09-28 19:14:12 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (7; app: pipelines-db) hook" 2020-09-28 19:14:12 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (7; app: pipelines-db) hook" 2020-09-28 19:14:12 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running mysql-relation-created hook 2020-09-28 19:14:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:13 INFO juju.worker.uniter.operation runhook.go:145 skipped "mysql-relation-created" hook (missing) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (7; app: pipelines-db) hook" 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (7; app: pipelines-db) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (8; app: pipelines-persistence) hook 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (8; app: pipelines-persistence) hook) 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (8; app: pipelines-persistence) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (8; app: pipelines-persistence) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (8; app: pipelines-persistence) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-created hook 2020-09-28 19:14:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:13 INFO juju.worker.uniter.operation runhook.go:145 skipped "pipelines-api-relation-created" hook (missing) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (8; app: pipelines-persistence) hook" 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (8; app: pipelines-persistence) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (11; app: pipelines-ui) hook 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (11; app: pipelines-ui) hook) 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (11; app: pipelines-ui) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (11; app: pipelines-ui) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (11; app: pipelines-ui) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:13 INFO juju.worker.uniter.operation runhook.go:145 skipped "pipelines-api-relation-created" hook (missing) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (11; app: pipelines-ui) hook" 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (11; app: pipelines-ui) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (12; app: pipelines-visualization) hook 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (12; app: pipelines-visualization) hook) 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (12; app: pipelines-visualization) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (12; app: pipelines-visualization) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (12; app: pipelines-visualization) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-visualization-relation-created hook 2020-09-28 19:14:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:13 INFO juju.worker.uniter.operation runhook.go:145 skipped "pipelines-visualization-relation-created" hook (missing) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (12; app: pipelines-visualization) hook" 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (12; app: pipelines-visualization) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (9; app: kubeflow-profiles) hook 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (9; app: kubeflow-profiles) hook) 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (9; app: kubeflow-profiles) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (9; app: kubeflow-profiles) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (9; app: kubeflow-profiles) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running kubeflow-profiles-relation-created hook 2020-09-28 19:14:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:13 INFO juju.worker.uniter.operation runhook.go:145 skipped "kubeflow-profiles-relation-created" hook (missing) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (9; app: kubeflow-profiles) hook" 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (9; app: kubeflow-profiles) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (10; app: minio) hook 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (10; app: minio) hook) 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (10; app: minio) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (10; app: minio) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (10; app: minio) hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running minio-relation-created hook 2020-09-28 19:14:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:13 INFO juju.worker.uniter.operation runhook.go:145 skipped "minio-relation-created" hook (missing) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (10; app: minio) hook" 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (10; app: minio) hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation accept leadership 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "accept leadership" 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "accept leadership" 2020-09-28 19:14:13 INFO juju.worker.uniter resolver.go:147 found queued "leader-elected" hook 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run leader-elected hook 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run leader-elected hook) 2020-09-28 19:14:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run leader-elected hook) 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run leader-elected hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run leader-elected hook" 2020-09-28 19:14:13 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running leader-elected hook 2020-09-28 19:14:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:14 INFO juju-log Reactive main running for hook leader-elected 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:15 DEBUG juju-log tracer> tracer: starting handler dispatch, 12 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:15 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:15 DEBUG juju-log tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:15 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:15 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:15 INFO juju.worker.uniter.operation runhook.go:142 ran "leader-elected" hook (via explicit, bespoke hook script) 2020-09-28 19:14:15 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run leader-elected hook" 2020-09-28 19:14:15 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run leader-elected hook) 2020-09-28 19:14:15 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:15 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:14:15 DEBUG juju.worker.uniter.operation executor.go:83 running operation run config-changed hook 2020-09-28 19:14:15 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run config-changed hook) 2020-09-28 19:14:15 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run config-changed hook) 2020-09-28 19:14:15 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run config-changed hook" 2020-09-28 19:14:15 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run config-changed hook" 2020-09-28 19:14:15 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running config-changed hook 2020-09-28 19:14:15 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:15 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:16 INFO juju-log Reactive main running for hook config-changed 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:16 DEBUG juju-log tracer> tracer: starting handler dispatch, 12 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:16 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:16 DEBUG juju-log tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:16 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:16 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:17 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:17 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:14:17 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:17 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:14:17 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:17 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:17 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {10 {map[minio/0:{2}] map[] []}} 2020-09-28 19:14:17 INFO juju.worker.uniter.operation runhook.go:142 ran "config-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:14:17 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run config-changed hook" 2020-09-28 19:14:17 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run config-changed hook) 2020-09-28 19:14:17 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:17 INFO juju.worker.uniter resolver.go:147 found queued "start" hook 2020-09-28 19:14:17 DEBUG juju.worker.uniter.operation executor.go:83 running operation run start hook 2020-09-28 19:14:17 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run start hook) 2020-09-28 19:14:17 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run start hook) 2020-09-28 19:14:17 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run start hook" 2020-09-28 19:14:17 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run start hook" 2020-09-28 19:14:17 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running start hook 2020-09-28 19:14:17 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:17 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:17 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:17 INFO juju-log Reactive main running for hook start 2020-09-28 19:14:17 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:18 DEBUG juju-log tracer> tracer: starting handler dispatch, 12 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:14:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:18 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:19 DEBUG juju-log tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:19 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:19 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:19 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:19 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:19 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:19 INFO juju.worker.uniter.operation runhook.go:142 ran "start" hook (via explicit, bespoke hook script) 2020-09-28 19:14:19 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run start hook" 2020-09-28 19:14:19 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run start hook) 2020-09-28 19:14:19 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:19 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:14:19 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (9; unit: kubeflow-profiles/0) hook 2020-09-28 19:14:19 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:14:19 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:14:19 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:14:19 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:14:19 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running kubeflow-profiles-relation-joined hook 2020-09-28 19:14:19 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:19 INFO juju-log kubeflow-profiles:9: Reactive main running for hook kubeflow-profiles-relation-joined 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:19 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:20 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: set flag endpoint.kubeflow-profiles.joined tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:20 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: set flag endpoint.kubeflow-profiles.changed tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:20 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:20 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.hostname 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:20 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:20 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.port 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:20 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.private-address 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: starting handler dispatch, 19 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 DEBUG juju-log kubeflow-profiles:9: tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 DEBUG juju-log kubeflow-profiles:9: tracer: set flag kubeflow-profiles.available 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:14:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:21 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:21 INFO juju.worker.uniter.operation runhook.go:142 ran "kubeflow-profiles-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:14:21 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:14:21 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:14:21 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:21 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:14:21 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (9; unit: kubeflow-profiles/0) hook 2020-09-28 19:14:21 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:14:21 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:14:21 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:14:22 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:14:22 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running kubeflow-profiles-relation-changed hook 2020-09-28 19:14:22 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:22 INFO juju-log kubeflow-profiles:9: Reactive main running for hook kubeflow-profiles-relation-changed 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:23 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: starting handler dispatch, 20 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:23 DEBUG juju-log kubeflow-profiles:9: tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:23 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:23 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:23 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:23 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:23 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:23 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:23 INFO juju.worker.uniter.operation runhook.go:142 ran "kubeflow-profiles-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:14:23 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:14:23 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:14:23 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:23 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:14:23 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (10; unit: minio/0) hook 2020-09-28 19:14:23 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (10; unit: minio/0) hook) 2020-09-28 19:14:23 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (10; unit: minio/0) hook) 2020-09-28 19:14:23 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (10; unit: minio/0) hook" 2020-09-28 19:14:23 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (10; unit: minio/0) hook" 2020-09-28 19:14:23 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running minio-relation-joined hook 2020-09-28 19:14:23 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:23 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:24 INFO juju-log minio:10: Reactive main running for hook minio-relation-joined 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:24 DEBUG juju-log minio:10: tracer> tracer: set flag endpoint.minio.joined tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:24 DEBUG juju-log minio:10: tracer> tracer: set flag endpoint.minio.changed tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:24 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.egress-subnets 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:24 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.hostname 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:24 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.ingress-address 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:24 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.port 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:24 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.private-address 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:24 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 DEBUG juju-log minio:10: tracer> tracer: starting handler dispatch, 27 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 DEBUG juju-log minio:10: tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 DEBUG juju-log minio:10: tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 DEBUG juju-log minio:10: tracer: set flag minio.available 2020-09-28 19:14:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:25 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:25 INFO juju.worker.uniter.operation runhook.go:142 ran "minio-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:14:25 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (10; unit: minio/0) hook" 2020-09-28 19:14:25 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (10; unit: minio/0) hook) 2020-09-28 19:14:25 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:25 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:14:25 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (10; unit: minio/0) hook 2020-09-28 19:14:25 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (10; unit: minio/0) hook) 2020-09-28 19:14:25 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (10; unit: minio/0) hook) 2020-09-28 19:14:25 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (10; unit: minio/0) hook" 2020-09-28 19:14:25 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (10; unit: minio/0) hook" 2020-09-28 19:14:25 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running minio-relation-changed hook 2020-09-28 19:14:26 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:26 INFO juju-log minio:10: Reactive main running for hook minio-relation-changed 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:26 DEBUG juju-log minio:10: tracer> tracer: starting handler dispatch, 28 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:26 DEBUG juju-log minio:10: tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:26 DEBUG juju-log minio:10: tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:27 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:27 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:27 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:14:27 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:14:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:27 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:14:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:27 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:27 INFO juju.worker.uniter.operation runhook.go:142 ran "minio-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:14:27 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (10; unit: minio/0) hook" 2020-09-28 19:14:27 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (10; unit: minio/0) hook) 2020-09-28 19:14:27 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:27 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:14:27 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] idle: 2020-09-28 19:14:27 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:14:35 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:14:35 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:14:35 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:14:35 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:14:35 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:14:35 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:14:35 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:14:35 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:14:35 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:14:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:35 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:14:35 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:35 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:36 DEBUG juju-log tracer> tracer: starting handler dispatch, 28 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:36 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:36 DEBUG juju-log tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:36 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:36 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:36 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:36 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:14:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:14:36 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:14:37 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:14:37 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:14:37 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:14:37 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:14:37 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:04 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:15:04 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:04 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:15:04 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:15:04 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:15:04 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:15:04 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:15:04 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:05 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:05 DEBUG juju-log tracer> tracer: starting handler dispatch, 28 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:05 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:05 DEBUG juju-log tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:05 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:06 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:15:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:06 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:06 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:06 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:06 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:06 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:15:06 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:15:06 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:15:06 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:15:06 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:06 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:07 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {11 {map[pipelines-ui/0:{0}] map[] []}} 2020-09-28 19:15:07 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:07 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (11; unit: pipelines-ui/0) hook 2020-09-28 19:15:07 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:07 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:07 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:07 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:07 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-joined hook 2020-09-28 19:15:07 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:07 INFO juju-log pipelines-api:11: Reactive main running for hook pipelines-api-relation-joined 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:07 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:07 DEBUG juju-log pipelines-api:11: tracer> tracer: set flag endpoint.pipelines-api.joined tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 DEBUG juju-log pipelines-api:11: tracer> tracer: starting handler dispatch, 29 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.pipelines-api.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 DEBUG juju-log pipelines-api:11: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 DEBUG juju-log pipelines-api:11: tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 DEBUG juju-log pipelines-api:11: tracer> tracer: set flag pipelines-api.available tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:08 INFO juju-log pipelines-api:11: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:09 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:09 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:09 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:09 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:09 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:15:09 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:09 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:09 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:09 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:09 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (11; unit: pipelines-ui/0) hook 2020-09-28 19:15:09 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:09 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:09 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:09 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:09 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-changed hook 2020-09-28 19:15:09 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:09 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:09 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:09 INFO juju-log pipelines-api:11: Reactive main running for hook pipelines-api-relation-changed 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:11 DEBUG juju-log pipelines-api:11: tracer> tracer: starting handler dispatch, 30 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.pipelines-api.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag pipelines-api.available 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:11 DEBUG juju-log pipelines-api:11: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:11 DEBUG juju-log pipelines-api:11: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:11 INFO juju-log pipelines-api:11: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:11 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {12 {map[pipelines-visualization/0:{0}] map[] []}} 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:11 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:11 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:11 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:11 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:11 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:12 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:12 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:15:12 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:12 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:12 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:12 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:12 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (12; unit: pipelines-visualization/0) hook 2020-09-28 19:15:12 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:12 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:12 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:12 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:12 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-visualization-relation-joined hook 2020-09-28 19:15:12 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:12 INFO juju-log pipelines-visualization:12: Reactive main running for hook pipelines-visualization-relation-joined 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:13 DEBUG juju-log pipelines-visualization:12: tracer> tracer: set flag endpoint.pipelines-visualization.joined tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:13 DEBUG juju-log pipelines-visualization:12: tracer> tracer: starting handler dispatch, 31 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag pipelines-api.available 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:13 DEBUG juju-log pipelines-visualization:12: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:13 DEBUG juju-log pipelines-visualization:12: tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:13 INFO juju-log pipelines-visualization:12: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:13 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:13 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:13 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:14 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:14 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:14 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:14 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:14 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:14 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-visualization-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:15:14 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:14 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:14 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:14 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:14 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (12; unit: pipelines-visualization/0) hook 2020-09-28 19:15:14 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:14 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:14 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:14 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:14 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-visualization-relation-changed hook 2020-09-28 19:15:14 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:15 INFO juju-log pipelines-visualization:12: Reactive main running for hook pipelines-visualization-relation-changed 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:15 DEBUG juju-log pipelines-visualization:12: tracer> tracer: starting handler dispatch, 31 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag pipelines-api.available 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:15 DEBUG juju-log pipelines-visualization:12: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:15 DEBUG juju-log pipelines-visualization:12: tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:16 INFO juju-log pipelines-visualization:12: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:16 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:16 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:16 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:16 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:16 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {12 {map[pipelines-visualization/0:{1}] map[] []}} 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:16 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-visualization-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:15:16 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:16 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:16 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:16 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:16 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (12; unit: pipelines-visualization/0) hook 2020-09-28 19:15:16 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:16 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:16 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:16 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:16 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:17 INFO juju-log pipelines-visualization:12: Reactive main running for hook pipelines-visualization-relation-changed 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:17 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {12 {map[pipelines-visualization/0:{2}] map[] []}} 2020-09-28 19:15:17 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer> tracer: set flag endpoint.pipelines-visualization.changed tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.hostname 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.ingress-address 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.port 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.private-address 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer> tracer: starting handler dispatch, 37 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag pipelines-api.available 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 DEBUG juju-log pipelines-visualization:12: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:18 INFO juju-log pipelines-visualization:12: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:18 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:19 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:19 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:19 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:19 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:19 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:19 DEBUG juju-log pipelines-visualization:12: tracer: set flag pipelines-visualization.available 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:19 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:20 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-visualization-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:15:20 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:20 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:20 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:20 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:20 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (12; unit: pipelines-visualization/0) hook 2020-09-28 19:15:20 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:20 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:20 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:20 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:20 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:20 INFO juju-log pipelines-visualization:12: Reactive main running for hook pipelines-visualization-relation-changed 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:20 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 DEBUG juju-log pipelines-visualization:12: tracer> tracer: starting handler dispatch, 38 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 DEBUG juju-log pipelines-visualization:12: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 DEBUG juju-log pipelines-visualization:12: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 INFO juju-log pipelines-visualization:12: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:21 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:21 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:22 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-visualization-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:15:22 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:15:22 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:15:22 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:22 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:22 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] idle: 2020-09-28 19:15:22 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:29 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:15:29 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:29 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:15:29 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:15:29 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:15:29 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:15:29 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:15:29 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:29 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:30 DEBUG juju-log tracer> tracer: starting handler dispatch, 38 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:15:30 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {11 {map[pipelines-ui/0:{1}] map[] []}} 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:30 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:30 DEBUG juju-log tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:30 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:30 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:30 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:30 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:31 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:31 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:31 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {7 {map[pipelines-db/0:{0}] map[] []}} 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:31 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:15:31 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:15:31 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:15:31 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:31 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:31 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (11; unit: pipelines-ui/0) hook 2020-09-28 19:15:31 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:31 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:31 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:31 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:31 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-changed hook 2020-09-28 19:15:31 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:31 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:31 INFO juju-log pipelines-api:11: Reactive main running for hook pipelines-api-relation-changed 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:32 DEBUG juju-log pipelines-api:11: tracer> tracer: set flag endpoint.pipelines-api.changed tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:32 DEBUG juju-log pipelines-api:11: tracer: set flag endpoint.pipelines-api.changed.egress-subnets 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:32 DEBUG juju-log pipelines-api:11: tracer: set flag endpoint.pipelines-api.changed.ingress-address 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:32 DEBUG juju-log pipelines-api:11: tracer: set flag endpoint.pipelines-api.changed.private-address 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:32 DEBUG juju-log pipelines-api:11: tracer> tracer: starting handler dispatch, 42 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:33 DEBUG juju-log pipelines-api:11: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:33 DEBUG juju-log pipelines-api:11: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:33 INFO juju-log pipelines-api:11: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:33 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:33 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:33 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:33 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:33 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:33 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:34 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:15:35 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:15:36 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:15:36 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:36 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:36 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (7; unit: pipelines-db/0) hook 2020-09-28 19:15:36 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:36 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:36 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:37 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:37 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running mysql-relation-joined hook 2020-09-28 19:15:37 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:37 INFO juju-log mysql:7: Reactive main running for hook mysql-relation-joined 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:37 DEBUG juju-log mysql:7: tracer> tracer: set flag endpoint.mysql.joined tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 DEBUG juju-log mysql:7: tracer> tracer: starting handler dispatch, 43 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 DEBUG juju-log mysql:7: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 DEBUG juju-log mysql:7: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 DEBUG juju-log mysql:7: tracer: set flag mysql.connected 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:38 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:39 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:39 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:39 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:39 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:39 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:39 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:39 INFO juju.worker.uniter.operation runhook.go:142 ran "mysql-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:15:40 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:43 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:43 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:43 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:43 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (7; unit: pipelines-db/0) hook 2020-09-28 19:15:43 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:43 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:43 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:44 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:44 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running mysql-relation-changed hook 2020-09-28 19:15:44 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:44 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:44 INFO juju-log mysql:7: Reactive main running for hook mysql-relation-changed 2020-09-28 19:15:44 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:44 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:44 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:44 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:45 DEBUG juju-log mysql:7: tracer> tracer: starting handler dispatch, 44 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:45 DEBUG juju-log mysql:7: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:45 DEBUG juju-log mysql:7: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:45 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:45 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:45 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:45 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:46 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:46 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:46 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:46 INFO juju.worker.uniter.operation runhook.go:142 ran "mysql-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:15:46 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:47 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:47 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:47 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:47 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] idle: 2020-09-28 19:15:47 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:47 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {7 {map[pipelines-db/0:{2}] map[] []}} 2020-09-28 19:15:47 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:47 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (7; unit: pipelines-db/0) hook 2020-09-28 19:15:47 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:47 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:47 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:47 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:47 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running mysql-relation-changed hook 2020-09-28 19:15:47 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:47 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 INFO juju-log mysql:7: Reactive main running for hook mysql-relation-changed 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer> tracer: set flag endpoint.mysql.changed tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.database 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.egress-subnets 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.host 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.ingress-address 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.password 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.port 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.private-address 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.root_password 2020-09-28 19:15:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:48 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.user 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 DEBUG juju-log mysql:7: tracer> tracer: starting handler dispatch, 54 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 DEBUG juju-log mysql:7: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 DEBUG juju-log mysql:7: tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 DEBUG juju-log mysql:7: tracer> tracer: set flag mysql.available tracer: ++ queue handler reactive/pipelines_api.py:54:start_charm 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 DEBUG juju-log mysql:7: tracer> tracer: set flag mysql.changed tracer: ++ queue handler reactive/pipelines_api.py:36:update_image 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:49 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:49 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:36:update_image 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 DEBUG juju-log mysql:7: tracer> tracer: cleared flag mysql.changed tracer: -- dequeue handler reactive/pipelines_api.py:36:update_image 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:54:start_charm 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "is-leader" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 INFO juju-log mysql:7: status-set: maintenance: configuring container 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "pod-spec-set" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "pod-spec-set" 2020-09-28 19:15:50 DEBUG juju.kubernetes.provider.specs decode.go:83 decoding stream as JSON 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 INFO juju-log mysql:7: status-set: maintenance: creating container 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 DEBUG juju-log mysql:7: tracer> tracer: set flag charm.started tracer: -- dequeue handler reactive/pipelines_api.py:54:start_charm 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:50 INFO juju-log mysql:7: status-set: active: 2020-09-28 19:15:51 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:15:51 INFO juju.worker.uniter.operation runhook.go:142 ran "mysql-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:15:51 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:15:51 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:15:51 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:51 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:51 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] idle: 2020-09-28 19:15:53 DEBUG juju.worker.uniter.remotestate watcher.go:630 got a relation units change: {8 {map[pipelines-persistence/0:{0}] map[] []}} 2020-09-28 19:15:53 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:53 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (8; unit: pipelines-persistence/0) hook 2020-09-28 19:15:53 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:15:53 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:15:53 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:15:53 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:15:53 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-joined hook 2020-09-28 19:15:53 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:53 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:53 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:53 INFO juju-log pipelines-api:8: Reactive main running for hook pipelines-api-relation-joined 2020-09-28 19:15:53 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:53 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:53 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:54 DEBUG juju-log pipelines-api:8: tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:54 DEBUG juju-log pipelines-api:8: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:54 DEBUG juju-log pipelines-api:8: tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:54 INFO juju-log pipelines-api:8: Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:54 INFO juju-log pipelines-api:8: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:54 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:55 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:55 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:55 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:55 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:55 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:55 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:55 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:56 INFO juju-log pipelines-api:8: status-set: active: 2020-09-28 19:15:56 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:15:56 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:15:57 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:15:57 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:15:57 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:15:57 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:15:57 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (8; unit: pipelines-persistence/0) hook 2020-09-28 19:15:57 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:15:57 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:15:57 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:15:57 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:15:57 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-changed hook 2020-09-28 19:15:57 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:15:57 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:15:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:57 INFO juju-log pipelines-api:8: Reactive main running for hook pipelines-api-relation-changed 2020-09-28 19:15:57 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:57 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:57 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:57 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:57 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:58 DEBUG juju-log pipelines-api:8: tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:58 DEBUG juju-log pipelines-api:8: tracer: hooks phase, 0 handlers queued 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:58 DEBUG juju-log pipelines-api:8: tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:58 INFO juju-log pipelines-api:8: Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:58 INFO juju-log pipelines-api:8: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:58 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:59 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:59 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:59 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:59 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:59 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:15:59 INFO juju-log pipelines-api:8: status-set: active: 2020-09-28 19:15:59 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:16:00 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:16:00 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:16:00 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:16:00 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:16:00 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:16:00 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] idle: 2020-09-28 19:16:01 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:16:01 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:16:01 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:16:01 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:16:01 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:16:01 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:16:01 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:16:01 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:01 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:16:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:03 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:03 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:03 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:03 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:03 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:03 INFO juju-log status-set: active: 2020-09-28 19:16:03 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:16:04 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:16:04 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:16:04 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:16:04 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:16:04 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:16:04 DEBUG juju.worker.uniter.remotestate watcher.go:544 got address change: ok=true, hashes=[408f888076b3d5f4a0b2430e4a08cd842ff59a0a97ff6b4528dd92ea8e150db3] 2020-09-28 19:16:04 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:16:04 DEBUG juju.worker.uniter.operation executor.go:83 running operation run config-changed hook 2020-09-28 19:16:04 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run config-changed hook) 2020-09-28 19:16:04 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run config-changed hook) 2020-09-28 19:16:04 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run config-changed hook" 2020-09-28 19:16:04 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:16:04 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run config-changed hook" 2020-09-28 19:16:04 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running config-changed hook 2020-09-28 19:16:05 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:05 INFO juju-log Reactive main running for hook config-changed 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:05 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:05 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:05 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:16:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:05 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:06 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:06 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:06 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:06 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:06 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:06 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:06 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:06 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:06 INFO juju-log status-set: active: 2020-09-28 19:16:06 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:16:07 INFO juju.worker.uniter.operation runhook.go:142 ran "config-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:16:07 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run config-changed hook" 2020-09-28 19:16:07 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run config-changed hook) 2020-09-28 19:16:07 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:16:07 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:16:07 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] idle: 2020-09-28 19:16:11 DEBUG juju.worker.caasoperator caasoperator.go:520 trigger running status for caas unit pipelines-api/0 2020-09-28 19:16:11 DEBUG juju.worker.uniter.remotestate watcher.go:491 got running status change 2020-09-28 19:16:11 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-0" cd3930cb-f085-48b3-869f-0a837deb2f33 2020-09-28 19:16:11 DEBUG juju.kubernetes.provider.exec exec.go:324 no pod named "cd3930cb-f085-48b3-869f-0a837deb2f33" found 2020-09-28 19:16:11 DEBUG juju.kubernetes.provider.exec exec.go:325 try get pod by UID for "cd3930cb-f085-48b3-869f-0a837deb2f33" 2020-09-28 19:16:11 DEBUG juju.worker.uniter.operation executor.go:83 running operation remote init 2020-09-28 19:16:11 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "remote init" 2020-09-28 19:16:12 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "remote init" 2020-09-28 19:16:12 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-0" pipelines-api-6ff7f99877-nnvbf 2020-09-28 19:16:12 DEBUG juju.worker.caasoperator caasoperator.go:659 remote init for "unit-pipelines-api-0" {PodName:pipelines-api-6ff7f99877-nnvbf Initialising:true InitialisingTime:2020-09-28 19:16:04 +0000 UTC Running:false} 2020-09-28 19:16:12 INFO juju.worker.caasoperator initializer.go:124 started pod init on "pipelines-api/0" 2020-09-28 19:16:12 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-6ff7f99877-nnvbf" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/cszdlepx.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-api-0503127813'; "] 2020-09-28 19:16:12 DEBUG juju.kubernetes.provider.exec copy.go:81 copying from {/var/lib/juju/agents/application-pipelines-api/charm } to {/tmp/unit-pipelines-api-0503127813 pipelines-api-6ff7f99877-nnvbf juju-pod-init} 2020-09-28 19:16:12 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-6ff7f99877-nnvbf" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/kxfvauaq.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0503127813'; "] 2020-09-28 19:16:13 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-6ff7f99877-nnvbf" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/lsstvdlh.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0503127813'; "] 2020-09-28 19:16:13 DEBUG juju.kubernetes.provider.exec copy.go:81 copying from {/tmp/unit-pipelines-api-0503127813/ca.crt } to {/tmp/unit-pipelines-api-0503127813/ca.crt pipelines-api-6ff7f99877-nnvbf juju-pod-init} 2020-09-28 19:16:13 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-6ff7f99877-nnvbf" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/ddlvswcs.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0503127813/ca.crt'; "] 2020-09-28 19:16:13 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-6ff7f99877-nnvbf" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/suaklmgm.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0503127813'; "] 2020-09-28 19:16:14 DEBUG juju.worker.caasoperator initializer.go:220 operator service address: 10.152.183.199 2020-09-28 19:16:14 DEBUG juju.kubernetes.provider.exec copy.go:81 copying from {/var/lib/juju/agents/unit-pipelines-api-0/operator-client-cache.yaml } to {/tmp/unit-pipelines-api-0503127813/operator-client-cache.yaml pipelines-api-6ff7f99877-nnvbf juju-pod-init} 2020-09-28 19:16:14 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-6ff7f99877-nnvbf" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/uotpwwyb.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-0503127813/operator-client-cache.yaml'; "] 2020-09-28 19:16:14 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-6ff7f99877-nnvbf" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/oparumlt.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-0503127813'; "] 2020-09-28 19:16:14 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-6ff7f99877-nnvbf" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/olsbhobj.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-api-0 --charm-dir /tmp/unit-pipelines-api-0503127813/charm --send --operator-file /tmp/unit-pipelines-api-0503127813/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-api-0503127813/ca.crt'; "] 2020-09-28 19:16:15 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "remote init" 2020-09-28 19:16:15 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:16:36 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:16:36 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:16:36 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:16:36 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:16:36 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:16:36 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:16:36 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:16:36 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:37 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:37 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:37 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:37 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:16:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:37 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:38 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:38 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:38 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:38 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:38 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:38 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:38 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:16:38 INFO juju-log status-set: active: 2020-09-28 19:16:38 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:16:40 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:16:40 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:16:40 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:16:40 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:16:40 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:17:05 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:17:12 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:17:12 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:17:12 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:17:12 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:17:12 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:17:12 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:17:12 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:17:12 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:12 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:13 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:17:13 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:17:14 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:17:14 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:17:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:14 INFO juju-log status-set: active: 2020-09-28 19:17:14 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:17:14 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:17:14 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:17:15 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:17:15 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:17:15 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:17:35 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:17:36 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:17:36 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:17:36 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:17:36 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:17:36 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:17:36 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:17:36 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:17:36 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:37 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:37 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:37 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:37 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:37 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:37 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:17:37 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:38 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:38 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:38 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:38 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:38 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:38 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:38 INFO juju-log status-set: active: 2020-09-28 19:17:38 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:17:39 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:17:39 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:17:39 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:17:39 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:17:39 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:17:55 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:17:55 DEBUG juju.worker.uniter uniter.go:212 starting uniter for "pipelines-api/1" 2020-09-28 19:17:55 DEBUG juju.worker.caasoperator runner.go:355 start "pipelines-api/1" 2020-09-28 19:17:55 INFO juju.worker.caasoperator runner.go:527 start "pipelines-api/1" 2020-09-28 19:17:55 DEBUG juju.worker.caasoperator runner.go:364 "pipelines-api/1" started 2020-09-28 19:17:55 DEBUG juju.worker.leadership tracker.go:125 pipelines-api/1 making initial claim for pipelines-api leadership 2020-09-28 19:17:55 INFO juju.worker.leadership tracker.go:217 pipelines-api leadership for pipelines-api/1 denied 2020-09-28 19:17:55 DEBUG juju.worker.leadership tracker.go:224 pipelines-api/1 waiting for pipelines-api leadership release 2020-09-28 19:17:55 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-api-1 2020-09-28 19:17:55 DEBUG juju.agent.tools symlinks.go:47 jujuc not found at /var/lib/juju/tools/unit-pipelines-api-1/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-api-1/jujud 2020-09-28 19:17:55 DEBUG juju.agent.tools symlinks.go:50 target tools path /var/lib/juju/tools/unit-pipelines-api-1/jujud 2020-09-28 19:17:55 DEBUG juju.worker.uniter uniter.go:775 starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-api-1/run.socket } 2020-09-28 19:17:55 INFO juju.worker.uniter uniter.go:298 unit "pipelines-api/1" started 2020-09-28 19:17:55 INFO juju.worker.uniter uniter.go:308 resuming charm install 2020-09-28 19:17:55 DEBUG juju.worker.uniter runlistener.go:116 juju-run listener running 2020-09-28 19:17:55 DEBUG juju.worker.uniter.operation executor.go:83 running operation install local:kubernetes/pipelines-api-0 2020-09-28 19:17:55 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "install local:kubernetes/pipelines-api-0" 2020-09-28 19:17:55 INFO juju.worker.uniter.charm bundles.go:79 downloading local:kubernetes/pipelines-api-0 from API server 2020-09-28 19:17:55 INFO juju.downloader download.go:111 downloading from local:kubernetes/pipelines-api-0 2020-09-28 19:17:55 INFO juju.downloader download.go:94 download complete ("local:kubernetes/pipelines-api-0") 2020-09-28 19:17:55 INFO juju.downloader download.go:174 download verified ("local:kubernetes/pipelines-api-0") 2020-09-28 19:17:55 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "install local:kubernetes/pipelines-api-0" 2020-09-28 19:17:55 DEBUG juju.worker.uniter.charm manifest_deployer.go:135 preparing to deploy charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:17:55 DEBUG juju.worker.uniter.charm manifest_deployer.go:124 deploying charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:17:56 DEBUG juju.worker.uniter.charm manifest_deployer.go:156 finishing deploy of charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:17:56 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "install local:kubernetes/pipelines-api-0" 2020-09-28 19:17:56 INFO juju.worker.uniter uniter.go:349 hooks are retried true 2020-09-28 19:17:56 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-1" 28c6a42c-6c21-4e2b-ba77-ac84893ab1cb 2020-09-28 19:17:56 DEBUG juju.kubernetes.provider.exec exec.go:324 no pod named "28c6a42c-6c21-4e2b-ba77-ac84893ab1cb" found 2020-09-28 19:17:56 DEBUG juju.kubernetes.provider.exec exec.go:325 try get pod by UID for "28c6a42c-6c21-4e2b-ba77-ac84893ab1cb" 2020-09-28 19:17:56 DEBUG juju.worker.leadership tracker.go:276 pipelines-api/1 is not pipelines-api leader 2020-09-28 19:17:56 DEBUG juju.worker.uniter.remotestate watcher.go:512 got config change: ok=true, hashes=[3da95c4ad53ef52c4e6d84e4744a140197f8ee389904ab3f4c9eaf6f3ce8e7c0] 2020-09-28 19:17:56 DEBUG juju.worker.uniter.remotestate watcher.go:593 got update status interval change: ok=true 2020-09-28 19:17:56 DEBUG juju.worker.uniter.remotestate watcher.go:523 got trust config change: ok=true, hashes=[7f9884bc998df3cb2761f63f838f8c09c8a7ce0ea62a095fae071ad6e1a23413] 2020-09-28 19:17:56 DEBUG juju.worker.uniter.remotestate watcher.go:555 got leader settings change: ok=true 2020-09-28 19:17:56 DEBUG juju.worker.uniter.remotestate watcher.go:573 got relations change: ok=true 2020-09-28 19:17:57 DEBUG juju.worker.uniter.remotestate watcher.go:565 got action change: [] ok=true 2020-09-28 19:17:57 DEBUG juju.worker.uniter.remotestate watcher.go:544 got address change: ok=true, hashes=[408f888076b3d5f4a0b2430e4a08cd842ff59a0a97ff6b4528dd92ea8e150db3] 2020-09-28 19:17:57 DEBUG juju.worker.uniter.remotestate watcher.go:461 got unit change 2020-09-28 19:17:57 DEBUG juju.worker.uniter.remotestate watcher.go:583 got storage change: [] ok=true 2020-09-28 19:17:57 INFO juju.worker.uniter resolver.go:147 found queued "install" hook 2020-09-28 19:17:57 DEBUG juju.worker.uniter.operation executor.go:83 running operation run install hook 2020-09-28 19:17:57 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run install hook) 2020-09-28 19:17:57 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run install hook) 2020-09-28 19:17:57 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run install hook" 2020-09-28 19:17:57 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run install hook" 2020-09-28 19:17:57 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running install hook 2020-09-28 19:17:57 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:57 INFO juju-log Reactive main running for hook install 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:57 DEBUG juju-log tracer> tracer: starting handler dispatch, 15 flags set tracer: set flag config.changed tracer: set flag config.changed.grpc-port tracer: set flag config.changed.http-port tracer: set flag config.changed.minio-access-key tracer: set flag config.changed.minio-bucket-name tracer: set flag config.changed.minio-secret-key tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:57 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:57 DEBUG juju-log tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/docker_resource.py:7:auto_fetch tracer: ++ queue handler reactive/pipelines_api.py:36:update_image 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:57 INFO juju-log Invoking reactive handler: reactive/docker_resource.py:7:auto_fetch 2020-09-28 19:17:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:57 DEBUG juju-log tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 DEBUG juju-log tracer> tracer: set flag layer.docker-resource.pending tracer: ++ queue handler reactive/docker_resource.py:18:fetch 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 DEBUG juju-log tracer> tracer: set flag layer.docker-resource.auto-fetched tracer: -- dequeue handler reactive/docker_resource.py:7:auto_fetch 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:36:update_image 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 DEBUG juju-log tracer> tracer: cleared flag config.changed tracer: -- dequeue handler reactive/pipelines_api.py:36:update_image 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 INFO juju-log Invoking reactive handler: reactive/docker_resource.py:18:fetch 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 INFO juju-log status-set: maintenance: fetching resource: oci-image 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "resource-get" 2020-09-28 19:17:58 DEBUG juju.worker.caasoperator caasoperator.go:520 trigger running status for caas unit pipelines-api/0 2020-09-28 19:17:58 DEBUG juju.worker.uniter.remotestate watcher.go:491 got running status change 2020-09-28 19:17:58 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-0" cd3930cb-f085-48b3-869f-0a837deb2f33 2020-09-28 19:17:58 DEBUG juju.kubernetes.provider.exec exec.go:324 no pod named "cd3930cb-f085-48b3-869f-0a837deb2f33" found 2020-09-28 19:17:58 DEBUG juju.kubernetes.provider.exec exec.go:325 try get pod by UID for "cd3930cb-f085-48b3-869f-0a837deb2f33" 2020-09-28 19:17:58 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 DEBUG juju-log tracer: set flag layer.docker-resource.oci-image.available 2020-09-28 19:17:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:17:58 DEBUG juju-log tracer> tracer: cleared flag layer.docker-resource.pending tracer: -- dequeue handler reactive/docker_resource.py:18:fetch 2020-09-28 19:17:59 INFO juju.worker.uniter.operation runhook.go:142 ran "install" hook (via explicit, bespoke hook script) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run install hook" 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run install hook) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:pipelines-visualization pipelines-visualization:pipelines-visualization" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:pipelines-visualization pipelines-visualization:pipelines-visualization" 2020-09-28 19:17:59 DEBUG juju.worker.leadership tracker.go:276 pipelines-api/1 is not pipelines-api leader 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:mysql pipelines-db:mysql" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:mysql pipelines-db:mysql" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:kubeflow-profiles kubeflow-profiles:kubeflow-profiles" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:kubeflow-profiles kubeflow-profiles:kubeflow-profiles" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:minio minio:minio" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:minio minio:minio" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-ui:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:17:59 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-ui:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (12; app: pipelines-visualization) hook 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (12; app: pipelines-visualization) hook) 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (12; app: pipelines-visualization) hook) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (12; app: pipelines-visualization) hook" 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (12; app: pipelines-visualization) hook" 2020-09-28 19:17:59 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-visualization-relation-created hook 2020-09-28 19:17:59 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:17:59 INFO juju.worker.uniter.operation runhook.go:145 skipped "pipelines-visualization-relation-created" hook (missing) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (12; app: pipelines-visualization) hook" 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (12; app: pipelines-visualization) hook) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (8; app: pipelines-persistence) hook 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (8; app: pipelines-persistence) hook) 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (8; app: pipelines-persistence) hook) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (8; app: pipelines-persistence) hook" 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (8; app: pipelines-persistence) hook" 2020-09-28 19:17:59 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-created hook 2020-09-28 19:17:59 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:17:59 INFO juju.worker.uniter.operation runhook.go:145 skipped "pipelines-api-relation-created" hook (missing) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (8; app: pipelines-persistence) hook" 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (8; app: pipelines-persistence) hook) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (9; app: kubeflow-profiles) hook 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (9; app: kubeflow-profiles) hook) 2020-09-28 19:17:59 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (9; app: kubeflow-profiles) hook) 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (9; app: kubeflow-profiles) hook" 2020-09-28 19:17:59 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (9; app: kubeflow-profiles) hook" 2020-09-28 19:17:59 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running kubeflow-profiles-relation-created hook 2020-09-28 19:17:59 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:17:59 INFO juju.worker.uniter.operation runhook.go:145 skipped "kubeflow-profiles-relation-created" hook (missing) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (9; app: kubeflow-profiles) hook" 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (9; app: kubeflow-profiles) hook) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (11; app: pipelines-ui) hook 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (11; app: pipelines-ui) hook) 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (11; app: pipelines-ui) hook) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (11; app: pipelines-ui) hook" 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (11; app: pipelines-ui) hook" 2020-09-28 19:18:00 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-created hook 2020-09-28 19:18:00 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:00 INFO juju.worker.uniter.operation runhook.go:145 skipped "pipelines-api-relation-created" hook (missing) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (11; app: pipelines-ui) hook" 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (11; app: pipelines-ui) hook) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (10; app: minio) hook 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (10; app: minio) hook) 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (10; app: minio) hook) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (10; app: minio) hook" 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (10; app: minio) hook" 2020-09-28 19:18:00 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running minio-relation-created hook 2020-09-28 19:18:00 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:00 INFO juju.worker.uniter.operation runhook.go:145 skipped "minio-relation-created" hook (missing) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (10; app: minio) hook" 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (10; app: minio) hook) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-created (7; app: pipelines-db) hook 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-created (7; app: pipelines-db) hook) 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-created (7; app: pipelines-db) hook) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-created (7; app: pipelines-db) hook" 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-created (7; app: pipelines-db) hook" 2020-09-28 19:18:00 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running mysql-relation-created hook 2020-09-28 19:18:00 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:00 INFO juju.worker.uniter.operation runhook.go:145 skipped "mysql-relation-created" hook (missing) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-created (7; app: pipelines-db) hook" 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-created (7; app: pipelines-db) hook) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:83 running operation run leader-settings-changed hook 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run leader-settings-changed hook) 2020-09-28 19:18:00 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run leader-settings-changed hook) 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run leader-settings-changed hook" 2020-09-28 19:18:00 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run leader-settings-changed hook" 2020-09-28 19:18:00 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running leader-settings-changed hook 2020-09-28 19:18:00 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:00 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:01 INFO juju-log Reactive main running for hook leader-settings-changed 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:01 DEBUG juju-log tracer> tracer: starting handler dispatch, 12 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:01 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:01 DEBUG juju-log tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:02 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:02 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:18:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:02 INFO juju.worker.uniter.operation runhook.go:142 ran "leader-settings-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:02 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run leader-settings-changed hook" 2020-09-28 19:18:02 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run leader-settings-changed hook) 2020-09-28 19:18:02 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:02 DEBUG juju.worker.uniter.operation executor.go:83 running operation remote init 2020-09-28 19:18:02 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "remote init" 2020-09-28 19:18:02 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "remote init" 2020-09-28 19:18:02 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-1" pipelines-api-747fcfc785-5j6xj 2020-09-28 19:18:02 DEBUG juju.worker.caasoperator caasoperator.go:659 remote init for "unit-pipelines-api-1" {PodName:pipelines-api-747fcfc785-5j6xj Initialising:true InitialisingTime:2020-09-28 19:17:28 +0000 UTC Running:false} 2020-09-28 19:18:02 INFO juju.worker.caasoperator initializer.go:124 started pod init on "pipelines-api/1" 2020-09-28 19:18:02 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-747fcfc785-5j6xj" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/aszuzgqp.pid; exec sh -c 'mkdir -p /tmp/unit-pipelines-api-1699662804'; "] 2020-09-28 19:18:02 DEBUG juju.kubernetes.provider.exec copy.go:81 copying from {/var/lib/juju/agents/application-pipelines-api/charm } to {/tmp/unit-pipelines-api-1699662804 pipelines-api-747fcfc785-5j6xj juju-pod-init} 2020-09-28 19:18:02 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-747fcfc785-5j6xj" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/qmizkzzj.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-1699662804'; "] 2020-09-28 19:18:03 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-747fcfc785-5j6xj" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/gtstjkbr.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-1699662804'; "] 2020-09-28 19:18:03 DEBUG juju.kubernetes.provider.exec copy.go:81 copying from {/tmp/unit-pipelines-api-1699662804/ca.crt } to {/tmp/unit-pipelines-api-1699662804/ca.crt pipelines-api-747fcfc785-5j6xj juju-pod-init} 2020-09-28 19:18:03 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-747fcfc785-5j6xj" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/yfbxsmai.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-1699662804/ca.crt'; "] 2020-09-28 19:18:03 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-747fcfc785-5j6xj" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/cmwjqeah.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-1699662804'; "] 2020-09-28 19:18:03 DEBUG juju.worker.caasoperator initializer.go:220 operator service address: 10.152.183.199 2020-09-28 19:18:03 DEBUG juju.kubernetes.provider.exec copy.go:81 copying from {/var/lib/juju/agents/unit-pipelines-api-1/operator-client-cache.yaml } to {/tmp/unit-pipelines-api-1699662804/operator-client-cache.yaml pipelines-api-747fcfc785-5j6xj juju-pod-init} 2020-09-28 19:18:03 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-747fcfc785-5j6xj" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/twfvzwjr.pid; exec sh -c 'test -d /tmp/unit-pipelines-api-1699662804/operator-client-cache.yaml'; "] 2020-09-28 19:18:04 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-747fcfc785-5j6xj" for cmd ["sh" "-c" "mkdir -p /tmp; echo $$ > /tmp/rrpnhumw.pid; exec sh -c 'tar -xmf - -C /tmp/unit-pipelines-api-1699662804'; "] 2020-09-28 19:18:04 DEBUG juju.kubernetes.provider.exec exec.go:176 exec on pod "pipelines-api-747fcfc785-5j6xj" for cmd ["sh" "-c" "cd /var/lib/juju; mkdir -p /tmp; echo $$ > /tmp/whhugalr.pid; exec sh -c '/var/lib/juju/tools/jujud caas-unit-init --unit unit-pipelines-api-1 --charm-dir /tmp/unit-pipelines-api-1699662804/charm --send --operator-file /tmp/unit-pipelines-api-1699662804/operator-client-cache.yaml --operator-ca-cert-file /tmp/unit-pipelines-api-1699662804/ca.crt'; "] 2020-09-28 19:18:04 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "remote init" 2020-09-28 19:18:04 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:04 DEBUG juju.worker.uniter.operation executor.go:83 running operation run config-changed hook 2020-09-28 19:18:04 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run config-changed hook) 2020-09-28 19:18:04 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run config-changed hook) 2020-09-28 19:18:04 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run config-changed hook" 2020-09-28 19:18:04 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run config-changed hook" 2020-09-28 19:18:04 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running config-changed hook 2020-09-28 19:18:04 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:04 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:04 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:04 INFO juju-log Reactive main running for hook config-changed 2020-09-28 19:18:04 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:04 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:04 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:04 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:05 DEBUG juju-log tracer> tracer: starting handler dispatch, 12 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:05 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:05 DEBUG juju-log tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:05 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:05 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:05 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:05 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:05 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:05 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:06 INFO juju.worker.uniter.operation runhook.go:142 ran "config-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:06 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run config-changed hook" 2020-09-28 19:18:06 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run config-changed hook) 2020-09-28 19:18:06 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:06 INFO juju.worker.uniter resolver.go:147 found queued "start" hook 2020-09-28 19:18:06 DEBUG juju.worker.uniter.operation executor.go:83 running operation run start hook 2020-09-28 19:18:06 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run start hook) 2020-09-28 19:18:06 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run start hook) 2020-09-28 19:18:06 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run start hook" 2020-09-28 19:18:06 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run start hook" 2020-09-28 19:18:06 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running start hook 2020-09-28 19:18:06 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:06 INFO juju-log Reactive main running for hook start 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:06 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:18:06 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:06 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:18:06 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:06 DEBUG juju-log tracer> tracer: starting handler dispatch, 12 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:06 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:06 DEBUG juju-log tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:15:broken:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:06 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:06 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:07 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:pipelines-api 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:07 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:07 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:07 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:07 INFO juju.worker.uniter.operation runhook.go:142 ran "start" hook (via explicit, bespoke hook script) 2020-09-28 19:18:07 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run start hook" 2020-09-28 19:18:07 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run start hook) 2020-09-28 19:18:07 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:07 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:18:07 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:07 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:18:07 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (8; unit: pipelines-persistence/0) hook 2020-09-28 19:18:07 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:18:07 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:18:07 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-0/agent.socket } 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:07 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:07 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:08 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:08 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:09 INFO juju-log status-set: active: 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:18:09 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:18:09 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:18:09 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:18:09 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:09 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:18:09 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:18:09 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:09 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:18:09 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-joined hook 2020-09-28 19:18:09 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:09 INFO juju-log pipelines-api:8: Reactive main running for hook pipelines-api-relation-joined 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:09 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 DEBUG juju-log pipelines-api:8: tracer> tracer: set flag endpoint.pipelines-api.joined tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 DEBUG juju-log pipelines-api:8: tracer> tracer: starting handler dispatch, 13 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.pipelines-api.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 DEBUG juju-log pipelines-api:8: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 DEBUG juju-log pipelines-api:8: tracer> tracer: main dispatch loop, 5 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 DEBUG juju-log pipelines-api:8: tracer> tracer: set flag pipelines-api.available tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:10 INFO juju-log pipelines-api:8: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:11 DEBUG juju.worker.caasoperator caasoperator.go:520 trigger running status for caas unit pipelines-api/1 2020-09-28 19:18:11 DEBUG juju.worker.uniter.remotestate watcher.go:491 got running status change 2020-09-28 19:18:11 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-1" 28c6a42c-6c21-4e2b-ba77-ac84893ab1cb 2020-09-28 19:18:11 DEBUG juju.kubernetes.provider.exec exec.go:324 no pod named "28c6a42c-6c21-4e2b-ba77-ac84893ab1cb" found 2020-09-28 19:18:11 DEBUG juju.kubernetes.provider.exec exec.go:325 try get pod by UID for "28c6a42c-6c21-4e2b-ba77-ac84893ab1cb" 2020-09-28 19:18:11 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:18:11 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:18:11 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:18:11 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:11 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:11 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (11; unit: pipelines-ui/0) hook 2020-09-28 19:18:11 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (11; unit: pipelines-ui/0) hook) 2020-09-28 19:18:11 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (11; unit: pipelines-ui/0) hook) 2020-09-28 19:18:11 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (11; unit: pipelines-ui/0) hook" 2020-09-28 19:18:11 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (11; unit: pipelines-ui/0) hook" 2020-09-28 19:18:11 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:11 INFO juju-log pipelines-api:11: Reactive main running for hook pipelines-api-relation-joined 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:12 DEBUG juju-log pipelines-api:11: tracer> tracer: set flag endpoint.pipelines-api.changed tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:12 DEBUG juju-log pipelines-api:11: tracer: set flag endpoint.pipelines-api.changed.egress-subnets 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:12 DEBUG juju-log pipelines-api:11: tracer: set flag endpoint.pipelines-api.changed.ingress-address 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:12 DEBUG juju-log pipelines-api:11: tracer: set flag endpoint.pipelines-api.changed.private-address 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:12 DEBUG juju-log pipelines-api:11: tracer> tracer: starting handler dispatch, 18 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag pipelines-api.available 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:12 DEBUG juju-log pipelines-api:11: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:12 DEBUG juju-log pipelines-api:11: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:12 INFO juju-log pipelines-api:11: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:13 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:13 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:13 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:13 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:13 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:13 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:18:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (11; unit: pipelines-ui/0) hook" 2020-09-28 19:18:13 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (11; unit: pipelines-ui/0) hook) 2020-09-28 19:18:13 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:13 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (8; unit: pipelines-persistence/0) hook 2020-09-28 19:18:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:18:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:18:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:18:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:18:13 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-api-relation-changed hook 2020-09-28 19:18:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:13 INFO juju-log pipelines-api:8: Reactive main running for hook pipelines-api-relation-changed 2020-09-28 19:18:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:14 DEBUG juju-log pipelines-api:8: tracer> tracer: starting handler dispatch, 18 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag pipelines-api.available 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:14 DEBUG juju-log pipelines-api:8: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:14 DEBUG juju-log pipelines-api:8: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:14 INFO juju-log pipelines-api:8: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:14 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:14 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:15 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:15 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:15 INFO juju-log pipelines-api:8: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:15 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:15 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (8; unit: pipelines-persistence/0) hook" 2020-09-28 19:18:15 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (8; unit: pipelines-persistence/0) hook) 2020-09-28 19:18:15 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:15 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:15 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (11; unit: pipelines-ui/0) hook 2020-09-28 19:18:15 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:18:15 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:18:15 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:18:15 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:18:15 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:15 INFO juju-log pipelines-api:11: Reactive main running for hook pipelines-api-relation-changed 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 DEBUG juju-log pipelines-api:11: tracer> tracer: starting handler dispatch, 18 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag pipelines-api.available 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 DEBUG juju-log pipelines-api:11: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 DEBUG juju-log pipelines-api:11: tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 INFO juju-log pipelines-api:11: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:16 INFO juju-log pipelines-api:11: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:17 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:17 INFO juju.worker.uniter uniter.go:282 unit "pipelines-api/0" shutting down: catacomb 0xc00053f400 is dying 2020-09-28 19:18:17 DEBUG juju.worker.uniter runlistener.go:129 juju-run listener stopping 2020-09-28 19:18:17 DEBUG juju.worker.uniter runlistener.go:148 juju-run listener stopped 2020-09-28 19:18:17 INFO juju.worker.caasoperator runner.go:558 stopped "pipelines-api/0", err: watcher has been stopped (stopped) 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:370 "pipelines-api/0" done: watcher has been stopped (stopped) 2020-09-28 19:18:17 ERROR juju.worker.caasoperator runner.go:442 exited "pipelines-api/0": watcher has been stopped (stopped) 2020-09-28 19:18:17 INFO juju.worker.caasoperator runner.go:517 restarting "pipelines-api/0" in 3s 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:360 stop "pipelines-api/0" 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:506 killing "pipelines-api/0" 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:259 killing runner 0xc000890900 2020-09-28 19:18:17 DEBUG juju.worker.uniter runlistener.go:129 juju-run listener stopping 2020-09-28 19:18:17 DEBUG juju.worker.uniter runlistener.go:148 juju-run listener stopped 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:370 "pipelines-api/0" done: 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:445 no restart, removing "pipelines-api/0" from known workers 2020-09-28 19:18:17 INFO juju.worker.caasoperator runner.go:349 runner is dying 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:506 killing "pipelines-api/1" 2020-09-28 19:18:17 DEBUG juju.worker.uniter runlistener.go:129 juju-run listener stopping 2020-09-28 19:18:17 DEBUG juju.worker.uniter runlistener.go:148 juju-run listener stopped 2020-09-28 19:18:17 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:17 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:17 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:17 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-api-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:17 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (11; unit: pipelines-ui/0) hook" 2020-09-28 19:18:17 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (11; unit: pipelines-ui/0) hook) 2020-09-28 19:18:17 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:17 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:17 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (12; unit: pipelines-visualization/0) hook 2020-09-28 19:18:17 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:18:17 ERROR juju.worker.uniter agent.go:31 resolver loop error: could not acquire lock: cancelled acquiring mutex 2020-09-28 19:18:17 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] failed: resolver loop error 2020-09-28 19:18:17 INFO juju.worker.uniter uniter.go:282 unit "pipelines-api/1" shutting down: could not acquire lock: cancelled acquiring mutex 2020-09-28 19:18:17 INFO juju.worker.caasoperator runner.go:558 stopped "pipelines-api/1", err: could not acquire lock: cancelled acquiring mutex 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:370 "pipelines-api/1" done: could not acquire lock: cancelled acquiring mutex 2020-09-28 19:18:17 ERROR juju.worker.caasoperator runner.go:442 exited "pipelines-api/1": could not acquire lock: cancelled acquiring mutex 2020-09-28 19:18:17 DEBUG juju.worker.caasoperator runner.go:445 no restart, removing "pipelines-api/1" from known workers 2020-09-28 19:18:17 DEBUG juju.worker.dependency engine.go:598 "operator" manifold worker stopped: permission denied 2020-09-28 19:18:17 ERROR juju.worker.dependency engine.go:671 "operator" manifold worker returned unexpected error: permission denied 2020-09-28 19:18:20 DEBUG juju.worker.dependency engine.go:564 "operator" manifold worker started at 2020-09-28 19:18:20.693847092 +0000 UTC 2020-09-28 19:18:20 DEBUG juju.worker.uniter.charm manifest_deployer.go:135 preparing to deploy charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:18:20 DEBUG juju.worker.uniter.charm manifest_deployer.go:124 deploying charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:18:22 DEBUG juju.worker.uniter.charm manifest_deployer.go:156 finishing deploy of charm "local:kubernetes/pipelines-api-0" 2020-09-28 19:18:22 DEBUG juju.worker.caasoperator caasoperator.go:373 starting caas operator juju-run listener on &{tcp :30666 0xc0009e7800} 2020-09-28 19:18:22 INFO juju.worker.caasoperator caasoperator.go:402 operator "pipelines-api" started 2020-09-28 19:18:22 DEBUG juju.worker.uniter runlistener.go:116 juju-run listener running 2020-09-28 19:18:22 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:18:22 DEBUG juju.worker.uniter uniter.go:212 starting uniter for "pipelines-api/1" 2020-09-28 19:18:22 DEBUG juju.worker.caasoperator runner.go:355 start "pipelines-api/1" 2020-09-28 19:18:22 DEBUG juju.worker.caasoperator caasoperator.go:520 trigger running status for caas unit pipelines-api/1 2020-09-28 19:18:22 INFO juju.worker.caasoperator runner.go:527 start "pipelines-api/1" 2020-09-28 19:18:22 DEBUG juju.worker.caasoperator runner.go:364 "pipelines-api/1" started 2020-09-28 19:18:22 DEBUG juju.worker.leadership tracker.go:125 pipelines-api/1 making initial claim for pipelines-api leadership 2020-09-28 19:18:22 INFO juju.worker.leadership tracker.go:217 pipelines-api leadership for pipelines-api/1 denied 2020-09-28 19:18:22 DEBUG juju.worker.leadership tracker.go:224 pipelines-api/1 waiting for pipelines-api leadership release 2020-09-28 19:18:22 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-pipelines-api-1 2020-09-28 19:18:22 DEBUG juju.agent.tools symlinks.go:47 jujuc not found at /var/lib/juju/tools/unit-pipelines-api-1/jujuc using jujud path /var/lib/juju/tools/unit-pipelines-api-1/jujud 2020-09-28 19:18:22 DEBUG juju.agent.tools symlinks.go:50 target tools path /var/lib/juju/tools/unit-pipelines-api-1/jujud 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-ui:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-ui:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:18:22 DEBUG juju.worker.leadership tracker.go:276 pipelines-api/1 is not pipelines-api leader 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:pipelines-visualization pipelines-visualization:pipelines-visualization" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:pipelines-visualization pipelines-visualization:pipelines-visualization" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:mysql pipelines-db:mysql" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:mysql pipelines-db:mysql" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-persistence:pipelines-api pipelines-api:pipelines-api" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:kubeflow-profiles kubeflow-profiles:kubeflow-profiles" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:kubeflow-profiles kubeflow-profiles:kubeflow-profiles" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:156 joining relation "pipelines-api:minio minio:minio" 2020-09-28 19:18:22 INFO juju.worker.uniter.relation statetracker.go:192 joined relation "pipelines-api:minio minio:minio" 2020-09-28 19:18:22 DEBUG juju.worker.uniter uniter.go:775 starting local juju-run listener on {unix /var/lib/juju/agents/unit-pipelines-api-1/run.socket } 2020-09-28 19:18:22 INFO juju.worker.uniter uniter.go:298 unit "pipelines-api/1" started 2020-09-28 19:18:22 DEBUG juju.worker.uniter runlistener.go:116 juju-run listener running 2020-09-28 19:18:22 INFO juju.worker.uniter uniter.go:349 hooks are retried true 2020-09-28 19:18:22 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:18:22 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-1" 28c6a42c-6c21-4e2b-ba77-ac84893ab1cb 2020-09-28 19:18:22 DEBUG juju.kubernetes.provider.exec exec.go:324 no pod named "28c6a42c-6c21-4e2b-ba77-ac84893ab1cb" found 2020-09-28 19:18:22 DEBUG juju.kubernetes.provider.exec exec.go:325 try get pod by UID for "28c6a42c-6c21-4e2b-ba77-ac84893ab1cb" 2020-09-28 19:18:22 DEBUG juju.worker.leadership tracker.go:276 pipelines-api/1 is not pipelines-api leader 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:573 got relations change: ok=true 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:593 got update status interval change: ok=true 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:512 got config change: ok=true, hashes=[3da95c4ad53ef52c4e6d84e4744a140197f8ee389904ab3f4c9eaf6f3ce8e7c0] 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:555 got leader settings change: ok=true 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:583 got storage change: [] ok=true 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:565 got action change: [] ok=true 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:544 got address change: ok=true, hashes=[408f888076b3d5f4a0b2430e4a08cd842ff59a0a97ff6b4528dd92ea8e150db3] 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:491 got running status change 2020-09-28 19:18:22 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-1" 28c6a42c-6c21-4e2b-ba77-ac84893ab1cb 2020-09-28 19:18:22 DEBUG juju.kubernetes.provider.exec exec.go:324 no pod named "28c6a42c-6c21-4e2b-ba77-ac84893ab1cb" found 2020-09-28 19:18:22 DEBUG juju.kubernetes.provider.exec exec.go:325 try get pod by UID for "28c6a42c-6c21-4e2b-ba77-ac84893ab1cb" 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:461 got unit change 2020-09-28 19:18:22 DEBUG juju.worker.uniter.remotestate watcher.go:523 got trust config change: ok=true, hashes=[7f9884bc998df3cb2761f63f838f8c09c8a7ce0ea62a095fae071ad6e1a23413] 2020-09-28 19:18:22 DEBUG juju.worker.uniter.operation executor.go:83 running operation run leader-settings-changed hook 2020-09-28 19:18:22 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run leader-settings-changed hook) 2020-09-28 19:18:22 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run leader-settings-changed hook) 2020-09-28 19:18:22 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run leader-settings-changed hook" 2020-09-28 19:18:22 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run leader-settings-changed hook" 2020-09-28 19:18:22 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running leader-settings-changed hook 2020-09-28 19:18:22 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:22 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:22 INFO juju-log Reactive main running for hook leader-settings-changed 2020-09-28 19:18:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 DEBUG juju-log tracer> tracer: starting handler dispatch, 18 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag pipelines-api.available 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 DEBUG juju-log tracer> tracer: main dispatch loop, 6 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:35:_handle_broken:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:35:_handle_broken:mysql 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:24 INFO juju.worker.uniter.operation runhook.go:142 ran "leader-settings-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run leader-settings-changed hook" 2020-09-28 19:18:24 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run leader-settings-changed hook) 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:83 running operation remote init 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "remote init" 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "remote init" 2020-09-28 19:18:24 DEBUG juju.worker.caasoperator caasoperator.go:615 request running status for "unit-pipelines-api-1" pipelines-api-747fcfc785-5j6xj 2020-09-28 19:18:24 DEBUG juju.worker.caasoperator caasoperator.go:659 remote init for "unit-pipelines-api-1" {PodName:pipelines-api-747fcfc785-5j6xj Initialising:false InitialisingTime:2020-09-28 19:17:28 +0000 UTC Running:true} 2020-09-28 19:18:24 DEBUG juju.worker.caasoperator caasoperator.go:676 no need to do remote-init for a running container 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "remote init" 2020-09-28 19:18:24 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (7; unit: pipelines-db/0) hook 2020-09-28 19:18:24 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (7; unit: pipelines-db/0) hook) 2020-09-28 19:18:24 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (7; unit: pipelines-db/0) hook) 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (7; unit: pipelines-db/0) hook" 2020-09-28 19:18:24 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (7; unit: pipelines-db/0) hook" 2020-09-28 19:18:24 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running mysql-relation-joined hook 2020-09-28 19:18:24 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:24 INFO juju-log mysql:7: Reactive main running for hook mysql-relation-joined 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:24 DEBUG juju.worker.caasoperator.remotestate watcher.go:137 got application change 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:24 DEBUG juju-log mysql:7: tracer> tracer: set flag endpoint.mysql.joined tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:24 DEBUG juju-log mysql:7: tracer> tracer: set flag endpoint.mysql.changed tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:24 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:24 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.database 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.egress-subnets 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.host 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.ingress-address 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.password 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.port 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.private-address 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.root_password 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: set flag endpoint.mysql.changed.user 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer> tracer: starting handler dispatch, 29 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag pipelines-api.available 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 DEBUG juju-log mysql:7: tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:25 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 DEBUG juju-log mysql:7: tracer: set flag mysql.connected 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 DEBUG juju-log mysql:7: tracer: set flag mysql.available 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 DEBUG juju-log mysql:7: tracer> tracer: set flag mysql.changed tracer: ++ queue handler reactive/pipelines_api.py:36:update_image 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:36:update_image 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:26 DEBUG juju-log mysql:7: tracer> tracer: cleared flag mysql.changed tracer: -- dequeue handler reactive/pipelines_api.py:36:update_image 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:26 INFO juju.worker.uniter.operation runhook.go:142 ran "mysql-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:18:26 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (7; unit: pipelines-db/0) hook" 2020-09-28 19:18:26 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (7; unit: pipelines-db/0) hook) 2020-09-28 19:18:26 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:26 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:26 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (7; unit: pipelines-db/0) hook 2020-09-28 19:18:26 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:18:26 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:18:26 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:18:26 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:18:26 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running mysql-relation-changed hook 2020-09-28 19:18:26 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:27 INFO juju-log mysql:7: Reactive main running for hook mysql-relation-changed 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:27 DEBUG juju-log mysql:7: tracer> tracer: starting handler dispatch, 31 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:27 DEBUG juju-log mysql:7: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:27 DEBUG juju-log mysql:7: tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:27 INFO juju-log mysql:7: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:27 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:28 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:28 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:28 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:28 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:28 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:28 INFO juju-log mysql:7: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:pipelines-visualization 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:28 INFO juju.worker.uniter.operation runhook.go:142 ran "mysql-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:28 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (7; unit: pipelines-db/0) hook" 2020-09-28 19:18:28 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (7; unit: pipelines-db/0) hook) 2020-09-28 19:18:28 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:28 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:28 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (12; unit: pipelines-visualization/0) hook 2020-09-28 19:18:28 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:18:28 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:18:28 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:18:28 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:18:28 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-visualization-relation-joined hook 2020-09-28 19:18:28 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:28 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 INFO juju-log pipelines-visualization:12: Reactive main running for hook pipelines-visualization-relation-joined 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 DEBUG juju-log pipelines-visualization:12: tracer> tracer: set flag endpoint.pipelines-visualization.joined tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 DEBUG juju-log pipelines-visualization:12: tracer> tracer: set flag endpoint.pipelines-visualization.changed tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.hostname 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.ingress-address 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.port 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 DEBUG juju-log pipelines-visualization:12: tracer: set flag endpoint.pipelines-visualization.changed.private-address 2020-09-28 19:18:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:29 DEBUG juju-log pipelines-visualization:12: tracer> tracer: starting handler dispatch, 38 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 DEBUG juju-log pipelines-visualization:12: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 DEBUG juju-log pipelines-visualization:12: tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 INFO juju-log pipelines-visualization:12: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 DEBUG juju-log pipelines-visualization:12: tracer: set flag pipelines-visualization.available 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:30 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:31 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-visualization-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:18:31 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:18:31 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:18:31 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:31 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:31 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (12; unit: pipelines-visualization/0) hook 2020-09-28 19:18:31 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:18:31 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:18:31 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:18:31 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:18:31 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running pipelines-visualization-relation-changed hook 2020-09-28 19:18:31 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:31 INFO juju-log pipelines-visualization:12: Reactive main running for hook pipelines-visualization-relation-changed 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:31 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 DEBUG juju-log pipelines-visualization:12: tracer> tracer: starting handler dispatch, 39 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 DEBUG juju-log pipelines-visualization:12: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 DEBUG juju-log pipelines-visualization:12: tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 INFO juju-log pipelines-visualization:12: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:kubeflow-profiles 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:32 INFO juju-log pipelines-visualization:12: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:32 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:33 INFO juju.worker.uniter.operation runhook.go:142 ran "pipelines-visualization-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:33 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (12; unit: pipelines-visualization/0) hook" 2020-09-28 19:18:33 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (12; unit: pipelines-visualization/0) hook) 2020-09-28 19:18:33 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:33 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:33 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (9; unit: kubeflow-profiles/0) hook 2020-09-28 19:18:33 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:18:33 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:18:33 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:18:33 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:18:33 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running kubeflow-profiles-relation-joined hook 2020-09-28 19:18:33 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:33 INFO juju-log kubeflow-profiles:9: Reactive main running for hook kubeflow-profiles-relation-joined 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:33 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: set flag endpoint.kubeflow-profiles.joined tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:33 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: set flag endpoint.kubeflow-profiles.changed tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:33 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:33 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.hostname 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:33 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:33 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.port 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:33 DEBUG juju-log kubeflow-profiles:9: tracer: set flag endpoint.kubeflow-profiles.changed.private-address 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:33 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: starting handler dispatch, 46 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 DEBUG juju-log kubeflow-profiles:9: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 DEBUG juju-log kubeflow-profiles:9: tracer: set flag kubeflow-profiles.available 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:34 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:34 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:35 INFO juju.worker.uniter.operation runhook.go:142 ran "kubeflow-profiles-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:18:35 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:18:35 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:18:35 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:35 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:35 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (9; unit: kubeflow-profiles/0) hook 2020-09-28 19:18:35 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:18:35 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:18:35 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:18:35 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:18:35 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running kubeflow-profiles-relation-changed hook 2020-09-28 19:18:35 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:35 INFO juju-log kubeflow-profiles:9: Reactive main running for hook kubeflow-profiles-relation-changed 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:35 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: starting handler dispatch, 47 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 DEBUG juju-log kubeflow-profiles:9: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 DEBUG juju-log kubeflow-profiles:9: tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/http/requires.py:15:broken:minio tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:36 INFO juju-log kubeflow-profiles:9: Invoking reactive handler: hooks/relations/http/requires.py:15:broken:minio 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:36 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:37 INFO juju.worker.uniter.operation runhook.go:142 ran "kubeflow-profiles-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:37 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (9; unit: kubeflow-profiles/0) hook" 2020-09-28 19:18:37 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (9; unit: kubeflow-profiles/0) hook) 2020-09-28 19:18:37 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:37 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:37 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-joined (10; unit: minio/0) hook 2020-09-28 19:18:37 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-joined (10; unit: minio/0) hook) 2020-09-28 19:18:37 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-joined (10; unit: minio/0) hook) 2020-09-28 19:18:37 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-joined (10; unit: minio/0) hook" 2020-09-28 19:18:37 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-joined (10; unit: minio/0) hook" 2020-09-28 19:18:37 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running minio-relation-joined hook 2020-09-28 19:18:37 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:37 INFO juju-log minio:10: Reactive main running for hook minio-relation-joined 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:37 DEBUG juju-log minio:10: tracer> tracer: set flag endpoint.minio.joined tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:37 DEBUG juju-log minio:10: tracer> tracer: set flag endpoint.minio.changed tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:18:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:37 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.egress-subnets 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.hostname 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.ingress-address 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.port 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 DEBUG juju-log minio:10: tracer: set flag endpoint.minio.changed.private-address 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 DEBUG juju-log minio:10: tracer> tracer: starting handler dispatch, 54 flags set tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 DEBUG juju-log minio:10: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 DEBUG juju-log minio:10: tracer> tracer: main dispatch loop, 7 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 INFO juju-log minio:10: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:38 INFO juju.worker.leadership tracker.go:194 pipelines-api/1 promoted to leadership of pipelines-api 2020-09-28 19:18:38 DEBUG juju.worker.uniter.remotestate watcher.go:620 got leadership change for pipelines-api/1: leader 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:38 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:39 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:39 DEBUG juju-log minio:10: tracer> tracer: set flag minio.available tracer: ++ queue handler reactive/pipelines_api.py:54:start_charm 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:39 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:39 INFO juju-log minio:10: Invoking reactive handler: reactive/pipelines_api.py:54:start_charm 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "is-leader" 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:39 INFO juju-log minio:10: status-set: maintenance: configuring container 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "pod-spec-set" 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "pod-spec-set" 2020-09-28 19:18:39 DEBUG juju.kubernetes.provider.specs decode.go:83 decoding stream as JSON 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:39 INFO juju-log minio:10: status-set: maintenance: creating container 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:39 DEBUG juju-log minio:10: tracer> tracer: set flag charm.started tracer: -- dequeue handler reactive/pipelines_api.py:54:start_charm 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:39 INFO juju-log minio:10: Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:39 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:40 INFO juju-log minio:10: status-set: active: 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:18:40 INFO juju.worker.uniter.operation runhook.go:142 ran "minio-relation-joined" hook (via explicit, bespoke hook script) 2020-09-28 19:18:40 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-joined (10; unit: minio/0) hook" 2020-09-28 19:18:40 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-joined (10; unit: minio/0) hook) 2020-09-28 19:18:40 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:40 DEBUG juju.worker.uniter.operation executor.go:83 running operation accept leadership 2020-09-28 19:18:40 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "accept leadership" 2020-09-28 19:18:40 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "accept leadership" 2020-09-28 19:18:40 INFO juju.worker.uniter resolver.go:147 found queued "leader-elected" hook 2020-09-28 19:18:40 DEBUG juju.worker.uniter.operation executor.go:83 running operation run leader-elected hook 2020-09-28 19:18:40 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run leader-elected hook) 2020-09-28 19:18:40 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run leader-elected hook) 2020-09-28 19:18:40 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run leader-elected hook" 2020-09-28 19:18:40 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run leader-elected hook" 2020-09-28 19:18:40 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running leader-elected hook 2020-09-28 19:18:40 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:40 INFO juju-log Reactive main running for hook leader-elected 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:41 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:18:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:42 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:42 INFO juju-log status-set: active: 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:18:42 INFO juju.worker.uniter.operation runhook.go:142 ran "leader-elected" hook (via explicit, bespoke hook script) 2020-09-28 19:18:42 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run leader-elected hook" 2020-09-28 19:18:42 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run leader-elected hook) 2020-09-28 19:18:42 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:42 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:42 DEBUG juju.worker.uniter.operation executor.go:83 running operation run relation-changed (10; unit: minio/0) hook 2020-09-28 19:18:42 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run relation-changed (10; unit: minio/0) hook) 2020-09-28 19:18:42 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run relation-changed (10; unit: minio/0) hook) 2020-09-28 19:18:42 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run relation-changed (10; unit: minio/0) hook" 2020-09-28 19:18:42 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run relation-changed (10; unit: minio/0) hook" 2020-09-28 19:18:42 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] executing: running minio-relation-changed hook 2020-09-28 19:18:42 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:42 INFO juju-log minio:10: Reactive main running for hook minio-relation-changed 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:42 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 DEBUG juju-log minio:10: tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 DEBUG juju-log minio:10: tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 DEBUG juju-log minio:10: tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 INFO juju-log minio:10: Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 INFO juju-log minio:10: Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:43 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:44 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:44 INFO juju-log minio:10: Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:44 INFO juju-log minio:10: status-set: active: 2020-09-28 19:18:44 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:18:44 INFO juju.worker.uniter.operation runhook.go:142 ran "minio-relation-changed" hook (via explicit, bespoke hook script) 2020-09-28 19:18:44 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run relation-changed (10; unit: minio/0) hook" 2020-09-28 19:18:44 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run relation-changed (10; unit: minio/0) hook) 2020-09-28 19:18:44 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:44 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:44 DEBUG juju.worker.uniter agent.go:20 [AGENT-STATUS] idle: 2020-09-28 19:18:44 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:50 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:18:50 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:18:50 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:18:50 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:18:50 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:18:50 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:18:50 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:18:50 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:50 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:50 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:18:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:50 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:18:51 INFO juju-log status-set: active: 2020-09-28 19:18:51 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:18:52 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:18:52 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:18:52 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:18:52 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:18:52 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:19:21 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:19:21 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:19:21 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:19:21 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:19:21 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:19:21 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:19:21 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:19:21 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:19:21 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:22 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:22 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:22 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:22 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:22 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:22 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:19:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:23 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:23 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:23 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:23 INFO juju-log status-set: active: 2020-09-28 19:19:23 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:19:24 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:19:24 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:19:24 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:19:24 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:19:24 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:19:56 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:19:56 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:19:56 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:19:56 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:19:56 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:19:56 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:19:56 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:19:56 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:56 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:56 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:57 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:19:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:19:58 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:19:58 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:19:58 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:19:58 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:19:58 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:19:58 INFO juju-log status-set: active: 2020-09-28 19:19:58 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:19:58 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:19:58 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:19:58 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:19:58 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:19:58 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:20:29 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:20:29 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:20:29 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:20:29 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:20:29 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:20:29 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:20:29 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:20:29 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:29 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:29 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:29 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:29 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:20:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:29 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:30 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:30 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:30 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:30 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:30 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:30 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:30 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:20:30 INFO juju-log status-set: active: 2020-09-28 19:20:30 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:20:31 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:20:31 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:20:31 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:20:31 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:20:31 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:21:02 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:21:02 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:21:02 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:21:02 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:21:02 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:21:02 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:21:02 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:21:02 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:03 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:03 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:03 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:03 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:03 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:03 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:03 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:04 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:04 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:04 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:04 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:04 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:04 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:04 INFO juju-log status-set: active: 2020-09-28 19:21:04 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:21:04 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:21:05 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:21:05 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:21:05 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:21:05 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:21:33 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:21:33 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:21:33 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:21:33 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:21:33 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:21:33 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:21:33 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:21:33 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:34 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:34 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:34 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:34 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:34 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:34 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:21:34 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:35 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:35 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:35 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:35 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:35 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:35 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:21:35 INFO juju-log status-set: active: 2020-09-28 19:21:35 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:21:36 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:21:36 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:21:36 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:21:36 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:21:36 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:22:00 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:22:00 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:22:00 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:22:00 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:22:00 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:22:00 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:22:00 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:22:00 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:00 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:00 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:01 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:01 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:02 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:02 INFO juju-log status-set: active: 2020-09-28 19:22:02 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:22:02 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:22:02 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:22:02 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:22:02 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:22:02 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:22:24 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:22:24 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:22:24 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:22:24 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:22:24 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:22:24 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:22:24 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:22:24 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:25 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:25 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:25 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:25 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:25 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:25 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:22:25 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:26 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:26 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:26 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:26 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:26 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:26 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:26 INFO juju-log status-set: active: 2020-09-28 19:22:26 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:22:27 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:22:27 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:22:27 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:22:27 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:22:27 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:22:50 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:22:50 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:22:50 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:22:50 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:22:50 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:22:50 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:22:50 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:22:50 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:22:50 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:22:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:50 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:22:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:51 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:52 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:22:52 INFO juju-log status-set: active: 2020-09-28 19:22:52 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:22:52 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:22:52 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:22:52 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:22:52 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:22:52 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:23:15 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:23:15 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:23:15 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:23:15 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:23:15 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:23:15 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:23:15 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:23:15 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:15 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:16 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:23:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:23:17 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:23:17 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:23:17 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:17 INFO juju-log status-set: active: 2020-09-28 19:23:17 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:23:17 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:23:17 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:23:17 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:23:17 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:23:17 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:23:43 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:23:43 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:23:43 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:23:43 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:23:43 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:23:43 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:23:43 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:23:43 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:23:43 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:23:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:43 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:23:43 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:43 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:43 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:43 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:44 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:45 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:23:45 INFO juju-log status-set: active: 2020-09-28 19:23:45 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:23:45 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:23:45 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:23:45 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:23:45 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:23:45 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:24:13 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:24:13 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:24:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:24:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:24:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:24:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:24:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:24:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:13 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:13 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:14 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:24:14 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:24:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:24:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:24:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:15 INFO juju-log status-set: active: 2020-09-28 19:24:15 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:24:15 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:24:15 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:24:15 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:24:15 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:24:15 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:24:47 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:24:47 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:24:47 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:24:47 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:24:47 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:24:47 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:24:47 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:24:47 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:24:47 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:24:47 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:47 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:24:47 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:47 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:47 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:47 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:48 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:48 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:49 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:24:49 INFO juju-log status-set: active: 2020-09-28 19:24:49 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:24:49 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:24:49 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:24:49 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:24:49 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:24:49 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:25:15 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:25:15 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:25:15 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:25:15 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:25:15 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:25:15 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:25:15 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:25:15 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:15 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:15 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:15 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:15 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:15 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:25:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:15 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:16 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:16 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:16 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:16 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:16 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:16 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:16 INFO juju-log status-set: active: 2020-09-28 19:25:16 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:25:17 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:25:17 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:25:17 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:25:17 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:25:17 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:25:42 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:25:42 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:25:42 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:25:42 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:25:42 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:25:42 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:25:42 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:25:42 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:42 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:42 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:43 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:25:43 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:25:44 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:25:44 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:25:44 INFO juju-log status-set: active: 2020-09-28 19:25:44 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:25:44 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:25:44 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:25:44 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:25:44 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:25:44 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:26:09 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:26:09 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:26:09 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:26:09 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:26:09 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:26:09 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:26:09 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:26:09 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:09 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:09 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:09 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:09 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:26:09 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:09 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:10 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:10 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:10 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:10 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:10 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:10 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:10 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:10 INFO juju-log status-set: active: 2020-09-28 19:26:10 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:26:11 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:26:11 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:26:11 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:26:11 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:26:11 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:26:35 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:26:35 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:26:35 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:26:35 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:26:35 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:26:35 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:26:35 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:26:35 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:26:35 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:26:35 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:35 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:26:35 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:36 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:36 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:36 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:36 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:36 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:36 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:36 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:36 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:36 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:37 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:37 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:37 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:26:37 INFO juju-log status-set: active: 2020-09-28 19:26:37 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:26:37 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:26:37 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:26:37 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:26:37 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:26:37 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:27:11 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:27:11 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:27:11 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:27:11 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:27:11 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:27:11 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:27:11 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:27:11 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:11 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:11 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:11 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:27:11 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:11 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:12 INFO juju-log status-set: active: 2020-09-28 19:27:12 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:27:13 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:27:13 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:27:13 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:27:13 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:27:13 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:27:38 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:27:38 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:27:38 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:27:38 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:27:38 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:27:38 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:27:38 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:27:38 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:27:38 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:27:38 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:38 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:27:38 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:38 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:38 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:38 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:39 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:27:39 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:40 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:40 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:27:40 INFO juju-log status-set: active: 2020-09-28 19:27:40 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:27:40 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:27:40 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:27:40 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:27:40 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:27:40 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:28:13 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:28:13 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:28:13 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:28:13 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:28:13 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:28:13 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:28:13 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:28:13 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:28:13 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:28:13 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:13 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:28:13 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:14 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:14 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:14 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:14 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:14 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:14 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:14 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:28:14 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:15 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:15 INFO juju-log status-set: active: 2020-09-28 19:28:15 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:28:15 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:28:15 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:28:15 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:28:15 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:28:15 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:28:40 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:28:40 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:28:40 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:28:40 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:28:40 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:28:40 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:28:40 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:28:40 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:40 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:28:40 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:40 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:28:41 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:28:41 INFO juju-log status-set: active: 2020-09-28 19:28:42 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:28:42 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:28:42 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:28:42 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:28:42 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:28:42 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:29:14 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:29:14 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:29:14 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:29:14 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:29:14 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:29:14 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:29:14 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:29:14 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:14 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:14 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:14 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:29:14 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:14 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:15 INFO juju-log status-set: active: 2020-09-28 19:29:15 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:29:16 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:29:16 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:29:16 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:29:16 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:29:16 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:29:45 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:29:45 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:29:45 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:29:45 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:29:45 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:29:45 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:29:45 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:29:45 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:45 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:45 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:46 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:29:46 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:29:47 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:29:47 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:29:47 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:29:47 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:29:47 INFO juju-log status-set: active: 2020-09-28 19:29:47 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:29:47 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:29:47 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:29:47 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:29:47 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:29:47 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:30:21 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:30:21 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:30:21 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:30:21 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:30:21 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:30:21 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:30:21 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:30:21 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:21 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:21 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:30:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:21 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:22 INFO juju-log status-set: active: 2020-09-28 19:30:22 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:30:23 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:30:23 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:30:23 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:30:23 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:30:23 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:30:55 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:30:55 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:30:55 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:30:55 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:30:55 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:30:55 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:30:55 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:30:55 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:30:55 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:30:55 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:55 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:30:55 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:55 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:55 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:55 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:56 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:30:56 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:57 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:57 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:30:57 INFO juju-log status-set: active: 2020-09-28 19:30:57 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:30:57 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:30:57 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:30:57 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:30:57 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:30:57 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:31:28 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:31:28 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:31:28 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:31:28 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:31:28 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:31:28 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:31:28 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:31:28 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:28 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:31:28 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:29 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:31:29 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:31:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:31:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:31:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:31:30 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:31:30 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:31:30 INFO juju-log status-set: active: 2020-09-28 19:31:30 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:31:30 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:31:30 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:31:30 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:31:30 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:31:30 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:32:00 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:32:00 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:32:00 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:32:00 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:32:00 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:32:00 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:32:00 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:32:00 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:32:00 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:32:00 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:00 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:01 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:01 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:01 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:01 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:01 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:01 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:01 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:32:01 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:02 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:02 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:02 INFO juju-log status-set: active: 2020-09-28 19:32:02 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:32:02 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:32:02 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:32:02 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:32:02 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:32:02 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:32:24 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:32:24 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:32:24 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:32:24 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:32:24 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:32:24 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:32:24 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:32:24 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:25 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:25 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:25 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:25 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:25 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:25 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:32:25 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:26 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:26 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:26 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:26 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:26 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:26 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:26 INFO juju-log status-set: active: 2020-09-28 19:32:26 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:32:27 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:32:27 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:32:27 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:32:27 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:32:27 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:32:50 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:32:50 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:32:50 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:32:50 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:32:50 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:32:50 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:32:50 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:32:50 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:50 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:50 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:51 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:51 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:52 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:52 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:32:52 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:32:52 INFO juju-log status-set: active: 2020-09-28 19:32:52 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:32:52 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:32:52 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:32:52 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:32:52 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:32:52 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:33:21 DEBUG juju.worker.uniter.remotestate watcher.go:636 update status timer triggered 2020-09-28 19:33:21 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes 2020-09-28 19:33:21 DEBUG juju.worker.uniter.operation executor.go:83 running operation run update-status hook 2020-09-28 19:33:21 DEBUG juju.machinelock machinelock.go:162 acquire machine lock for uniter (run update-status hook) 2020-09-28 19:33:21 DEBUG juju.machinelock machinelock.go:172 machine lock acquired for uniter (run update-status hook) 2020-09-28 19:33:21 DEBUG juju.worker.uniter.operation executor.go:130 preparing operation "run update-status hook" 2020-09-28 19:33:21 DEBUG juju.worker.uniter.operation executor.go:130 executing operation "run update-status hook" 2020-09-28 19:33:21 DEBUG juju.worker.uniter.runner runner.go:705 starting jujuc server {unix @/var/lib/juju/agents/unit-pipelines-api-1/agent.socket } 2020-09-28 19:33:21 DEBUG jujuc server.go:211 running hook tool "config-get" 2020-09-28 19:33:21 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:21 INFO juju-log Reactive main running for hook update-status 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-ids" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-list" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:22 DEBUG juju-log tracer> tracer: starting handler dispatch, 56 flags set tracer: set flag charm.started tracer: set flag config.default.grpc-port tracer: set flag config.default.http-port tracer: set flag config.default.minio-access-key tracer: set flag config.default.minio-bucket-name tracer: set flag config.set.grpc-port tracer: set flag config.set.http-port tracer: set flag config.set.minio-access-key tracer: set flag config.set.minio-bucket-name tracer: set flag config.set.minio-secret-key tracer: set flag endpoint.kubeflow-profiles.changed tracer: set flag endpoint.kubeflow-profiles.changed.egress-subnets tracer: set flag endpoint.kubeflow-profiles.changed.hostname tracer: set flag endpoint.kubeflow-profiles.changed.ingress-address tracer: set flag endpoint.kubeflow-profiles.changed.port tracer: set flag endpoint.kubeflow-profiles.changed.private-address tracer: set flag endpoint.kubeflow-profiles.joined tracer: set flag endpoint.minio.changed tracer: set flag endpoint.minio.changed.egress-subnets tracer: set flag endpoint.minio.changed.hostname tracer: set flag endpoint.minio.changed.ingress-address tracer: set flag endpoint.minio.changed.port tracer: set flag endpoint.minio.changed.private-address tracer: set flag endpoint.minio.joined tracer: set flag endpoint.mysql.changed tracer: set flag endpoint.mysql.changed.database tracer: set flag endpoint.mysql.changed.egress-subnets tracer: set flag endpoint.mysql.changed.host tracer: set flag endpoint.mysql.changed.ingress-address tracer: set flag endpoint.mysql.changed.password tracer: set flag endpoint.mysql.changed.port tracer: set flag endpoint.mysql.changed.private-address tracer: set flag endpoint.mysql.changed.root_password tracer: set flag endpoint.mysql.changed.user tracer: set flag endpoint.mysql.joined tracer: set flag endpoint.pipelines-api.changed tracer: set flag endpoint.pipelines-api.changed.egress-subnets tracer: set flag endpoint.pipelines-api.changed.ingress-address tracer: set flag endpoint.pipelines-api.changed.private-address tracer: set flag endpoint.pipelines-api.joined tracer: set flag endpoint.pipelines-visualization.changed tracer: set flag endpoint.pipelines-visualization.changed.egress-subnets tracer: set flag endpoint.pipelines-visualization.changed.hostname tracer: set flag endpoint.pipelines-visualization.changed.ingress-address tracer: set flag endpoint.pipelines-visualization.changed.port tracer: set flag endpoint.pipelines-visualization.changed.private-address tracer: set flag endpoint.pipelines-visualization.joined tracer: set flag kubeflow-profiles.available tracer: set flag layer.docker-resource.auto-fetched tracer: set flag layer.docker-resource.oci-image.available tracer: set flag layer.docker-resource.oci-image.fetched tracer: set flag minio.available tracer: set flag mysql.available tracer: set flag mysql.connected tracer: set flag pipelines-api.available tracer: set flag pipelines-visualization.available 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:22 DEBUG juju-log tracer: hooks phase, 0 handlers queued 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:22 DEBUG juju-log tracer> tracer: main dispatch loop, 8 handlers queued tracer: ++ queue handler hooks/relations/http/provides.py:11:joined:pipelines-api tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:kubeflow-profiles tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:minio tracer: ++ queue handler hooks/relations/http/requires.py:10:changed:pipelines-visualization tracer: ++ queue handler hooks/relations/mysql/requires.py:21:_handle_joined:mysql tracer: ++ queue handler hooks/relations/mysql/requires.py:26:_handle_changed:mysql tracer: ++ queue handler reactive/pipelines_api.py:26:charm_ready tracer: ++ queue handler reactive/pipelines_api.py:31:configure_http 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:22 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:26:charm_ready 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:22 INFO juju-log Invoking reactive handler: reactive/pipelines_api.py:31:configure_http 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:22 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:21:_handle_joined:mysql 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:22 INFO juju-log Invoking reactive handler: hooks/relations/mysql/requires.py:26:_handle_changed:mysql 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:33:22 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:22 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:11:joined:pipelines-api 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:kubeflow-profiles 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:minio 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:23 INFO juju-log Invoking reactive handler: hooks/relations/http/requires.py:10:changed:pipelines-visualization 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "relation-set" 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-09-28 19:33:23 INFO juju-log status-set: active: 2020-09-28 19:33:23 DEBUG jujuc server.go:211 running hook tool "status-set" 2020-09-28 19:33:23 INFO juju.worker.uniter.operation runhook.go:142 ran "update-status" hook (via explicit, bespoke hook script) 2020-09-28 19:33:23 DEBUG juju.worker.uniter.operation executor.go:130 committing operation "run update-status hook" 2020-09-28 19:33:23 DEBUG juju.machinelock machinelock.go:186 machine lock released for uniter (run update-status hook) 2020-09-28 19:33:23 DEBUG juju.worker.uniter.operation executor.go:119 lock released 2020-09-28 19:33:23 DEBUG juju.worker.uniter resolver.go:170 no operations in progress; waiting for changes