################
Running on Maas 3.4/stable, deploying via lxd (5.19) kvm
################
command:
juju deploy -n 3 ceph-osd --storage osd-devices=32G,2 --storage osd-journals=8G,1
################
debug log:
unit-ceph-osd-0: 12:17:49 INFO juju Starting unit workers for "ceph-osd/0"
unit-ceph-osd-0: 12:17:49 INFO juju.worker.apicaller [18406b] "unit-ceph-osd-0" successfully connected to "192.168.2.216:17070"
unit-ceph-osd-0: 12:17:49 INFO juju.worker.apicaller [18406b] password changed for "unit-ceph-osd-0"
unit-ceph-osd-0: 12:17:49 INFO juju.worker.apicaller [18406b] "unit-ceph-osd-0" successfully connected to "192.168.2.216:17070"
unit-ceph-osd-0: 12:17:49 INFO juju.worker.upgrader no waiter, upgrader is done
unit-ceph-osd-0: 12:17:49 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-ceph-osd-0: 12:17:49 INFO juju.worker.logger logger worker started
unit-ceph-osd-0: 12:17:49 INFO juju.worker.uniter unit "ceph-osd/0" started
unit-ceph-osd-0: 12:17:49 INFO juju.worker.uniter resuming charm install
unit-ceph-osd-0: 12:17:49 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/ceph-osd-580 from API server
unit-ceph-osd-0: 12:17:50 ERROR juju.worker.meterstatus error running "meter-status-changed": charm missing from disk
unit-ceph-osd-2: 12:17:59 INFO juju Starting unit workers for "ceph-osd/2"
unit-ceph-osd-2: 12:17:59 INFO juju.worker.apicaller [18406b] "unit-ceph-osd-2" successfully connected to "192.168.2.216:17070"
unit-ceph-osd-2: 12:17:59 INFO juju.worker.apicaller [18406b] password changed for "unit-ceph-osd-2"
unit-ceph-osd-2: 12:17:59 INFO juju.worker.apicaller [18406b] "unit-ceph-osd-2" successfully connected to "192.168.2.216:17070"
unit-ceph-osd-2: 12:17:59 INFO juju.worker.upgrader no waiter, upgrader is done
unit-ceph-osd-2: 12:17:59 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-ceph-osd-2: 12:17:59 INFO juju.worker.logger logger worker started
unit-ceph-osd-2: 12:17:59 INFO juju.worker.uniter unit "ceph-osd/2" started
unit-ceph-osd-2: 12:17:59 INFO juju.worker.uniter resuming charm install
unit-ceph-osd-2: 12:17:59 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/ceph-osd-580 from API server
unit-ceph-osd-2: 12:17:59 ERROR juju.worker.meterstatus error running "meter-status-changed": charm missing from disk
unit-ceph-osd-1: 12:18:03 INFO juju Starting unit workers for "ceph-osd/1"
unit-ceph-osd-1: 12:18:03 INFO juju.worker.apicaller [18406b] "unit-ceph-osd-1" successfully connected to "192.168.2.216:17070"
unit-ceph-osd-1: 12:18:04 INFO juju.worker.apicaller [18406b] password changed for "unit-ceph-osd-1"
unit-ceph-osd-1: 12:18:04 INFO juju.worker.apicaller [18406b] "unit-ceph-osd-1" successfully connected to "192.168.2.216:17070"
unit-ceph-osd-1: 12:18:04 INFO juju.worker.upgrader no waiter, upgrader is done
unit-ceph-osd-1: 12:18:04 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-ceph-osd-1: 12:18:04 INFO juju.worker.logger logger worker started
unit-ceph-osd-1: 12:18:05 INFO juju.worker.uniter unit "ceph-osd/1" started
unit-ceph-osd-1: 12:18:05 INFO juju.worker.uniter resuming charm install
unit-ceph-osd-1: 12:18:05 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/ceph-osd-580 from API server
unit-ceph-osd-1: 12:18:05 ERROR juju.worker.meterstatus error running "meter-status-changed": charm missing from disk
unit-ceph-osd-0: 12:18:05 INFO juju.worker.uniter hooks are retried true
unit-ceph-osd-2: 12:18:18 INFO juju.worker.uniter hooks are retried true
unit-ceph-osd-1: 12:18:19 INFO juju.worker.uniter hooks are retried true
################
machine-log:
2024-03-04 18:17:58 INFO juju.cmd supercommand.go:56 running jujud [3.4.0 e1b7dcd7390348c37f8b860011e7436e6ed3f4cc gc go1.20.13]
2024-03-04 18:17:58 DEBUG juju.cmd supercommand.go:57 args: []string{"/var/lib/juju/tools/machine-34/jujud", "machine", "--data-dir", "/var/lib/juju", "--machine-id", "34", "--debug"}
2024-03-04 18:17:58 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 2
2024-03-04 18:17:58 DEBUG juju.agent agent.go:600 read agent config, format "2.0"
2024-03-04 18:17:58 INFO juju.worker.upgradesteps worker.go:60 upgrade steps for 3.4.0 have already been run.
2024-03-04 18:17:58 DEBUG juju.cmd.jujud runner.go:416 start "engine"
2024-03-04 18:17:58 INFO juju.cmd.jujud runner.go:592 start "engine"
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "upgrade-steps-gate" manifold worker started at 2024-03-04 18:17:58.661446625 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "upgrade-steps-flag" manifold worker started at 2024-03-04 18:17:58.661826364 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "upgrade-check-gate" manifold worker started at 2024-03-04 18:17:58.664211064 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.introspection worker.go:127 introspection worker listening on "@jujud-machine-34"
2024-03-04 18:17:58 DEBUG juju.cmd.jujud runner.go:424 "engine" started
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "syslog" manifold worker started at 2024-03-04 18:17:58.669964281 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "termination-signal-handler" manifold worker started at 2024-03-04 18:17:58.670659565 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "clock" manifold worker started at 2024-03-04 18:17:58.670845488 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "agent" manifold worker started at 2024-03-04 18:17:58.67092269 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "charmhub-http-client" manifold worker started at 2024-03-04 18:17:58.67108853 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.introspection worker.go:152 stats worker now serving
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "upgrade-check-flag" manifold worker started at 2024-03-04 18:17:58.67630416 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.apicaller connect.go:129 connecting with old password
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "state-config-watcher" manifold worker started at 2024-03-04 18:17:58.68348853 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "api-config-watcher" manifold worker started at 2024-03-04 18:17:58.683707115 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "is-not-controller-flag" manifold worker started at 2024-03-04 18:17:58.694774497 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "is-controller-flag" manifold worker started at 2024-03-04 18:17:58.694941565 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://192.168.2.216:17070/model/18406b16-dbaa-49e9-8fe0-bc6fe7fae672/api"
2024-03-04 18:17:58 INFO juju.api apiclient.go:707 connection established to "wss://192.168.2.216:17070/model/18406b16-dbaa-49e9-8fe0-bc6fe7fae672/api"
2024-03-04 18:17:58 INFO juju.worker.apicaller connect.go:163 [18406b] "machine-34" successfully connected to "192.168.2.216:17070"
2024-03-04 18:17:58 DEBUG juju.worker.apicaller connect.go:255 changing password...
2024-03-04 18:17:58 INFO juju.worker.apicaller connect.go:260 [18406b] password changed for "machine-34"
2024-03-04 18:17:58 DEBUG juju.api monitor.go:35 RPC connection died
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:621 "api-caller" manifold worker stopped: restart immediately
2024-03-04 18:17:58 DEBUG juju.worker.apicaller connect.go:116 connecting with current password
2024-03-04 18:17:58 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://192.168.2.216:17070/model/18406b16-dbaa-49e9-8fe0-bc6fe7fae672/api"
2024-03-04 18:17:58 INFO juju.api apiclient.go:707 connection established to "wss://192.168.2.216:17070/model/18406b16-dbaa-49e9-8fe0-bc6fe7fae672/api"
2024-03-04 18:17:58 INFO juju.worker.apicaller connect.go:163 [18406b] "machine-34" successfully connected to "192.168.2.216:17070"
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "api-caller" manifold worker started at 2024-03-04 18:17:58.876967825 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "upgrader" manifold worker started at 2024-03-04 18:17:58.887017033 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "upgrade-steps-runner" manifold worker started at 2024-03-04 18:17:58.89261417 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "migration-inactive-flag" manifold worker started at 2024-03-04 18:17:58.892674478 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:603 "upgrade-steps-runner" manifold worker completed successfully
2024-03-04 18:17:58 INFO juju.worker.upgrader upgrader.go:166 desired agent binary version: 3.4.0
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:618 "upgrade-check-flag" manifold worker stopped: gate unlocked
stack trace:
github.com/juju/juju/worker/gate.init:95: gate unlocked
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "valid-credential-flag" manifold worker started at 2024-03-04 18:17:58.914574578 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "upgrade-check-flag" manifold worker started at 2024-03-04 18:17:58.926367229 +0000 UTC
2024-03-04 18:17:58 INFO juju.worker.deployer nested.go:160 new context: units "", stopped ""
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "deployer" manifold worker started at 2024-03-04 18:17:58.937210202 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "migration-fortress" manifold worker started at 2024-03-04 18:17:58.938078618 +0000 UTC
2024-03-04 18:17:58 INFO juju.worker.deployer deployer.go:161 checking unit "ceph-osd/2"
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "migration-minion" manifold worker started at 2024-03-04 18:17:58.94905561 +0000 UTC
2024-03-04 18:17:58 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "proxy-config-updater" manifold worker started at 2024-03-04 18:17:58.962647114 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "host-key-reporter" manifold worker started at 2024-03-04 18:17:58.962893074 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.logger logger.go:65 initial log config: "<root>=DEBUG"
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "logging-config-updater" manifold worker started at 2024-03-04 18:17:58.963054728 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "api-address-updater" manifold worker started at 2024-03-04 18:17:58.963191226 +0000 UTC
2024-03-04 18:17:58 INFO juju.worker.logger logger.go:120 logger worker started
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "reboot-executor" manifold worker started at 2024-03-04 18:17:58.963366132 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "machine-action-runner" manifold worker started at 2024-03-04 18:17:58.9638561 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "storage-provisioner" manifold worker started at 2024-03-04 18:17:58.964099148 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "upgrade-series" manifold worker started at 2024-03-04 18:17:58.964211694 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "state-converter" manifold worker started at 2024-03-04 18:17:58.964304591 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "disk-manager" manifold worker started at 2024-03-04 18:17:58.9644123 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "log-sender" manifold worker started at 2024-03-04 18:17:58.964505553 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.dependency engine.go:580 "ssh-authkeys-updater" manifold worker started at 2024-03-04 18:17:58.964665086 +0000 UTC
2024-03-04 18:17:58 DEBUG juju.worker.machinesetup startupmanifold.go:54 Starting machine setup requiring an API connection
2024-03-04 18:17:58 DEBUG juju.worker.fanconfigurer fanconfigurer.go:42 Processing new fan config
2024-03-04 18:17:58 INFO juju.worker.deployer deployer.go:204 deploying unit "ceph-osd/2"
2024-03-04 18:17:59 INFO juju.worker.diskmanager diskmanager.go:67 block devices changed: []storage.BlockDevice{storage.BlockDevice{DeviceName:"loop0", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x3f, FilesystemType:"squashfs", InUse:true, MountPoint:"/snap/core20/2182", SerialId:""}, storage.BlockDevice{DeviceName:"loop1", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x56, FilesystemType:"squashfs", InUse:true, MountPoint:"/snap/lxd/27037", SerialId:""}, storage.BlockDevice{DeviceName:"loop2", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x28, FilesystemType:"squashfs", InUse:true, MountPoint:"/snap/snapd/20671", SerialId:""}, storage.BlockDevice{DeviceName:"sda", DeviceLinks:[]string{"/dev/disk/by-dname/sda", "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_root", "/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_root", "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:0:1"}, Label:"", UUID:"", HardwareId:"scsi-0QEMU_QEMU_HARDDISK_lxd_root", WWN:"", BusAddress:"scsi@0:0.0.1", Size:0x1dcd, FilesystemType:"", InUse:true, MountPoint:"", SerialId:"0QEMU_QEMU_HARDDISK_lxd_root"}, storage.BlockDevice{DeviceName:"sda1", DeviceLinks:[]string{"/dev/disk/by-dname/sda-part1", "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_root-part1", "/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_root-part1", "/dev/disk/by-label/efi", "/dev/disk/by-partuuid/5ffcc408-5006-48ea-9b15-797b0e1c304f", "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:0:1-part1", "/dev/disk/by-uuid/DBAA-210C"}, Label:"efi", UUID:"DBAA-210C", HardwareId:"scsi-0QEMU_QEMU_HARDDISK_lxd_root", WWN:"", BusAddress:"scsi@0:0.0.1", Size:0x200, FilesystemType:"vfat", InUse:true, MountPoint:"/boot/efi", SerialId:"0QEMU_QEMU_HARDDISK_lxd_root"}, storage.BlockDevice{DeviceName:"sda2", DeviceLinks:[]string{"/dev/disk/by-dname/sda-part2", "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_root-part2", "/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_root-part2", "/dev/disk/by-label/root", "/dev/disk/by-partuuid/d0a2d22e-eaa0-428e-b27f-f8d26a6a9aa0", "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:0:1-part2", "/dev/disk/by-uuid/abd1f52f-a42c-431c-be56-f2372a1022f3"}, Label:"root", UUID:"abd1f52f-a42c-431c-be56-f2372a1022f3", HardwareId:"scsi-0QEMU_QEMU_HARDDISK_lxd_root", WWN:"", BusAddress:"scsi@0:0.0.1", Size:0x1bcc, FilesystemType:"ext4", InUse:true, MountPoint:"/", SerialId:"0QEMU_QEMU_HARDDISK_lxd_root"}, storage.BlockDevice{DeviceName:"sdb", DeviceLinks:[]string{"/dev/disk/by-dname/sdb", "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk1", "/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_disk1", "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:1:1"}, Label:"", UUID:"", HardwareId:"scsi-0QEMU_QEMU_HARDDISK_lxd_disk1", WWN:"", BusAddress:"scsi@0:0.1.1", Size:0x7735, FilesystemType:"", InUse:false, MountPoint:"", SerialId:"0QEMU_QEMU_HARDDISK_lxd_disk1"}, storage.BlockDevice{DeviceName:"sdc", DeviceLinks:[]string{"/dev/disk/by-dname/sdc", "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk2", "/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_disk2", "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:2:1"}, Label:"", UUID:"", HardwareId:"scsi-0QEMU_QEMU_HARDDISK_lxd_disk2", WWN:"", BusAddress:"scsi@0:0.2.1", Size:0x7735, FilesystemType:"", InUse:false, MountPoint:"", SerialId:"0QEMU_QEMU_HARDDISK_lxd_disk2"}, storage.BlockDevice{DeviceName:"sdd", DeviceLinks:[]string{"/dev/disk/by-dname/sdd", "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_lxd_disk3", "/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_lxd_disk3", "/dev/disk/by-path/pci-0000:02:00.0-scsi-0:0:3:1"}, Label:"", UUID:"", HardwareId:"scsi-0QEMU_QEMU_HARDDISK_lxd_disk3", WWN:"", BusAddress:"scsi@0:0.3.1", Size:0x1dcd, FilesystemType:"", InUse:false, MountPoint:"", SerialId:"0QEMU_QEMU_HARDDISK_lxd_disk3"}}
2024-03-04 18:17:59 DEBUG juju.worker.logger logger.go:93 reconfiguring logging from "<root>=DEBUG" to "<root>=INFO"
2024-03-04 18:17:59 ERROR juju.worker.dependency engine.go:695 "broker-tracker" manifold worker returned unexpected error: no container types determined
2024-03-04 18:17:59 ERROR juju.worker.dependency engine.go:695 "lxd-container-provisioner" manifold worker returned unexpected error: container types not yet available
2024-03-04 18:17:59 ERROR juju.worker.dependency engine.go:695 "kvm-container-provisioner" manifold worker returned unexpected error: container types not yet available
2024-03-04 18:17:59 INFO juju.worker.machiner machiner.go:163 setting addresses for "machine-34" to [local-machine:127.0.0.1 local-cloud:192.168.2.241 local-machine:::1]
2024-03-04 18:17:59 INFO juju.worker.upgradeseries worker.go:162 no series upgrade lock present
2024-03-04 18:17:59 INFO juju.worker.deployer unit_agent.go:125 creating new agent config for "ceph-osd/2"
2024-03-04 18:17:59 INFO juju.worker.deployer nested.go:367 starting workers for "ceph-osd/2"
2024-03-04 18:17:59 INFO juju.worker.deployer runner.go:592 start "ceph-osd/2"
2024-03-04 18:17:59 INFO juju.worker.machiner machiner.go:113 "machine-34" started
2024-03-04 18:17:59 INFO juju.api apiclient.go:707 connection established to "wss://192.168.2.216:17070/model/18406b16-dbaa-49e9-8fe0-bc6fe7fae672/api"
2024-03-04 18:17:59 INFO juju.worker.authenticationworker worker.go:96 "machine-34" key updater worker started
2024-03-04 18:17:59 INFO juju.api apiclient.go:707 connection established to "wss://192.168.2.216:17070/model/18406b16-dbaa-49e9-8fe0-bc6fe7fae672/api"
2024-03-04 18:17:59 INFO juju.worker.leadership tracker.go:217 ceph-osd leadership for ceph-osd/2 denied
2024-03-04 18:17:59 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-ceph-osd-2
2024-03-04 18:17:59 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/3.4.0-ubuntu-amd64
2024-03-04 18:17:59 INFO juju.downloader download.go:112 downloading from ch:amd64/jammy/ceph-osd-580
2024-03-04 18:17:59 INFO juju.downloader download.go:95 download complete ("ch:amd64/jammy/ceph-osd-580")
2024-03-04 18:17:59 INFO juju.downloader download.go:175 download verified ("ch:amd64/jammy/ceph-osd-580")
2024-03-04 18:18:02 INFO juju.container interface.go:81 unused config option: "container-networking-method" -> "provider"
I cannot find any OSD-related errors - it appears to be a MAAS+Juju deployment kind of issue. Can you tell me if the problem persists if you remove the 'storage' directives from the command?