Looks like there is a lot of contention on a futex and the underlying database too. Perf record and perf report can show that most of the issues are more to do with lxd, database and futex lock contention. Once these are resolved, I'll be happy to re-analyze the entire ZFS + lxd stack, but I believe the main issue is lxd eating cycles and not ZFS per se.
Looks like there is a lot of contention on a futex and the underlying database too. Perf record and perf report can show that most of the issues are more to do with lxd, database and futex lock contention. Once these are resolved, I'll be happy to re-analyze the entire ZFS + lxd stack, but I believe the main issue is lxd eating cycles and not ZFS per se.
- 27.21% 1.05% lxd [kernel.kallsyms] [k] entry_SYSCALL_ 64_fastpath 64_fastpath return_ slowpath 1.8-std. so.1 [.] runtime.goexit ▒ operation) .Run.func1 Image.func1 reateFromImage reateInternal XCCreate
- 11.30% main.networkUpd ateStatic
- 9.16% main.containerL oadByName
- 4.79% main.dbContainerGet
+ 2.15% main.dbContaine rConfig
+ 1.13% main.dbQueryRowScan
+ 0.82% main.dbContaine rProfiles
+ 0.59% main.dbDevices
+ 4.33% main.containerL XCLoad
+ 1.98% main.(* containerLXC) .fillNetworkDev ice
+ 0.86% main.(* containerLXC) .ConfigKeySet
0.67% main.(* storageZfs) .ContainerCreat eFromImage tatePut. func1 tListener (*conn) .serve sql.(*Rows) .awaitDone operation) .Run.func1 ▒ operation) .Run.func1 Image.func1 reateFromImage reateInternal XCCreate ateStatic
- 9.16% main.containerL oadByName
- 4.79% main.dbContainerGet
+ 2.15% main.dbContaine rConfig
+ 1.13% main.dbQueryRowScan
+ 0.82% main.dbContaine rProfiles
+ 0.59% main.dbDevices
- 26.16% entry_SYSCALL_
- 19.28% sys_futex
- 18.91% do_futex
- 12.70% futex_wake
+ 9.91% wake_up_q
+ 0.90% _raw_spin_lock
+ 5.00% futex_wait
+ 1.48% sys_select
+ 0.73% sys_newstat
+ 0.57% syscall_
+ 0.54% sys_pread64
- 25.16% 0.00% lxd libgolang-
- runtime.goexit
- 14.98% main.(*
- 13.92% main.createFrom
- 13.86% main.containerC
- 12.26% main.containerC
- 12.24% main.containerL
+ 0.70% main.dbImageGet
+ 1.05% main.containerS
+ 2.37% main.deviceEven
+ 2.21% runtime.mcall
+ 1.70% net/http.
+ 1.27% database/
+ 0.82% runtime.systemstack
+ 19.33% 0.33% lxd [kernel.kallsyms] [k] sys_futex
+ 18.96% 0.43% lxd [kernel.kallsyms] [k] do_futex
- 14.98% 0.00% lxd lxd [.] main.(*
- main.(*
- 13.92% main.createFrom
- 13.86% main.containerC
- 12.26% main.containerC
- 12.24% main.containerL
- 11.30% main.networkUpd