Comment 10 for bug 1567557

Looks like there is a lot of contention on a futex and the underlying database too. Perf record and perf report can show that most of the issues are more to do with lxd, database and futex lock contention. Once these are resolved, I'll be happy to re-analyze the entire ZFS + lxd stack, but I believe the main issue is lxd eating cycles and not ZFS per se.

- 27.21% 1.05% lxd [kernel.kallsyms] [k] entry_SYSCALL_64_fastpath
   - 26.16% entry_SYSCALL_64_fastpath
      - 19.28% sys_futex
         - 18.91% do_futex
            - 12.70% futex_wake
               + 9.91% wake_up_q
               + 0.90% _raw_spin_lock
            + 5.00% futex_wait
      + 1.48% sys_select
      + 0.73% sys_newstat
      + 0.57% syscall_return_slowpath
      + 0.54% sys_pread64
- 25.16% 0.00% lxd libgolang-1.8-std.so.1 [.] runtime.goexit ▒
   - runtime.goexit
      - 14.98% main.(*operation).Run.func1
         - 13.92% main.createFromImage.func1
            - 13.86% main.containerCreateFromImage
               - 12.26% main.containerCreateInternal
                  - 12.24% main.containerLXCCreate
                     - 11.30% main.networkUpdateStatic
                        - 9.16% main.containerLoadByName
                           - 4.79% main.dbContainerGet
                              + 2.15% main.dbContainerConfig
                              + 1.13% main.dbQueryRowScan
                              + 0.82% main.dbContainerProfiles
                              + 0.59% main.dbDevices
                           + 4.33% main.containerLXCLoad
                        + 1.98% main.(*containerLXC).fillNetworkDevice
                     + 0.86% main.(*containerLXC).ConfigKeySet
               + 0.70% main.dbImageGet
                 0.67% main.(*storageZfs).ContainerCreateFromImage
         + 1.05% main.containerStatePut.func1
      + 2.37% main.deviceEventListener
      + 2.21% runtime.mcall
      + 1.70% net/http.(*conn).serve
      + 1.27% database/sql.(*Rows).awaitDone
      + 0.82% runtime.systemstack
+ 19.33% 0.33% lxd [kernel.kallsyms] [k] sys_futex
+ 18.96% 0.43% lxd [kernel.kallsyms] [k] do_futex
- 14.98% 0.00% lxd lxd [.] main.(*operation).Run.func1 ▒
   - main.(*operation).Run.func1
      - 13.92% main.createFromImage.func1
         - 13.86% main.containerCreateFromImage
            - 12.26% main.containerCreateInternal
               - 12.24% main.containerLXCCreate
                  - 11.30% main.networkUpdateStatic
                     - 9.16% main.containerLoadByName
                        - 4.79% main.dbContainerGet
                           + 2.15% main.dbContainerConfig
                           + 1.13% main.dbQueryRowScan
                           + 0.82% main.dbContainerProfiles
                           + 0.59% main.dbDevices