Looks like there is a lot of contention on a futex and the underlying database too. Perf record and perf report can show that most of the issues are more to do with lxd, database and futex lock contention. Once these are resolved, I'll be happy to re-analyze the entire ZFS + lxd stack, but I believe the main issue is lxd eating cycles and not ZFS per se.

-   27.21%     1.05%  lxd              [kernel.kallsyms]                              [k] entry_SYSCALL_64_fastpath                 
   - 26.16% entry_SYSCALL_64_fastpath                                                                                                   
      - 19.28% sys_futex                                                                                                                
         - 18.91% do_futex                                                                                                              
            - 12.70% futex_wake                                                                                                         
               + 9.91% wake_up_q                                                                                                        
               + 0.90% _raw_spin_lock                                                                                                  
            + 5.00% futex_wait                                                                                                         
      + 1.48% sys_select                                                                                                                
      + 0.73% sys_newstat                                                                                                              
      + 0.57% syscall_return_slowpath                                                                                                   
      + 0.54% sys_pread64                                                                                                               
-   25.16%     0.00%  lxd              libgolang-1.8-std.so.1                         [.] runtime.goexit                                ▒
   - runtime.goexit                                                                                                                     
      - 14.98% main.(*operation).Run.func1                                                                                              
         - 13.92% main.createFromImage.func1                                                                                            
            - 13.86% main.containerCreateFromImage                                                                                      
               - 12.26% main.containerCreateInternal                                                                                    
                  - 12.24% main.containerLXCCreate                                                                                      
                     - 11.30% main.networkUpdateStatic                                                                                  
                        - 9.16% main.containerLoadByName                                                                                
                           - 4.79% main.dbContainerGet                                                                                  
                              + 2.15% main.dbContainerConfig                                                                            
                              + 1.13% main.dbQueryRowScan                                                                               
                              + 0.82% main.dbContainerProfiles                                                                          
                              + 0.59% main.dbDevices                                                                                    
                           + 4.33% main.containerLXCLoad                                                                               
                        + 1.98% main.(*containerLXC).fillNetworkDevice                                                                  
                     + 0.86% main.(*containerLXC).ConfigKeySet                                                                          
               + 0.70% main.dbImageGet                                                                                                  
                 0.67% main.(*storageZfs).ContainerCreateFromImage                                                                      
         + 1.05% main.containerStatePut.func1                                                                                           
      + 2.37% main.deviceEventListener                                                                                                  
      + 2.21% runtime.mcall                                                                                                             
      + 1.70% net/http.(*conn).serve                                                                                                    
      + 1.27% database/sql.(*Rows).awaitDone                                                                                            
      + 0.82% runtime.systemstack                                                                                                       
+   19.33%     0.33%  lxd              [kernel.kallsyms]                              [k] sys_futex                                     
+   18.96%     0.43%  lxd              [kernel.kallsyms]                              [k] do_futex                                      
-   14.98%     0.00%  lxd              lxd                                            [.] main.(*operation).Run.func1                   ▒
   - main.(*operation).Run.func1                                                                                                        
      - 13.92% main.createFromImage.func1                                                                                               
         - 13.86% main.containerCreateFromImage                                                                                         
            - 12.26% main.containerCreateInternal                                                                                       
               - 12.24% main.containerLXCCreate                                                                                         
                  - 11.30% main.networkUpdateStatic                                                                                     
                     - 9.16% main.containerLoadByName                                                                                   
                        - 4.79% main.dbContainerGet                                                                                     
                           + 2.15% main.dbContainerConfig                                                                               
                           + 1.13% main.dbQueryRowScan                                                                                  
                           + 0.82% main.dbContainerProfiles                                                                             
                           + 0.59% main.dbDevices