carry over from IRC

<admcleod>»·just putting this here for tomorrow: https://bugs.launchpad.net/charm-nova-compute/+bug/1891203
<cpaelzer>»·looked at the bug - doesn't ring a known bell                        
<cpaelzer>»·maybe a dependency change                                            
<cpaelzer>»·so this is Ussuri - so we are talking about the Focal backported to Bionic here right?
<admcleod>»·bionic ussuri, right - focal ussuri works fine with that same version
<cpaelzer>»·also if you could compare dpkg -l 'qemu*' 'libvirt*' on the working focal-ussuri vs bionuc-ussuri
<admcleod>»·will do                                                              
<cpaelzer>»·I'm concerned if just an extra qemu-something-arm might be missing    
<admcleod>»·heh right                                                            
<cpaelzer>»·and for arm that also includes packages like ovmf or such            
<cpaelzer>»·what else then ovmf might come to ming ...                            
<cpaelzer>»·mind                                                                 
<cpaelzer>»·maybe just be brutal and Diff full `dpkg -l` among the systemd       
<cpaelzer>»·systems                                                              
<cpaelzer>»·in particular                                                        
<cpaelzer>»·your armv6l has this                                                    
<cpaelzer>»·      <emulator>/usr/bin/qemu-system-arm</emulator>                  
<cpaelzer>»·whatever the good system has as emulator for aarch64 really needs to be there
<cpaelzer>»·also I've seen the capability prober processes fail, so if nothing of the above helps check the logs of libvirt for failing qemu processes
<admcleod>»·right if i downgrade to prev version                                 
<admcleod>»·i do have   <guest>                                                     
<admcleod>»·    <os_type>hvm</os_type>                                           
<admcleod>»·    <arch name='armv7l'>                                             
<admcleod>»·      <wordsize>32</wordsize>                                        
<admcleod>»·      <emulator>/usr/bin/qemu-system-aarch64</emulator>
<admcleod> bionic has libvirt 6.0.0-0ubuntu8.2 and focal 6.0.0-0ubuntu8.3
<admcleod> nothing else obvious w qemu or libvirt packages
<cpaelzer> hmm
<cpaelzer> there were a few things I mentioned to james that need to be removed/reverted on backport
<cpaelzer> I mostly send these mails to allow my brain forgetting about them     
* cpaelzer tries to search what I have mentioned there                           
<admcleod> yeah updating libvirt to that latest (...8.3) didnt work              
<admcleod> there are no libvirt/qemu logs either                                 
<admcleod> come tot hink about it, it feels as if there is a binary which reports the version and that aarch64 that is missing somehow 
<admcleod> that how it 'feels'                                                   
<admcleod> .....without understanding how it 'works'                             
<cpaelzer> on startup libvirt probes for binaries of qemu                        
<cpaelzer> then it calls all those asking for HW/Features they support           
<cpaelzer> that is what builds that "capabilities" output                        
<admcleod> right so if it cant probe those bins, it wont get the version or the caps
<cpaelzer> hence I'm saying you might want to debug a restart of libvirtd        
<admcleod> cos the error is 'not min ver' but it is                              
<cpaelzer> maybe one of these processes dies off                                 
<admcleod> right                                                                 
<admcleod> how debug?·                                                           
<cpaelzer> I found my old mails BTW - we were only aware of x86 changes that need to be reverted ont he backport
<cpaelzer> enable this https://wiki.libvirt.org/page/DebugLogs                   
<admcleod> ah                                                                    
<cpaelzer> then delete all files in /var/cache/libvirt/qemu/capabilities/        
<cpaelzer> that forces libvirt to re-probe for sure                              
<cpaelzer> then systemctl restart libvirtd                                       
<cpaelzer> then read the log and hope to find something                          
<cpaelzer> it is quite verbose, so you have to search a while (that is normal)   
<cpaelzer> you look for qemu processes started for this probing in the log       
<cpaelzer> and how they worked out                                               
<cpaelzer> best maybe is to do the same on F-ussuri vs B-ussuri                  
<cpaelzer> as that will help to ignore the bits that didn't change