Activity log for bug #997978

Date Who What changed Old value New value Message
2012-05-11 09:33:43 Jonathan Tullett bug added bug
2012-05-11 15:28:16 Serge Hallyn bug task added bridge-utils (Ubuntu)
2012-05-11 15:28:31 Serge Hallyn libvirt (Ubuntu): importance Undecided High
2012-05-11 15:37:42 Serge Hallyn libvirt (Ubuntu): status New Incomplete
2012-05-16 19:38:45 Serge Hallyn libvirt (Ubuntu): status Incomplete New
2012-05-16 19:38:48 Serge Hallyn bridge-utils (Ubuntu): status New Incomplete
2012-05-16 19:38:52 Serge Hallyn libvirt (Ubuntu): status New Incomplete
2012-05-16 19:38:59 Serge Hallyn bridge-utils (Ubuntu): importance Undecided High
2012-05-21 15:57:34 Serge Hallyn bug task added qemu-kvm (Ubuntu)
2012-05-21 15:57:45 Serge Hallyn qemu-kvm (Ubuntu): status New Confirmed
2012-05-21 15:57:50 Serge Hallyn bridge-utils (Ubuntu): status Incomplete Invalid
2012-05-21 15:57:55 Serge Hallyn qemu-kvm (Ubuntu): importance Undecided High
2012-05-24 08:41:43 Slawomir Skowron bug added subscriber Slawomir Skowron
2012-06-05 12:21:25 Juan Gonzalez bug added subscriber Juan Gonzalez
2012-06-11 18:43:19 bradleyd bug added subscriber bradleyd
2012-06-12 14:59:36 Georg Leciejewski bug watch added https://bugzilla.redhat.com/show_bug.cgi?id=571991
2012-06-12 14:59:36 Georg Leciejewski bug watch added https://bugzilla.redhat.com/show_bug.cgi?id=583139
2012-06-13 15:17:41 Federico Castagnini bug added subscriber Federico Castagnini
2012-06-19 12:13:24 Serge Hallyn libvirt (Ubuntu): status Incomplete Confirmed
2012-06-22 09:06:04 lacroix bug added subscriber lacroix
2012-06-27 00:43:17 andrew bezella bug added subscriber andrew bezella
2012-06-27 01:31:38 Chet Burgess bug added subscriber Chet Burgess
2012-06-28 07:27:37 Jian Wen bug added subscriber Jian Wen
2012-06-28 21:48:33 Alex Dioso bug watch added http://bugzilla.kernel.org/show_bug.cgi?id=42829
2012-06-28 22:53:04 Camille bug added subscriber Camille
2012-07-03 09:10:32 Juan Gonzalez removed subscriber Juan Gonzalez
2012-07-05 02:02:23 YAMAMOTO Hirotaka bug added subscriber YAMAMOTO Hirotaka
2012-07-05 13:43:29 Serge Hallyn bug task added linux (Ubuntu)
2012-07-05 13:43:40 Serge Hallyn linux (Ubuntu): status New Confirmed
2012-07-05 13:43:44 Serge Hallyn linux (Ubuntu): importance Undecided High
2012-07-05 13:44:48 Serge Hallyn tags kvm networking kernel-kvm kvm networking
2012-07-15 03:20:27 Alex Dioso bug added subscriber Alex Dioso
2012-08-01 18:18:59 Peter Matulis bug added subscriber Peter Matulis
2012-08-02 16:20:42 Tom Martinson bug added subscriber Tom Martinson
2012-08-10 22:27:15 Serge Hallyn bug task added ifenslave (Ubuntu)
2012-08-13 12:44:31 Launchpad Janitor ifenslave (Ubuntu): status New Confirmed
2012-08-13 12:45:08 Simon Sprünker bug added subscriber Simon Sprünker
2012-08-14 07:59:35 Eugene Nelen bug added subscriber Eugene Nelen
2012-08-14 18:42:17 Gary Cuozzo bug added subscriber Gary Cuozzo
2012-08-15 00:26:11 kraig bug added subscriber kraig
2012-08-15 14:29:10 Thomas Vachon bug task added nova
2012-08-20 14:18:29 Thomas Vachon bug added subscriber Thomas Vachon
2012-08-27 10:02:59 Vassilis Vatikiotis bug added subscriber Vassilis Vatikiotis
2012-08-28 00:04:45 Elmer Rivera bug added subscriber Elmer Rivera
2012-08-28 00:22:58 David Mitchell bug added subscriber David Mitchell
2012-08-31 11:45:11 Salih Sen bug added subscriber Salih Sen
2012-09-01 00:41:16 Gustave Hellman bug added subscriber Gustave Hellman
2012-09-01 12:20:37 Peng Yong bug added subscriber Peng Yong
2012-09-08 00:05:36 mrf bug added subscriber mrf
2012-09-11 06:22:01 Matthias Eichstaedt bug added subscriber Matthias Eichstaedt
2012-09-11 14:40:35 Thierry Carrez nova: status New Invalid
2012-09-23 17:45:22 Joe T bug added subscriber Joe T
2012-09-24 19:57:05 Stéphane Graber bug task deleted ifenslave (Ubuntu)
2012-09-24 19:57:15 Stéphane Graber bug task deleted libvirt (Ubuntu)
2012-09-24 19:57:19 Stéphane Graber bug task deleted bridge-utils (Ubuntu)
2012-09-24 19:57:22 Stéphane Graber bug task deleted linux (Ubuntu)
2012-09-24 19:59:46 Serge Hallyn nominated for series Ubuntu Precise
2012-09-24 19:59:46 Serge Hallyn bug task added qemu-kvm (Ubuntu Precise)
2012-09-24 19:59:58 Serge Hallyn qemu-kvm (Ubuntu): status Confirmed Fix Released
2012-09-24 20:00:03 Serge Hallyn qemu-kvm (Ubuntu Precise): status New In Progress
2012-09-24 20:00:07 Serge Hallyn qemu-kvm (Ubuntu Precise): importance Undecided High
2012-09-24 20:05:17 Serge Hallyn description System: ----------- Dell R410 Dual processor 2.4Ghz w/16G RAM Distributor ID: Ubuntu Description: Ubuntu 12.04 LTS Release: 12.04 Codename: precise Setup: --------- We're running 3 KVM guests, all Ubuntu 12.04 LTS using bridged networking. From the host: # cat /etc/network/interfaces auto br0 iface br0 inet static address 212.XX.239.98 netmask 255.255.255.240 gateway 212.XX.239.97 bridge_ports eth0 bridge_fd 9 bridge_hello 2 bridge_maxage 12 bridge_stp off # ifconfig eth0 eth0 Link encap:Ethernet HWaddr d4:ae:52:84:2d:5a UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:11278363 errors:0 dropped:3128 overruns:0 frame:0 TX packets:14437384 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:4115980743 (4.1 GB) TX bytes:5451961979 (5.4 GB) Interrupt:36 Memory:da000000-da012800 # ifconfig br0 br0 Link encap:Ethernet HWaddr d4:ae:52:84:2d:5a inet addr:212.XX.239.98 Bcast:212.XX.239.111 Mask:255.255.255.240 inet6 addr: fe80::d6ae:52ff:fe84:2d5a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1720861 errors:0 dropped:0 overruns:0 frame:0 TX packets:1708622 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:210152198 (210.1 MB) TX bytes:300858508 (300.8 MB) # brctl show bridge name bridge id STP enabled interfaces br0 8000.d4ae52842d5a no eth0 I have no default network configured to autostart in libvirt as we're using bridged networking: # virsh net-list --all Name State Autostart ----------------------------------------- default inactive no # arp Address HWtype HWaddress Flags Mask Iface mailer03.xxxx.com ether 52:54:00:82:5f:0f C br0 mailer01.xxxx.com ether 52:54:00:d2:f7:31 C br0 mailer02.xxxx.com ether 52:54:00:d3:8f:91 C br0 dxi-gw2.xxxx.com ether 00:1a:30:2a:b1:c0 C br0 From one of the guests: <domain type='kvm' id='4'> <name>mailer01</name> <uuid>d41d1355-84e8-ae23-e84e-227bc0231b97</uuid> <memory>2097152</memory> <currentMemory>2097152</currentMemory> <vcpu>1</vcpu> <os> <type arch='x86_64' machine='pc-1.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/mapper/vg_main-mailer01--root'/> <target dev='hda' bus='ide'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' unit='0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/dev/mapper/vg_main-mailer01--swap'/> <target dev='hdb' bus='ide'/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' unit='1'/> </disk> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='bridge'> <mac address='52:54:00:d2:f7:31'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='apparmor' relabel='yes'> <label>libvirt-d41d1355-84e8-ae23-e84e-227bc0231b97</label> <imagelabel>libvirt-d41d1355-84e8-ae23-e84e-227bc0231b97</imagelabel> </seclabel> </domain> From within the guest: # cat /etc/network/interfaces # The primary network interface auto eth0 iface eth0 inet static address 212.XX.239.100 netmask 255.255.255.240 network 212.XX.239.96 broadcast 212.XX.239.111 gateway 212.XX.239.97 # ifconfig eth0 Link encap:Ethernet HWaddr 52:54:00:d2:f7:31 inet addr:212.XX.239.100 Bcast:212.XX.239.111 Mask:255.255.255.240 inet6 addr: fe80::5054:ff:fed2:f731/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:5631830 errors:0 dropped:0 overruns:0 frame:0 TX packets:6683416 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2027322829 (2.0 GB) TX bytes:2076698690 (2.0 GB) A commandline which starts the KVM guest: /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name mailer01 -uuid d41d1355-84e8-ae23-e84e-227bc0231b97 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/mailer01.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/dev/mapper/vg_main-mailer01--root,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/dev/mapper/vg_main-mailer01--swap,if=none,id=drive-ide0-0-1,format=raw -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=18,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d2:f7:31,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 Problem: ------------ Periodically (at least once a day), one or more of the guests lose network connectivity. Ping responds with 'host unreachable', even from the dom host. Logging in via the serial console shows no problems: eth0 is up, can ping the local host, but no outside connectivity. Restart the network (/etc/init.d/networking restart) does nothing. Reboot the machine and it comes alive again. I've verified there's no arp games going on on the primary host (the arp tables remain the same before - when it had connectivity - and after - when it doesn't. This is a critical issue affecting production services on the latest LTS release of Ubuntu. It's similar to an issue which was 'resolved' in 10.04 but appears to have risen its ugly head again. ========================================= SRU Justification: 1. Impact: networking breaks after awhile in kvm guests using virtio networking 2. Development fix: The bug was fixed upstream and the fix picked up in a new merge. 3. Stable fix: 3 virtio patches are cherrypicked from upstream: a821ce5 virtio: order index/descriptor reads 92045d8 virtio: add missing mb() on enable notification a281ebc virtio: add missing mb() on notification 4. Test case: Create a bridge enslaving the real NIC, and use that as the bridge for a kvm instance with virtio networking. See comment #44 for specific test case. 5. Regression potential: Should be low as several people have tested the fixed package under heavy load. ========================================= System: ----------- Dell R410 Dual processor 2.4Ghz w/16G RAM Distributor ID: Ubuntu Description: Ubuntu 12.04 LTS Release: 12.04 Codename: precise Setup: --------- We're running 3 KVM guests, all Ubuntu 12.04 LTS using bridged networking. From the host: # cat /etc/network/interfaces auto br0 iface br0 inet static         address 212.XX.239.98         netmask 255.255.255.240         gateway 212.XX.239.97         bridge_ports eth0         bridge_fd 9         bridge_hello 2         bridge_maxage 12         bridge_stp off # ifconfig eth0 eth0 Link encap:Ethernet HWaddr d4:ae:52:84:2d:5a           UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1           RX packets:11278363 errors:0 dropped:3128 overruns:0 frame:0           TX packets:14437384 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:1000           RX bytes:4115980743 (4.1 GB) TX bytes:5451961979 (5.4 GB)           Interrupt:36 Memory:da000000-da012800 # ifconfig br0 br0 Link encap:Ethernet HWaddr d4:ae:52:84:2d:5a           inet addr:212.XX.239.98 Bcast:212.XX.239.111 Mask:255.255.255.240           inet6 addr: fe80::d6ae:52ff:fe84:2d5a/64 Scope:Link           UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1           RX packets:1720861 errors:0 dropped:0 overruns:0 frame:0           TX packets:1708622 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:0           RX bytes:210152198 (210.1 MB) TX bytes:300858508 (300.8 MB) # brctl show bridge name bridge id STP enabled interfaces br0 8000.d4ae52842d5a no eth0 I have no default network configured to autostart in libvirt as we're using bridged networking: # virsh net-list --all Name State Autostart ----------------------------------------- default inactive no # arp Address HWtype HWaddress Flags Mask Iface mailer03.xxxx.com ether 52:54:00:82:5f:0f C br0 mailer01.xxxx.com ether 52:54:00:d2:f7:31 C br0 mailer02.xxxx.com ether 52:54:00:d3:8f:91 C br0 dxi-gw2.xxxx.com ether 00:1a:30:2a:b1:c0 C br0 From one of the guests: <domain type='kvm' id='4'>   <name>mailer01</name>   <uuid>d41d1355-84e8-ae23-e84e-227bc0231b97</uuid>   <memory>2097152</memory>   <currentMemory>2097152</currentMemory>   <vcpu>1</vcpu>   <os>     <type arch='x86_64' machine='pc-1.0'>hvm</type>     <boot dev='hd'/>   </os>   <features>     <acpi/>   </features>   <clock offset='utc'/>   <on_poweroff>destroy</on_poweroff>   <on_reboot>restart</on_reboot>   <on_crash>destroy</on_crash>   <devices>     <emulator>/usr/bin/kvm</emulator>     <disk type='file' device='disk'>       <driver name='qemu' type='raw'/>       <source file='/dev/mapper/vg_main-mailer01--root'/>       <target dev='hda' bus='ide'/>       <alias name='ide0-0-0'/>       <address type='drive' controller='0' bus='0' unit='0'/>     </disk>     <disk type='file' device='disk'>       <driver name='qemu' type='raw'/>       <source file='/dev/mapper/vg_main-mailer01--swap'/>       <target dev='hdb' bus='ide'/>       <alias name='ide0-0-1'/>       <address type='drive' controller='0' bus='0' unit='1'/>     </disk>     <controller type='ide' index='0'>       <alias name='ide0'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>     </controller>     <interface type='bridge'>       <mac address='52:54:00:d2:f7:31'/>       <source bridge='br0'/>       <target dev='vnet0'/>       <model type='virtio'/>       <alias name='net0'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>     </interface>     <serial type='pty'>       <source path='/dev/pts/0'/>       <target port='0'/>       <alias name='serial0'/>     </serial>     <console type='pty' tty='/dev/pts/0'>       <source path='/dev/pts/0'/>       <target type='serial' port='0'/>       <alias name='serial0'/>     </console>     <input type='mouse' bus='ps2'/>     <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>       <listen type='address' address='127.0.0.1'/>     </graphics>     <video>       <model type='cirrus' vram='9216' heads='1'/>       <alias name='video0'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>     </video>     <memballoon model='virtio'>       <alias name='balloon0'/>       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>     </memballoon>   </devices>   <seclabel type='dynamic' model='apparmor' relabel='yes'>     <label>libvirt-d41d1355-84e8-ae23-e84e-227bc0231b97</label>     <imagelabel>libvirt-d41d1355-84e8-ae23-e84e-227bc0231b97</imagelabel>   </seclabel> </domain> From within the guest: # cat /etc/network/interfaces # The primary network interface auto eth0 iface eth0 inet static         address 212.XX.239.100         netmask 255.255.255.240         network 212.XX.239.96         broadcast 212.XX.239.111         gateway 212.XX.239.97 # ifconfig eth0 Link encap:Ethernet HWaddr 52:54:00:d2:f7:31           inet addr:212.XX.239.100 Bcast:212.XX.239.111 Mask:255.255.255.240           inet6 addr: fe80::5054:ff:fed2:f731/64 Scope:Link           UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1           RX packets:5631830 errors:0 dropped:0 overruns:0 frame:0           TX packets:6683416 errors:0 dropped:0 overruns:0 carrier:0           collisions:0 txqueuelen:1000           RX bytes:2027322829 (2.0 GB) TX bytes:2076698690 (2.0 GB) A commandline which starts the KVM guest: /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 2048 -smp 1,sockets=1,cores=1,threads=1 -name mailer01 -uuid d41d1355-84e8-ae23-e84e-227bc0231b97 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/mailer01.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/dev/mapper/vg_main-mailer01--root,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/dev/mapper/vg_main-mailer01--swap,if=none,id=drive-ide0-0-1,format=raw -device ide-drive,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev tap,fd=18,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d2:f7:31,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 Problem: ------------ Periodically (at least once a day), one or more of the guests lose network connectivity. Ping responds with 'host unreachable', even from the dom host. Logging in via the serial console shows no problems: eth0 is up, can ping the local host, but no outside connectivity. Restart the network (/etc/init.d/networking restart) does nothing. Reboot the machine and it comes alive again. I've verified there's no arp games going on on the primary host (the arp tables remain the same before - when it had connectivity - and after - when it doesn't. This is a critical issue affecting production services on the latest LTS release of Ubuntu. It's similar to an issue which was 'resolved' in 10.04 but appears to have risen its ugly head again.
2012-09-24 20:05:45 Serge Hallyn bug added subscriber Ubuntu Stable Release Updates Team
2012-09-25 16:08:15 Matt Hilt bug added subscriber Matt Hilt
2012-09-27 15:26:14 Hannes Koschier bug added subscriber Hannes Koschier
2012-10-01 13:38:06 Andreas Ntaflos bug added subscriber Andreas Ntaflos
2012-10-03 17:18:48 Adam Conrad qemu-kvm (Ubuntu Precise): status In Progress Fix Committed
2012-10-03 17:18:54 Adam Conrad bug added subscriber SRU Verification
2012-10-03 17:18:59 Adam Conrad tags kernel-kvm kvm networking kernel-kvm kvm networking verification-needed
2012-10-03 17:20:28 Adam Conrad qemu-kvm (Ubuntu Precise): status Fix Committed In Progress
2012-10-03 18:33:33 Simon Déziel bug added subscriber Simon Déziel
2012-10-04 00:37:38 Adam Conrad qemu-kvm (Ubuntu Precise): status In Progress Fix Committed
2012-10-08 15:08:57 Robert Dupont qemu-kvm (Ubuntu Precise): status Fix Committed Fix Released
2012-10-08 15:27:43 Jean-Baptiste Lallement qemu-kvm (Ubuntu): status Fix Released Fix Committed
2012-10-08 15:27:48 Jean-Baptiste Lallement qemu-kvm (Ubuntu): status Fix Committed Fix Released
2012-10-08 15:27:52 Jean-Baptiste Lallement qemu-kvm (Ubuntu Precise): status Fix Released Fix Committed
2012-10-08 17:32:42 Serge Hallyn tags kernel-kvm kvm networking verification-needed kernel-kvm kvm networking verification-done
2012-10-10 18:43:49 Adam Conrad removed subscriber Ubuntu Stable Release Updates Team
2012-10-10 18:44:19 Launchpad Janitor qemu-kvm (Ubuntu Precise): status Fix Committed Fix Released
2012-12-02 12:39:14 Davide Guerri bug added subscriber Davide Guerri
2012-12-11 08:40:30 Qiu Yu bug added subscriber unicell
2012-12-12 18:59:26 Mike Cronnelly bug added subscriber Mike Cronnelly
2013-01-01 21:31:40 hanasaki bug added subscriber hanasaki
2013-01-05 19:37:29 Scobo bug added subscriber Scobo
2013-03-18 21:37:05 Julian Clark bug added subscriber Julian Clark
2013-09-03 14:37:35 Andrea Giardini bug added subscriber Andrea Giardini
2014-03-28 15:49:46 Russell McOrmond bug added subscriber Russell McOrmond
2014-07-10 17:53:28 Armindo Silva bug added subscriber Armindo Silva
2016-03-30 19:50:50 Kundai Andrew Midzi bug added subscriber Kundai Andrew Midzi