2017-12-05 11:44:02 |
James Page |
bug |
|
|
added bug |
2017-12-05 11:47:19 |
James Page |
description |
Reproducable on bionic using the autopkgtest's from openvswitch on i386:
Dec 5 11:41:05 autopkgtest kernel: [ 41.392680] device ovs-system left promiscuous mode
Dec 5 11:41:05 autopkgtest kernel: [ 41.420568] BUG: unable to handle kernel NULL pointer dereference at (null)
Dec 5 11:41:05 autopkgtest kernel: [ 41.421000] IP: igmp_group_dropped+0x21/0x220
Dec 5 11:41:05 autopkgtest kernel: [ 41.421246] *pdpt = 000000001d62c001 *pde = 0000000000000000
Dec 5 11:41:05 autopkgtest kernel: [ 41.421248]
Dec 5 11:41:05 autopkgtest kernel: [ 41.421659] Oops: 0000 [#1] SMP
Dec 5 11:41:05 autopkgtest kernel: [ 41.421852] Modules linked in: veth openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c 9p fscache ppdev kvm_intel kvm 9pnet_virtio irqbypass input_leds joydev 9pnet parport_pc serio_raw parport i2c_piix4 qemu_fw_cfg mac_hid sch_fq_codel ip_tables x_tables autofs4 btrfs xor raid6_pq psmouse virtio_blk virtio_net pata_acpi floppy
Dec 5 11:41:05 autopkgtest kernel: [ 41.423855] CPU: 0 PID: 5 Comm: kworker/u2:0 Tainted: G W 4.13.0-18-generic #21-Ubuntu
Dec 5 11:41:05 autopkgtest kernel: [ 41.424355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
Dec 5 11:41:05 autopkgtest kernel: [ 41.424849] Workqueue: netns cleanup_net
Dec 5 11:41:05 autopkgtest kernel: [ 41.425071] task: db8fba80 task.stack: dba10000
Dec 5 11:41:05 autopkgtest kernel: [ 41.425346] EIP: igmp_group_dropped+0x21/0x220
Dec 5 11:41:05 autopkgtest kernel: [ 41.425656] EFLAGS: 00010202 CPU: 0
Dec 5 11:41:05 autopkgtest kernel: [ 41.425864] EAX: 00000000 EBX: dd726360 ECX: dba11e6c EDX: 00000002
Dec 5 11:41:05 autopkgtest kernel: [ 41.426335] ESI: 00000000 EDI: dd4db500 EBP: dba11dcc ESP: dba11d94
Dec 5 11:41:05 autopkgtest kernel: [ 41.426687] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Dec 5 11:41:05 autopkgtest kernel: [ 41.426990] CR0: 80050033 CR2: 00000000 CR3: 1e6d6d60 CR4: 000006f0
Dec 5 11:41:05 autopkgtest kernel: [ 41.427340] Call Trace:
Dec 5 11:41:05 autopkgtest kernel: [ 41.427485] ? __wake_up+0x36/0x40
Dec 5 11:41:05 autopkgtest kernel: [ 41.427680] ip_mc_down+0x27/0x90
Dec 5 11:41:05 autopkgtest kernel: [ 41.427869] inetdev_event+0x398/0x4e0
Dec 5 11:41:05 autopkgtest kernel: [ 41.428082] ? skb_dequeue+0x5b/0x70
Dec 5 11:41:05 autopkgtest kernel: [ 41.428286] ? wireless_nlevent_flush+0x4c/0x90
Dec 5 11:41:05 autopkgtest kernel: [ 41.428541] notifier_call_chain+0x4e/0x70
Dec 5 11:41:05 autopkgtest kernel: [ 41.428772] raw_notifier_call_chain+0x11/0x20
Dec 5 11:41:05 autopkgtest kernel: [ 41.429023] call_netdevice_notifiers_info+0x2a/0x60
Dec 5 11:41:05 autopkgtest kernel: [ 41.429301] dev_close_many+0x9d/0xe0
Dec 5 11:41:05 autopkgtest kernel: [ 41.429509] rollback_registered_many+0xd7/0x380
Dec 5 11:41:05 autopkgtest kernel: [ 41.429768] unregister_netdevice_many.part.102+0x10/0x80
Dec 5 11:41:05 autopkgtest kernel: [ 41.430075] default_device_exit_batch+0x134/0x160
Dec 5 11:41:05 autopkgtest kernel: [ 41.430344] ? do_wait_intr_irq+0x80/0x80
Dec 5 11:41:05 autopkgtest kernel: [ 41.430650] ops_exit_list.isra.8+0x4d/0x60
Dec 5 11:41:05 autopkgtest kernel: [ 41.430886] cleanup_net+0x18e/0x260
Dec 5 11:41:05 autopkgtest kernel: [ 41.431090] process_one_work+0x1a0/0x390
Dec 5 11:41:05 autopkgtest kernel: [ 41.431317] worker_thread+0x37/0x450
Dec 5 11:41:05 autopkgtest kernel: [ 41.431525] kthread+0xf3/0x110
Dec 5 11:41:05 autopkgtest kernel: [ 41.431714] ? process_one_work+0x390/0x390
Dec 5 11:41:05 autopkgtest kernel: [ 41.431941] ? kthread_create_on_node+0x20/0x20
Dec 5 11:41:05 autopkgtest kernel: [ 41.432187] ret_from_fork+0x19/0x24
Dec 5 11:41:05 autopkgtest kernel: [ 41.432382] Code: 90 90 90 90 90 90 90 90 90 90 3e 8d 74 26 00 55 89 e5 57 56 53 89 c3 83 ec 2c 8b 33 65 a1 14 00 00 00 89 45 f0 31 c0 80 7b 4b 00 <8b> 06 8b b8 20 03 00 00 8b 43 04 0f 85 5e 01 00 00 3d e0 00 00
Dec 5 11:41:05 autopkgtest kernel: [ 41.433405] EIP: igmp_group_dropped+0x21/0x220 SS:ESP: 0068:dba11d94
Dec 5 11:41:05 autopkgtest kernel: [ 41.433750] CR2: 0000000000000000
Dec 5 11:41:05 autopkgtest kernel: [ 41.433961] ---[ end trace 595db54cab84070c ]---
system then becomes unresponsive; no further interfaces can be created. |
Reproducable on bionic using the autopkgtest's from openvswitch on i386:
[ 41.420568] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 41.421000] IP: igmp_group_dropped+0x21/0x220
[ 41.421246] *pdpt = 000000001d62c001 *pde = 0000000000000000
[ 41.421659] Oops: 0000 [#1] SMP
[ 41.421852] Modules linked in: veth openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c 9p fscache ppdev kvm_intel kvm 9pnet_virtio irqbypass input_leds joydev 9pnet parport_pc serio_raw parport i2c_piix4 qemu_fw_cfg mac_hid sch_fq_codel ip_tables x_tables autofs4 btrfs xor raid6_pq psmouse virtio_blk virtio_net pata_acpi floppy
[ 41.423855] CPU: 0 PID: 5 Comm: kworker/u2:0 Tainted: G W 4.13.0-18-generic #21-Ubuntu
[ 41.424355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 41.424849] Workqueue: netns cleanup_net
[ 41.425071] task: db8fba80 task.stack: dba10000
[ 41.425346] EIP: igmp_group_dropped+0x21/0x220
[ 41.425656] EFLAGS: 00010202 CPU: 0
[ 41.425864] EAX: 00000000 EBX: dd726360 ECX: dba11e6c EDX: 00000002
[ 41.426335] ESI: 00000000 EDI: dd4db500 EBP: dba11dcc ESP: dba11d94
[ 41.426687] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 41.426990] CR0: 80050033 CR2: 00000000 CR3: 1e6d6d60 CR4: 000006f0
[ 41.427340] Call Trace:
[ 41.427485] ? __wake_up+0x36/0x40
[ 41.427680] ip_mc_down+0x27/0x90
[ 41.427869] inetdev_event+0x398/0x4e0
[ 41.428082] ? skb_dequeue+0x5b/0x70
[ 41.428286] ? wireless_nlevent_flush+0x4c/0x90
[ 41.428541] notifier_call_chain+0x4e/0x70
[ 41.428772] raw_notifier_call_chain+0x11/0x20
[ 41.429023] call_netdevice_notifiers_info+0x2a/0x60
[ 41.429301] dev_close_many+0x9d/0xe0
[ 41.429509] rollback_registered_many+0xd7/0x380
[ 41.429768] unregister_netdevice_many.part.102+0x10/0x80
[ 41.430075] default_device_exit_batch+0x134/0x160
[ 41.430344] ? do_wait_intr_irq+0x80/0x80
[ 41.430650] ops_exit_list.isra.8+0x4d/0x60
[ 41.430886] cleanup_net+0x18e/0x260
[ 41.431090] process_one_work+0x1a0/0x390
[ 41.431317] worker_thread+0x37/0x450
[ 41.431525] kthread+0xf3/0x110
[ 41.431714] ? process_one_work+0x390/0x390
[ 41.431941] ? kthread_create_on_node+0x20/0x20
[ 41.432187] ret_from_fork+0x19/0x24
[ 41.432382] Code: 90 90 90 90 90 90 90 90 90 90 3e 8d 74 26 00 55 89 e5 57 56 53 89 c3 83 ec 2c 8b 33 65 a1 14 00 00 00 89 45 f0 31 c0 80 7b 4b 00 <8b> 06 8b b8 20 03 00 00 8b 43 04 0f 85 5e 01 00 00 3d e0 00 00
[ 41.433405] EIP: igmp_group_dropped+0x21/0x220 SS:ESP: 0068:dba11d94
[ 41.433750] CR2: 0000000000000000
[ 41.433961] ---[ end trace 595db54cab84070c ]---
system then becomes unresponsive; no further interfaces can be created. |
|
2017-12-05 12:00:05 |
Ubuntu Kernel Bot |
linux (Ubuntu): status |
New |
Incomplete |
|
2017-12-05 12:00:06 |
Ubuntu Kernel Bot |
tags |
|
artful |
|
2017-12-05 15:55:57 |
James Page |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
2017-12-05 20:43:43 |
Joseph Salisbury |
linux (Ubuntu): importance |
Undecided |
Medium |
|
2017-12-05 20:43:50 |
Joseph Salisbury |
nominated for series |
|
Ubuntu Bionic |
|
2017-12-05 20:43:50 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Bionic) |
|
2017-12-05 20:44:10 |
Joseph Salisbury |
linux (Ubuntu Bionic): importance |
Medium |
High |
|
2017-12-05 20:44:19 |
Joseph Salisbury |
tags |
artful |
artful kernel-key |
|
2017-12-05 20:45:54 |
Joseph Salisbury |
linux (Ubuntu Bionic): status |
Confirmed |
Triaged |
|
2017-12-11 15:49:09 |
Joseph Salisbury |
tags |
artful kernel-key |
artful kernel-da-key |
|
2017-12-13 12:44:34 |
James Page |
bug task added |
|
openvswitch (Ubuntu) |
|
2017-12-18 21:31:58 |
Joseph Salisbury |
nominated for series |
|
Ubuntu Artful |
|
2017-12-18 21:31:58 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Artful) |
|
2017-12-18 21:31:58 |
Joseph Salisbury |
bug task added |
|
openvswitch (Ubuntu Artful) |
|
2017-12-18 21:32:06 |
Joseph Salisbury |
linux (Ubuntu Artful): status |
New |
In Progress |
|
2017-12-18 21:32:08 |
Joseph Salisbury |
linux (Ubuntu Artful): importance |
Undecided |
High |
|
2017-12-18 21:32:12 |
Joseph Salisbury |
linux (Ubuntu Artful): assignee |
|
Joseph Salisbury (jsalisbury) |
|
2017-12-18 21:32:20 |
Joseph Salisbury |
linux (Ubuntu Bionic): status |
Triaged |
Fix Committed |
|
2017-12-18 21:32:24 |
Joseph Salisbury |
linux (Ubuntu Bionic): assignee |
|
Joseph Salisbury (jsalisbury) |
|
2017-12-18 21:32:32 |
Joseph Salisbury |
tags |
artful kernel-da-key |
artful bionic kernel-da-key |
|
2018-01-03 17:07:34 |
Launchpad Janitor |
openvswitch (Ubuntu): status |
New |
Confirmed |
|
2018-01-03 17:07:34 |
Launchpad Janitor |
openvswitch (Ubuntu Artful): status |
New |
Confirmed |
|
2018-01-03 20:09:39 |
Steve Langasek |
summary |
openvswitch: kernel opps destroying interfaces on i386 |
openvswitch: kernel oops destroying interfaces on i386 |
|
2018-01-04 07:17:06 |
Christian Ehrhardt |
bug |
|
|
added subscriber ChristianEhrhardt |
2018-01-04 19:21:51 |
Joseph Salisbury |
attachment added |
|
hang.jpg https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1736390/+attachment/5031209/+files/hang.jpg |
|
2018-01-31 22:29:41 |
Joseph Salisbury |
linux (Ubuntu Bionic): status |
Fix Committed |
In Progress |
|
2018-02-06 09:53:10 |
James Page |
openvswitch (Ubuntu Artful): status |
Confirmed |
Invalid |
|
2018-02-06 09:53:12 |
James Page |
openvswitch (Ubuntu Bionic): status |
Confirmed |
Invalid |
|
2018-07-24 09:31:43 |
Andy Whitcroft |
linux (Ubuntu Artful): status |
In Progress |
Won't Fix |
|
2018-09-12 13:41:01 |
Joseph Salisbury |
nominated for series |
|
Ubuntu Cosmic |
|
2018-09-12 13:41:01 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Cosmic) |
|
2018-09-12 13:41:01 |
Joseph Salisbury |
bug task added |
|
openvswitch (Ubuntu Cosmic) |
|
2019-01-19 17:22:37 |
Joseph Salisbury |
linux (Ubuntu Bionic): assignee |
Joseph Salisbury (jsalisbury) |
|
|
2019-01-19 17:22:42 |
Joseph Salisbury |
linux (Ubuntu Cosmic): assignee |
Joseph Salisbury (jsalisbury) |
|
|
2019-01-19 17:22:46 |
Joseph Salisbury |
linux (Ubuntu Artful): assignee |
Joseph Salisbury (jsalisbury) |
|
|
2019-01-19 17:22:49 |
Joseph Salisbury |
linux (Ubuntu): assignee |
Joseph Salisbury (jsalisbury) |
|
|
2019-03-11 07:10:34 |
Juerg Haefliger |
nominated for series |
|
Ubuntu Disco |
|
2019-03-11 07:10:34 |
Juerg Haefliger |
bug task added |
|
linux (Ubuntu Disco) |
|
2019-03-11 07:10:34 |
Juerg Haefliger |
bug task added |
|
openvswitch (Ubuntu Disco) |
|
2019-03-11 07:35:05 |
Juerg Haefliger |
description |
Reproducable on bionic using the autopkgtest's from openvswitch on i386:
[ 41.420568] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 41.421000] IP: igmp_group_dropped+0x21/0x220
[ 41.421246] *pdpt = 000000001d62c001 *pde = 0000000000000000
[ 41.421659] Oops: 0000 [#1] SMP
[ 41.421852] Modules linked in: veth openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c 9p fscache ppdev kvm_intel kvm 9pnet_virtio irqbypass input_leds joydev 9pnet parport_pc serio_raw parport i2c_piix4 qemu_fw_cfg mac_hid sch_fq_codel ip_tables x_tables autofs4 btrfs xor raid6_pq psmouse virtio_blk virtio_net pata_acpi floppy
[ 41.423855] CPU: 0 PID: 5 Comm: kworker/u2:0 Tainted: G W 4.13.0-18-generic #21-Ubuntu
[ 41.424355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 41.424849] Workqueue: netns cleanup_net
[ 41.425071] task: db8fba80 task.stack: dba10000
[ 41.425346] EIP: igmp_group_dropped+0x21/0x220
[ 41.425656] EFLAGS: 00010202 CPU: 0
[ 41.425864] EAX: 00000000 EBX: dd726360 ECX: dba11e6c EDX: 00000002
[ 41.426335] ESI: 00000000 EDI: dd4db500 EBP: dba11dcc ESP: dba11d94
[ 41.426687] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 41.426990] CR0: 80050033 CR2: 00000000 CR3: 1e6d6d60 CR4: 000006f0
[ 41.427340] Call Trace:
[ 41.427485] ? __wake_up+0x36/0x40
[ 41.427680] ip_mc_down+0x27/0x90
[ 41.427869] inetdev_event+0x398/0x4e0
[ 41.428082] ? skb_dequeue+0x5b/0x70
[ 41.428286] ? wireless_nlevent_flush+0x4c/0x90
[ 41.428541] notifier_call_chain+0x4e/0x70
[ 41.428772] raw_notifier_call_chain+0x11/0x20
[ 41.429023] call_netdevice_notifiers_info+0x2a/0x60
[ 41.429301] dev_close_many+0x9d/0xe0
[ 41.429509] rollback_registered_many+0xd7/0x380
[ 41.429768] unregister_netdevice_many.part.102+0x10/0x80
[ 41.430075] default_device_exit_batch+0x134/0x160
[ 41.430344] ? do_wait_intr_irq+0x80/0x80
[ 41.430650] ops_exit_list.isra.8+0x4d/0x60
[ 41.430886] cleanup_net+0x18e/0x260
[ 41.431090] process_one_work+0x1a0/0x390
[ 41.431317] worker_thread+0x37/0x450
[ 41.431525] kthread+0xf3/0x110
[ 41.431714] ? process_one_work+0x390/0x390
[ 41.431941] ? kthread_create_on_node+0x20/0x20
[ 41.432187] ret_from_fork+0x19/0x24
[ 41.432382] Code: 90 90 90 90 90 90 90 90 90 90 3e 8d 74 26 00 55 89 e5 57 56 53 89 c3 83 ec 2c 8b 33 65 a1 14 00 00 00 89 45 f0 31 c0 80 7b 4b 00 <8b> 06 8b b8 20 03 00 00 8b 43 04 0f 85 5e 01 00 00 3d e0 00 00
[ 41.433405] EIP: igmp_group_dropped+0x21/0x220 SS:ESP: 0068:dba11d94
[ 41.433750] CR2: 0000000000000000
[ 41.433961] ---[ end trace 595db54cab84070c ]---
system then becomes unresponsive; no further interfaces can be created. |
== SRU Justification ==
Commit 120645513f55 ("openvswitch: Add eventmask support to CT action.") introduced a regression on i386. Simply running the following commands in a loop will trigger a crash rather quickly:
ovs-vsctl add-br test
ovs-vsctl del-br test
== Fix ==
Disable the logic introduced by the above commit on i386.
== Regression Potential ==
Low, the above commit introduced a new feature. Per upstream, the result of not having this feature results in higher CPU usage and potential buffering issues in user space.
== Test Case ==
See SRU justification above.
Original bug description:
Reproducable on bionic using the autopkgtest's from openvswitch on i386:
[ 41.420568] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 41.421000] IP: igmp_group_dropped+0x21/0x220
[ 41.421246] *pdpt = 000000001d62c001 *pde = 0000000000000000
[ 41.421659] Oops: 0000 [#1] SMP
[ 41.421852] Modules linked in: veth openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c 9p fscache ppdev kvm_intel kvm 9pnet_virtio irqbypass input_leds joydev 9pnet parport_pc serio_raw parport i2c_piix4 qemu_fw_cfg mac_hid sch_fq_codel ip_tables x_tables autofs4 btrfs xor raid6_pq psmouse virtio_blk virtio_net pata_acpi floppy
[ 41.423855] CPU: 0 PID: 5 Comm: kworker/u2:0 Tainted: G W 4.13.0-18-generic #21-Ubuntu
[ 41.424355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1ubuntu1 04/01/2014
[ 41.424849] Workqueue: netns cleanup_net
[ 41.425071] task: db8fba80 task.stack: dba10000
[ 41.425346] EIP: igmp_group_dropped+0x21/0x220
[ 41.425656] EFLAGS: 00010202 CPU: 0
[ 41.425864] EAX: 00000000 EBX: dd726360 ECX: dba11e6c EDX: 00000002
[ 41.426335] ESI: 00000000 EDI: dd4db500 EBP: dba11dcc ESP: dba11d94
[ 41.426687] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 41.426990] CR0: 80050033 CR2: 00000000 CR3: 1e6d6d60 CR4: 000006f0
[ 41.427340] Call Trace:
[ 41.427485] ? __wake_up+0x36/0x40
[ 41.427680] ip_mc_down+0x27/0x90
[ 41.427869] inetdev_event+0x398/0x4e0
[ 41.428082] ? skb_dequeue+0x5b/0x70
[ 41.428286] ? wireless_nlevent_flush+0x4c/0x90
[ 41.428541] notifier_call_chain+0x4e/0x70
[ 41.428772] raw_notifier_call_chain+0x11/0x20
[ 41.429023] call_netdevice_notifiers_info+0x2a/0x60
[ 41.429301] dev_close_many+0x9d/0xe0
[ 41.429509] rollback_registered_many+0xd7/0x380
[ 41.429768] unregister_netdevice_many.part.102+0x10/0x80
[ 41.430075] default_device_exit_batch+0x134/0x160
[ 41.430344] ? do_wait_intr_irq+0x80/0x80
[ 41.430650] ops_exit_list.isra.8+0x4d/0x60
[ 41.430886] cleanup_net+0x18e/0x260
[ 41.431090] process_one_work+0x1a0/0x390
[ 41.431317] worker_thread+0x37/0x450
[ 41.431525] kthread+0xf3/0x110
[ 41.431714] ? process_one_work+0x390/0x390
[ 41.431941] ? kthread_create_on_node+0x20/0x20
[ 41.432187] ret_from_fork+0x19/0x24
[ 41.432382] Code: 90 90 90 90 90 90 90 90 90 90 3e 8d 74 26 00 55 89 e5 57 56 53 89 c3 83 ec 2c 8b 33 65 a1 14 00 00 00 89 45 f0 31 c0 80 7b 4b 00 <8b> 06 8b b8 20 03 00 00 8b 43 04 0f 85 5e 01 00 00 3d e0 00 00
[ 41.433405] EIP: igmp_group_dropped+0x21/0x220 SS:ESP: 0068:dba11d94
[ 41.433750] CR2: 0000000000000000
[ 41.433961] ---[ end trace 595db54cab84070c ]---
system then becomes unresponsive; no further interfaces can be created. |
|
2019-04-05 07:38:01 |
Andrea Righi |
bug |
|
|
added subscriber Andrea Righi |
2019-04-08 08:25:39 |
Juerg Haefliger |
marked as duplicate |
|
1813244 |
|
2019-07-24 20:21:10 |
Brad Figg |
tags |
artful bionic kernel-da-key |
artful bionic cscc kernel-da-key |
|