Xenial: ARM64: Unable to handle kernel NULL pointer dereference at virtual address 00000038

[Ming Lei]

When running 'stress-ng --all 64 -t 800 -v' on Xenial/ARM64, the following kernel oops is triggered.

[ 93.309158] Unable to handle kernel NULL pointer dereference at virtual address 00000038
[ 93.309160] pgd = ffff8007a5914000
[ 93.309163] [00000038] *pgd=00000047a5b15003, *pud=00000047a5b16003, *pmd=0000000000000000
[ 93.309167] Internal error: Oops: 96000006 [#1] SMP
[ 93.309202] Modules linked in: wp512 rmd320 rmd256 rmd160 rmd128 md4 algif_hash af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nls_iso8859_1 shpchp ghash_ce sha2_ce sha1_ce xgene_rng i2c_xgene_slimpro xgene_edac dwc3 edac_core udc_core i2c_designware_platform ulpi i2c_designware_core uio_pdrv_genirq uio gpio_keys rtc_efi autofs4 tg3 ptp pps_core sdhci_of_arasan ahci_xgene gpio_dwapb sdhci_pltfm xgene_enet libahci_platform sdhci libahci gpio_xgene_sb
[ 93.309208] CPU: 5 PID: 2062 Comm: stress-ng-remap Not tainted 4.4.0-15.31-generic #31+clk.1
[ 93.309209] Hardware name: AppliedMicro Mustang/Mustang, BIOS 2.04.08-beta Feb 2 2016
[ 93.309211] task: ffff8007a58c5b00 ti: ffff8007a7a30000 task.ti: ffff8007a7a30000
[ 93.309217] PC is at fput+0x20/0xd0
[ 93.309222] LR is at vma_do_fput+0x24/0x48
[ 93.309223] pc : [<ffff800000257578>] lr : [<ffff8000001fe1f4>] pstate: 60000145
[ 93.309224] sp : ffff8007a7a33e20
[ 93.309226] x29: ffff8007a7a33e20 x28: ffff8007a7a30000
[ 93.309228] x27: 0000000000000003 x26: ffff8007a59886e8
[ 93.309229] x25: ffff800746248ed8 x24: 0000000000008011
[ 93.309231] x23: 0000000000000000 x22: ffff80000091b780
[ 93.309232] x21: 0000000000001000 x20: 0000000000000000
[ 93.309233] x19: 0000000000000000 x18: 0000000000001140
[ 93.309235] x17: 0000ffff8915f2c0 x16: ffff800000133de0
[ 93.309236] x15: 0000ffff893c2000 x14: 0000000000000000
[ 93.309237] x13: 00000003e8000000 x12: 0000ffff734b0200
[ 93.309239] x11: 00000000003d0f00 x10: 0000000000000930
[ 93.309240] x9 : 0000000000000000 x8 : 0000000000000000
[ 93.309242] x7 : ffff80074645ac10 x6 : 0068000000000f43
[ 93.309243] x5 : 0068000000000f4f x4 : 0060000000000004
[ 93.309245] x3 : 0068000000000f53 x2 : 0000000000000038
[ 93.309246] x1 : ffff80000091b780 x0 : ffff8000001fe1f4

[ 93.309248] Process stress-ng-remap (pid: 2062, stack limit = 0xffff8007a7a30020)
[ 93.309250] Stack: (0xffff8007a7a33e20 to 0xffff8007a7a34000)
[ 93.309252] 3e20: ffff8007a7a33e40 ffff8000001fe1f4 0000000000000000 ffff8007a59886e8
[ 93.309253] 3e40: ffff8007a7a33e60 ffff80000020c1c0 0000ffff88300000 ffff800000d8e000
[ 93.309255] 3e60: 0000ffffc85b2280 ffff800000085c70 0000000000000000 0000ffff88301000
[ 93.309257] 3e80: ffffffffffffffff 0000ffff89161ea8 0000000080000000 0000000000000015
[ 93.309258] 3ea0: 000000000000011d 00000000000000ea ffff800000901000 ffff8007a7a30000
[ 93.309260] 3ec0: 0000000000001000 cb88537fdc8ba606 0000ffff88300000 0000000000001000
[ 93.309261] 3ee0: 0000000000000000 0000000000000000 0000000000000000 0000000000000200
[ 93.309262] 3f00: 00000000000000d1 00000000000000d1 00000000000000ea 0000000000002626
[ 93.309264] 3f20: 0101010101010101 000000000000001e 0000000000000018 00000003e8000000
[ 93.309266] 3f40: 0000000000000000 0000ffff893c2cc0 00000000004828c0 0000ffff89161ea0
[ 93.309267] 3f60: 0000000000001140 0000000000000050 0000ffff88301000 0000000000001000
[ 93.309269] 3f80: 0000000000001000 0000ffffc85b2368 0000ffff88d96fd0 0000000000004650
[ 93.309270] 3fa0: 00000000004558f0 0000ffffc85b34b8 000000000048f000 0000ffffc85b2280
[ 93.309271] 3fc0: 000000000042ec60 0000ffffc85b2280 0000ffff89161ea8 0000000080000000
[ 93.309273] 3fe0: 0000ffff88300000 00000000000000ea ffff8007a7a33fe8 0000000000000000
[ 93.309274] Call trace:
[ 93.309277] [<ffff800000257578>] fput+0x20/0xd0
[ 93.309280] [<ffff8000001fe1f4>] vma_do_fput+0x24/0x48
[ 93.309283] [<ffff80000020c1c0>] SyS_remap_file_pages+0x258/0x2a0
[ 93.309287] [<ffff800000085c70>] el0_svc_naked+0x24/0x28
[ 93.309289] Code: aa1e03e0 d503201f 9100e262 f9800051 (c85f7c40)
[ 93.309308] ---[ end trace e42a31bddbea7038 ]---
[ 93.672808] Unable to handle kernel NULL pointer dereference at virtual address 00000038
[ 93.672809] pgd = ffff8007d8f22000
[ 93.672813] [00000038] *pgd=00000047d9123003, *pud=00000047d9124003, *pmd=0000000000000000
[ 93.672817] Internal error: Oops: 96000006 [#2] SMP
[ 93.672853] Modules linked in: wp512 rmd320 rmd256 rmd160 rmd128 md4 algif_hash af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nls_iso8859_1 shpchp ghash_ce sha2_ce sha1_ce xgene_rng i2c_xgene_slimpro xgene_edac dwc3 edac_core udc_core i2c_designware_platform ulpi i2c_designware_core uio_pdrv_genirq uio gpio_keys rtc_efi autofs4 tg3 ptp pps_core sdhci_of_arasan ahci_xgene gpio_dwapb sdhci_pltfm xgene_enet libahci_platform sdhci libahci gpio_xgene_sb
[ 93.672858] CPU: 0 PID: 1613 Comm: stress-ng-remap Tainted: G D 4.4.0-15.31-generic #31+clk.1
[ 93.672859] Hardware name: AppliedMicro Mustang/Mustang, BIOS 2.04.08-beta Feb 2 2016
[ 93.672860] task: ffff8007d8ea4e00 ti: ffff8007d8f40000 task.ti: ffff8007d8f40000
[ 93.672868] PC is at fput+0x20/0xd0
[ 93.672872] LR is at vma_do_fput+0x24/0x48
[ 93.672873] pc : [<ffff800000257578>] lr : [<ffff8000001fe1f4>] pstate: 60000145
[ 93.672874] sp : ffff8007d8f43e20
[ 93.672876] x29: ffff8007d8f43e20 x28: ffff8007d8f40000
[ 93.672878] x27: 0000000000000003 x26: ffff8007d8ed0d68
[ 93.672879] x25: ffff80077ebe2d48 x24: 0000000000008011
[ 93.672881] x23: 000000000000018d x22: ffff80000091b780
[ 93.672882] x21: 0000000000001000 x20: 0000000000000000
[ 93.672884] x19: 0000000000000000 x18: 0000000000001140
[ 93.672885] x17: 0000ffff892089a0 x16: ffff800000255e18
[ 93.672887] x15: 0000ffff893c2cc0 x14: 0000000000000000
[ 93.672888] x13: 00000003e8000000 x12: 0000ffffc85b3330
[ 93.672890] x11: 0000000000000f11 x10: 0000000000000930
[ 93.672891] x9 : ffff8007d8f43c50 x8 : 000000000000018d
[ 93.672893] x7 : ffff80077a29af30 x6 : 0068000000000f43
[ 93.672894] x5 : 0068000000000f4f x4 : 0060000000000004
[ 93.672896] x3 : 0068000000000f53 x2 : 0000000000000038
[ 93.672897] x1 : ffff80000091b780 x0 : ffff8000001fe1f4

[ 93.672899] Process stress-ng-remap (pid: 1613, stack limit = 0xffff8007d8f40020)
[ 93.672901] Stack: (0xffff8007d8f43e20 to 0xffff8007d8f44000)
[ 93.672903] 3e20: ffff8007d8f43e40 ffff8000001fe1f4 0000000000000000 ffff8007d8ed0d68
[ 93.672905] 3e40: ffff8007d8f43e60 ffff80000020c1c0 0000ffff88426000 ffff800000d8e000
[ 93.672907] 3e60: 0000ffffc85b2280 ffff800000085c70 0000000000000000 0000ffff88427000
[ 93.672908] 3e80: ffffffffffffffff 0000ffff89161ea8 0000000080000000 0000000000000015
[ 93.672910] 3ea0: 000000000000011d 00000000000000ea ffff800000901000 ffff8007d8f40000
[ 93.672911] 3ec0: 0000000000001000 cb88537fdc8ba606 0000ffff88426000 0000000000001000
[ 93.672913] 3ee0: 0000000000000000 000000000000018d 0000000000000000 0000000000000200
[ 93.672914] 3f00: 000000003d71bd5b 0000000024f659ca 00000000000000ea 00000000000022b2
[ 93.672916] 3f20: 0101010101010101 000000000000001e 0000000000000018 00000003e8000000
[ 93.672917] 3f40: 0000000000000000 0000ffff893c2cc0 00000000004828c0 0000ffff89161ea0
[ 93.672919] 3f60: 0000000000001140 0000000000000176 0000ffff88427000 0000000000001000
[ 93.672921] 3f80: 0000000000001000 0000ffffc85b2368 0000ffff88d96860 0000000000004650
[ 93.672922] 3fa0: 00000000004558f0 0000ffffc85b34b8 000000000048f000 0000ffffc85b2280
[ 93.672924] 3fc0: 000000000042ec60 0000ffffc85b2280 0000ffff89161ea8 0000000080000000
[ 93.672925] 3fe0: 0000ffff88426000 00000000000000ea 0000000000000000 000000000000b700
[ 93.672926] Call trace:
[ 93.672929] [<ffff800000257578>] fput+0x20/0xd0
[ 93.672932] [<ffff8000001fe1f4>] vma_do_fput+0x24/0x48
[ 93.672935] [<ffff80000020c1c0>] SyS_remap_file_pages+0x258/0x2a0
[ 93.672939] [<ffff800000085c70>] el0_svc_naked+0x24/0x28
[ 93.672942] Code: aa1e03e0 d503201f 9100e262 f9800051 (c85f7c40)
[ 93.672960] ---[ end trace e42a31bddbea7039 ]---

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1575506

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ming Lei (tom-leiming)]

The issue can be reproduced on '4.4.0-22-generic #38' too

[Ming Lei (tom-leiming)]

Upstream 4.6-rc6 hasn't this problem

[Ming Lei (tom-leiming)]

On Wed, Apr 27, 2016 at 4:31 PM, Ming Lei <email address hidden> wrote:
> Upstream 4.6-rc6 hasn't this problem
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1575506
>
> Title:
> Xenial: ARM64: Unable to handle kernel NULL pointer dereference at
> virtual address 00000038
>
> Status in linux package in Ubuntu:
> Confirmed
>
> Bug description:
>
> When running 'stress-ng --all 64 -t 800 -v' on Xenial/ARM64, the following kernel oops is triggered.
>
>
> [ 93.309158] Unable to handle kernel NULL pointer dereference at virtual address 00000038

The issue is caused by the following commit:

commit 3df6b7b72fd8cb3fd8dd1bee6916b1c91c74d441
Author: Tim Gardner <email address hidden>
Date: Wed Dec 2 07:28:37 2015 -0700

    UBUNTU: SAUCE: AUFS

    Signed-off-by: Tim Gardner <email address hidden>

Once it is reverted, no oops is observed any more.

Thanks,

[Tim Gardner (timg-tpi)]

Ming - please try the kernel at http://people.canonical.com/~rtg/lp1575506/ - I've updated AUFS to the latest stable branch. Source at git://kernel.ubuntu.com/rtg/ubuntu-xenial.git aufs

[Ming Lei (tom-leiming)]

On Thu, Apr 28, 2016 at 9:55 PM, Tim Gardner <email address hidden> wrote:
> Ming - please try the kernel at
> http://people.canonical.com/~rtg/lp1575506/ - I've updated AUFS to the
> latest stable branch. Source at git://kernel.ubuntu.com/rtg/ubuntu-
> xenial.git aufs

Looks no difference by installing the new kernel of 'Linux ubuntu
4.4.0-23-generic #39' on arm64.

Kernel oops can happen immediately by running the following command:

       stress-ng --remap 64 --timeout 60 -v

Thanks,
Ming

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1575506
>
> Title:
> Xenial: ARM64: Unable to handle kernel NULL pointer dereference at
> virtual address 00000038
>
> Status in linux package in Ubuntu:
> Confirmed
>
> Bug description:
>
> When running 'stress-ng --all 64 -t 800 -v' on Xenial/ARM64, the following kernel oops is triggered.
>
>
> [ 93.309158] Unable to handle kernel NULL pointer dereference at virtual address 00000038
> [ 93.309160] pgd = ffff8007a5914000
> [ 93.309163] [00000038] *pgd=00000047a5b15003, *pud=00000047a5b16003, *pmd=0000000000000000
> [ 93.309167] Internal error: Oops: 96000006 [#1] SMP
> [ 93.309202] Modules linked in: wp512 rmd320 rmd256 rmd160 rmd128 md4 algif_hash af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables nls_iso8859_1 shpchp ghash_ce sha2_ce sha1_ce xgene_rng i2c_xgene_slimpro xgene_edac dwc3 edac_core udc_core i2c_designware_platform ulpi i2c_designware_core uio_pdrv_genirq uio gpio_keys rtc_efi autofs4 tg3 ptp pps_core sdhci_of_arasan ahci_xgene gpio_dwapb sdhci_pltfm xgene_enet libahci_platform sdhci libahci gpio_xgene_sb
> [ 93.309208] CPU: 5 PID: 2062 Comm: stress-ng-remap Not tainted 4.4.0-15.31-generic #31+clk.1
> [ 93.309209] Hardware name: AppliedMicro Mustang/Mustang, BIOS 2.04.08-beta Feb 2 2016
> [ 93.309211] task: ffff8007a58c5b00 ti: ffff8007a7a30000 task.ti: ffff8007a7a30000
> [ 93.309217] PC is at fput+0x20/0xd0
> [ 93.309222] LR is at vma_do_fput+0x24/0x48
> [ 93.309223] pc : [<ffff800000257578>] lr : [<ffff8000001fe1f4>] pstate: 60000145
> [ 93.309224] sp : ffff8007a7a33e20
> [ 93.309226] x29: ffff8007a7a33e20 x28: ffff8007a7a30000
> [ 93.309228] x27: 0000000000000003 x26: ffff8007a59886e8
> [ 93.309229] x25: ffff800746248ed8 x24: 0000000000008011
> [ 93.309231] x23: 0000000000000000 x22: ffff80000091b780
> [ 93.309232] x21: 0000000000001000 x20: 0000000000000000
> [ 93.309233] x19: 0000000000000000 x18: 0000000000001140
> [ 93.309235] x17: 0000ffff8915f2c0 x16: ffff800000133de0
> [ 93.309236] x15: 0000ffff893c2000 x14: 0000000000000000
> [ 93.309237] x13: 00000003e8000000 x12: 0000ffff734b0200
> [ 93.309239] x11: 00000000003d0f00 x10: 0000000000000930
> [ 93.309240] x9 : 0000000000000000 x8 : 0000000000000000
> [ 93.309242] x7 : ffff80074645ac10 x6 : 0068000000000f...