pcie failed to assign [mem size 0x00900000]

[Fred Kimmy]

[Bug Description]
pcie failed to assign [mem size 0x00900000]

[Steps to Reproduce]
root@ubuntu-8:~# dmesgecho 1 > /sys/bus/pci/devices/0000\:7d\:00.0/remove
root@ubuntu-8:~# dmesg
[65515.491437] hns3 0000:7d:00.0 eno1: link down
[65516.196296] pci 0000:7d:00.0: Removing from iommu group 27
root@ubuntu-8:~#
root@ubuntu-8:~# echo 1 > /sys/bus/pci/devices/0000\:7d\:00.0/remove rescan
root@ubuntu-8:~# dmesg

[Actual Results]
[65538.929105] pci 0000:7c:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus 7d] add_size 700000 add_align 100000
[65538.929122] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00900000]
[65538.929130] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00900000]
[65538.929140] pci 0000:7c:00.0: BAR 14: no space for [mem size 0x00200000]
[65538.929147] pci 0000:7c:00.0: BAR 14: failed to assign [mem size 0x00200000]
[65538.929168] pci 0000:7d:00.0: BAR 2: assigned [mem 0x120000000-0x1200fffff 64bit pref]
[65538.929183] pci 0000:7d:00.0: BAR 9: assigned [mem 0x120100000-0x1207fffff 64bit pref]
[65538.929195] pci 0000:7d:00.0: BAR 0: assigned [mem 0x122000000-0x12200ffff 64bit pref]
[65538.929209] pci 0000:7d:00.0: BAR 7: assigned [mem 0x122010000-0x12207ffff 64bit pref

[Expected Results]
no fail

[Reproducibility]

[Additional information]
(Firmware version, kernel version, affected hardware, etc. if required):
OS:Ubuntu20.04.1
Linux ubuntu-8 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

[Resolution]

内核commit ID：c13704f5685d ("PCI: Avoid double hpmemsize MMIO window assignment")
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.5.y&id=c13704f5685deb7d6eb21e293233e0901ed77377

[Ike Panhc (ikepanhc)]

I can not reproduce with 5.4.0-84.94

$ cat /proc/version
Linux version 5.4.0-84-generic (buildd@bos02-arm64-020) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #94-Ubuntu SMP Thu Aug 26 19:28:28 UTC 2021
$ echo 1 | sudo tee /sys/bus/pci/devices/0000:bd:00.1/remove
1
$ dmesg
[ 787.925286] hns3 0000:bd:00.1 enp189s0f1: link down
$ echo 1 | sudo tee /sys/devices/pci0000:bc/0000:bc:00.0/rescan
1
$ dmesg
[ 787.925286] hns3 0000:bd:00.1 enp189s0f1: link down
[ 957.878375] pci 0000:bd:00.1: [19e5:a222] type 00 class 0x020000
[ 957.878386] pci 0000:bd:00.1: reg 0x10: [mem 0x200120840000-0x20012084ffff 64bit pref]
[ 957.878390] pci 0000:bd:00.1: reg 0x18: [mem 0x200120400000-0x2001204fffff 64bit pref]
[ 957.878426] pci 0000:bd:00.1: reg 0x224: [mem 0x200120850000-0x20012085ffff 64bit pref]
[ 957.878428] pci 0000:bd:00.1: VF(n) BAR0 space: [mem 0x200120850000-0x20012087ffff 64bit pref] (contains BAR0 for 3 VFs)
[ 957.878432] pci 0000:bd:00.1: reg 0x22c: [mem 0x200120500000-0x2001205fffff 64bit pref]
[ 957.878434] pci 0000:bd:00.1: VF(n) BAR2 space: [mem 0x200120500000-0x2001207fffff 64bit pref] (contains BAR2 for 3 VFs)
[ 957.878547] pci 0000:bd:00.1: BAR 2: assigned [mem 0x200120400000-0x2001204fffff 64bit pref]
[ 957.878551] pci 0000:bd:00.1: BAR 9: assigned [mem 0x200120500000-0x2001207fffff 64bit pref]
[ 957.878553] pci 0000:bd:00.1: BAR 0: assigned [mem 0x200120840000-0x20012084ffff 64bit pref]
[ 957.878556] pci 0000:bd:00.1: BAR 7: assigned [mem 0x200120850000-0x20012087ffff 64bit pref]
[ 957.879382] hns3 0000:bd:00.1: The firmware version is 1.8.11.0
[ 957.883597] libphy: hisilicon MII bus: probed
[ 957.936529] hns3 0000:bd:00.1: hclge driver initialization finished.
[ 957.937875] Marvell 88E1510 mii-0000:bd:00.1:01: attached PHY driver [Marvell 88E1510] (mii_bus:phy_addr=mii-0000:bd:00.1:01, irq=POLL)

[Ike Panhc (ikepanhc)]

Upstream patch c13704f5685d ("PCI: Avoid double hpmemsize MMIO window assignment") has been backported to Ubuntu focal GA kernel since Ubuntu-5.4.0-55.61

$ git tag --contains 9d50ba193535 | head -1
Ubuntu-5.4.0-55.61

I will try early kernel version to make sure and close this bug.

[Ike Panhc (ikepanhc)]

I can reproduce with 5.4.0-54.60. This issue has been fix since 5.4.0-55.61. Update kernel will fix this issue.

$ sudo dmesg --clear
$ cat /proc/version
Linux version 5.4.0-54-generic (buildd@bos02-arm64-055) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #60-Ubuntu SMP Fri Nov 6 10:42:16 UTC 2020
$ echo 1 | sudo tee /sys/bus/pci/devices/0000:bd:00.1/remove
1
$ dmesg
[ 534.458623] hns3 0000:bd:00.1 enp189s0f1: link down
$ echo 1 | sudo tee /sys/devices/pci0000:bc/0000:bc:00.0/rescan
1
$ dmesg
[ 534.458623] hns3 0000:bd:00.1 enp189s0f1: link down
[ 548.673838] pci 0000:bd:00.1: [19e5:a222] type 00 class 0x020000
[ 548.673854] pci 0000:bd:00.1: reg 0x10: [mem 0x200120840000-0x20012084ffff 64bit pref]
[ 548.673858] pci 0000:bd:00.1: reg 0x18: [mem 0x200120400000-0x2001204fffff 64bit pref]
[ 548.673901] pci 0000:bd:00.1: reg 0x224: [mem 0x200120850000-0x20012085ffff 64bit pref]
[ 548.673903] pci 0000:bd:00.1: VF(n) BAR0 space: [mem 0x200120850000-0x20012087ffff 64bit pref] (contains BAR0 for 3 VFs)
[ 548.673906] pci 0000:bd:00.1: reg 0x22c: [mem 0x200120500000-0x2001205fffff 64bit pref]
[ 548.673908] pci 0000:bd:00.1: VF(n) BAR2 space: [mem 0x200120500000-0x2001207fffff 64bit pref] (contains BAR2 for 3 VFs)
[ 548.674034] pci 0000:bc:00.0: bridge window [mem 0x00100000-0x002fffff] to [bus bd] add_size 300000 add_align 100000
[ 548.674040] pci 0000:bc:00.0: BAR 14: no space for [mem size 0x00500000]
[ 548.674041] pci 0000:bc:00.0: BAR 14: failed to assign [mem size 0x00500000]
[ 548.674043] pci 0000:bc:00.0: BAR 14: no space for [mem size 0x00200000]
[ 548.674044] pci 0000:bc:00.0: BAR 14: failed to assign [mem size 0x00200000]
[ 548.674049] pci 0000:bd:00.1: BAR 2: assigned [mem 0x200120400000-0x2001204fffff 64bit pref]
[ 548.674053] pci 0000:bd:00.1: BAR 9: assigned [mem 0x200120500000-0x2001207fffff 64bit pref]
[ 548.674055] pci 0000:bd:00.1: BAR 0: assigned [mem 0x200120840000-0x20012084ffff 64bit pref]
[ 548.674057] pci 0000:bd:00.1: BAR 7: assigned [mem 0x200120850000-0x20012087ffff 64bit pref]
[ 548.674650] hns3 0000:bd:00.1: The firmware version is 1.8.11.0
[ 548.678180] libphy: hisilicon MII bus: probed
[ 548.729620] hns3 0000:bd:00.1: hclge driver initialization finished.
[ 548.730959] Marvell 88E1510 mii-0000:bd:00.1:01: attached PHY driver [Marvell 88E1510] (mii_bus:phy_addr=mii-0000:bd:00.1:01, irq=POLL)
[ 548.757202] cma_alloc: 275 callbacks suppressed
[ 548.757204] cma: cma_alloc: alloc failed, req-size: 4 pages, ret: -12
[ 548.765038] cma: cma_alloc: alloc failed, req-size: 2 pages, ret: -12