fix non-8-bit x86 displacements breaking AVX512 builds on Bionic

[Christian Ehrhardt ]

[Impact]

 * the assembler scales non 8 bit cases which was identified
   to break e.g. some AVX512 code. It is nasty as it isn't a compile/link/
   time error. Instead the instructions might silently be corrupted until
   running. Things might even work on some but fail on other systems if
   e.g. the AVX code paths only run on newer chips.

  * The fix is upstream for a while and not re-changed again. Furthermore
    it is in several Ubuntu releases without bugs due to that, which should
    make the backport rather safe.

[Test Case]

 * Simple example to trigger the bug:

echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps

The expected output is that the objdump output matches the vmovaps instruction input. When using binutils with the bug, the initial 0x40 will be incorrect.

Working:
$ echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
   0: 62 f1 7c 48 28 04 05 vmovaps 0x40(,%rax,1),%zmm0
Failing:
$ echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
   0: 62 f1 7c 48 28 04 05 vmovaps 0x1(,%rax,1),%zmm0

[Regression Potential]

 * Well, this is a double edged sword. On one hand this is fortunately a
   small change and only affects something formerly clearly broken. So it
   should be good and only change cases formerly being bad.
   But OTOH binutils areused in so many cases that I feel unable to say
   "nothing will happen". The change goes to the gnu assembler, so that is
   the place to look out for regressions.

[Other Info]

 * needs a sponsor experienced with binutils to check potential pitfalls

---

Hi,
DPDK has run into some issues in the past
 https://bugs.dpdk.org/show_bug.cgi?id=97
 https://bugs.dpdk.org/show_bug.cgi?id=249

Eventually the issues got resolved in binutils via
 https://sourceware.org/bugzilla/show_bug.cgi?id=23465

After binutils is fixed people rebuilding DPDK themselve can use
 http://patches.dpdk.org/patch/71679/
to gain more performance while on Bionics bintuils level.

Note: Bionic is on DPDK 17.11.x which will not get further stable release afaik. But quite often people build their own DPDK. In fact this came up as a request from Openvswitch upstream/Intel to allow such builds on Bionic.
I'd ping those people about the bug and ask them to participate in the verification if this becomes an SRU.

[Harry van Haaren (Intel) (harryvanhaaren)]

Hi All,

I contacted Christian to mention/report, just noting that here.
Happy to test if we get an SRU, I'll get email notifications once I post this :)

Regards, -Harry

[Bruce Richardson (Intel) (bricha3)]

Command to reproduce the issue:

echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps

The expected output is that the objdump output matches the vmovaps instruction input. When using binutils with the bug, the initial 0x40 will be incorrect.

From an Ubuntu 20.04 system:

$ echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
   0: 62 f1 7c 48 28 04 05 vmovaps 0x40(,%rax,1),%zmm0

On an Ubuntu 18.04 system:

$ echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
   0: 62 f1 7c 48 28 04 05 vmovaps 0x1(,%rax,1),%zmm0

[Harry van Haaren (Intel) (harryvanhaaren)]

Proposed patch on OVS Mailing list to check status at configure time for OVS 2.14 using the method Bruce mentioned above:
https://patchwork.ozlabs.org/project<email address hidden>/

[Christian Ehrhardt  (paelzer)]

Added an SRU Teamplate entry in the description, thanks Bruce for the test.

The changes build fine, all self tests on build are good [1].
The dep8 sniff tests have just started at [2] and will later be available there.
The changes are rather small as seen on [3]

But while all that shall help by being good preparation I have no experience on binutils.
Therefore I'd now assign to doko so that he can ack&sponsor or comment why this won't be able to happen.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4105/+packages
[2]: https://bileto.ubuntu.com/#/ticket/4105
[3]: https://code.launchpad.net/~paelzer/ubuntu/+source/binutils/+git/binutils/+merge/386073

[Christian Ehrhardt  (paelzer)]

A few dep8 tests failed in the sniff, but all of them are existing known overrides in Bionic:

ubuntu-release:9:force-badtest dh-ada-library/6.12
ubuntu-release:18:force-badtest lintian/2.5.81ubuntu1
pitti:2:force-badtest rpmlint/1.9-6
ubuntu-sru:17:force-badtest snapcraft/2.43.1+18.04/amd64 snapcraft/2.43.1+18.04/i386
ubuntu-sru:20:force-badtest snapcraft/2.43.1+18.04/armhf snapcraft/2.43.1+18.04/arm64
ubuntu-release:90:force-badtest linux-hwe/all
ubuntu-release:91:force-badtest linux-hwe-edge/all

So this should be good as-is, but needs Doko to sign-off and sponsor with the experience of potential side effects of a binutils update.

[Christian Ehrhardt  (paelzer)]

Tested in Bionic Container:

no PPA:
root@b:~# echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
   0: 62 f1 7c 48 28 04 05 vmovaps 0x1(,%rax,1),%zmm0

with PPA:
root@b:~# echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
   0: 62 f1 7c 48 28 04 05 vmovaps 0x40(,%rax,1),%zmm0

Seems to work as intended

[Christian Ehrhardt  (paelzer)]

I got doko to have a quick look (as he is unavailable atm) which seemed ok.
This was mostly to hear from him if this was an overall horrendous idea (it isn't) and not to check every line, so this should be ok as-is then.
Further as stated above all other pre-checks against the PPA are good as well.

I uploaded to bionic-unapproved for consideration by the SRU Team.

[Harry van Haaren (Intel) (harryvanhaaren)]

Hey Folks,

Apologies I'm not familiar with the SRU testing process. I'd like to help - have an 18.04 setup for testing here. Is there an easy way to add the test-build PPA and just do an apt update && apt install binutils?

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4105/+packages

Regards, -Harry

[Christian Ehrhardt  (paelzer)]

Hi Harry,
the real action/verification will be needed ocne the SRU team has accepted that into proposed [1].
If you want to test the PPA the following will enable it on your system:
  $ sudo add-apt-repository ppa:ci-train-ppa-service/4105

The PPA tells you that on its front page [2] as well, I just happen to link the sub-page with the package details as I more often want to check build/version details. I beg your pardon that was my fault not realizing you could make better use the entry page instead.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates
[2]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4105

[Harry van Haaren (Intel) (harryvanhaaren)]

Hi Christian,

Thanks for the additional info! Confirm that with the PPA the proposed checks now pass:

$ echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
  8: 0: 62 f1 7c 48 28 04 05 vmovaps 0x40(,%rax,1),%zmm0

Also the proposed OVS automake build-system changes will enable avx512 correctly:
checking binutils avx512 assembler checks passing... yes

I'll wait for the SRU team to accept into proposed. Regards, -Harry

[Matthias Klose (doko)]

yes, this looks sensible as a SRU

[Christian Ehrhardt  (paelzer)]

On first check by the SRU team I got:
[18:34] <sil2100> cpaelzer: ok, looking at the binutils SRU, so you talked to do_ko about it, right? Was he +1 on the change?

He was in a private IRC, I now asked him to state it here as well so that it is visible to the SRU team. And he already did so - thanks!

[18:35] <sil2100> cpaelzer: also, since this is a toolchain package, we need to first build it in a -security only PPA, as we need it to go to both -updates and -security

Hmm, I didn't know that is a common pre-check for toolchain packages - thanks for the hint.
I have opened a security only all-bionic-arch enabled PPA for this and started a build at:
=> https://launchpad.net/~paelzer/+archive/ubuntu/lp-1883880-binutils-sec-only
The first few builds are completed by now.

I hope this helps to remove the remaining blockers to get this into bionic-proposed

[Christian Ehrhardt  (paelzer)]

To have a bileto PPA with security only (thanks Sil2100 for stopping it to reconfigure itself) look at:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4151

[Robie Basak (racb)]

Hello Christian, or anyone else affected,

Accepted binutils into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/binutils/2.30-21ubuntu1~18.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted binutils (2.30-21ubuntu1~18.04.4) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

glibc/2.27-3ubuntu1.2 (armhf)
apport/2.20.9-0ubuntu7.15 (i386, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#binutils

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Christian Ehrhardt  (paelzer)]

with:
2.30-21ubuntu1~18.04.3

root@b:~# echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
   0: 62 f1 7c 48 28 04 05 vmovaps 0x1(,%rax,1),%zmm0

with:
2.30-21ubuntu1~18.04.4

root@b:~# echo "vmovaps 0x40(,%rax,1),%zmm0" | as --64 -o avx.o && objdump -d avx.o | grep vmovaps
   0: 62 f1 7c 48 28 04 05 vmovaps 0x40(,%rax,1),%zmm0

Setting verified, but please feel free to
a) keep this in proposed a bit longer
b) @Harry could you test the proposed changes to the OVS build system again?

[Christian Ehrhardt  (paelzer)]

Autopkgtest issues resolved, all good now.

[Harry van Haaren (Intel) (harryvanhaaren)]

Hi All,

I have enabled the bionic-proposed repo as described in link. OVS-master is used for testing, containing the build-time binutils bug check as described above.

1) Can confirm that old binutils version (2.30-21ubuntu1~18.04.3) had issue, and OVS was correctly DISabling AVX512.

2) Can confirm that with -proposed binutils (2.30-21ubuntu1~18.04.4), OVS build system is correctly ENabling AVX512 support, as linker checks are passing.

After the configure step, I have tested OVS, running traffic and enabling the AVX512 optimizations. It is working as expected, with AVX512 instructions running in the datapath.

I believe you have already added the "verification-done" and bionic versions Christian, so I'll not change the status... please let me know if I misunderstood the process.

Thanks Robie & Christian for your support on this issue! -Harry

[Harry van Haaren (Intel) (harryvanhaaren)]

Hey Folks,

Does the -proposed changes get automatically integrated to -updates after a certain time has passed, or does this require somebody to take action?

It would be great to have this update rolled out to 18.04 before OVS 2.14 is released in a week or two.

Regards, -Harry

[Łukasz Zemczak (sil2100)]

It is a manual process which happens after 7+ days of aging in -proposed - but I have just confirmed the verification and will be proceeding to release this into -updates. Thank you for your testing!

[Launchpad Janitor (janitor)]

This bug was fixed in the package binutils - 2.30-21ubuntu1~18.04.4

---------------
binutils (2.30-21ubuntu1~18.04.4) bionic; urgency=medium

  * d/p/lp-1883880-x86-don-t-mistakenly-scale-non-8-bit-displacements.patch:
    avoid scaling displacements that breaks e.g. some AVX512 code (LP: #1883880)

 -- Christian Ehrhardt <email address hidden> Thu, 18 Jun 2020 13:03:55 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for binutils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Harry van Haaren (Intel) (harryvanhaaren)]

Thanks Lukasz, Christian, Robie, Matthias, and the various robots and launchpad janitors: appreciate your help!