Add GCC atomic support (-moutline-atomics) for arm64 on Focal

[Salvatore]

[ Impact ]

When using nginx on arm64, the CPU may go under heavy load and drop packets when being stressed.

A compile-time optimization to use hardware acceleration could be included to help alleviate the CPU utilization for systems that have atomic instructions available.

[ Test Plan ]

The test plan requires a decent amount of setup. This setup is covered in full at https://github.com/mitchdz/aws-nginx-testbed.

In short, the setup involves a focal based arm64 nginx server, 4 amd64 focal endpoints running a nodejs applicaiton, and 1 amd64 focal server to initiate the workloads.

This test is an E2E test which tests both functionality and performance improvements.

1) Set up the system as described in https://github.com/mitchdz/aws-nginx-testbed

2) Initiate the workload on the DRV instance
$ ./wrk/wrk -t36 -c512 -d60s http://172.31.33.24:80/ # Private IP of SUT

3) While workload is running, capture results on SUT
$ sudo perf record -a -e r6e sleep 20s

4) After the test is ran, you should have metrics on both the DRV and SUT system.

5) Install new nginx with -moutline atomics
$ sudo add-apt-repository ppa:mitchdz/lp2024019-moutline-atomic
$ sudo apt update
$ Sudo apt install -y nginx
$ dpkg -s nginx | grep Version
Version: 1.18.0-0ubuntu1.5~focal1

6) ensure outline-atomics is installed (ctrl+f to see it, since this is not a user friendly wall of text)
$ nginx -V |& grep moutline
configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-v8ZFDO/nginx-1.18.0=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2 -moutline-atomics' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module

7) re-run wrk test as shown in steps 2-3

Results:
- without patch:
   - wrk output:
    ```
      ubuntu@ip-172-31-43-203:~/wrk$ ./wrk -t36 -c512 -d60s http://172.31.40.247:80/
      Running 1m test @ http://172.31.40.247:80/
        36 threads and 512 connections
        Thread Stats Avg Stdev Max +/- Stdev
          Latency 3.14ms 2.03ms 211.63ms 72.12%
          Req/Sec 4.67k 0.89k 23.92k 76.44%
        10056815 requests in 1.00m, 5.20GB read
      Requests/sec: 167336.42
      Transfer/sec: 88.57MB
    ```
  - STREX count: Samples: 4M of event 'r6e', Event count (approx.): 364457784
  - SUT CPU utiliz: 100%

- with patch:
  - wrk output:
    ```
      ubuntu@ip-172-31-43-203:~/wrk$ ./wrk -t36 -c512 -d60s http://172.31.40.247:80/
      Running 1m test @ http://172.31.40.247:80/
        36 threads and 512 connections
        Thread Stats Avg Stdev Max +/- Stdev
          Latency 1.02ms 1.04ms 210.87ms 99.52%
          Req/Sec 13.94k 327.01 16.95k 79.69%
        30006090 requests in 1.00m, 15.51GB read
      Requests/sec: 499273.74
      Transfer/sec: 264.26MB
    ```
  - STREX count: No Samples
  - SUT CPU utiliz: 20%

[ Where Problems Could Occur ]
Performance Trade-offs:
* While this will decrease the CPU load, it will increase the utilization of atomic operations.
* outlining atomics can make debugging more complicated, especially in the case of concurrency debugging.
* This improves application performance through offloading instructions. This can reveal bugs that were not possible before such as race conditions, deadlocks, or incorrect synchronization.
* This optimization adds a run-time check for the availability of atomic instructions. If atomics instructions are not found, ARMv8.0 compatible code is executed, so overhead will be added to systems that do not have atomic instructions.

[ Other Info ]
* Why is -moutline-atomics not enabled already?
Focal uses gcc-9 which does not enable -moutline-atomics by default. gcc-10 is when it became enabled by default.

https://devdocs.io/gcc~9/aarch64-options
vs
https://devdocs.io/gcc~10/aarch64-options

gcc-9 changelog showing the addition of -moutline-atomics: https://gcc.gnu.org/gcc-9/changes.html

Here is the thread with the discussion to enable by default: https://gcc.gnu.org/pipermail/gcc/2020-April/000490.html

A lengthy discussion also happened in Debian to include this flag by default - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956418

[Christian Ehrhardt  (paelzer)]

I talked with Fabio (now subscribed), he will track and ensure this is done together with all the other "enable -moutline-atomics" cases.

[Salvatore (dipietro-salvatore)]

Do you have any update regarding this change?

[Thomas Ward (teward)]

@paelzer if you could update it that'd be great.

@Salvatore Note that @sil2100 made a 'Needs information' request on the pull request from here, and that needs attention, you might see more stuff there about problems, etc. introduced that're blocking this.

[Fabio Augusto Miranda Martins (fabio.martins)]

This request was previously blocked by the GCC SRU for focal (LP#1994020) which is now released. I'm discussing this internally with Paelzer's team and we'll update this launchpad with the progress.