Bug #1864221 “Layered Build: Main controller fails after unlock ...” : Bugs : StarlingX

Cristopher Lemus (cjlemusc) on 2020-02-21

description:

updated

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2020-02-21:

#1

stx.4.0 / high priority - sanity issue w/ layered build

summary:	- Main controller fails after unlock - drbd error on manifest + Layered Build: Main controller fails after unlock - drbd error on + manifest
Changed in starlingx:
importance:	Undecided → High
status:	New → Triaged
tags:	added: stx.4.0 stx.build stx.sanity
Changed in starlingx:
assignee:	nobody → Scott Little (slittle1)

Revision history for this message

Cristopher Lemus (cjlemusc) wrote on 2020-02-26:

#2

This issue is still appearing on a new build, 20200225T194204Z, it was reproduced on Simplex and Duplex configurations: http://paste.openstack.org/show/790036/

Full collect from duplex baremetal: https://files.starlingx.kube.cengn.ca/download_file/50

Revision history for this message

Mihail-Laurentiu Trica (mtrica) wrote on 2020-02-27:

#3

The issue is still reproducible with build 20200226T090221Z, on both Simplex and Duplex configurations.

Revision history for this message

Nicolae Jascanu (njascanu-intel) wrote on 2020-02-28:

#4

The issue is still reproducible with build 20200227T122131Z, on both Simplex and Duplex configurations.

Revision history for this message

Frank Miller (sensfan22) wrote on 2020-03-04:

#5

We are unable to reproduce this issue when testing the ISO generated by build layering. Using the /mirror/starlingx/master/centos/flock/20200227T122131Z/outputs/iso/ load resulted in sanity passing for both AIO-SX and AIO-DX when sanity was run in our hardware labs.

Next step is to determine why you are seeing issues in your lab. Please set up a zoom session so we can perform a joint debug session. Ideally would like to see you reproduce the issue during the debug session so it can be debugged in real time.

Revision history for this message

Frank Miller (sensfan22) wrote on 2020-03-04:

#6

From the puppet logs this is an issue with being out of memory:

2020-02-20T23:13:49.329 ^[[mNotice: 2020-02-20 23:13:49 +0000 /Stage[main]/Platform::Drbd::Platform/Platform::Drbd::Filesystem[drbd-platform]/Drbd::Resource[drbd-platform]/Drbd::Resource::Enable[drbd-platform]/Drbd::Resource::Up[drbd-platform]/Exec[reuse existing DRBD resource drbd-platform]/returns: drbd-platform: Failure: (122) kmalloc() failed. Out of memory?^[[0m
2020-02-20T23:13:49.332 ^[[mNotice: 2020-02-20 23:13:49 +0000 /Stage[main]/Platform::Drbd::Platform/Platform::Drbd::Filesystem[drbd-platform]/Drbd::Resource[drbd-platform]/Drbd::Resource::Enable[drbd-platform]/Drbd::Resource::Up[drbd-platform]/Exec[reuse existing DRBD resource drbd-platform]/returns: Command 'drbdsetup new-resource drbd-platform --cpu-mask=300000003' terminated with exit code 10^[[0m

It is also reported in https://bugs.launchpad.net/starlingx/+bug/1863362

The bug is not related to the layered build but an intermittent issue that is not yet understood. Has this same lab been used to run sanity with a regular ISO (not layered build) built on the same day?

Revision history for this message

Cristopher Lemus (cjlemusc) wrote on 2020-03-05:

#7

Hi Frank,

We used the latest layered build: http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20200305T111629Z/

However, we still face the exact same issue with Simplex and Duplex

LOGS
=======
Simplex:
Some outputs: http://paste.openstack.org/show/790365/
Collect: https://files.starlingx.kube.cengn.ca/download_file/54

Duplex:
Some outputs: http://paste.openstack.org/show/790368/
Collect: https://files.starlingx.kube.cengn.ca/download_file/55

HARDWARE DETAILS
=======
Here are some hardware details for our simplex system: http://paste.openstack.org/show/790370/

Both duplex systems have the same hardware specs.

COMMANDS EXECUTED
=======
These are the commands that we ran against the simplex system before doing the unlock:
http://paste.openstack.org/show/790373/
After that, the system boots, but that's when we receive the error "Openstack Admin credentials can only be loaded from the active controller."

NOTE: The exact same steps are followed up for both, monolithic and layered build. Is there a different command that needs to be executed?

QUESTIONS
=======
So regarding the questions:
- We are using the exact same server for Simplex on Monolithic and Layered build.
- Another pair of servers is used on Duplex for Monolithic and Layered build.
- Latest monolithic build doesn't face this error on Simplex, neither on Duplex, Sanity report was sent as green.
- We can arrange the meeting, however, could you please let us know at which point do you want the servers to be left? Just with the Image installed? Or, will it be better to execute every command, just before the "system host-unlock controller-0"? In that way, we can check the status before the unlock and monitor how it progress.

Revision history for this message

Chris Friesen (cbf123) wrote on 2020-03-09:

#8

1) Can we get a "collect" from the monolithic system as well?

2) Also, I was wondering if you could try the following commands before the initial unlock while installing each of the monolithic and layered builds:

First, run "drbdsetup new-resource blah --cpu-mask=300000003|dmesg|tail" and check whether you see an error in the kernel logs that looks like "Overflow in bitmap_parse(300000003), truncating to 256 bits"

Next, run "drbdsetup new-resource blah2 --cpu-mask=3,00000003|dmesg|tail" and see if any new drbd-related errors have been added to the kernel logs.

3) Lastly, could you run "grep DRBD /boot/config-3.10.0*" and "find /lib/modules/3.10.0* -name drbd.ko" on both the monolithic and layered builds?

In the one monolithic lab I checked, the drbd kernel module is provided by the kmod-drbd-rt-8.4.11-1.tis.0.x86_64 package, which will emit the above "overflow" message to the kernel logs instead of returning an error code back up to userspace.

The logs from https://files.starlingx.kube.cengn.ca/download_file/50 however have the following:
2020-02-25T23:03:01.537 controller-0 kernel: warning [ 370.541536] d-con drbd-platform: bitmap_parse() failed with -75
2020-02-25T23:03:01.634 controller-0 kernel: warning [ 370.638097] d-con drbd-rabbit: bitmap_parse() failed with -75
2020-02-25T23:03:01.689 controller-0 kernel: warning [ 370.693070] d-con drbd-extension: bitmap_parse() failed with -75
2020-02-25T23:03:02.137 controller-0 kernel: warning [ 371.139836] d-con drbd-etcd: bitmap_parse() failed with -75
2020-02-25T23:03:02.212 controller-0 kernel: warning [ 371.214433] d-con drbd-dockerdistribution: bitmap_parse() failed with -75
2020-02-25T23:06:02.802 controller-0 kernel: warning [ 551.381727] d-con drbd-pgsql: bitmap_parse() failed with -75

These logs make it seem like the layered build is somehow using the kernel implementation rather than the out-of-tree implementation. Alternately, something is messed up and it's not recognizing that EOVERFLOW corresponds to 75.

1) Can we get a "collect" from the monolithic system as well?

2) Also, I was wondering if you could try the following commands before the initial unlock while installing each of the monolithic and layered builds:

First, run "drbdsetup new-resource blah --cpu-mask=300000003|dmesg|tail" and check whether you see an error in the kernel logs that looks like "Overflow in bitmap_parse(300000003), truncating to 256 bits"

Next, run "drbdsetup new-resource blah2  --cpu-mask=3,00000003|dmesg|tail" and see if any new drbd-related errors have been added to the kernel logs.

3) Lastly, could you run "grep DRBD /boot/config-3.10.0*" and "find /lib/modules/3.10.0* -name drbd.ko" on both the monolithic and layered builds?

In the one monolithic lab I checked, the drbd kernel module is provided by the kmod-drbd-rt-8.4.11-1.tis.0.x86_64 package, which will emit the above "overflow" message to the kernel logs instead of returning an error code back up to userspace.

The logs from https://files.starlingx.kube.cengn.ca/download_file/50 however have the following:
2020-02-25T23:03:01.537 controller-0 kernel: warning [  370.541536] d-con drbd-platform: bitmap_parse() failed with -75
2020-02-25T23:03:01.634 controller-0 kernel: warning [  370.638097] d-con drbd-rabbit: bitmap_parse() failed with -75
2020-02-25T23:03:01.689 controller-0 kernel: warning [  370.693070] d-con drbd-extension: bitmap_parse() failed with -75
2020-02-25T23:03:02.137 controller-0 kernel: warning [  371.139836] d-con drbd-etcd: bitmap_parse() failed with -75
2020-02-25T23:03:02.212 controller-0 kernel: warning [  371.214433] d-con drbd-dockerdistribution: bitmap_parse() failed with -75
2020-02-25T23:06:02.802 controller-0 kernel: warning [  551.381727] d-con drbd-pgsql: bitmap_parse() failed with -75

These logs make it seem like the layered build is somehow using the kernel implementation rather than the out-of-tree implementation. Alternately, something is messed up and it's not recognizing that EOVERFLOW corresponds to 75.

Revision history for this message

Cristopher Lemus (cjlemusc) wrote on 2020-03-10:

#9

Hi Chris,

From the Layered Build, I got the following:

drbdsetup new-resource blah --cpu-mask=300000003|dmesg|tail
returns an error:
blah: Failure: (122) kmalloc() failed. Out of memory?
and
[ 4079.525275] d-con blah: bitmap_parse() failed with -75

drbdsetup new-resource blah2 --cpu-mask=3,00000003|dmesg|tail
doesn't return an error on kernel logs.

After 15-20mins, the last entry on dmesg is still:
[ 4079.525275] d-con blah: bitmap_parse() failed with -75

Full outputs here:

http://paste.openstack.org/show/790506/

Tomorrow, I'll install this same hardware with the monolithic build and provide the same output and collect.

Thanks a lot for your help!

Revision history for this message

Nicolae Jascanu (njascanu-intel) wrote on 2020-03-10:

#10

Hi Chris,
We checked the monolithic build on bare-metal and we found similar errors as we are expecting.

Please take a look at:

http://paste.openstack.org/show/790517/

Revision history for this message

Chris Friesen (cbf123) wrote on 2020-03-10:

#11

On the layered build, can you check whether there's a "drbd.ko" file under the "extra" kernel modules directory? It should look something like this:

controller-1:/home/sysadmin# ls -l /lib/modules/3.10.0-1062*/extra/drbd/drbd.ko
-rw-r--r--. 1 root root 684157 Feb 13 03:51 /lib/modules/3.10.0-1062.1.2.el7.2.tis.x86_64/extra/drbd/drbd.ko

Revision history for this message

Chris Friesen (cbf123) wrote on 2020-03-10:

#12

Looking at the drbd driver version in the kernel log, it's reporting version 8.4.3:

-bash-4.2$ grep drbd kern.log |grep Version
2020-02-25T22:33:53.116 localhost kernel: info [ 555.710962] drbd: initialized. Version: 8.4.3 (api:1/proto:86-101)
2020-02-25T23:00:52.074 controller-0 kernel: info [ 241.382367] drbd: initialized. Version: 8.4.3 (api:1/proto:86-101)
2020-02-25T23:02:09.308 controller-0 kernel: info [ 318.434910] drbd: initialized. Version: 8.4.3 (api:1/proto:86-101)

The out-of-tree driver (which is what we should be using) is version 8.4.11.

Is the kmod-drbd-8.4.11-1.tis.0.x86_64.rpm being installed as part of the layered loadbuild?

Revision history for this message

Cristopher Lemus (cjlemusc) wrote on 2020-03-10:

#13

Hi Chris,

Here are the outputs from the layered build before the unlock:

controller-0:~$ cat /etc/build.info |egrep "BUILD_ID|JOB"
BUILD_ID="20200307T090226Z"
JOB="STX_build_layer_flock_master_master"
FLOCK_JOB="STX_build_layer_flock_master_master"
DISTRO_JOB="STX_build_layer_distro_master_master"
COMPILER_JOB="STX_build_layer_compiler_master_master"

controller-0:~$ ls -l /lib/modules/3.10.0-1062*/extra/drbd/drbd.ko
-rw-r--r--. 1 root root 684157 ene 7 23:56 /lib/modules/3.10.0-1062.1.2.el7.1.tis.x86_64/extra/drbd/drbd.ko

controller-0:~$ rpm -qa |grep kmod-drbd
kmod-drbd-8.4.11-1.tis.0.x86_64

Kernel parameters:
controller-0:~$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-1062.1.2.el7.2.tis.x86_64 root=UUID=6f98e090-a0de-40b3-907c-b43ac5e63227 ro security_profile=standard module_blacklist=integrity,ima audit=0 tboot=false crashkernel=auto biosdevname=0 console=ttyS0,115200 iommu=pt usbcore.autosuspend=-1 hugepagesz=2M hugepages=0 default_hugepagesz=2M isolcpus=2,34,3,35 rcu_nocbs=2-31,34-63 kthread_cpus=0,32,1,33 irqaffinity=0,32,1,33 selinux=0 enforcing=0 nmi_watchdog=panic,1 softlockup_panic=1 intel_iommu=on user_namespace.enable=1

Should I try to do the unlock and check the outputs? The failure is during the unlock activity, I assume that something might be different after the unlock.

Revision history for this message

Frank Miller (sensfan22) wrote on 2020-03-10:

#14

After some further debug by Chris, the drbd command is failing because it is using an in-tree kmod instead of an out-of-tree kmod.

Don Penney looked into the output for the build layering build and sees that the distro layer is building the kmod-drbd RPM correctly, but the flock layer is not picking up the right RPM. This is likely due to the version being the same, and whatever repoquery command is being used is finding a very old version.

It looks like the latest flock ISO build has a kmod-drbd built in January, rather than a recently built distro build.

Next step will be for Scott to update how the Flock build is pulling in the rpms for kmods so it aligns with the monolithic build.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-03-19: Fix proposed to tools (master)

#15

Fix proposed to branch: master
Review: https://review.opendev.org/713968

Changed in starlingx:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-14: Fix merged to tools (master)

#16

Reviewed: https://review.opendev.org/717755
Committed: https://git.openstack.org/cgit/starlingx/tools/commit/?id=bbd79fcd380992192027601b9444edd668803ed4
Submitter: Zuul
Branch: master

commit bbd79fcd380992192027601b9444edd668803ed4
Author: Scott Little <email address hidden>
Date: Sat Apr 4 19:16:28 2020 -0400

Fast download of lower layer rpms

    The default yumdownloader approach to downloading files
    does so one at a time, and is very slow. Too slow for layered
    build which is built around the assumption that files can
    be downloaded faster than they can be built.

    This update will switch from yumdownloader to reposync
    for the download of lower layer rpms. It exploits the
    fact that each layer and build type publishes it's own lst file
    of rpms to download. The lst file is transformed into an'
    includepkgs directive in a custom yum.conf which is passed
    to reposync, so we only sync the desited rpms.

    Reposync won't redownload rpms that it already has, even if the
    upstream repodata indicates that the files checksum has changed.
    Forceing the redownload of these rpms requires that we manually
    download the upstream repodata and that we use verifytree to
    identify and delete the new obsolete rpms.

    Also including two small bug fixes found while investivgating an
    alternative solution to launchpad 1864221.
    - incorrect userid in a chown
    - Bug in get_url, --url pass being passed in the wrong place

    Story: 2006166
    Task: 39307
    Closes-Bug: 1864221
    Change-Id: If12b98ff4f5f24d9318250356920f397419f0f80
    Signed-off-by: Scott Little <email address hidden>

Changed in starlingx:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-04-20: Change abandoned on tools (master)

#17

Change abandoned by Scott Little (<email address hidden>) on branch: master
Review: https://review.opendev.org/713968

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-21: Fix proposed to tools (f/centos8)

#18

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/729840

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-21: Fix merged to tools (f/centos8)

#19

Download full text (20.8 KiB)

Reviewed: https://review.opendev.org/729840
Committed: https://git.openstack.org/cgit/starlingx/tools/commit/?id=05ba65cb054e71ef7c2669668831f50ee334e768
Submitter: Zuul
Branch: f/centos8

commit d923525441f034c31f0231d15e0f55e3ac5c37b6
Author: Don Penney <email address hidden>
Date: Wed May 20 14:20:35 2020 -0400

Update 7.7.1908 repo to point to vault

    The Centos 7.7.1908 repo has been deprecated and moved to the vault.
    This commit updates the repo config file to point to the new baseurl
    locations.

Change-Id: I0d9b98eea925ad05544c7a8bd626ee5dcc48d103
Signed-off-by: Don Penney <email address hidden>

commit daff8261d18e7512c2ebc1d42fa4a75ddbaddda8
Author: Ran An <email address hidden>
Date: Fri May 15 01:34:46 2020 +0000

Add python3 rpms required by python3-daemon runtime

This reverts commit 6fa137af2312abbdfb0f0f015869d65c29e36b82.

Add python3 rpms required by python3-daemon runtime
adding python3 pkgs to support python3-daemon build

Story: 2007106
Task: 39353

Signed-off-by: SidneyAn <email address hidden>
Change-Id: I12e1ed08381cbc45480c6c02e4d93c8e04fac2dc

commit e12c55e47ed72fc4e4c38c11525769d89d1f9eed
Author: Ran An <email address hidden>
Date: Fri May 15 01:34:42 2020 +0000

python3: add python3 and python3-devel

This reverts commit 79abb72562b53c8682ea0d854e1071540b38a398.

    following pkgs were added:
    python3
    python3-lib
    python3-devel
    python3-rpm-generators (python3-devel dependency)
    python3-rpm-macros (python3-devel dependency)

Story: 2007106
Task: 39147

Signed-off-by: SidneyAn <email address hidden>
Change-Id: I127453a728a0ce1bbc517a5505c0b187fc945207

commit 008685698ae3e520dbc5178e76fa5956697192ed
Author: Shuicheng Lin <email address hidden>
Date: Wed Apr 29 04:25:59 2020 +0800

Upgrade kata containers to 1.11.0

Include several fixs for IPv6 support in Kata.

Test:
Could create kata container in both IPv4 and IPv6 environment.

Story: 2006145
Task: 39077

Change-Id: I0fcd2b516f9003a851e601493ae7c323ceab8df5
Signed-off-by: Shuicheng Lin <email address hidden>

commit 79abb72562b53c8682ea0d854e1071540b38a398
Author: Ran An <email address hidden>
Date: Thu May 14 11:42:07 2020 +0000

Revert "python3: add python3-devel dep pkgs"

This reverts commit 50bff514030d51dece7d23c2e1b64284ea5cda27.

Change-Id: I50d5926f3ead6e773fa8e86ed2ef72f86ff9f343

commit 6fa137af2312abbdfb0f0f015869d65c29e36b82
Author: Ran An <email address hidden>
Date: Thu May 14 11:42:01 2020 +0000

Revert "Add python3 rpms required by python3-daemon runtime"

This reverts commit b2b481a7b0d348cf332e8002ec522457fbb80814.

Change-Id: Ieb7d2dae22efa31730d3f5560c17162b668eca58

commit 07e819c768456022cf2668ebd7f2fb1b6596b9dc
Author: Poornima <email address hidden>
Date: Tue Apr 21 06:32:32 2020 +0530

Added mirror folder creation

Added mirror folder creation in the docker automation script

Depends-on: https://review.opendev.org/#/c/721517/

    Story: 2007580
    Task: 39501
    Change-Id: Ib...

Reviewed:  https://review.opendev.org/729840
Committed: https://git.openstack.org/cgit/starlingx/tools/commit/?id=05ba65cb054e71ef7c2669668831f50ee334e768
Submitter: Zuul
Branch:    f/centos8

commit d923525441f034c31f0231d15e0f55e3ac5c37b6
Author: Don Penney <don.penney@windriver.com>
Date:   Wed May 20 14:20:35 2020 -0400

Update 7.7.1908 repo to point to vault
    
    The Centos 7.7.1908 repo has been deprecated and moved to the vault.
    This commit updates the repo config file to point to the new baseurl
    locations.
    
    Change-Id: I0d9b98eea925ad05544c7a8bd626ee5dcc48d103
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit daff8261d18e7512c2ebc1d42fa4a75ddbaddda8
Author: Ran An <ran1.an@intel.com>
Date:   Fri May 15 01:34:46 2020 +0000

Add python3 rpms required by python3-daemon runtime
    
    This reverts commit 6fa137af2312abbdfb0f0f015869d65c29e36b82.
    
    Add python3 rpms required by python3-daemon runtime
    adding python3 pkgs to support python3-daemon build
    
    Story: 2007106
    Task: 39353
    
    Signed-off-by: SidneyAn <ran1.an@intel.com>
    Change-Id: I12e1ed08381cbc45480c6c02e4d93c8e04fac2dc

commit e12c55e47ed72fc4e4c38c11525769d89d1f9eed
Author: Ran An <ran1.an@intel.com>
Date:   Fri May 15 01:34:42 2020 +0000

python3: add python3 and python3-devel
    
    This reverts commit 79abb72562b53c8682ea0d854e1071540b38a398.
    
    following pkgs were added:
    python3
    python3-lib
    python3-devel
    python3-rpm-generators (python3-devel dependency)
    python3-rpm-macros (python3-devel dependency)
    
    Story: 2007106
    Task: 39147
    
    Signed-off-by: SidneyAn <ran1.an@intel.com>
    Change-Id: I127453a728a0ce1bbc517a5505c0b187fc945207

commit 008685698ae3e520dbc5178e76fa5956697192ed
Author: Shuicheng Lin <shuicheng.lin@intel.com>
Date:   Wed Apr 29 04:25:59 2020 +0800

Upgrade kata containers to 1.11.0
    
    Include several fixs for IPv6 support in Kata.
    
    Test:
    Could create kata container in both IPv4 and IPv6 environment.
    
    Story: 2006145
    Task: 39077
    
    Change-Id: I0fcd2b516f9003a851e601493ae7c323ceab8df5
    Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>

commit 79abb72562b53c8682ea0d854e1071540b38a398
Author: Ran An <ran1.an@intel.com>
Date:   Thu May 14 11:42:07 2020 +0000

Revert "python3: add python3-devel dep pkgs"
    
    This reverts commit 50bff514030d51dece7d23c2e1b64284ea5cda27.
    
    Change-Id: I50d5926f3ead6e773fa8e86ed2ef72f86ff9f343

commit 6fa137af2312abbdfb0f0f015869d65c29e36b82
Author: Ran An <ran1.an@intel.com>
Date:   Thu May 14 11:42:01 2020 +0000

Revert "Add python3 rpms required by python3-daemon runtime"
    
    This reverts commit b2b481a7b0d348cf332e8002ec522457fbb80814.
    
    Change-Id: Ieb7d2dae22efa31730d3f5560c17162b668eca58

commit 07e819c768456022cf2668ebd7f2fb1b6596b9dc
Author: Poornima <poornima.y.n@intel.com>
Date:   Tue Apr 21 06:32:32 2020 +0530

Added mirror folder creation
    
    Added mirror folder creation in the docker automation script
    
    Depends-on: https://review.opendev.org/#/c/721517/
    
    Story: 2007580
    Task: 39501
    Change-Id: Ib007fe30cf7a7632fa4df0a6664d7c2c09691781
    Signed-off-by: Poornima <poornima.y.n@intel.com>

commit b7ab60320f58d39a9be103583dc3b89ade32004c
Author: Dongqi Chen <chen.dq@neusoft.com>
Date:   Tue May 12 14:14:47 2020 +0800

Remove tpmdd/integrity script
    
    Now use the kernel built-in tpmdd
    and integrity is no longer supported.
    
    Change-Id: I64dd3db829978f2a7adea60b5049604d866f213b
    Story: 2007308
    Task: 39742
    Depends-On: https://review.opendev.org/#/c/726999/
    Signed-off-by: Dongqi Chen <chen.dq@neusoft.com>

commit b2b481a7b0d348cf332e8002ec522457fbb80814
Author: SidneyAn <ran1.an@intel.com>
Date:   Wed May 13 16:21:08 2020 +0800

Add python3 rpms required by python3-daemon runtime
    
    adding python3 pkgs to support python3-daemon build
    
    Story: 2007106
    Task: 39353
    
    Change-Id: I05a31b92df7234be80e4a81b006be0b8d0c555b0
    Signed-off-by: SidneyAn <ran1.an@intel.com>

commit 50bff514030d51dece7d23c2e1b64284ea5cda27
Author: SidneyAn <ran1.an@intel.com>
Date:   Wed May 13 20:46:18 2020 +0800

python3: add python3-devel dep pkgs
    
    add python3-devel dep pkgs python3-rpm-generators
    and python3-rpm-macros
    
    Story: 2007106
    Task: 39147
    
    Change-Id: Ia144af8a5f1fb491937726398c2becf0335dc556
    Signed-off-by: SidneyAn <ran1.an@intel.com>

commit d8db3ee2c4ba4a2955bb25fb2dc140b51456965c
Author: Shuicheng Lin <shuicheng.lin@intel.com>
Date:   Tue May 5 11:01:53 2020 +0800

Add perl-generators to distro layer
    
    4.18 kernel build requires perl-generators.
    
    Story: 2007308
    Task: 39645
    Change-Id: I28123577eede1a558476eb3b6c97019f9982136f
    Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>

commit 43cd8641e62310b057526b4eb49eaa1b7551c154
Author: Shuicheng Lin <shuicheng.lin@intel.com>
Date:   Sat May 2 19:27:54 2020 +0800

Correct rt kernel srpm name to el8_1
    
    It is to align the package name in srpm_path
    
    Story: 2007308
    Task: 39617
    
    Change-Id: I106024a15c2916dbd25c86ecac73cd05d4c15dfc
    Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>

commit 2661a4df94a52c9c03b09e98703a8575f9329276
Author: Sun Austin <austin.sun@intel.com>
Date:   Mon May 4 16:17:35 2020 +0800

add kernel-headers in 3rdparties list
    
    although kernel-headers-4.18.0-147.3.1.el8_1.x86_64.rpm is in
    http://mirror.centos.org/centos/8.1.1911 , but it was not in centos
    repo database anymore, so to avoid mirror download failure , download
    it from 3rdparties directly.
    
    Test:
    download compiler layer mirror succeeded.
    
    Story: 2007308
    Task: 39625
    
    Change-Id: I5b810d4d882ea5e4e275e860280f2871c4fa9fdb
    Signed-off-by: Sun Austin <austin.sun@intel.com>

commit 757d64a13dd5ca95993804915aae08066d5cf021
Author: Saul Wold <sgw@linux.intel.com>
Date:   Fri May 1 13:20:35 2020 -0700

download_mirror: fix failure with custom yum.conf
    
    When using upstream only and a custom yum.conf the wrong configuration
    is sent to the lower_level_rpm_downloader script.
    
    Closes-Bug: 1876216
    Change-Id: I442a4fbddabc223cd0f27df329c04cad7bd159d4
    Signed-off-by: Saul Wold <sgw@linux.intel.com>

commit 0e02ad6f65f2760b693055a55ffe51b40fcd81c2
Author: Shuicheng Lin <shuicheng.lin@intel.com>
Date:   Thu Mar 5 12:41:05 2020 +0800

Get tpm/integrity source code from CentOS 8.1 kernel tarball
    
    The CentOS 4.18.0-147.3.1 kernel is not the same as upstream 4.18
    kernel, and cause tpm/integrity from upstream cannot pass build.
    There are many patches need be cherry-picked or reverted in order
    to pass build with CentOS 8.1 kernel.
    So switch to get tpm/integrity code from CentOS 8.1 kernel tarball.
    With it, we just need focus on STX's patch.
    
    Story: 2007308
    Task: 38950
    Depends-On: https://review.opendev.org/720325
    
    Change-Id: I8090bc1aff7d5fe194567520093738099464b572
    Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>

commit 0328a964126bd0d7d0c5f4fbfdf2c85ae673a46a
Author: Shuicheng Lin <shuicheng.lin@intel.com>
Date:   Wed Mar 4 13:02:42 2020 +0800

Upgrade mellanox driver to 4.7-3.2.9.0 to support CentOS 8.1
    
    tarball download script is updated with the new srpm name.
    
    Story: 2007308
    Task: 38926
    Depends-On: https://review.opendev.org/720325
    Change-Id: Ie1caef3083b9c7ab6aca2873ceef2fd4a86ef4b0
    Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>

commit 6b2912a6aa96862673cdb3fe17dad9870f1c83c6
Author: fuyong <fuyong@neusoft.com>
Date:   Fri Apr 24 11:33:36 2020 +0800

dl_tarball: Add support for creating rt-setup SRPM
    
    This is for downloading the rt-setup tarball from CentOS
    and then use it to create the SRPM via the CentOS common tooling.
    
    This is needed as CentOS does not currently create a rt-setup.src.rpm
    package.
    
    We clone the CentOS rpm/rt-setup which contains the rt-setup.spec
    and use the into_srpm.sh tool which then downloads the correct linux
    tarball from CentOS and generates the .src.rpm.  Since we use the rpm
    name in the .lst file it will only be re-created if the rpm is not in
    the downloads directory.
    
    Signed-off-by: Yong Fu <fuyong@neusoft.com>
    Story: 2007308
    Task: 39558
    Change-Id: Ie43ee5c6b345bf338ea63b9c2a8bfc2cb1430387

commit d030bbb6b3463ab3e69829d9ffcf2019112ae507
Author: Scott Little <scott.little@windriver.com>
Date:   Wed Apr 29 15:09:03 2020 -0400

Fix distro build error, need python2-sphinxcontrib-apidoc
    
    python2-sphinxcontrib-apidoc is required to build python-keystoneauth1.
    
    Closes-Bug: 1875945
    Change-Id: I0a7c7cf459d1cd48060d4d29ccf7f81c7b764246
    Signed-off-by: Scott Little <scott.little@windriver.com>

commit eeb7e4187d2becea8b036ec55783601f0503f071
Author: Don Penney <don.penney@windriver.com>
Date:   Tue Apr 28 14:53:49 2020 -0400

Fix train source repo config
    
    The StarlingX_CentOS-OpenStack-train.repo had a copy/paste error,
    where the train source repo was misconfigured with the stein label.
    This conflicted with the Stein repo config, causing the Train source
    repo not to be mirrored properly. This, in turn, causes build failures
    because updated src rpm references cannot be found.
    
    Change-Id: I323ffbce037a183802302c372b0e9510757b58b1
    Signed-off-by: Don Penney <don.penney@windriver.com>

commit faea86cf1654c77eb8d5b8bc8b9aa8daa5487ed6
Author: Scott Little <scott.little@windriver.com>
Date:   Fri Feb 21 14:21:02 2020 -0500

Fix starlingx_add_pkgs.sh for use in a layered environment.
    
    1) Make starlingx_add_pkgs.sh aware of the extra rpm repositories
    required for a layered build.
    
    2) Don't spew errors if we come accross a dependency on rpmlib(*).
    These aren't dependencies on packages, but rather internal features of
    rpm.
    
    3) Don't get baffled by epochs.  feature_to_pkg returns a package
    version that might included an epoch.  The queries within
    pkg_in_cgcsrepo were baffled by this. I add a second query
    stripping the epoch from the query.
    
    4) Favor packages we already have over newer content from upstream.
    pkg_to_dependencies favors upstream packages if they look to be
    a higher version, so I dropped it's use. Instead I do a recursive
    search, with a modified feature_to_pkg that prefers existing sources.
    
    Story: 2006166
    Task: 37103
    
    Change-Id: I2b199c809267904fef08d9e93383dd2475adcb94
    Signed-off-by: Scott Little <scott.little@windriver.com>

commit 60b8a2c765bbab88f8afff16a9fe6d802be8af01
Author: Long Li <lilong-neu@neusoft.com>
Date:   Tue Apr 28 11:46:16 2020 +0800

Remove redundant python36-psutils lst
    
    keep one python36-psutil-5.6.7-1.el7.x86_64.rpm
    in flock/rpms_centos.lst
    and list with alphabetic sequence
    
    Story: 2007106
    Task: 39353
    
    Change-Id: Ic6147bdea21f217ea87fb9f04b07911de2f6ee4c
    Signed-off-by: Long Li <lilong-neu@neusoft.com>

commit 4401c63893a0ccd7d65375a9b682979bbe8c7581
Author: Saul Wold <sgw@linux.intel.com>
Date:   Wed Apr 22 08:21:45 2020 -0700

dl_tarball: Add support for creating kernel-rt SRPM
    
    This is for downloading the linux-rt tarball from CentOS
    and then use it to create the SRPM via the CentOS common tooling.
    
    This is needed as CentOS does not currently create a kernel-rt.src.rpm
    package.
    
    We clone the CentOS rpm/kernel-rt which contains the kernel-rt.spec
    and use the into_srpm.sh tool which then downloads the correct linux
    tarball from CentOS and generates the .src.rpm.  Since we use the rpm
    name in the .lst file it will only be re-created if the rpm is not in
    the downloads directory.
    
    Story: 2007308
    Task: 39570
    Change-Id: Ieb0443fae4431b78173ef3fa42933b3d795a0816
    Signed-off-by: Saul Wold <sgw@linux.intel.com>

commit e6f0b1bcc77814358f0067f27c4a8c62be1bfb0b
Author: Long Li <lilong-neu@neusoft.com>
Date:   Fri Apr 10 13:17:33 2020 +0800

Add python3 rpms required by python-daemon runtime
    
    Following python3 rpms added:
    python3-lockfile
    python36-docutils
    python36-mock
    python36-pbr
    
    Story: 2007106
    Task: 39353
    Change-Id: Ia23314bd5f5d8529331781426ed54951477835e8
    Signed-off-by: Long Li <lilong-neu@neusoft.com>

commit 8bee1704848debc04941f87235c1ed8d97646826
Author: Poornima <poornima.y.n@intel.com>
Date:   Mon Apr 20 04:23:39 2020 +0530

Correction in Local mirror location
    
    Corrected HOST_MIRROR_DIR build env variable.
    Updated the README.rst file as well.
    
    Closes-Bug: 1874160
    Change-Id: Ib1144c858bbe9fd71b56ac97382cb4c571fd8dbd
    Signed-off-by: Poornima <poornima.y.n@intel.com>

commit ae486fa9afadab21907ea9a9779f950987b31a40
Author: Saul Wold <sgw@linux.intel.com>
Date:   Fri Apr 10 13:12:45 2020 -0700

Update filter to track won't fix or invalid bugs
    
    The Security team wants to be able to mark bugs as Won't Fix or
    Invalid when the CVE pertains to an upstream component or driver
    that we do not include in StarlingX
    
    Story: 2007538
    Task: 39357
    Change-Id: I030f4ea0a7b9132b23796d7acf6c2650770b1d44
    Signed-off-by: Saul Wold <sgw@linux.intel.com>

commit 3853fb86dc5e5822d7bad4136b4474313c4c81b1
Author: Scott Little <scott.little@windriver.com>
Date:   Tue Apr 14 22:49:01 2020 -0400

Fixes for fast layer download
    
    1) Pass the correct yum.conf to dl_lower_layer_rpms.sh. One that is
    consistent with the -s -S -u -U flag we pass.
    
    2) Allow dl_lower_layer_rpms.sh to accept a yum.conf that includes
    a path element.
    
    Story: 2006166
    Task: 39307
    Change-Id: Ib1a434feb7ce2aa068816d31aa8d858441360ac7
    Signed-off-by: Scott Little <scott.little@windriver.com>

commit f6755e179a030f03d0e1b37e505099923b47a48b
Author: Scott Little <scott.little@windriver.com>
Date:   Wed Mar 4 12:24:47 2020 -0500

Enable build of docker images, charts and wheels from a layered build
    
    1) Add config for the container layer.
    
    2) Add support for the new config file 'required_layer_wheel_inc.cfg',
    Which will specify the default urls for lower layer wheels.inc files.
    These files must be downloaded by download_mirror.sh, and linked by
    generate-cgcs-centos-repo.sh.
    
    3) Add options to download_mirror.sh and generate-cgcs-centos-repo.sh
    allowing overrides or additions to the set of url's for lower layer
    wheels.inc
    
    4) Changes to allow build scripts running within a docker container
    to launch a sister (not daughter) container.  This will allow
    build-wheels to run within a container.
    
    Story: 2006166
    Task: 38980
    Depends-On: https://review.opendev.org/711773
    Depends-On: https://review.opendev.org/717755
    Change-Id: I95c71a0327dc9d919ed835d3661a6f1ec7ce30c7
    Signed-off-by: Scott Little <scott.little@windriver.com>

commit bbd79fcd380992192027601b9444edd668803ed4
Author: Scott Little <scott.little@windriver.com>
Date:   Sat Apr 4 19:16:28 2020 -0400

Fast download of lower layer rpms
    
    The default yumdownloader approach to downloading files
    does so one at a time, and is very slow.  Too slow for layered
    build which is built around the assumption that files can
    be downloaded faster than they can be built.
    
    This update will switch from yumdownloader to reposync
    for the download of lower layer rpms.  It exploits the
    fact that each layer and build type publishes it's own lst file
    of rpms to download.  The lst file is transformed into an'
    includepkgs directive in a custom yum.conf which is passed
    to reposync, so we only sync the desited rpms.
    
    Reposync won't redownload rpms that it already has, even if the
    upstream repodata indicates that the files checksum has changed.
    Forceing the redownload of these rpms requires that we manually
    download the upstream repodata and that we use verifytree to
    identify and delete the new obsolete rpms.
    
    Also including two small bug fixes found while investivgating an
    alternative solution to launchpad  1864221.
    - incorrect userid in a chown
    - Bug in get_url, --url pass being passed in the wrong place
    
    Story: 2006166
    Task: 39307
    Closes-Bug: 1864221
    Change-Id: If12b98ff4f5f24d9318250356920f397419f0f80
    Signed-off-by: Scott Little <scott.little@windriver.com>

commit 72ca5f4ce517fd07577b0a139e9980871d809dda
Author: yuchengde <yu.chengde@99cloud.net>
Date:   Mon Apr 13 17:52:14 2020 +0800

Add python3 rpms to build openstack docker image
    
    following rpms are added:
    python36-virtualenv  -- for ussuri opesntack image
    rh-python36-mod_wsgi -- for stx-keystone apache
    
    Do alphabetical ordering from python3* to python36*
    
    Change-Id: I62a823565785b4efd994f3b4100eb8724ec3f1f5

commit 988b0fd88ad4db11ec0b2c9f026ccd4562c9c593
Author: Long Li <lilong-neu@neusoft.com>
Date:   Fri Mar 20 18:39:55 2020 +0800

Add python3 rpms required by engtools runtime
    
    python36-psutil and python3-devel added
    remove fc rpms and use el7 instead
    python3-pip-9.0.3-5.el7.noarch.rpm
    python3-wheel-0.31.1-4.el7.noarch.rpm
    
    Story: 2007106
    Task: 39115
    Depends-on: https://review.opendev.org/#/c/712218/
    Change-Id: Ifd2a00740b04e871b419d1e4d92f7ea1220bac20
    Signed-off-by: Long Li <lilong-neu@neusoft.com>

commit 66479595174bc93335f4ea077be9a16bed2c21ac
Author: SidneyAn <ran1.an@intel.com>
Date:   Wed Mar 11 09:48:04 2020 +0800

add python3 rpms required by nfv runtime
    
    add rpms to support nfv servies runnint in python3 env
    
    following rpms are added:
     python36-six
     python36-netaddr
     python36-iso8601
     python36-sqlalchemy
    
    Change-Id: I97ec668f2d8c6acc42ac212b5989124a017604f3
    Story: 2007106
    Task: 39024
    Signed-off-by: SidneyAn <ran1.an@intel.com>

commit be47bd5b814d31008fe95271af4b0905bc878031
Author: Chen, Yan <yan.chen@intel.com>
Date:   Mon Mar 23 09:10:19 2020 +0000

stx-tools: update dependencies of Train
    
    Update dependencies of Train keystone/barbican
    
    Story: 2007502
    Task: 39248
    
    Depends-on: https://review.opendev.org/#/c/716818
    Change-Id: I705c1bf3c81ef3d79b889fdfb53c61535e8f66c5
    Signed-off-by: Chen, Yan <yan.chen@intel.com>

commit a1fe4b996d82cf54ff65f0cf7f5d6c4d6a7e5639
Author: SidneyAn <ran1.an@intel.com>
Date:   Thu Apr 2 15:52:16 2020 +0800

adding the Train centos repo
    
    adding the Train centos repo to our configured mirror repos.
    
    Change-Id: Iaf11db06357206b7ba24733043fb4431270a72b5
    Story: 2007502
    Task: 39249
    Signed-off-by: SidneyAn <ran1.an@intel.com>

commit 1d2e79449801397c774bcd0ba5063a866707972d
Author: Dongqi Chen <chen.dq@neusoft.com>
Date:   Tue Mar 31 16:22:06 2020 +0800

[kernel-rt 4.18] Add dependence for building rt-setup
    
    1. Add annobin dependence for rt-setup
    
    2. Upgrade binutils and binutils-devel
    Upgrade binutils-2.27-34 to binutils-2.27-41 and
    binutils-devel-2.27-34 to binutils-devel-2.27-41
    to fix rt-setup build error.
    Error: unknown pseudo-op: `.attach_to_group'
    
    3. Remove rt-setup-1.59-5.el7.noarch.rpm
    
    Story: 2007308
    Task: 39211
    
    Change-Id: If95c1b8bbca1ac05ffc4b3b511abd8e082373f35
    Signed-off-by: Dongqi Chen <chen.dq@neusoft.com>

commit 2c36d713c8b729985ee02d5a69330b89a78ac54a
Author: Shuicheng Lin <shuicheng.lin@intel.com>
Date:   Tue Mar 3 09:59:14 2020 +0800

Add CentOS 8 repo for 4.18.0-147.3.1 kernel from CentOS 8.1
    
    This is the 1st step for the incremental upgrade of CentOS 8.
    CentOS 8 repo is added in order to fetch the srpm and rpm.
    kernel module drivers will be upgraded together with kernel,
    while all other modules will be kept unchanged.
    GCC also will be kept using 4.8.5 version from CentOS 7.
    
    Story: 2007308
    Task: 38916
    
    Change-Id: I3262c10fa56620dbf8af6f72999043f3984c83cd
    Signed-off-by: Shuicheng Lin <shuicheng.lin@intel.com>

commit d722dc6a9bb032cf9afc435e89e1bd4c0048ebae
Author: Mingyuan Qi <mingyuan.qi@intel.com>
Date:   Tue Dec 3 06:47:55 2019 +0000

Add user application generation tool
    
    This commit adds the user app gen tool for customer application
    development. This tool completely decouples app development
    from stx build, which means the app developers no longer need
    to fetch stx code/build tool nor to build app by stx build system.
    
    Features:
    
    1. One command to package chart, generate manifest, checksum and
       package app.
    2. Supports local dir, git repo and tarball as chart source.
    3. The app manifest abstracts a few important fields from armada
       schema for user to lower the learning curve of armada.
    4. Static value overrides allowed in app manifest
    
    Story: 2006974
    Task: 37704
    
    Change-Id: I25a85e9fd7bdd0130041499eb9b1fc1350a63756
    Signed-off-by: Mingyuan Qi <mingyuan.qi@intel.com>

tags:

added: in-f-centos8

StarlingX

Layered Build: Main controller fails after unlock - drbd error on manifest

Bug Description

Other bug subscribers

Remote bug watches