R5.0-micro-services - The core files of the micro-services are not stored under /var/crashes

Bug #1757340 reported by Sudheendra Rao
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Released
Critical
Andrey Pavlov

Bug Description

The core files of the micro-services are not stored under /var/crashes (it goes into / of container) and they are generated with random name instead of with service name.
This is the behaviour for vrouter-agent, i don't know about other services.
Also it would be good to mount the /var/crashes of container on the host.

[root@nodei2 /]# ps -eaf | grep vrouter
root 5637 5615 15 Mar20 ? 02:49:48 /usr/bin/contrail-vrouter-agent
root 8392 8373 0 Mar16 ? 00:43:25 /usr/bin/python /usr/bin/contrail-nodemgr --nodetype=contrail-vrouter
root 10052 10009 0 05:16 ? 00:00:00 docker exec -it vrouter_vrouter-agent_1 bash
root 10117 10078 0 05:16 ? 00:00:00 grep --color=auto vrouter
[root@nodei2 /]#
[root@nodei2 /]# kill -3 5637
[root@nodei2 /]#
[root@nodei2 /]# ls -l /var/crashes
total 0
[root@nodei2 /]#
[root@nodei2 /]# pwd
/
[root@nodei2 /]# ls -l core*
-rw------- 1 root root 381034496 Mar 21 05:16 core.5637
[root@nodei2 /]#

Jeba Paulaiyan (jebap)
tags: added: sanityblocker
Revision history for this message
Hari Prasad Killi (haripk) wrote :

This is not vrouter-agent specific behavior. The file /proc/sys/kernel in the container should be set to /var/crashes/core.%e.%p.%h.%t ?

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

Please specify build number and deployment method.

Revision history for this message
Sudheendra Rao (sudheendra-k) wrote :

This is seen on all R5.0 builds with micro-services, to be specific ocata-master-34, using ansible deployer.

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

@Sudheendra - can you please run 'sysctl kernel.core_pattern' in vrouter-agent container?

I have deployment by kolla where it works well
an I have deployment by helm where it points to something strange

but I don't have ansible-deployer to check

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

ansible has another strange thing even if file is present in sysctl

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/40905
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/40907
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/40905
Committed: http://github.com/Juniper/contrail-container-builder/commit/776f8c40d9bdd87e069bbba91c51ae48c5a80d48
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 776f8c40d9bdd87e069bbba91c51ae48c5a80d48
Author: Andrey Pavlov <email address hidden>
Date: Wed Mar 21 16:25:09 2018 +0300

apply core pattern setting to all containers

Change-Id: I688dcdbacb06049efcf0a17a0ac6150c839ae768
Closes-Bug: #1757340

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/40907
Committed: http://github.com/Juniper/contrail-ansible-deployer/commit/84a6d685365cd8faab03ea0542b8700ff71fe47f
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 84a6d685365cd8faab03ea0542b8700ff71fe47f
Author: Andrey Pavlov <email address hidden>
Date: Wed Mar 21 16:37:43 2018 +0300

add mounting of /var/crashes

Change-Id: Icf5c4164e60840301545cb2c8f8b3fa7b7b18c11
Partial-Bug: #1757340

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

current solution doesn't work

[root@node-10-3-71-149 /]# sysctl -p /etc/sysctl.d/10-core-pattern.conf
sysctl: setting key "kernel.core_pattern": Read-only file system

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/41185
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/41185
Committed: http://github.com/Juniper/contrail-container-builder/commit/a57024cd94afe8a3eceacdda4adc9ff7e1a2f579
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit a57024cd94afe8a3eceacdda4adc9ff7e1a2f579
Author: Andrey Pavlov <email address hidden>
Date: Thu Mar 29 16:20:28 2018 +0300

introduce node-init container

This container configures Host's OS for common Contrail needs:
- tune sysctl
- creates contrail-status utility
- can create certificates

This container must be run in each POD in privileged mode.

Change-Id: If4b93845218dbbf7ec218adeccc5b84309d00360
Partial-Bug: #1757340

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/41241
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/41247
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/41248
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/41248
Committed: http://github.com/Juniper/openstack-helm/commit/2e0053585362ee89cf698651d71b058b6b7cde67
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 2e0053585362ee89cf698651d71b058b6b7cde67
Author: Andrey Pavlov <email address hidden>
Date: Fri Mar 30 23:00:34 2018 +0300

change build from 43 to 46

node-init is new image that is present starting from build 46
to make changes for node-init build has to changed

Change-Id: I442f27d5a02b32dfad3f7a58884f2ee9a2553fa6
Partial-Bug: #1757340

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/41247
Committed: http://github.com/Juniper/contrail-helm-deployer/commit/b7ffa6faae39234c60ca8f513c595301972ec2e4
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit b7ffa6faae39234c60ca8f513c595301972ec2e4
Author: Andrey Pavlov <email address hidden>
Date: Fri Mar 30 22:54:34 2018 +0300

change build from 43 to 46

node-init is new image that is present starting from build 46
to make changes for node-init build has to changed

Change-Id: I9e00f74d991caaa50a5f93df816aaebb1ad68073
Partial-Bug: #1757340

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/41241
Committed: http://github.com/Juniper/contrail-ansible-deployer/commit/941c9df6d6b99fd65186a66188cf4ce9365b6896
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 941c9df6d6b99fd65186a66188cf4ce9365b6896
Author: Andrey Pavlov <email address hidden>
Date: Fri Mar 30 18:01:31 2018 +0300

add node-init container to pods

This container configures Host's OS for common Contrail needs:
- tune sysctl
- creates contrail-status utility
- can create certificates
This container must be run in each POD in privileged mode.

Change-Id: I5ce04acc8f783dd6e9e9ec21391fa556cfb53b4d
Related-Change: If4b93845218dbbf7ec218adeccc5b84309d00360
Partial-Bug: #1757340

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/41266
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/41266
Committed: http://github.com/Juniper/contrail-helm-deployer/commit/573efb8fe24b09c27063440c068171b4396ec33c
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 573efb8fe24b09c27063440c068171b4396ec33c
Author: Andrey Pavlov <email address hidden>
Date: Sat Mar 31 10:01:05 2018 +0300

add node-init container to pods

This container configures Host's OS for common Contrail needs:
- tune sysctl (core_pattern: /var/crashes should be mounted without paths)
- creates contrail-status utility
- can create certificates
This container must be run in each POD in privileged mode.

Change-Id: Ib1bca26924ab0729981294e9433ba21db48d6e92
Related-Change: If4b93845218dbbf7ec218adeccc5b84309d00360
Partial-Bug: #1757340

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.