/var/log/upstart is masked by the mapping of /dev/mapper/logs-log on controllers

[guillaume thouvenin]

With MOS 7.0 there is no directory /var/log/upstart.

It seems that this directory is created before mounting the /dev/mapper/logs-log. Once the /dev/mapper/logs-log is mounted on /var/log, it hides the upstart directory and the consequence is that the upstart process is writing logs where we cannot see them.

If I create a /var/log/upstart then upstart is writing log again for newly started process.

[deactivateduser (deactivateduser-deactivatedaccount)]

Can you please add steps to reproduce?

[guillaume thouvenin (guillaume-thouvenin)]

Here are the steps to reproduce the bug:

1) Install MOS 7.0
2) Connect to a node (controller for example)
3) look in /var/log, upstart directory is missing for me.

I'm currently deploying another environmen so I will be able to confirm if the upstart directory is missing for all roles or only for controllers.

[guillaume thouvenin (guillaume-thouvenin)]

Just checked on a newly created environment and the /var/log/upstart directory is missing on controller.

[Matthew Mosesohn (raytrac3r)]

This bug looks like upstart is writing to /var/log/upstart before /var/log is mounted.
We should consider if this critical for 7.0 or not. Definitely high priority for 8.0.

[Matthew Mosesohn (raytrac3r)]

Marked as critical for 7.0 too.

The solution is to apply one of the two commands (preferrably during fuel-image stage):
mkdir /var/log/upstart
or
apt-get reinstall -y upstart

Passing to Vladimir Kozhukalov for the fix.

[Mike Scherbakov (mihgen)]

Why do you consider this bug as a blocker for 7.0? Can you please clarify user impact here?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/224347

[Vladimir Kozhukalov (kozhukalov)]

The thing is that we build images for / and /boot file systems. We don't have separate images for nodes or roles. One image set for a cluster. But for some roles we have /var/log file system. This file system is mounted into the /var/log directory on root file system where there are files, including upstart log directory. When starting upstart begins to write log messages into directory and then mounts /var/log file system and log is still written on root file system and we can not see those logs and can not rotate them.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/224460

[Vladimir Kozhukalov (kozhukalov)]

The issue does not seem to be critical because it does not break deployment and any other piece of UX. The only symptom is that we can not see some log messages and thus can not rotate them. So, I'm changing the priority to high and we will release the fix via updates channel.

[Vladimir Kuklin (vkuklin)]

Vladimir, we do not need full-scale solution here. Let's just do mkdir here and deliver the full fix with updates.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/224579

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/7.0
Review: https://review.openstack.org/224581

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224579
Committed: https://git.openstack.org/cgit/stackforge/fuel-agent/commit/?id=8f39e1a8486603a69f3147357f3d82edfa5cd2b4
Submitter: Jenkins
Branch: master

commit 8f39e1a8486603a69f3147357f3d82edfa5cd2b4
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 17 14:24:45 2015 +0300

    Temporary fix for lost upstart directory

    When we have just os-root fs image and don't have
    os-var-log fs image while / and /var/log are supposed to be
    separate file systems and os-var-log is mounted into
    non-empty directory on the / file system, those files in /var/log
    directory become unavailable.
    The thing is that among those file there is /var/log/upstart
    where upstart daemon writes its logs. We have specific upstart job
    which is to flush open files once all file systems are mounted.
    This job needs upstart directory to be available on os-var-log
    file system.
    This is just a temporary fix and correct fix will be available soon
    via updates.

    Change-Id: Idfea34541e9aabdd22455456c082d78da42657f9
    Related-Bug: #1494684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/224581
Committed: https://git.openstack.org/cgit/stackforge/fuel-agent/commit/?id=50e90af6e3d560e9085ff71d2950cfbcca91af67
Submitter: Jenkins
Branch: stable/7.0

commit 50e90af6e3d560e9085ff71d2950cfbcca91af67
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 17 14:24:45 2015 +0300

    Temporary fix for lost upstart directory

    When we have just os-root fs image and don't have
    os-var-log fs image while / and /var/log are supposed to be
    separate file systems and os-var-log is mounted into
    non-empty directory on the / file system, those files in /var/log
    directory become unavailable.
    The thing is that among those file there is /var/log/upstart
    where upstart daemon writes its logs. We have specific upstart job
    which is to flush open files once all file systems are mounted.
    This job needs upstart directory to be available on os-var-log
    file system.
    This is just a temporary fix and correct fix will be available soon
    via updates.

    Change-Id: Idfea34541e9aabdd22455456c082d78da42657f9
    Related-Bug: #1494684

[Vladimir Kozhukalov (kozhukalov)]

It turned out that the issue is rather critical as far as it can lead to log loss as well as to non-functional node due to full root file system.

[Vladimir Kozhukalov (kozhukalov)]

This patch https://review.openstack.org/224579 does not properly fixes the issue but still it is a workaround. So, the issue for 8.0 should not be closed, it is still High priority issue and the proper fix needs to be implemented.

[Andrey Sledzinskiy (asledzinskiy)]

verified on
{

    "build_id": "299",
    "build_number": "299",
    "release_versions":

{

    "2015.1.0-7.0":

{

    "VERSION":

{

    "build_id": "299",
    "build_number": "299",
    "api": "1.0",
    "fuel-library_sha": "6efc6698dd3d4b07e90a97a49c702128ccc88de7",
    "nailgun_sha": "d590b26dbb09785b8a8b3651b0ef69746fcf9991",
    "feature_groups":

            [
                "mirantis"
            ],
            "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
            "openstack_version": "2015.1.0-7.0",
            "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67",
            "production": "docker",
            "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
            "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012",
            "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c",
            "release": "7.0",
            "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "6efc6698dd3d4b07e90a97a49c702128ccc88de7",
"nailgun_sha": "d590b26dbb09785b8a8b3651b0ef69746fcf9991",
"feature_groups":

    [
        "mirantis"
    ],
    "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
    "openstack_version": "2015.1.0-7.0",
    "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67",
    "production": "docker",
    "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
    "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012",
    "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c",
    "release": "7.0",
    "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

}

[Roman Vyalov (r0mikiam)]

decrease to medium because workaround was merged everything works fine.

[Vladimir Kozhukalov (kozhukalov)]

fuel-agent features development has been frozen in favor of bareon, so this bug should not be fixed in fuel-agent taking into account that workaround works just fine.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: master
Review: https://review.openstack.org/224347
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Vitaly Sedelnik (vsedelnik)]

Targeted to 6.1-mu-6 as this fix is needed for https://bugs.launchpad.net/fuel/+bug/1537699

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/302054

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/305714

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexey Stupnikov (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/302054
Reason: This patch will not work for MOS6.1: it will break ntpd, and lead to missing logs.

[Alexey Stupnikov (astupnikov)]

This bug was fixed in MOS7 by temporary patch. The issue will be fully resolved in bug #1537699, so closing this bug as a duplicate.