Activity log for bug #1124384

Date Who What changed Old value New value Message
2013-02-13 18:26:56 Ben Howard bug added bug
2013-02-13 18:27:34 Ben Howard affects ubuntu cloud-init (Ubuntu)
2013-02-13 20:00:27 Scott Moser bug added subscriber Scott Moser
2013-02-13 21:31:21 Scott Moser affects cloud-init (Ubuntu) mountall (Ubuntu)
2013-02-13 22:43:05 Steve Langasek affects mountall (Ubuntu) cloud-init (Ubuntu)
2013-02-14 14:15:42 Scott Moser summary cloud-init parses yaml incorrectly reload-configuration can confuse upstart
2013-02-14 14:42:49 Scott Moser description Example from failure: https://jenkins.qa.ubuntu.com/job/raring-server-ec2/ARCH=amd64,REGION=eu-west-1,STORAGE=ebs,TEST=multi-part-ud,label=ubuntu-server-ec2-testing/17/console Cloud-init is parsing YAML incorrectly runcmd: ... - [ sh, -c, "echo ============= $(date) ': hello world!' ============= " ] ... - [ sh, -c, 'echo "$(date)" > /root/runcmd.date.txt' ] ... - [ wget, "http://slashdot.org", -O, /tmp/index.html ] ... - touch /tmp/done Fails, while changing "- touch /tmp/done" to " - [ touch /tmp/done ]" works. Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: | #!/bin/sh echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: | #upstart-job description "a test upstart job" start on stopped rc RUNLEVEL=[2345] console output task script echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs: * bug 1080841: should reload configuration if an upstart job is added * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image
2013-02-14 14:42:59 Scott Moser bug task added upstart (Ubuntu)
2013-02-14 14:54:43 Scott Moser bug added subscriber James Hunt
2013-02-14 14:55:48 Scott Moser attachment added dmesg when 'initctl reload-configuration' is used https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1124384/+attachment/3528558/+files/dmesg.reload.txt
2013-02-14 14:56:23 Scott Moser attachment added dmesg when 'initctl reload-configuration' is *not* used https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/1124384/+attachment/3528559/+files/dmesg.no-reload.txt
2013-02-14 14:56:33 Scott Moser cloud-init (Ubuntu): status New Confirmed
2013-02-14 14:56:36 Scott Moser upstart (Ubuntu): status New Confirmed
2013-02-14 14:56:40 Scott Moser upstart (Ubuntu): importance Undecided Medium
2013-02-14 15:30:12 Scott Moser attachment added dmesg with --debug when 'initctl reload-configuration' is used https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+attachment/3528635/+files/dmesg-debug-reload.txt
2013-02-14 15:32:11 Scott Moser attachment added dmesg with --debug when 'initctl reload-configuration' is *not* used https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+attachment/3528636/+files/dmesg-debug-no-reload.txt
2013-02-14 17:19:40 Ubuntu QA Website tags iso-testing
2013-02-27 18:07:57 Scott Moser description Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: | #!/bin/sh echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: | #upstart-job description "a test upstart job" start on stopped rc RUNLEVEL=[2345] console output task script echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs: * bug 1080841: should reload configuration if an upstart job is added * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: | #boothook #!/bin/sh touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: |    #!/bin/sh    echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: |    #upstart-job    description "a test upstart job"    start on stopped rc RUNLEVEL=[2345]    console output    task    script    echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log    end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs:  * bug 1080841: should reload configuration if an upstart job is added  * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image
2013-02-27 18:08:21 Scott Moser description Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: | #boothook #!/bin/sh touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: |    #!/bin/sh    echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: |    #upstart-job    description "a test upstart job"    start on stopped rc RUNLEVEL=[2345]    console output    task    script    echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log    end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs:  * bug 1080841: should reload configuration if an upstart job is added  * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: |    #cloud-boothook    #!/bin/sh    touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: |    #!/bin/sh    echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: |    #upstart-job    description "a test upstart job"    start on stopped rc RUNLEVEL=[2345]    console output    task    script    echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log    end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs:  * bug 1080841: should reload configuration if an upstart job is added  * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image
2013-04-23 18:01:10 Stéphane Graber summary reload-configuration can confuse upstart Configuration reload clears event that others jobs may be waiting on
2013-04-23 18:07:57 Stéphane Graber bug added subscriber Stéphane Graber
2013-04-26 15:33:07 James Hunt attachment added upstart-bug-1124384-minimal-fix.diff https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1124384/+attachment/3654975/+files/upstart-bug-1124384-minimal-fix.diff
2013-04-26 15:33:27 James Hunt upstart (Ubuntu): assignee James Hunt (jamesodhunt)
2013-04-26 16:19:23 Ubuntu Foundations Team Bug Bot tags iso-testing iso-testing patch
2013-04-30 16:41:49 Steve Langasek bug added subscriber Steve Langasek
2013-04-30 22:00:33 Launchpad Janitor branch linked lp:~jamesodhunt/ubuntu/saucy/upstart/bug-1124384
2013-04-30 22:21:02 Launchpad Janitor branch linked lp:~xnox/ubuntu/saucy/upstart/fix-reload-upgrade-bug
2013-05-15 20:41:52 Scott Moser bug task added cloud-init
2013-05-15 20:42:28 Scott Moser cloud-init: status New Confirmed
2013-05-15 20:42:30 Scott Moser cloud-init: importance Undecided High
2013-06-06 11:40:04 Dimitri John Ledkov nominated for series Ubuntu Raring
2013-06-06 11:40:04 Dimitri John Ledkov bug task added upstart (Ubuntu Raring)
2013-06-06 11:40:04 Dimitri John Ledkov bug task added cloud-init (Ubuntu Raring)
2013-06-06 11:40:04 Dimitri John Ledkov nominated for series Ubuntu Saucy
2013-06-06 11:40:04 Dimitri John Ledkov bug task added upstart (Ubuntu Saucy)
2013-06-06 11:40:04 Dimitri John Ledkov bug task added cloud-init (Ubuntu Saucy)
2013-06-06 14:59:17 Dimitri John Ledkov description Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: |    #cloud-boothook    #!/bin/sh    touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: |    #!/bin/sh    echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: |    #upstart-job    description "a test upstart job"    start on stopped rc RUNLEVEL=[2345]    console output    task    script    echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log    end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs:  * bug 1080841: should reload configuration if an upstart job is added  * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image [Impact] * The status of blocked events was not preserved, when upstart performed stateful re-execution or configuration reload. Thus jobs with complex start/stop conditions (one or more "and" clauses), with at least one event emitted before re-exec/reload, may not execute when remaining conditions are finally satisfied. * Above may prevent certain system to function correctly, and in the cases similar to cloud-init instances may even cause failure to boot. * The fix includes incriminating reference counts on blocked events, whilst job configuration is reloaded and fully serialising all upstart objects, including blocked events, during stateful re-execution. * Since previous versions of upstart, do not serialise blocked events the upgrade needs special casing. On upgrade upstart will perform stateful re-execution, unless runlevel 2 has been already reached. Instead upstart, will re-executed at system shutdown. This should allow upgrading upstart during early boot of cloud-init instances. But do note, that old instance of upstart will still be running as init and the running machine will still be affected by the bug described here. [Test Case] * Create a sample job /etc/init/foo.conf similar to this: start on (event1 and event2) task exec date * Test reload configuration works correctly: $ sudo status foo foo stop/waiting $ sudo initctl emit -n event1 $ sudo initctl reload-configuration $ sudo initctl emit -n event2 $ sudo tail /var/log/upstart/foo.log At the end one should see a timestamp appended in the foo.log. * Test stateful re-exec works correctly: $ sudo initctl emit -n event1 $ sudo telinit u $ sudo initctl emit -n event2 $ sudo tail /var/log/upstart/foo.log [Regression Potential] * The bug fix introduced here is fairly large (approx 1.5k line diff) but comes with comprehensive set of test-suites to verify the two bug fixes as well as all possible combinations of stateful re-execution serialisation formats. Majority of code changes are for additional [de]serialisation, which follow existing well tested code pattern. And changes to reference counting have been carefully reviewed and tested by multiple developers. * While the bug report indicates a severe problem, it was not noticed until recently, as the system must be under heavy race conditions to become affected by this bug. Since systems reaching stable state, with little or no blocked events left, would not normally be affected. * Overall regression potential is deemed low. [Original Bug Report] Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: |    #cloud-boothook    #!/bin/sh    touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: |    #!/bin/sh    echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: |    #upstart-job    description "a test upstart job"    start on stopped rc RUNLEVEL=[2345]    console output    task    script    echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log    end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs:  * bug 1080841: should reload configuration if an upstart job is added  * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image
2013-06-13 12:08:06 Dimitri John Ledkov upstart (Ubuntu Saucy): status Confirmed Fix Released
2013-06-17 13:01:33 Dimitri John Ledkov description [Impact] * The status of blocked events was not preserved, when upstart performed stateful re-execution or configuration reload. Thus jobs with complex start/stop conditions (one or more "and" clauses), with at least one event emitted before re-exec/reload, may not execute when remaining conditions are finally satisfied. * Above may prevent certain system to function correctly, and in the cases similar to cloud-init instances may even cause failure to boot. * The fix includes incriminating reference counts on blocked events, whilst job configuration is reloaded and fully serialising all upstart objects, including blocked events, during stateful re-execution. * Since previous versions of upstart, do not serialise blocked events the upgrade needs special casing. On upgrade upstart will perform stateful re-execution, unless runlevel 2 has been already reached. Instead upstart, will re-executed at system shutdown. This should allow upgrading upstart during early boot of cloud-init instances. But do note, that old instance of upstart will still be running as init and the running machine will still be affected by the bug described here. [Test Case] * Create a sample job /etc/init/foo.conf similar to this: start on (event1 and event2) task exec date * Test reload configuration works correctly: $ sudo status foo foo stop/waiting $ sudo initctl emit -n event1 $ sudo initctl reload-configuration $ sudo initctl emit -n event2 $ sudo tail /var/log/upstart/foo.log At the end one should see a timestamp appended in the foo.log. * Test stateful re-exec works correctly: $ sudo initctl emit -n event1 $ sudo telinit u $ sudo initctl emit -n event2 $ sudo tail /var/log/upstart/foo.log [Regression Potential] * The bug fix introduced here is fairly large (approx 1.5k line diff) but comes with comprehensive set of test-suites to verify the two bug fixes as well as all possible combinations of stateful re-execution serialisation formats. Majority of code changes are for additional [de]serialisation, which follow existing well tested code pattern. And changes to reference counting have been carefully reviewed and tested by multiple developers. * While the bug report indicates a severe problem, it was not noticed until recently, as the system must be under heavy race conditions to become affected by this bug. Since systems reaching stable state, with little or no blocked events left, would not normally be affected. * Overall regression potential is deemed low. [Original Bug Report] Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: |    #cloud-boothook    #!/bin/sh    touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: |    #!/bin/sh    echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: |    #upstart-job    description "a test upstart job"    start on stopped rc RUNLEVEL=[2345]    console output    task    script    echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log    end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs:  * bug 1080841: should reload configuration if an upstart job is added  * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image [Impact]  * The status of blocked events was not preserved, when upstart    performed stateful re-execution or configuration reload. Thus jobs    with complex start/stop conditions (one or more "and" clauses),    with at least one event emitted before re-exec/reload, may not    execute when remaining conditions are finally satisfied.  * Above may prevent certain system to function correctly, and in the    cases similar to cloud-init instances may even cause failure to    boot.  * The fix includes incriminating reference counts on blocked events,    whilst job configuration is reloaded and fully serialising all    upstart objects, including blocked events, during stateful    re-execution.  * Since previous versions of upstart, do not serialise blocked events    the upgrade needs special casing. On upgrade upstart will perform    stateful re-execution, unless runlevel 2 has been already    reached. Instead upstart, will re-executed at system shutdown. This    should allow upgrading upstart during early boot of cloud-init    instances. But do note, that old instance of upstart will still be    running as init and the running machine will still be affected by    the bug described here. [Test Case]  * Create a sample job /etc/init/foo.conf similar to this: start on (event1 and event2) task exec date  * Test reload configuration works correctly: $ sudo status foo foo stop/waiting $ sudo initctl emit -n event1 $ sudo initctl reload-configuration $ sudo initctl emit -n event2 $ sudo tail /var/log/upstart/foo.log At the end one should see a timestamp appended in the foo.log.  * Test stateful re-exec works correctly: $ sudo initctl emit -n event1 $ sudo telinit u $ sudo initctl emit -n event2 $ sudo tail /var/log/upstart/foo.log * Start an ubuntu-cloud image (in lxc or cloud) with apt-get update & upgrade enabled going from upstart version without this fix included to a one that does have it. Cloud-final should finish and boot-finished under /var/lib/cloud/instances/*/boot-finished. Please note this test should be performed in isolation from dbus security update that does partial stateful re-exec at the moment. [Regression Potential]  * The bug fix introduced here is fairly large (approx 1.5k line diff)    but comes with comprehensive set of test-suites to verify the two    bug fixes as well as all possible combinations of stateful    re-execution serialisation formats. Majority of code changes are    for additional [de]serialisation, which follow existing well tested    code pattern. And changes to reference counting have been carefully    reviewed and tested by multiple developers.  * While the bug report indicates a severe problem, it was not noticed    until recently, as the system must be under heavy race conditions    to become affected by this bug. Since systems reaching stable state,    with little or no blocked events left, would not normally be    affected.  * Overall regression potential is deemed low. [Original Bug Report] Under bug 1080841 we made cloud-init invoke 'initctl reload-configuration' after it wrote a upstart job. This was necessary because inotify is not supported on all filesystems (overlayfs being the one of most current interst). This seems to be causing upstart some pain, and resulting in cloud-final (and 'rc') not being run. Easy user-data to reproduce the problem is: #cloud-config-archive - content: |    #cloud-boothook    #!/bin/sh    touch /run/cloud-init-upstart-reload # hack, see trunk commit 783 - content: |    #!/bin/sh    echo "==== $(date -R): user-script run ===" | tee /run/user-script.log - content: |    #upstart-job    description "a test upstart job"    start on stopped rc RUNLEVEL=[2345]    console output    task    script    echo "==== $(date -R): upstart job run ===" | tee /run/upstart-job.log    end script You should (and do on quantal) end up with 2 files written to /run. I've verified that the same behavior is true on quantal. If you change cloud-init to notify upstart about a job immediately after it writes it, then quantal's upstart gets confused also. Related bugs:  * bug 1080841: should reload configuration if an upstart job is added  * bug 1103881: cloud-final is never executed if upstart is upgraded during initialization of the image
2013-06-18 13:52:03 Launchpad Janitor cloud-init (Ubuntu Raring): status New Confirmed
2013-06-18 13:52:03 Launchpad Janitor upstart (Ubuntu Raring): status New Confirmed
2013-06-28 00:40:58 Steve Langasek upstart (Ubuntu Raring): status Confirmed Fix Committed
2013-06-28 00:41:02 Steve Langasek bug added subscriber Ubuntu Stable Release Updates Team
2013-06-28 00:41:05 Steve Langasek bug added subscriber SRU Verification
2013-06-28 00:41:07 Steve Langasek tags iso-testing patch iso-testing patch verification-needed
2013-06-29 20:02:52 Ubuntu Foundations Team Bug Bot bug added subscriber Brian Murray
2013-06-29 20:02:58 Ubuntu Foundations Team Bug Bot tags iso-testing patch verification-needed iso-testing patch verification-failed verification-needed
2013-07-01 15:04:21 Dimitri John Ledkov upstart (Ubuntu Raring): assignee Dmitrijs Ledkovs (xnox)
2013-07-01 16:18:36 Steve Langasek tags iso-testing patch verification-failed verification-needed iso-testing patch verification-needed
2013-07-04 03:25:28 Mike Milner bug added subscriber Mike Milner
2013-07-04 10:05:53 Martin Packman bug added subscriber Martin Packman
2013-07-09 06:27:50 Launchpad Janitor branch linked lp:cloud-init
2013-07-10 14:58:49 Steve Langasek tags iso-testing patch verification-needed iso-testing patch verification-failed
2013-07-10 18:29:05 Launchpad Janitor cloud-init (Ubuntu Saucy): status Confirmed Fix Released
2013-07-12 09:53:55 Launchpad Janitor branch linked lp:~ubuntu-core-dev/ubuntu/raring/upstart/raring
2013-07-23 15:30:22 Launchpad Janitor branch linked lp:ubuntu/cloud-init
2013-07-24 21:54:10 Simon Belluzzo bug added subscriber Simon Belluzzo
2013-08-01 22:10:02 Launchpad Janitor branch linked lp:~ubuntu-branches/ubuntu/saucy/cloud-init/saucy-proposed
2013-11-07 03:13:15 Steve Langasek upstart (Ubuntu Raring): status Fix Committed Won't Fix
2014-12-05 06:08:52 Rolf Leggewie cloud-init (Ubuntu Raring): status Confirmed Won't Fix
2014-12-05 15:05:18 Brian Murray removed subscriber Brian Murray
2015-11-09 19:58:42 Stéphane Graber removed subscriber Stéphane Graber