pollinate can take a long time

[Patricia Gaughen]

[Impact]

* Due to code inefficiences, pollinate takes a significant amount of time during firstboot.

* Customers expect minimal boot-times, particularly wrt. pollinate which is described as having an impact in seconds.

[Test Case]

* Boot an Amazon instance, e.g., and examine the systemd-analyze output with and without the fix.

* Without the fix, pollinate will account for 15-60s of time. With the fix, that drops to ~6s.

[Regression Potential]

* While a large number of changes to pollinate:

 changelog | 35 ++++++++++++++++++++++++++++++++++-
 pollinate | 54 ++++++++++++++++++++++++++++++++--------------------
 pollinate.default | 2 +-
 3 files changed, 67 insertions(+), 24 deletions(-)

the releases changes have been well-tested manually in yakkety and represent the next logical release for pollinate relative to what has already been SRU'd (precise upload currently is in -proposed)..

* The highest likelihood of regression stems from possible behaviour dependencies on the old code, but that seems a) unlikely to be the case and b) fixable during the -proposed testing, if encountered.

---

backport to xenial fixes in pollinate that address slow run times (see http://bazaar.launchpad.net/~pollinate/pollinate/trunk/revision/325).

To see the issue launch an AMI, and look at systemd-analyze blame. pollinate was taking anywhere from 12-50 seconds to run during first boot.

[Brian Murray (brian-murray)]

Hello Patricia, or anyone else affected,

Accepted pollinate into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pollinate/4.23-0ubuntu1~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Patricia, or anyone else affected,

Accepted pollinate into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pollinate/4.23-0ubuntu1~14.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Patricia, or anyone else affected,

Accepted pollinate into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pollinate/4.23-0ubuntu1~12.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dustin Kirkland  (kirkland)]

The package in xenial-updates passes my post-boot testing. Appears to be working fine here. To really complete this testing, though, Nish will need to try this out in a custom built AMI with pollinate running for the first time at boot in a public cloud.

[Joshua Powers (powersj)]

After a little fighting with what directories to remove and creating the
AMI I think I was able to verify on all three releases. Below are the
steps that I completed:

* Launched all three LTS AMI
* For Xenial captured `systemd-analyze` and `systemd-analyze blame`
* Installed the proposed version of pollinate
* sudo rm -Rf /var/lib/cloud/instance/* /var/cache/pollinate/* /var/log/cloud-init*
* Shut the instance down and create a new AMI
* Launch the new AMI
* On Xenial, verify the systemd-analyze output was improved
* Verify /var/cache/polinate/seeded existed and /var/log/cloud-init* logs existed
* Verified syslog has pollinate entries and were sucessful

Below is the Xenial systemd output:

Xenial ami-204e0040
Startup finished in 9.240s (kernel) + 51.217s (userspace) = 1min 457ms
29.245s pollinate.service

Xenial ami-204e0040 + New Pollinate
Startup finished in 5.051s (kernel) + 7.946s (userspace) = 12.998s
3.755s pollinate.service

[Joshua Powers (powersj)]

Versions tested:
Precise: 4.23-0ubuntu1~12.04
Trusty: 4.23-0ubuntu1~14.04
Xenial: 4.23-0ubuntu1~16.04

[Launchpad Janitor (janitor)]

This bug was fixed in the package pollinate - 4.23-0ubuntu1~12.04

---------------
pollinate (4.23-0ubuntu1~12.04) precise; urgency=medium

  [ Dustin Kirkland ]
  * New upstream release (LP: #1621280):
    - pollinate:
      + revert revision r300, as this was the wrong fix to the slow
        pollinate problem; as it turns out, it was the user_agent
        function, which was running apt-cache very early in boot, before
        the apt database had been created
      + as it turns out, we need the curl timeout options in order for
        curl to work properly and be resilient against issues with the
        network coming up early in boot
    - debian/pollinate.default, pollinate
      + fix a couple of bugs affecting how long pollinate takes, and if
        it actually completes successfully
      + dpkg -l is way faster than apt-cache, when there is no apt cache
      + wait a maximum of 10 seconds
      + only log to stderr if in an interactive terminal; otherwise,
        just log to syslog
        - this fixes the odd double-printing to /var/log/syslog
      + optimize obtaining version strings by saving one pipe per call
      + source /etc/lsb/release and use $DISTRIB_DESCRIPTION, rather
        than calling lsb-release (python, can be slow)
      + add -m (max-timeout) back to curl options; removing this option
        has caused network failures in various strange ways
      + when network fails due to timeout, log accordingly
      + ignore cloud-init package not found; it's optional

 -- Nishanth Aravamudan <email address hidden> Wed, 28 Sep 2016 14:13:33 -0700

[Martin Pitt (pitti)]

The verification of the Stable Release Update for pollinate has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pollinate - 4.23-0ubuntu1~14.04

---------------
pollinate (4.23-0ubuntu1~14.04) trusty; urgency=medium

  [ Dustin Kirkland ]
  * New upstream release (LP: #1621280):
    - pollinate:
      + revert revision r300, as this was the wrong fix to the slow
        pollinate problem; as it turns out, it was the user_agent
        function, which was running apt-cache very early in boot, before
        the apt database had been created
      + as it turns out, we need the curl timeout options in order for
        curl to work properly and be resilient against issues with the
        network coming up early in boot
    - debian/pollinate.default, pollinate
      + fix a couple of bugs affecting how long pollinate takes, and if
        it actually completes successfully
      + dpkg -l is way faster than apt-cache, when there is no apt cache
      + wait a maximum of 10 seconds
      + only log to stderr if in an interactive terminal; otherwise,
        just log to syslog
        - this fixes the odd double-printing to /var/log/syslog
      + optimize obtaining version strings by saving one pipe per call
      + source /etc/lsb/release and use $DISTRIB_DESCRIPTION, rather
        than calling lsb-release (python, can be slow)
      + add -m (max-timeout) back to curl options; removing this option
        has caused network failures in various strange ways
      + when network fails due to timeout, log accordingly
      + ignore cloud-init package not found; it's optional

 -- Nishanth Aravamudan <email address hidden> Wed, 28 Sep 2016 14:12:37 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package pollinate - 4.23-0ubuntu1~16.04

---------------
pollinate (4.23-0ubuntu1~16.04) xenial; urgency=medium

  [ Dustin Kirkland ]
  * New upstream release (LP: #1621280):
    - pollinate:
      + revert revision r300, as this was the wrong fix to the slow
        pollinate problem; as it turns out, it was the user_agent
        function, which was running apt-cache very early in boot, before
        the apt database had been created
      + as it turns out, we need the curl timeout options in order for
        curl to work properly and be resilient against issues with the
        network coming up early in boot
    - debian/pollinate.default, pollinate
      + fix a couple of bugs affecting how long pollinate takes, and if
        it actually completes successfully
      + dpkg -l is way faster than apt-cache, when there is no apt cache
      + wait a maximum of 10 seconds
      + only log to stderr if in an interactive terminal; otherwise,
        just log to syslog
        - this fixes the odd double-printing to /var/log/syslog
      + optimize obtaining version strings by saving one pipe per call
      + source /etc/lsb/release and use $DISTRIB_DESCRIPTION, rather
        than calling lsb-release (python, can be slow)
      + add -m (max-timeout) back to curl options; removing this option
        has caused network failures in various strange ways
      + when network fails due to timeout, log accordingly
      + ignore cloud-init package not found; it's optional

 -- Nishanth Aravamudan <email address hidden> Wed, 28 Sep 2016 14:01:48 -0700