Feature Freeze Exception request for Upstart in Natty

[James Hunt]

Binary package hint: upstart

= Overview =

The Upstart work for Natty comprises 4 branches to be merged first into
an Upstream version of Upstart, and then merged down into an
Ubuntu-specific branch for Natty. The branches are:

1) lp:~canonical-scott/upstart/bridges
2) lp:~canonical-scott/upstart/session-support

3) lp:~jamesodhunt/upstart/override-support
4) lp:~jamesodhunt/upstart/upstart-job-visualisation

As the names suggest, Scott worked on (1) and (2), whilst I worked on
(3) and (4). However, the timing of when Scott left meant that (1) and
(2) were in an incomplete state (although the basic coding was
complete). Branches (1) and (2)...

  a) Were not documented
  b) Were not fully tested
  c) Did not have their own set of Upstart unit/functional tests.

I have worked with cjwatson to merge branches (1)-(4) into an upstream
version of Upstart. Due to the issues outlined above, this has taken a
lot longer than anticipated, so at this point, we have only merged
branches (1)-(3) inclusive into:

  lp:~upstart-devel/upstart/0.9

= Feature Freeze Exception Request =

I would like to apply for a Feature Freeze Exception for the inclusion
of all *four* branches into Natty in 2 staged releases:

- Stage 1 comprising branches : (1)-(3)

- Stage 2 comprising branch : (1)-(4)
  This encompasses:

  - the addition of Branch (4).
  - the addition of test code and documentation for branches (1) and
    (2).

== Rationale ==

All 4 branches are deemed to have highly desirable features for users.

=== Branch 1 ===

Branch (1) provides a "socket activation" feature. It has minimal impact
on users in case of problems in that users would have to migrate their
jobs (.conf files) to use the new features.

JUSTIFICATION:

- By default, this migration will not happen for Natty, thus the risk
  associated with the branch is deemed to be low.

=== Branch 2 ===

Branch (2) provides:

  - chroot support
  - session support
  - user-session support

These features are highly desirable, particularly the chroot support at
this point. However, unlike Branch (1), due to the nature of the session
support, there are *potential* security considerations (for example a
non-privileged user potentially being able to star/stop a system
service.

JUSTIFICATION

This branch contains very useful features for users. However, the view
is that in light of the amount of testing that has been performed on
this code, Branch (2) should be added to Natty ** with session and
user-session support disabled by default **. This support could be
re-enabled by users if they so wished, or by Ubuntu developers (via
apt/dpkg) at a later date.

=== Branch 3 ===

Branch (3) provides override file support which is a feature which users
have been requesting for some time.

JUSTIFICATION

- Branch (3) does modify the Upstart init daemon itself and has
  appropriate documentation and test cases. Thus, the perceived risk of
  adding this is deemed to be low.

- Users can choose not to use the override facility if problems are
  experienced in its operation.

=== Branch 4 ===

Branch (4) provides job and to some degree event visualisation, again a
feature users have been after for a long time (particularly on Ubuntu
Server).

The core functionality for Branch (4) is fully complete, but it was determined
that with some extra effort a very useful "configuration check" option
could be added which adds to the existing job visualisation work. This
"check-config" option is 90-95% complete (and would have been completed
had we not experienced such issues merging branches (1) and (2)).

Branch (4) should be completable within one week, hence the request to
delay the inclusion of this code.

JUSTIFICATION:

- Branch (4) does not impact the init daemon itself - the
  code changes are entirely to the utility program "initctl" so the impact
  of a late(r) delivery of this feature is low.

- Branch (4) contains very useful functionality for users (particularly
  sysadmins) so we really do want this in Natty.

= Timing =

== Stage 1 ==

At this stage, the view is that we should be able to land Branches
(1)-(3) by Monday 28 February / Tue 1st March.

== Stage 2 ==

This should be possible by 7 March, being the first Monday after Alpha 3
(March 3).

= Testing =

We will get various teams to help out with additional test resources and
the plan is to create a PPA for the new code as soon as possible to
allow users access to the new functionality.

= REFERENCES =

- https://wiki.ubuntu.com/FoundationsTeam/Specs/MaverickFinishUpstart

- https://blueprints.launchpad.net/ubuntu/+spec/packageselection-foundations-n-finish-upstart

[James Hunt (jamesodhunt)]

We now have a PPA for natty including branches (1)-(3) inclusive:

    https://launchpad.net/~jamesodhunt/+archive/upstart-testing

Do *NOT* use this PPA on systems you care about as it has not been fully tested yet. However, if you would like to help the test effort, feedback would be appreciated.

[James Hunt (jamesodhunt)]

Branches (1)-(3) inclusive have now been proposed for merging in the following:

  lp:~jamesodhunt/ubuntu/natty/upstart/proposed

[Kate Stewart (kate.stewart)]

Timing for A3 is tight at this point, since there is churn with A3 itself.

After discussions with jhunt and cjwatson, additional following facts emerged:
1) main risky part is branch 2
2) main user will be server team
3) documentation will be made available on usage of new features by Thursday
4) the features being added are all new ones, and shouldn't affect existing code
5) ad hoc testing has been done by others on team since comment on 02/25, no significant issues found
6) having it in A3 will better enable wider system and configuration testing, and basis for branch 4

Server team members are aware of risk, but lean to having it in, rather than not at this stage.
Significant part of the server team is away this week at sprint, so responses may be delayed.

Based on above facts, decision was made to accept this FFE in for A3.

[James Hunt (jamesodhunt)]

Branches (1)-(3) inclusive are now in the Natty archive as Upstart version 0.9.0-1ubuntu3.

I have now documented Scotts socket bridge branch with 2 new man pages (not yet released) and the requisite .conf file. These can go in the next update.

I have now completed tests for Branch (4). Plan now is to:

- Merge this into the ubuntu Upstart branch.
- Update the PPA (https://launchpad.net/~jamesodhunt/+archive/upstart-testing) as soon as possible
  to give users a chance to try the new features.
- Propose it for merging on Monday 7 March.

What remains:

1) documenting Branch (2)
    This is in progress.

2) Writing test code for branches (1) and (2)
    Also in progress.

At this stage, my feeling is that *comprehensive* and automated test code for Branches (1) and (2) probably won't be finished by 7 March since:

a) the Upstart test framework doesn't have infrastructure to cope with reconfiguring dbus (required to re-enable the session support).
b) there are rather too many scenarios to code in the time remaining.
c) the Upstart test framework is going to have to be strained to handle the socket bridge daemon.

That said, we should be able to produce a minimal set of tests (possibly manual) for Monday with a view to adding in the remaining tests as soon as possible after that date.

[James Hunt (jamesodhunt)]

Updated https://wiki.ubuntu.com/NattyNarwhal/TechnicalOverview with new features.

Branch (4) has now been merged into lp:~upstart-devel/upstart/0.9. Now working on adding doc for socket bridge, sessions and chroots (to be followed by lp:ubuntu/upstart merge + PPA).

Note that Chroot support seems to be unreliable. I've raise bug 728531 on this issue and updated the tech overview above.

[James Hunt (jamesodhunt)]

Updated lp:~upstart-devel/upstart/0.9 with doc for sessions and chroots (Branch (2)).

[Martin Pitt (pitti)]

Thanks James for your detailled description! I concur that it's fine to merge these into natty now, but disable user session support for now. Out of interest, how does this get enabled? ./configure option? config file option? Installing a separate package?

[James Hunt (jamesodhunt)]

Hi Martin - to re-enable user sessions, users would need to install the file dbus/Upstart.conf from the upstream version of Upstart as /etc/dbus-1/system.d/Upstart.conf.

[James Hunt (jamesodhunt)]

An updated branch for natty containing the visualisation support (Branch (4)) is here:

    lp:~jamesodhunt/ubuntu/natty/upstart/proposed-stage2

My PPA is currently building packages based on this branch:

    ppa:jamesodhunt/upstart-testing

Testers welcome!

[Dave Walker (davewalker)]

@James, Well i'm using your updated PPA... but not quite sure how to use the visualisation support, so haven't been able to test that. However, it hasn't eaten my machine and services seem to be handled correctly.

[James Hunt (jamesodhunt)]

@Dave - great! Thanks for trying this out. The vis support doc is below. This will be added to the Natty Tech Overview when it lands:

==== Basic Job/Event Visualisation ====

Upstart provides a new Python script `initctl2dot` which converts the
output of the new `initctl show-config` command to
[[http://www.graphviz.org/|GraphViz]] format. By default, all job
configuration files are considered and the links between jobs and events
are displayed graphically. Additionally, it is possible to list a set of
jobs to graph.

==== New Initctl Commands ====

Two new `initctl` commands have been added:

 1. `show-config`
 1. `check-config`

The `show-config` command displays core job configuration details,
namely the `start on`, `stop on` and `emits` stanza information. This is
useful since it allows users to see how Upstart has parsed the job
configuration files. Additionally, the `show-config` command accepts an
optional `--enumerate` option which makes it easy to see which elements
of complex conditions are jobs, which are events and which are
environment details. This option forms the basis of the Visualisation
feature above.

The `check-config` command is useful for System Adminstrators and
tooling to ensure that all jobs are theoretically startable/stoppable.
For example, if a job configuration file specified the following complex
condition:

{{{
  start on (A and (B or (starting C or (starting D or starting E))))
}}}

The `check-config` command would flag an error if for example none of
the jobs 'C', 'D' or 'E' were available since that would indicate
the job in question could never be automatically started (since the
start on condition could never be true). Similar checks are performed on
events, so if jobs 'C', 'D' and 'E' are available but events 'A' and 'B'
are not advertised as being emitted by any job, 'check-config' will
generate an error. If no errors are detected, `check-config` displays no
output and returns zero. If errors are detected for a job, each
condition that is unsatisfiable is displayed with a message.

[James Hunt (jamesodhunt)]

Minor updates to Branch 4 (man pages and minor script tweaks). Merge proposal raised for branch:

    lp:~jamesodhunt/ubuntu/natty/upstart/proposed-stage2

Latest code published to PPA below:

    ppa:jamesodhunt/upstart-testing

[James Hunt (jamesodhunt)]

@Dave: The just-updated PPA now contains a man page for the visualisation command "initctl2dot".

Also, new man pages...

  init-checkconf
  net-device-added (link to upstart-udev-bridge)
  net-device-removed (link to upstart-udev-bridge)
  graphics-device-added (link to upstart-udev-bridge)
  drm-device-added (link to upstart-udev-bridge)

Also, this man pages has been updated with some examples (we should add more):

  upstart-udev-bridge

[James Hunt (jamesodhunt)]

All review comments on merge proposal implemented, bar two:

* debian/manpages/upstart-events.7: This man page isn't available
  upstream since it documents the "Ubuntu way", so I think the URL is
  correct I think.

* init/main/init.5:
  > +Note that a user job configuration file cannot have the same name as a
  > +system job configuration file.
  Bug 732656 raised to investigate this issue.

[James Hunt (jamesodhunt)]

Re-submitted merge proposal due to a failing "make distcheck" in upstream branch.

[James Hunt (jamesodhunt)]

Stage 2 has now been merged.

[James Hunt (jamesodhunt)]

Stage 2 now pushed to users. I've updated the doc below to reflect this:

   https://wiki.ubuntu.com/NattyNarwhal/TechnicalOverview

[Colin Watson (cjwatson)]

Closing; there is still work to be done (notably tests, bug 728531, et al), but the core feature work has landed.