FFe: Support for Third Party Driver Installation

[Andres Rodriguez]

[Problem]
Various systems fail to successfully install Ubuntu due to the lack of the ability to install Thrid Party Drivers via MAAS. This is because the installer cannot automatically install these Third Part Drivers on PXE booting scenarios. ISO installs, using a USB drive to install these drivers is the current approach that works.

[Solution]
The proposed solutions allows the installation of these third party drivers from any location, provided they supply this in a configuration file and the installation of these drivers is allowed by the administrator via MAAS.

This solution requires the administrator to confirm whether they would like to install proprietary drivers, if available, on the MAAS settings page, by turning on this option. MAAS will then obtain the driver name and location of this third party driver from a config file and proceed with its installation when necessary.

[Dave Walker (davewalker)]

I really struggle to support this change. It is a clearly impactful change. FFe raised on the day of Final Freeze. Feels rushed. There is no commentary on the regression potential, or the testing done.

Not to mention that RC1 was happily released with a broken MAAS installer, that took 2 weeks to fix. This lowers general confidence in the attention to detail, and this upload could be similar.

This feels like a hardware enablement change that is better suited as a post-release SRU once it is well tested and the general churn has calmed. I do not see a valid reason why this needs to be on the release ISO.. it isn't an install time requirement?

I'm stopping short of outright NACK'ing this, but am rejecting the upload from the queue to avoid it being accepted by mistake.

[Andres Rodriguez (andreserl)]

Hi Dave,

It is a certification requirement hence needs to be on the ISO. We had
discussed this with Adam and he had been reviewing the code and gave us
pointer to address. He unofficially approved the FFe before final freeze.
On Apr 11, 2014 4:50 AM, "Dave Walker" <email address hidden> wrote:

> I really struggle to support this change. It is a clearly impactful
> change. FFe raised on the day of Final Freeze. Feels rushed. There is
> no commentary on the regression potential, or the testing done.
>
> Not to mention that RC1 was happily released with a broken MAAS
> installer, that took 2 weeks to fix. This lowers general confidence in
> the attention to detail, and this upload could be similar.
>
> This feels like a hardware enablement change that is better suited as a
> post-release SRU once it is well tested and the general churn has
> calmed. I do not see a valid reason why this needs to be on the release
> ISO.. it isn't an install time requirement?
>
> I'm stopping short of outright NACK'ing this, but am rejecting the
> upload from the queue to avoid it being accepted by mistake.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1305839
>
> Title:
> FFe: Support for Third Party Driver Installation
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1305839/+subscriptions
>

[Andres Rodriguez (andreserl)]

There's no regression potential per se because this doesn't affect the
current Maas operation as this will only be in effect if the user enables
the capability. Other than that this has been QA'd and testes on the field.
On Apr 11, 2014 4:50 AM, "Dave Walker" <email address hidden> wrote:

> I really struggle to support this change. It is a clearly impactful
> change. FFe raised on the day of Final Freeze. Feels rushed. There is
> no commentary on the regression potential, or the testing done.
>
> Not to mention that RC1 was happily released with a broken MAAS
> installer, that took 2 weeks to fix. This lowers general confidence in
> the attention to detail, and this upload could be similar.
>
> This feels like a hardware enablement change that is better suited as a
> post-release SRU once it is well tested and the general churn has
> calmed. I do not see a valid reason why this needs to be on the release
> ISO.. it isn't an install time requirement?
>
> I'm stopping short of outright NACK'ing this, but am rejecting the
> upload from the queue to avoid it being accepted by mistake.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1305839
>
> Title:
> FFe: Support for Third Party Driver Installation
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1305839/+subscriptions
>

[Adam Conrad (adconrad)]

I wouldn't say I "unofficially approved the FFe", that's twisting my words and the content of our conversations. I gave you the reasons why I would flat out reject the upload (FFe or not), and even petition to have it removed from the archive if the "install non-free software by default" feature slipped in.

You addressed those concerns, which is great, but none of that relates to the bit where you're trying to land new features a week before release, and a month and a half after feature freeze.

[Andres Rodriguez (andreserl)]

Hi Adam,

Sorry for the misunderstanding but to me the answer to my question ("Yeah,
I think that would satisfy me.") as to whether this was enough to get the
FFe approved sounds like an unofficial ACK.

So the question is now... Will this be ACK'd or NACK'd?
On Apr 11, 2014 8:16 AM, "Adam Conrad" <[e-mail]> wrote:

> I wouldn't say I "unofficially approved the FFe", that's twisting my
> words and the content of our conversations. I gave you the reasons why
> I would flat out reject the upload (FFe or not), and even petition to
> have it removed from the archive if the "install non-free software by
> default" feature slipped in.
>
> You addressed those concerns, which is great, but none of that relates
> to the bit where you're trying to land new features a week before
> release, and a month and a half after feature freeze.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1305839
>
> Title:
> FFe: Support for Third Party Driver Installation
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1305839/+subscriptions
>

[Adam Conrad (adconrad)]

Can you test the crap out of the new codepaths here, especially in any way that they interact with existing functionality (ie: the web UI, etc)? If you guys are satisfied that this won't make anything any more broken than it already is, I might be inclined to be a Very Nice Person, and let you slip it in. I'm still on the fence there, but any reassurances you can give would go a long way. Is there a test methodology you use to validate maas, can we see results, etc?

[Julian Edwards (julian-edwards)]

On Friday 11 Apr 2014 12:26:59 Adam Conrad wrote:
> Can you test the crap out of the new codepaths here, especially in any
> way that they interact with existing functionality (ie: the web UI,
> etc)? If you guys are satisfied that this won't make anything any more
> broken than it already is, I might be inclined to be a Very Nice Person,
> and let you slip it in. I'm still on the fence there, but any
> reassurances you can give would go a long way. Is there a test
> methodology you use to validate maas, can we see results, etc?

MAAS has a CI suite in the lab which runs a few times day, but it does not
have the hardware required to test this particular change. It does, however,
gives us regression mitigation confidence.

Andres, I think it would help folk if there were some evidence of running this
code on the hardware in question that requires the driver.

[Mark Shuttleworth (sabdfl)]

Thanks all for the comments and discussion.

Responding to some key points:

 * building confidence in code changes both for this FFE and subsequent SRUs is important, the archive and RM teams have a mandate to seek comfort on that front before ack'ing an upload under either circumstances

 * in other words - thank you for calling out the late change :)

 * I have asked and the MAAS team have assured that all SRU changes will come with appropriate test suites

 * I would expect to see a test suite for this change and would expect that test suite to have been run on the appropriate hardware. Separately, we'll arrange to have MAAS CI and testing integrated to the OIL lab, which has a rapidly growing range of hardware.

 * installing the necessary drivers is the norm in the linux environment; warning the user about the consequences of this is the norm in the Ubuntu environment.

 * I'd ask that we accept this FFE once tests are in place, or if needed, with a commitment to tests as a zero-day SRU

 * I'd ask that the MAAS team commit to flagging the installation of such drivers in the MAAS node UI, possibly also with a flag on the node listing

 * all of this assumes the existence of a setting which enables a system administrator to tell MAAS not to do this by default on any given machine without explicit approval

As general guidance, this is an area that will evolve fast in the coming months and years, so I've asked the MAAS team to keep developing on 14.04 LTS as their primary platform, introducing no dependencies that are not in 14.04, so we can SRU. If we need to move to universe to make that more comfortable for the release team, let's do that. We got stuck in 12.04 with old versions of key tools that should have been moved - stuck because of bad thinking on the deps front, and perceived dogma on the RM front. We can do better this time by requiring rigorous CI and testing on SRUs, and being sensible about deps. That goes for maas, juju and related tools, all of which need to be current and useful in the coming two years.

[Mark Shuttleworth (sabdfl)]

Also, thank you Adam for pointing out that we need to do the same sort of ubiquity- and jockey-like calling out of the issues associated with binary blobs on servers that we do on the PC. I'll get the MAAS folks to make that very clear on the node page as a way of socialising the benefits of open drivers.

[Adam Conrad (adconrad)]

Right, so the key here (on the software freedom front), as it was with ubiquity when we discussed it, is that the option needs to be off by default. If a sysadmin turns it on (and I don't doubt that many will), that's entirely fine, but they need to be the ones explicitly making the call.

If we made the call for them, and then noted "oh, by the way, there's some non-free junk installed", that's too late in the process for good-faith here.

And, even if this wasn't about the software freedom aspect of "something in main installing non-free stuff by default", I'd be wary of installing anything from an external repository by default without a toggle (defaulting to off) anyway. Thankfully, most paranoid sysadmins would firewall this to oblivion and notice immediately that it's not working when it tries to call out, but we can do better than requiring people to discover weird third-party server hits via draconian firewalls.

[Adam Conrad (adconrad)]

So, it's been noted that this is slightly different from the desktop/ubiquity case only in that the ubiquity case presents the user up-front with the "do you want non-free stuff?" toggle on an installer page, while maas is putting it in a settings page.

So, for starters, I think the settings page is the *right* place for it to live. It's a setting. That's where I should go if I want to turn it off and on.

That said, it's a valid point that this is a setting most people won't go looking for, and it'll just be weirdly confusing when they go to install on certain hardware and it just doesn't work. Since we can't, apparently, talk certain hardware vendors (*cough*) into shipping free drivers, I proposed the following:

1) Keep the master toggle in the settings page. That's where it belongs. That's where I'd go to turn it off and on.
2) When installing the maas master node (ie: the maas package itself), present my with a debconf question that reads similarly to the ubiquity prompt, "blah blah non-free stuff, blah blah, you might need it for certain hardware support" (list specific HP servers and other things, if that's helpful to make people know they want/need the feature), default that to off, but set the priority high enough that they'll see the question.
3) Make the response to that debconf question twiddle the same config file that the settings page in the web UI does.
4) Profit.

This would, I think, satisfy the visibility issue, while not turning this into a "do we need to take this to the TB to vote it down and then try to come up with ways to compromise our ideals against usibility" argument.

[Adam Conrad (adconrad)]

(To be clear to people following along, I'm fine with Mark's assessment and explicit ACK of the FFe, and we'll happily accept the feature being uploaded under his conditions of being heavily tested, etc, the above quibbling is only about if the feature is on or off in a default setup)

[Mark Shuttleworth (sabdfl)]

The key difference in my mind is recoverability. In the server case, the install is by nature largely automated, and often will fail altogether if you don't for example have the ability to configure your hard drives.

Perhaps an analogy for the desktop would be to ask the question - what if a popular, common device that was required for system installation in a large percentage of cases required a non-free driver to be on the disk and used by default. We would certainly opt to make that device installable; consider our approach on phones, where we accept binary blobs and invest a lot to deliver a free userspace.

I don't think we're crossing a significant new line here. There are many pieces of no-charge proprietary software we COULD be installing to "make the product better", but we don't, given our values. In this case, I see no moral high ground in crippling the install unless someone answered an ideological question. We often forget how hard it is for new users to come up to speed - we know all the permutations and combinations and tradeoffs, most users do not.

Consider also how comfortable we are enabling people to run Ubuntu on proprietary clouds - where they will never see the code beneath the code, as it were.

Having said all this, I didn't clarify earlier that this mechanism should only be used for hardware which has no free driver. We should always silently select the free driver if such a driver exists, and if people want to go hunting for a proprietary one, ensure they can find them in a standard, supportable, upgradeable fashion, but not use it by default.

[Dave Walker (davewalker)]

Hi,

I think everyone is largely aligned with the freedom aspects of this. This is certainly not a concern for myself at least.

What is concerning to me, is that this feature feels rushed and somewhat unfinished. I just had a quick look at the code that landed for this feature:

There is no support for third party drivers in enlistment, commissioning or fastpath/curtin. I am guessing the first two can be ignored, depending on the nature of the driver; but the fastpath seems a big loss.

The drivers.yaml file is a monolithic file that ships with the code-base. I might have expected this to be a yaml per-driver file, allowing easier stacking, probably managed via an external mechanism (simplestream?). I am guessing drivers.yaml would appear to be a config file that I can maintain as a sysadmin. Am I allowed to add extra modalias as new hardware comes out, myself? What about third party drivers I wish to add? What experience should I expect on upgrade? This, if added now, feels like something that is needlessly hard to upgrade from.

It seems the binary drivers udeb for the reference implementation is pulled from both plain http (via wget, without validating against a signed Release file), AND when the PPA is added the signing key is also retrieved over plain http. This would imply a trust failure on both parts of this.

It matches the kernel of the current install. Assuming that the pxe installer is still using the netboot from the current installer, is there any assurance that without warning - installs will suddenly stop working, until the third-party repository has updated it?

The modalias matching seems to be quite nice. It looks like I can only support one third party driver per node. Is there likely to be a scenario where multiple drivers may be required?

Only amd64 is supported as part of this feature. Is it not anticipated that other arches will require third-party drivers?

I don't think the archive pocket matters weather this is Universe or Main. By current process, the only thing moving to Universe would achieve is removal from the install media (which could equally be done if in main). Meaning that squeezing in prior to release is even less relevant.

HWE Kernel FFe bug 1281765, is left unfinished. Is this something also planned to land?

To be clear, I am not against this feature. I was proposing that if landed as a HWE feature post release, where this can be fully implemented without encumbering the upgrade process to be needlessly complicated.

[Adam Conrad (adconrad)]

I'm not sure our approach on phones should be held up as a benchmark here, it was a pragmatic solution to rapid iteration on devices that weren't built for Ubuntu. It's also code that we "control" (from the point of view of us shipping it in the archive, us being able to audit and fix it, etc).

Even if we take away the non-free part of the argument, is it really sane to have an installer silently grabbing things from third-party non-Ubuntu repositories without the sysadmin first confirming that this is what they want? I'm not arguing this be done on a per-node basis, that's madness. As you say, server installations are automated, and should remain that way.

But, automating installation of nodes doesn't mean that installing and configuring the master needs to also be so silent as to not point out to the user that we intend to fetch drivers from third parties, should their hardware not work out of the box with Ubuntu.

As I pointed out, this feature will fail hard in a well-firewalled network anyway (how many server farms that poke a hole for archive.ubuntu.com, or have a local mirror, will have a hole poked for ppa.launchpad.net? My bet is pretty close to zero). So, by warning them up front of our intent, we can also point out that they might need to read the config and poke some firewall holes too (or mirror the content locally and change the config to point to it).

I understand the urge to automate this all, but I not only feel it's dangerously close to dishonest, I also suspect it won't at all work in practice, except for test farms on Canonical employees' networks that already have unfettered access to the hosts in question.

[Jason Hobbs (jason-hobbs)]

Some responses to Dave's comments:

* Yes, it was done quickly and isn't perfect. It's a minimum useful set of functionality to address a real use case and that improvements can be added iteratively with the benefit of getting feedback from users in the meanwhile.

* fastpath can be added without a lot of extra work; we're not doing anything that prevents it from being added, I'd just rather get one install path working reasonably well before trying to get two working.

* The yaml file can be edited by an admin to add additional drivers. Yes, some sort of simplestreams like approach will probably be used to replace or augment the yaml file in the future, and yes, there will be some complication in the upgrade to whatever the new source is as we'll have to maintain user additions/edits. I don't see that as insurmountable though.

* I've posted a branch now that inserts the key directly into the yaml instead of retrieving it via http. Still working on a way to securely retrieve the udeb. The repo has a sha1 on the udeb; just need to work out how to validate the repo's sha1 now.

* There's always a possibility for multiple drivers to be required. We've done nothing that precludes adding that capability in the future, but it isn't a supported use case for today (and I haven't seen any real use cases that require it yet).

* Other arches could require it - adding support for other arches shouldn't be very difficult, but wasn't required to be useful here and is something that can added iteratively.

* HWE Kernel FFe bug 1281765: I don't know what the plans are here.

[Robbie Williamson (robbiew)]

I'm in favor of the approach suggested in comment #12, whereby we prompt the user during the first install of MAAS, as to whether they want to allow the install of non-free drivers for such cases where there are no free ones available, e.g. <HP example here>. Once they answer, we point out that the setting can be reversed in the "Settings" menu if the MAAS GUI. Could we move forward with this approach?

[Adam Conrad (adconrad)]

> I've posted a branch now that inserts the key directly into the yaml instead of retrieving it via http.
> Still working on a way to securely retrieve the udeb. The repo has a sha1 on the udeb; just need to
> work out how to validate the repo's sha1 now.

The whole point of having the key is to give you a trusty path to the (u)debs, surely?

[Robbie Williamson (robbiew)]

So I was informed that prompting for non-free during installation would break unmanned installs of MAAS, which causes problems for our cloud install plans. I'm all for supporting our commitment to non-free, but at the end of the day, we're also committed to ensuring our users have a great experience with Ubuntu. If users are unable to install because their drives aren't recognized, then the entire non-free argument becomes irrelevant. The argument of telling our partners to opensource their drivers, is valid, but will take awhile, e.g. Nvidia on the desktop...and could possibly never occur. I'll leave the decision up to the engineers and release team, but I hope we can come up with something that allows our users the freedom to use Ubuntu on the hardware of their choice.

[Andres Rodriguez (andreserl)]

Would it be a better approach to simply display a notice in the MAAS Web UI
and maybe when we do the initial MAAS setup, notifying the user that this
setting is enabled by default?

On Mon, Apr 14, 2014 at 9:24 AM, Robbie Williamson <
<email address hidden>> wrote:

> I'm in favor of the approach suggested in comment #12, whereby we prompt
> the user during the first install of MAAS, as to whether they want to
> allow the install of non-free drivers for such cases where there are no
> free ones available, e.g. <HP example here>. Once they answer, we point
> out that the setting can be reversed in the "Settings" menu if the MAAS
> GUI. Could we move forward with this approach?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1305839
>
> Title:
> FFe: Support for Third Party Driver Installation
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1305839/+subscriptions
>

--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

[Jason Hobbs (jason-hobbs)]

On 04/14/2014 08:45 AM, Adam Conrad wrote:
> The whole point of having the key is to give you a trusty path to the
> (u)debs, surely?

Yes - we need it for the udebs, and for setting up the repository in the
installed system. I'm still trying to work out the details on how to
verify the udebs given the public key. The installer initrd includes
gpgv to verify files/sigs against a keyring; I have a public key and
need to turn that into a keyring. The public key is ascii armored and
so it's easy to include it in a shell script. gpgv seems to expect a non
ascii armored keyring, which can't be easily included in a shell script.
There are no text->binary decoders included in the installer image,
afaict (cjwatson agrees).

Jason

[Jason Hobbs (jason-hobbs)]

Looks like ash can support hexadecimal escaped strings - so that's a way forward for me.

[Jason Hobbs (jason-hobbs)]

For context, here's the branch I'm working on. It handles the insecure key retrieval problem. I plan on handling the udeb problem by inserting the keyring into the preseed, retrieving Release/Release.gpg for the repo, using the keyring to verify the Release file, then using the sha sums to verify the Packages and udeb files.

https://code.launchpad.net/~jason-hobbs/maas/use-key-text/+merge/215517

[Mark Shuttleworth (sabdfl)]

To Robbie's point - yes, it makes no sense to make the install break
mysteriously when we can get it to work, any more so on a server or a
phone. It does make sense to flag in the UI when we have used drivers
outside of the normal Ubuntu kernel set. We're not installing
proprietary applications, I don't see a significant moral issue.

Mark

[Adam Conrad (adconrad)]

> So I was informed that prompting for non-free during installation would
> break unmanned installs of MAAS, which causes problems for our cloud
> install plans.

I'm trying to sort out what this means. Do we actually care that the maas *controller* be installable with zero intervention? Why is this an important feature?

Again, this has nothing to do with installing nodes. Nodes should install with zero intervention and Just Work. Why do we expect nobody to ever need to interact with or configure the controller, however?

[Daniel Westervelt (danwest)]

It has to do with automating the installation of MAAS environments, which includes automating the installation of MAAS itself. In particular this is important to our Openstack cloud-installer and for bootstrapping 'other' environments.

[Graham Binns (gmb)]

If you land code for this feature without tests then *please* file a bug (tagged tech-debt) and let us on the MAAS team know so that we can either write the tests or help you write the tests. If you don't file a bug, chances are it's going to get forgotten about.

I've already filed bug 1307906 for adding tests for the web UI warning; it's not a big change and I'll take care of adding the tests today.

[Launchpad Janitor (janitor)]

This bug was fixed in the package maas - 1.5+bzr2252-0ubuntu1

---------------
maas (1.5+bzr2252-0ubuntu1) trusty; urgency=medium

  * New upstream release
    - Add support to install Third Party Drivers. In order for this to be
      used the user will have to go to the Settings page to enable the
      installation of these drivers. (LP: #1305839)
    - Use release images instead of daily. (LP: #1306701)
    - Quote interface name in dhcpd.template, otherwise DHCP server fails
      to start. (LP: #1306335)
    - Fix IntegrityError, when multiple processes are trying to register
      the same component. (LP: #1307415)
    - Add missing armhf commissioning template (LP: #1307780)
  * debian/maas-region-controller-min.install: Install drivers.yaml.
  * debian/maas-region-controller.postinst: No longer show the
    installation note by default. (LP: #1284652)
 -- Andres Rodriguez <email address hidden> Wed, 09 Apr 2014 19:02:00 -0400