Need better error than "simplestreams.util.SignatureMissingException: No signature found"

[Jason Hobbs]

I'm modifying boot-source-selections and calling boot-resources import to start an import. However, no import starts, and my regiond.log has this error:

http://paste.ubuntu.com/p/m5MpGVnyg3/

This is with 2.3.1-6470-g036d646-0ubuntu1~16.04.1

[Andres Rodriguez (andreserl)]

Hi Jason,

Can you provide a step by step so we can attempt to reproduce this?

That said, what the issue could be:

1. a simplestreams issue
2. images.maas.io is being updated.

[Andres Rodriguez (andreserl)]

I /think/ this issue would indicate that the streams are not signed, which could mean that the streams were published before they were fully signed.

[Andres Rodriguez (andreserl)]

seems that the streams have just been updated: https://images.maas.io/ephemeral-v3/daily/streams/v1/

[Jason Hobbs (jason-hobbs)]

The streams were updated 3+ hours ago. It's 17:37 UTC now and 13:58 is the latest update time from there:

http://paste.ubuntu.com/p/KQyc8C24v9/

[Jason Hobbs (jason-hobbs)]

Steps I'm taking:

- update boot source selection for my one boot source (maas.io - the default) to include new architectures, such as ppc64el, armhf, and s390x. I'm using the CLI to do this.

- run 'maas <profile> boot-resource import'

The error occurs in the log after that.

[Jason Hobbs (jason-hobbs)]

We're running this through a proxy internal to Canonical. I believe the lack of Cache-Control settings for the simplestreams on maas.io are causing issues with that cache:

https://github.com/canonical-websites/maas.io/issues/236

[Jason Hobbs (jason-hobbs)]

That said, it's hard to tell, because the error from MAAS/simplestreams is not very useful. It would be great if it said what it's looking for that's missing...

[Jason Hobbs (jason-hobbs)]

Actually, this seems like a maas bug. I restarted regiond and it's working now.

[Andres Rodriguez (andreserl)]

I'm marking this incomplete as I'm unable to reproduce the issue.

That said, judging by the error it would seem the streams are not signed, or when simplestreams downloaded the streams they were not signed (e.g. the update happened while new images were being copied).

On the other hand, it could also be a bug in simplestreams as MAAS simply calls simplestreams, and the error is being raised by simplestreams itself. So it is simplestreams that raises the error. On restart, its doing a completely new import of simplestreams library, which is probably why you saw this get fixed.

If you are able to reproduce it, please do let me know. I would also like to know what the simplestreams folks think about this.

[Andres Rodriguez (andreserl)]

The bionic occurrence of this is [1]. Based on the logs attached in [1], this happened on "2018-04-10 01:29:39"

Looking at images.maas.io [2] and [3], I see images got updated on

20180410/ 11-Apr-2018 02:19 -
20180410/ 10-Apr-2018 21:07 -

respectively. So, given that this is something that has just recently come up, and apparently during update of images, I'm wondering this could actually be related to how the images are being synced (as we know we've had issues before).

[1] https://bugs.launchpad.net/maas/+bug/1764830
[2]: https://images.maas.io/ephemeral-v3/daily/xenial/amd64/
[3]: https://images.maas.io/ephemeral-v3/daily/bionic/amd64/

[Ryan Harper (raharper)]

The traceback indicates that the json document that was loaded did not include a PGP header.

No changes to simplestreams/util.py since 2015; so I expect there was some issue w.r.t the content loaded.

Simplestreams itself does not cache URL data; it fetches remotely first, and then mirrors content that's present in the streams data from URL and writes it locally.

If there is a proxy between simplestreams and the endpoint, I would suspect some issue there.

There are only 4 sjson files that it could be looking at:

http://images.maas.io/ephemeral-v3/daily/streams/v1/index.sjson
http://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:v3:download.sjson
http://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:centos-bases-download.sjson
http://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:1:bootloader-download.sjson

I suggest firing up curl/wget on the endpoint where it's failing to see if you get the same contents as you do from some other place outside of the failure.

[Jason Hobbs (jason-hobbs)]

A couple of things that make me think it's not a proxy issue:

1) We hit this behavior with the proxy enabled and disabled in MAAS
2) After occurring for over an hour, it immediately stopped after restarting maas-regiond

[Andres Rodriguez (andreserl)]

FWIW, I found a really old IRC log (1.4 times) talking about this similar
issue. Not saying it’s related or the same, but just bringing it up in case
this could lead somewhere.

[08:33] <gmb> rvba: That could work. We can't specify https_proxy *at
all* for simplestreams, or else it breaks.
[08:34] <gmb> If you give it an http url by accident (I just did) you get
[08:34] <gmb> simplestreams.util.SignatureMissingException: No signature found!

On Tue, Apr 17, 2018 at 3:31 PM Ryan Harper <email address hidden>
wrote:

> The traceback indicates that the json document that was loaded did not
> include a PGP header.
>
> No changes to simplestreams/util.py since 2015; so I expect there was
> some issue w.r.t the content loaded.
>
> Simplestreams itself does not cache URL data; it fetches remotely first,
> and then mirrors content that's present in the streams data from URL and
> writes it locally.
>
> If there is a proxy between simplestreams and the endpoint, I would
> suspect some issue there.
>
> There are only 4 sjson files that it could be looking at:
>
>
> http://images.maas.io/ephemeral-v3/daily/streams/v1/index.sjson
>
> http://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:v3:download.sjson
>
> http://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:centos-bases-download.sjson
>
> http://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:1:bootloader-download.sjson
>
> I suggest firing up curl/wget on the endpoint where it's failing to see
> if you get the same contents as you do from some other place outside of
> the failure.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1761813
>
> Title:
> can't import boot resources:
> "simplestreams.util.SignatureMissingException: No signature found"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1761813/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; milestone=2.4.0beta3; status=Incomplete;
> importance=Undecided; assignee=None;
> Launchpad-Bug: product=simplestreams; status=New; importance=Undecided;
> assignee=None;
> Launchpad-Bug-Tags: cdo-qa cdo-qa-blocker foundations-engine
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl jason-hobbs jog raharper
> Launchpad-Bug-Reporter: Jason Hobbs (jason-hobbs)
> Launchpad-Bug-Modifier: Ryan Harper (raharper)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

[Andres Rodriguez (andreserl)]

@Jason,

Instead of restarting regiond have you forced the import of images either
via the API or via the UI?

MAAS uses simple streams to download all streams and simple streams scans
for signatures. The error you see is a simplestreams error.

The way how the process work is that MAAS only parses data after simple
streams downloads all descriptions and verifies signatures.

So this is either simplestreams failing due to a proxy issue (as you still
use the internal proxy, there is no direct internet access) or the
signatures are not yet updated by the time you access the repository which
causes the issues.

On Tue, Apr 17, 2018 at 3:31 PM Jason Hobbs <email address hidden>
wrote:

> A couple of things that make me think it's not a proxy issue:
>
> 1) We hit this behavior with the proxy enabled and disabled in MAAS
> 2) After occurring for over an hour, it immediately stopped after
> restarting maas-regiond
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1761813
>
> Title:
> can't import boot resources:
> "simplestreams.util.SignatureMissingException: No signature found"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1761813/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; milestone=2.4.0beta3; status=Incomplete;
> importance=Undecided; assignee=None;
> Launchpad-Bug: product=simplestreams; status=New; importance=Undecided;
> assignee=None;
> Launchpad-Bug-Tags: cdo-qa cdo-qa-blocker foundations-engine
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl jason-hobbs jog raharper
> Launchpad-Bug-Reporter: Jason Hobbs (jason-hobbs)
> Launchpad-Bug-Modifier: Jason Hobbs (jason-hobbs)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

[Jason Hobbs (jason-hobbs)]

On Tue, Apr 17, 2018 at 2:50 PM, Andres Rodriguez
<email address hidden> wrote:
> @Jason,
>
> Instead of restarting regiond have you forced the import of images either
> via the API or via the UI?

Yes, as indicated in comment 6:

https://bugs.launchpad.net/maas/+bug/1761813/comments/6

> MAAS uses simple streams to download all streams and simple streams scans
> for signatures. The error you see is a simplestreams error.
>
> The way how the process work is that MAAS only parses data after simple
> streams downloads all descriptions and verifies signatures.
>
> So this is either simplestreams failing due to a proxy issue (as you still
> use the internal proxy, there is no direct internet access) or the

No, like I said, I tested with the proxy disabled and it still
occurred. We don't have to use the proxy to get to maas.io.

> signatures are not yet updated by the time you access the repository which
> causes the issues.

Nope, 3 hours later - the signatures were definitely updated and it
was working fine on other maas servers.

Jason

[Lee Trager (ltrager)]

@Jason,

Can you confirm there is no proxy at all(including transparent) between the region and and maas.io?

Can you run the following on the system running the region?
$ curl -s https://images.maas.io/ephemeral-v3/daily/streams/v1/index.sjson | sha256sum
af785dea5e7bf34b7bc17ba5902bd76e59da12ca3ca1e8753fd2e5cf1af70cf6 -
$ curl -s https://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:v3:download.sjson | sha256sum
c1f7f0314c158751c491a63401fe75864c9816773c80affc21f1a23baf1fada5 -
$ curl -s https://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:1:bootloader-download.sjson | sha256sum
8ac2b1f5ea563f2b170206451203d844debfab9d393b4bed29f2af6bfd0d01b4 -
$ curl -s https://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:centos-bases-download.json | sha256sum
7c41eaedf7cb91ce776184554d378890563cc8a970b479c524c87eef5d1a8319 -

Can you post the output of maas <profile> boot-sources read

[Jason Hobbs (jason-hobbs)]

I'm sorry, this bug is not reproduced right now so I can't do those things.

On Tue, Apr 17, 2018 at 3:39 PM, Lee Trager <email address hidden> wrote:
> @Jason,
>
> Can you confirm there is no proxy at all(including transparent) between
> the region and and maas.io?
>
> Can you run the following on the system running the region?
> $ curl -s https://images.maas.io/ephemeral-v3/daily/streams/v1/index.sjson | sha256sum
> af785dea5e7bf34b7bc17ba5902bd76e59da12ca3ca1e8753fd2e5cf1af70cf6 -
> $ curl -s https://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:v3:download.sjson | sha256sum
> c1f7f0314c158751c491a63401fe75864c9816773c80affc21f1a23baf1fada5 -
> $ curl -s https://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:1:bootloader-download.sjson | sha256sum
> 8ac2b1f5ea563f2b170206451203d844debfab9d393b4bed29f2af6bfd0d01b4 -
> $ curl -s https://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:centos-bases-download.json | sha256sum
> 7c41eaedf7cb91ce776184554d378890563cc8a970b479c524c87eef5d1a8319 -
>
> Can you post the output of maas <profile> boot-sources read
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1761813
>
> Title:
> can't import boot resources:
> "simplestreams.util.SignatureMissingException: No signature found"
>
> Status in MAAS:
> Incomplete
> Status in simplestreams:
> New
>
> Bug description:
> I'm modifying boot-source-selections and calling boot-resources import
> to start an import. However, no import starts, and my regiond.log has
> this error:
>
> http://paste.ubuntu.com/p/m5MpGVnyg3/
>
> This is with 2.3.1-6470-g036d646-0ubuntu1~16.04.1
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1761813/+subscriptions

[David Britton (dpb)]

Thanks Jason -- please set back to new when you can. we don't see anything systemic in the simplestreams library right now, so we will need a way to reproduce.

[Andres Rodriguez (andreserl)]

Marking this as invalid for MAAS because:

1. MAAS asks simplestreams to import image descriptions (using an external proxy)
2. simplestreams is denied access to the image descriptions by the external proxy. https://pastebin.canonical.com/p/6rk2MvTzYc/
3. Since 3, MAAS cannot download images.

That said, simplestreams should do a better job at telling MAAS that it cannot access streams. That would require simplestreams to better do info logging. I'm marking this New in simplestreams to address that instead.

https://pastebin.canonical.com/p/6rk2MvTzYc/

[Jason Hobbs (jason-hobbs)]

While simplestreams could do a better job telling MAAS what's wrong, MAAS could also do a better job telling the user what might be wrong. There are various possibilities from simplestreams - can't connect, missing signature, etc.

Currently, this error is completely unreported in the MAAS UI and API. MAAS doesn't have to know which of those it is to tell the user there was a problem with simplestreams and to suggest some of those causes, which can help the user troubleshoot.

Once simplestreams implements more specific errors, MAAS can also report more specific causes.

[Andres Rodriguez (andreserl)]

Sure we can treat that as maas adding some more error surface. That said, MAAS already logs as much as it cans and surfaces errors from MAAS itself into the UI.

This issue or the lack of logging from this issue comes from simplestreams. So simplestreams needs to surface errors.

Also, I believe since this issue is actually rather an issue with the environment, this is no longer a cdo-qa-blocker, as such, removing such tag.

[Jason Hobbs (jason-hobbs)]

It's not an issue with the environment that maas never says anything like:

"You're using a proxy; validate that you can access images.maas.io through the proxy"

Which it totally could, without any changes to simplestreams.

You can prioritize it however you'd like, but given it took 2 weeks and several engineers to figure it out internally, I wonder how many people externally just give up when they hit stuff like this.

[Andres Rodriguez (andreserl)]

Service checking as you describe is a known wishlist enhacement, but I'll
be happy to receive contributions if you have the cycles.

That said, just to be clear, it took a couple hours to figure out this
issue *based* on the assumption that your environment was correct. If we
would have known more about your environment beforehand (before we actually
had access to it), it would have been much easier to identify the issue.

On Wed, Apr 18, 2018 at 4:14 PM, Jason Hobbs <email address hidden>
wrote:

> It's not an issue with the environment that maas never says anything
> like:
>
> "You're using a proxy; validate that you can access images.maas.io
> through the proxy"
>
> Which it totally could, without any changes to simplestreams.
>
> You can prioritize it however you'd like, but given it took 2 weeks and
> several engineers to figure it out internally, I wonder how many people
> externally just give up when they hit stuff like this.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1761813
>
> Title:
> Simplestreams doesn't surface errors or messages about being unable to
> access a mirror but errors with:
> "simplestreams.util.SignatureMissingException: No signature found"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1761813/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; milestone=2.4.x; status=Triaged;
> importance=Undecided; assignee=None;
> Launchpad-Bug: product=simplestreams; status=New; importance=Undecided;
> assignee=None;
> Launchpad-Bug-Tags: cdo-qa error-surface foundations-engine
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl davidpbritton jason-hobbs jog ltrager
> raharper
> Launchpad-Bug-Reporter: Jason Hobbs (jason-hobbs)
> Launchpad-Bug-Modifier: Jason Hobbs (jason-hobbs)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>

--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer