configure_sources fails badly with misformatted configuration.

[Nicolas Thomas]

 must use local copy of the ppa (opnecontrail or contrail) and won't
have access to the keyserver etc..

I nailed the issue as follow :

in add_source(source, key=None): in juju helpers python:
 I try to pass a block with the gpg key starting with -----BEGIN PGP
PUBLIC KEY BLOCK----- and it fails on safe_loader ..

but if I do manually :
config-get install-keys | apt-key add -
                         OK
it works ..

I might have badly formed it but could not figure out .. seems related to safe_loader

Backtrace :
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Traceback (most recent call last):
  File "./hooks/install", line 406, in <module>
    main()
  File "./hooks/install", line 336, in main
    hooks.execute(sys.argv)
  File "/var/lib/juju/agents/unit-contrail-configuration-0/charm/hooks/charmhelpers/core/hookenv.py",
line 672, in execute
    self._hooks[hook_name]()
  File "./hooks/install", line 314, in install
    configure_sources(True, "install-sources", "install-keys")
  File "/var/lib/juju/agents/unit-contrail-configuration-0/charm/hooks/charmhelpers/fetch/__init__.py",
line 339, in configure_sources
    keys = safe_load((config(keys_var) or '').strip()) or None
  File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 93,
in safe_load
    return load(stream, SafeLoader)
  File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 71, in load
    return loader.get_single_data()
  File "/usr/lib/python2.7/dist-packages/yaml/constructor.py", line
37, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 36,
in get_single_node
    document = self.compose_document()
  File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 58,
in compose_document
    self.get_event()
  File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 118, in get_event
    self.current_event = self.state()
  File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 193, in
parse_document_end
    token = self.peek_token()
  File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 128,
in peek_token
    self.fetch_more_tokens()
  File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 220,
in fetch_more_tokens
    return self.fetch_value()
  File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 576,
in fetch_value
    self.get_mark())
yaml.scanner.ScannerError: mapping values are not allowed here
  in "<unicode string>", line 2, column 9:
    Version\: GnuPG v1
            ^
Looks like safe_loader want to interpret it

[Stuart Bishop (stub)]

I think you missed a layer of quoting. The install_keys config item needs to be a YAML list of strings encoded as a string, and I don't think your list of strings was valid YAML.

The following syntax should work and the most readable I have found. The indentation and two levels of '|' quoting is vital.

install_keys: |
    - |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        Version: GnuPG v1

        mQGiBD97vyYRBADC5JRFuEj+oVWY/60ljKpdt5Byv7I0XAnvR2s588Tmwi8T/Rv5
        YIiorQSP/ZU6zTrzNv4ZH+7Z1rivHLZAVOSCwwwFprkpmFut6uCmo7P2iR09hb94
        -----END PGP PUBLIC KEY BLOCK-----
    - |
        -----BEGIN PGP PUBLIC KEY BLOCK-----
        Version: GnuPG v1

        mQGiBD97vyYRBADC5JRFuEj+oVWY/60ljKpdt5Byv7I0XAnvR2s588Tmwi8T/Rv5
        YIiorQSP/ZU6zTrzNv4ZH+7Z1rivHLZAVOSCwwwFprkpmFut6uCmo7P2iR09hb94
        -----END PGP PUBLIC KEY BLOCK-----

[Nicolas Thomas (thomnico)]

I agree and can confirm this Stuart.

Meanwhile as it mandate a list of keys even if only passing one .. I
would suggest to turn this into a documentation issue .. (which would
be find)

On Fri, Nov 13, 2015 at 8:26 AM, Stuart Bishop
<email address hidden> wrote:
> ** Changed in: charm-helpers
> Status: New => Invalid
>
> ** Changed in: charm-helpers
> Status: Invalid => New
>
> ** Changed in: charm-helpers
> Status: New => Incomplete
>
> ** Changed in: charm-helpers
> Status: Incomplete => Opinion
>
> ** Changed in: charm-helpers
> Status: Opinion => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1515699
>
> Title:
> add_source fails if passing a full block key
>
> Status in Charm Helpers:
> Invalid
>
> Bug description:
> must use local copy of the ppa (opnecontrail or contrail) and won't
> have access to the keyserver etc..
>
> I nailed the issue as follow :
>
> in add_source(source, key=None): in juju helpers python:
> I try to pass a block with the gpg key starting with -----BEGIN PGP
> PUBLIC KEY BLOCK----- and it fails on safe_loader ..
>
> but if I do manually :
> config-get install-keys | apt-key add -
> OK
> it works ..
>
> I might have badly formed it but could not figure out .. seems related
> to safe_loader
>
>
> Backtrace :
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> Traceback (most recent call last):
> File "./hooks/install", line 406, in <module>
> main()
> File "./hooks/install", line 336, in main
> hooks.execute(sys.argv)
> File "/var/lib/juju/agents/unit-contrail-configuration-0/charm/hooks/charmhelpers/core/hookenv.py",
> line 672, in execute
> self._hooks[hook_name]()
> File "./hooks/install", line 314, in install
> configure_sources(True, "install-sources", "install-keys")
> File "/var/lib/juju/agents/unit-contrail-configuration-0/charm/hooks/charmhelpers/fetch/__init__.py",
> line 339, in configure_sources
> keys = safe_load((config(keys_var) or '').strip()) or None
> File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 93,
> in safe_load
> return load(stream, SafeLoader)
> File "/usr/lib/python2.7/dist-packages/yaml/__init__.py", line 71, in load
> return loader.get_single_data()
> File "/usr/lib/python2.7/dist-packages/yaml/constructor.py", line
> 37, in get_single_data
> node = self.get_single_node()
> File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 36,
> in get_single_node
> document = self.compose_document()
> File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 58,
> in compose_document
> self.get_event()
> File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 118, in get_event
> self.current_event = self.state()
> File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 193, in
> parse_document_end
> token = self.peek_token()
> File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 128,
> in peek_token
> sel...

[Stuart Bishop (stub)]

Yes, docs could certainly be improved. Also, I'm not sure why install_sources and install_keys needs to be so tightly coupled and always be the same length.

[Stuart Bishop (stub)]

And a nicer error message if the YAML fragment can't be decoded, because this is a common issue and the current traceback doesn't indicate the source of the problem.