ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list

Bug #1735950 reported by james beedy on 2017-12-02
38
This bug affects 8 people
Affects Status Importance Assigned to Milestone
MAAS
Low
Andres Rodriguez
2.3
Low
Unassigned
cloud-init
Undecided
Unassigned
curtin
Undecided
Unassigned

Bug Description

All nodes have these same failed events:

Node post-installation failure - 'cloudinit' running modules for config

Node post-installation failure - 'cloudinit' running config-apt-configure with frequency once-per-instance

Experiencing odd issues with the squid proxy not being reachable.

From a deployed node that had the event errors.

$ sudo cat /var/log/cloud-init.log | http://paste.ubuntu.com/26098787/
$ sudo cat /var/log/cloud-init-output.log | http://paste.ubuntu.com/26098802/

ubuntu@os-util-00:~$ sudo apt install htop
sudo: unable to resolve host os-util-00
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  htop
0 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 76.4 kB of archives.
After this operation, 215 kB of additional disk space will be used.
Err:1 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 htop amd64 2.0.1-1ubuntu1
  Could not connect to 10.10.0.110:8000 (10.10.0.110). - connect (113: No route to host)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/h/htop/htop_2.0.1-1ubuntu1_amd64.deb Could not connect to 10.10.0.110:8000 (10.10.0.110). - connect (113: No route to host)

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

Not sure if these things are related (the proxy not being reachable, and the node event errors) .... something is not right.

Related branches

james beedy (jamesbeedy) wrote :
summary: - failed events, borked nodes
+ failed events, borked nodes - ValueError: Old and New apt format defined
+ with unequal values True vs False @ apt_preserve_sources_list

oooh found it!

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 793, in _run_modules
    freq=freq)
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 274, in handle
    ocfg = convert_to_v3_apt_format(ocfg)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 758, in convert_to_v3_apt_format
    cfg = convert_v2_to_v3_apt_format(cfg)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 724, in convert_v2_to_v3_apt_format
    oldkey))

james beedy (jamesbeedy) wrote :

when I set the "Proxy" setting to "Don't use a proxy" I still get both of the errors in the events log, but I am able to successfully install packages

http://paste.ubuntu.com/26099075/

james beedy (jamesbeedy) on 2017-12-03
summary: - failed events, borked nodes - ValueError: Old and New apt format defined
- with unequal values True vs False @ apt_preserve_sources_list
+ ValueError: Old and New apt format defined with unequal values True vs
+ False @ apt_preserve_sources_list
Mike Pontillo (mpontillo) wrote :

Setting this to Incomplete for MAAS, since it's not clear from if bad input data from MAAS is causing this traceback in cloud-init, or if the bug is only in cloud-init.

@jamesbeedy, can you tell us about how you have configured/customized your apt sources in MAAS?

Changed in maas:
status: New → Incomplete
Scott Moser (smoser) wrote :

ValueError: Old and New apt format defined with unequal values True vs False @ apt_preserve_sources_list

Thats the error.
MAAS or the user sent configuration that had
 apt_preserve_sources_list: True
and
 apt:
   preserve_sources_list: False

Those are the same setting in 'Old' and 'New' config format. Cloud-init can't determine which one you meant, so it raised exception and recorded failure.

The curtin config would be good as well as any user-data that was provided.

Changed in cloud-init:
status: New → Incomplete
james beedy (jamesbeedy) wrote :

@mponillo I have the curtin daily added, but not enabled (see screenshot)

@smoser I hadn't made any customizations to user-data or curtin config ... just what maas gives the node when it deploys.

I can deploy a node and reproduce the error at anytime, please let me know what files you want off the machine if any and I'll boot one up and swipe them for you.

Mike Pontillo (mpontillo) wrote :

Thanks for the clarification.

Changed in maas:
status: Incomplete → Triaged
importance: Undecided → Critical
milestone: none → 2.4.0alpha1
james beedy (jamesbeedy) wrote :

@pmontillo do you want a separate bug for the proxy issue?

Scott Moser (smoser) wrote :

Hi, in order to further investigate this bug we'll need to get some information about the node.
Please collect and attach the storage configuration for the node. You can get this information with:
 * maas 1.9 via cli
   maas <session> node get-curtin-config <system-id>
   maas <session> maas set-config name=curtin_verbose value=true

 * maas 2.0 via cli
   maas <session> machine get-curtin-config <system-id>

* Web UI: On the node details page in the installation output section at the bottom of the page

Also, in the installed environment, could you please collect /var/lib/cloud/ and /var/log/cloud-init*
thanks.

Chad Smith (chad.smith) wrote :

On MAAS 2.3 I still see this issue:

MAAS version: 2.3.0 (6434-gd354690-0ubuntu1~16.04.1

If I block curtin ephemeral environment I can see the competing config provided to debconf_selections:

/curtin/configs/config-001.cfg
debconf_selections: {grub2: grub2 grub2/update_nvram boolean false, maas: 'cloud-init cloud-init/datasources multiselect
    MAAS

    cloud-init cloud-init/maas-metadata-url string http://10.5.0.5/MAAS/metadata/

    cloud-init cloud-init/maas-metadata-credentials string oauth_token_key=W9pshubrZ8StJAw5ss&oauth_consumer_key=r29m8b2F38n5jCEYZX&oauth_token_secret=G7ecZSKG8wsMHckxHCy9aE9Xh76U8bV4

    cloud-init cloud-init/local-cloud-config string apt:\n preserve_sources_list:
    false\n primary:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\n proxy:
    http://10.5.0.5:8000/\n security:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\napt_preserve_sources_list:
    true\napt_proxy: http://10.5.0.5:8000/\nmanage_etc_hosts: false\nmanual_cache_clean:
    true\nreporting:\n maas: {consumer_key: <REDACTED>, endpoint: ''http://10.5.0.5/MAAS/metadata/status/hhess8'',\n token_key:
    <REDACTED>, token_secret: <REDACTED>,\n type:
    webhook}\nsystem_info:\n package_mirrors:\n - arches: [i386, amd64]\n failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n search:\n primary:
    [''http://archive.ubuntu.com/ubuntu'']\n security: [''http://archive.ubuntu.com/ubuntu'']\n -
    arches: [default]\n failsafe: {primary: ''http://ports.ubuntu.com/ubuntu-ports'',
    security: ''http://ports.ubuntu.com/ubuntu-ports''}\n search:\n primary:
    [''http://ports.ubuntu.com/ubuntu-ports'']\n security: [''http://ports.ubuntu.com/ubuntu-ports'']\n

    '}

Chad Smith (chad.smith) wrote :

Here is the curtin-config provided by MAAS to the node which also shows the competing apt_preserve_sources-list values on lines 49 and 51 of the paste

https://pastebin.ubuntu.com/26355967/

Chad Smith (chad.smith) wrote :

I tested the following fix suggestion on MAAS 2.3 (xenial image deployment) and it seems to bahave well with cloud-init, and avoids providing duplicate/conflicting apt_preferences to cloud-init via curtin

diff --git a/src/maasserver/compose_preseed.py b/src/maasserver/compose_preseed.py
index e15b3b2..2a41fbc 100644
--- a/src/maasserver/compose_preseed.py
+++ b/src/maasserver/compose_preseed.py
@@ -230,10 +230,9 @@ def compose_cloud_init_preseed(
         # See bug 1087183 for details.
         "manage_etc_hosts": False,
         "apt_preserve_sources_list": True,
- # Prevent the node from requesting cloud-init data on every reboot.
- # This is done so a machine does not need to contact MAAS every time
- # it reboots.
- "manual_cache_clean": True,
+ # On ubuntu >= xenial (cloud-init 17.2) manual_cache_clean is no
+ # longer required. But, you might want to set it for < xenial
+ # "manual_cache_clean": True,
     }
     # This is used as preseed for a node that's been installed.
     # This will allow cloud-init to be configured with reporting for
@@ -248,11 +247,6 @@ def compose_cloud_init_preseed(
         node.get_boot_rack_controller(), default_region_ip=default_region_ip)
     if apt_proxy:
         config['apt_proxy'] = apt_proxy
- # Add APT configuration for new cloud-init (>= 0.7.7-17)
- config.update(
- get_archive_config(
- node=node, preserve_sources=False,
- default_region_ip=default_region_ip))

     local_config_yaml = yaml.safe_dump(config)
     # this is debconf escaping

Chad Smith (chad.smith) wrote :

the above patch also tries addressing lp:1742479

Chad Smith (chad.smith) wrote :

The reason for dropping the config.update(get_archive_config) is because maas is already telling cloud-init to apt_preserve_sources_list: True above in line 230-ish. We don't need to resend curtin's apt configuration to the cloud_init_preseed config because that introduces conflicting apt settings.

MAAS provides curtin itself with the following apt configuration, which shows up in /curtin/configs/config-002.cfg
apt:
  preserve_sources_list: false
  primary:
  - arches: [default]
    uri: http://archive.ubuntu.com/ubuntu
  proxy: http://10.5.0.5:8000/
  security:
  - arches: [default]
    uri: http://archive.ubuntu.com/ubuntu
~

(the cloud-init deb_selections is provded via /curtin/configs/config-001.cfg)

The issue is that the deb_selections for cloud-init concatonated both curtin's apt config and cloud-init's apt config which says to cloud-init preserve apt sources: True. (This is seen in the cloud-init's local-cloud-config setting).

To block maas from rebooting out of curtin's ephemeral environment to test this, add the following to /etc/maas/preseeds/curtin_userdata:

early_commands:
  disable_reboot: touch /run/block-curtin-poweroff /tmp/block-poweroff /tmp/block-reboot

You can ssh into the instance with the ip address MAAS auto-allocates to the node.

Chad Smith (chad.smith) wrote :

Marking invalid for cloud-init as cloud-init's Traceback and error is 'desired' as it informs the cloud-init user of invalid cloud-config provided to cloud-init.

Changed in cloud-init:
status: Incomplete → Invalid
Changed in maas:
milestone: 2.4.0alpha1 → 2.4.0alpha2
Changed in maas:
milestone: 2.4.0alpha2 → 2.4.0beta1
Changed in maas:
importance: Critical → Low
Changed in maas:
milestone: 2.4.0beta1 → 2.4.0beta2
Changed in maas:
milestone: 2.4.0beta2 → 2.4.0beta3
Changed in maas:
milestone: 2.4.0beta3 → 2.4.0beta4
Changed in maas:
milestone: 2.4.0beta4 → 2.4.x
Changed in maas:
milestone: 2.4.x → 2.5.0beta2
studentz (studenz) wrote :

After install MAAS vanilla without any additional configuration or any extra repository and
MAASBuild-in proxy the error persist.

In addition the fix suggested in #12 works for MAAS 2.3 but NOT for MAAS 2.4

Configuration:

- Maas 2.4.2

- cloud-init --version
/usr/bin/cloud-init 18.3-9-g2e62cb8a-0ubuntu1~16.04.2

- cloud-init.log
https://pastebin.ubuntu.com/p/5vC6Zp4kvc/

- cloud-init-output.log
https://pastebin.ubuntu.com/p/bPcYVhXsM8/

- curtin-config
https://pastebin.ubuntu.com/p/Ys5qXPcCRb/

Thanks.

Changed in maas:
milestone: 2.5.0beta2 → 2.5.0rc1
Changed in maas:
milestone: 2.5.0rc1 → 2.5.x
Eduardo Pérez (eperezf) wrote :

Is there any temporal fix for this error? I'm able to deploy Ubuntu 18.04 LTS but I get both errors described above:

Node post-installation failure - 'cloudinit' running modules for config

Node post-installation failure - 'cloudinit' running config-apt-configure with frequency once-per-instance

As I'm installing OpenStack using conjure-up, the installation fails.

Andres Rodriguez (andreserl) wrote :

Eduardo,

If the installation fails, its definitely not because of the bug above otherwise all ubuntu installations would fail.

This to me seems like a different issue, because the above message is only surface *after* the machine has already been marked as "Deployed". MAAS marks the machines deployed when cloud-init accesses the metadata *not* after cloud-init has run the metadata. This message is sent while cloud-init has bene running the metadata.

I would suggest you file a new bug report and include relevant information, including:

1. /var/log/maas/*.log
2. /var/log/maas/rsyslog/<machine-name>/<date>/messages
3. /var/log/cloud-init* from the deployed system
4. confirm if you are using Ubuntu mirrors or custom mirrors.

Changed in maas:
assignee: nobody → Andres Rodriguez (andreserl)
milestone: 2.5.x → 2.5.0rc1
Andres Rodriguez (andreserl) wrote :

@Scott,

So it seems that curtin is always writing apt_preserve_sources_list: True. To test this, I'm ensuring that MAAS never sends apt_preserve_source_list (and this is reflected in the PR attached to this branch).

With those changes, the resulting curtin config is:

http://paste.ubuntu.com/p/s8GsHRQMhX/

As you can see, we never set apt_preserve_source_list. Hoewver, the deployed system results with:

ubuntu@guided-osprey:~$ cat /etc/cloud/cloud.cfg.d/curtin-preserve-sources.cfg
{apt_preserve_sources_list: true}

So it seems it is curtin that's adding an old cloud-init config style.

Scott Moser (smoser) wrote :

I spoke in IRC to Andres in response to his comment #19. curtin is
correct in telling cloud-init in the installed system to not re-render
/etc/apt/sources.list. However, it would be better if curtin did so using
the newer format where possible.

So, we will update curtin to use the following when cloud-init in the target allows:

apt:
  preserve_sources_list: true

Also, I have written a 'hackmd' doc at
https://hackmd.io/Gx5mVd_nRIixmAOkA2XMAA
that provides information on hte preserve_sources_list setting that
I will also attach here for posterity.

Please read it as I believe it has a complete discussion of this
setting wrt maas's use.

Changed in maas:
status: Triaged → Fix Committed
Changed in maas:
milestone: 2.5.0rc1 → 2.5.0beta4
status: Fix Committed → Fix Released

This bug is fixed with commit 40e6cc35 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=40e6cc35

Changed in curtin:
status: New → Fix Committed

This bug is believed to be fixed in curtin in version 18.2. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers