If puppet host uid/gid is changed we cannot puppet apply anymore.

[Sofer Athlan-Guyot]

Hi,

this happened during a queens upgrade from pike but I believe it's still the case.

The client for some reason, changed the puppet uid/gid on the host. Then during upgrade he/she got:

 "Error running [\'docker\', \'run\', \'--name\', \'neutron_ovs_bridge\', \'--label\', \'config_id=tripleo_step3\', \'--label\', \'container_name=neutron_ovs_bridge\', \'--label\', \'managed_by=paunch\', \'--label\', \'config_data={\\"image\\": \\"XXXXXXXXXX/rhosp13/openstack-neutron-server:13.0-63\\", \\"pid\\": \\"host\\", \\"environment\\": [\\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\\", \\"TRIPLEO_CONFIG_HASH=97bd63f3e60339f5a0467f97a337b84b\\"], \\"command\\": [\\"puppet\\", \\"apply\\", \\"--modulepath\\", \\"/etc/puppet/modules:/usr/share/openstack-puppet/modules\\", \\"--tags\\", \\"file,file_line,concat,augeas,neutron::plugins::ovs::bridge,vs_config\\", \\"-v\\", \\"-e\\", \\"include neutron::agents::ml2::ovs\\"], \\"user\\": \\"root\\", \\"volumes\\": [\\"/etc/hosts:/etc/hosts:ro\\", \\"/etc/localtime:/etc/localtime:ro\\", \\"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\\", \\"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\\", \\"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\\", \\"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\\", \\"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\\", \\"/dev/log:/dev/log\\", \\"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/var/lib/kolla/config_files/neutron_ovs_agent.json:/var/lib/kolla/config_files/config.json:ro\\", \\"/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro\\", \\"/lib/modules:/lib/modules:ro\\", \\"/run/openvswitch:/run/openvswitch\\", \\"/etc/puppet:/etc/puppet:ro\\", \\"/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro\\", \\"/var/run/openvswitch/:/var/run/openvswitch/\\"], \\"net\\": \\"host\\", \\"detach\\": false, \\"privileged\\": true}\', \'--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\', \'--env=TRIPLEO_CONFIG_HASH=97bd63f3e60339f5a0467f97a337b84b\', \'--net=host\', \'--pid=host\', \'--privileged=true\', \'--user=root\', \'--volume=/etc/hosts:/etc/hosts:ro\', \'--volume=/etc/localtime:/etc/localtime:ro\', \'--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\', \'--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\', \'--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\', \'--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\', \'--volume=/dev/log:/dev/log\', \'--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\', \'--volume=/etc/puppet:/etc/puppet:ro\', \'--volume=/var/lib/kolla/config_files/neutron_ovs_agent.json:/var/lib/kolla/config_files/config.json:ro\', \'--volume=/var/lib/config-data/puppet-generated/neutron/:/var/lib/kolla/config_files/src:ro\', \'--volume=/lib/modules:/lib/modules:ro\', \'--volume=/run/openvswitch:/run/openvswitch\', \'--volume=/etc/puppet:/etc/puppet:ro\', \'--volume=/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro\', \'--volume=/var/run/openvswitch/:/var/run/openvswitch/\', \'XXXXXXXXXX/:5000/rhosp13/openstack-neutron-server:13.0-63\', \'puppet\', \'apply\', \'--modulepath\', \'/etc/puppet/modules:/usr/share/openstack-puppet/modules\', \'--tags\', \'file,file_line,concat,augeas,neutron::plugins::ovs::bridge,vs_config\', \'-v\', \'-e\', \'include neutron::agents::ml2::ovs\']. [1]", ',
 u' "stdout: \\u001b[mNotice: /File[/etc/puppet/ssl/certs]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "\\u001b[mNotice: /File[/etc/puppet/ssl/public_keys]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "\\u001b[mNotice: /File[/etc/puppet/ssl/certificate_requests]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "\\u001b[mNotice: /File[/etc/puppet/ssl/private_keys]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "\\u001b[mNotice: /File[/etc/puppet/ssl/private]: Dependency File[/etc/puppet/ssl] has failures: true\\u001b[0m", ',
 u' "stderr: \\u001b[1;31mError: Failed to set owner to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "\\u001b[1;31mError: /File[/etc/puppet/ssl]/owner: change from 31076 to puppet failed: Failed to set owner to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "\\u001b[1;31mError: Failed to set group to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "\\u001b[1;31mError: /File[/etc/puppet/ssl]/group: change from 10609 to puppet failed: Failed to set group to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/certs]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/public_keys]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/certificate_requests]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/private_keys]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;33mWarning: /File[/etc/puppet/ssl/private]: Skipping because of failed dependencies\\u001b[0m", ',
 u' "\\u001b[1;31mError: Could not prepare for execution: Got 1 failure(s) while initializing: File[/etc/puppet/ssl]: change from 31076 to puppet failed: Failed to set owner to \'52\': Read-only file system - /etc/puppet/ssl; File[/etc/puppet/ssl]: change from 10609 to puppet failed: Failed to set group to \'52\': Read-only file system - /etc/puppet/ssl\\u001b[0m", ',
 u' "stderr: Option \\"logdir\\" from group \\"DEFAULT\\" is deprecated. Use option \\"log-dir\\" from group \\"DEFAULT\\".", ',
 u' "stdout: Upgraded database to: queens_expand01, current revision(s): pike_contract01, queens_expand01", ',
[...]

The problem is that /etc/puppet is mounted read-only and that puppet apply still want to make sure that the /etc/puppet/ssl directory is correct. The mounted directory has uid/gid 31076 while inside the container puppet is 52/52. It tries to fix them and fails.

This is a requirements that I don't think should be there as the /etc/puppet/ssl in this use case is not used (puppet apply vs puppet master).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/625676

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So, according to https://tickets.puppetlabs.com/browse/PUP-9366 we can disable this by using manage_internal_file_permissions=false in puppet configuration[1]

[1] https://puppet.com/docs/puppet/6.1/configuration.html#manageinternalfilepermissions

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/654407

[OpenStack Infra (hudson-openstack)]

Change abandoned by Athlan-Guyot sofer (<email address hidden>) on branch: master
Review: https://review.opendev.org/625676
Reason: in favor for https://review.opendev.org/#/c/654407/1

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.opendev.org/654407
Reason: This review is > 90 days without comment, and failed Zuul the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results. For more details check policy https://specs.openstack.org/openstack/tripleo-specs/specs/policy/patch-abandonment.html