Brief Description
------------------
Failures related to lighttpd.service seen only in controller-1's puppet logs.
2024-07-13T03:25:20.399 Notice: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm/Exec[restart lighttpd for helm]/returns: Job for lighttpd.service failed because the control process exited with error code.
2024-07-13T03:25:20.402 Notice: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm/Exec[restart lighttpd for helm]/returns: See "systemctl status lighttpd.service" and "journalctl -xe" for details.
2024-07-13T03:25:20.404 Error: 2024-07-13 03:25:20 +0000 'systemctl restart lighttpd.service' returned 1 instead of one of [0]
2024-07-13T03:25:20.407 /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
2024-07-13T03:25:20.409 /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:168:in `sync'
2024-07-13T03:25:20.411 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:239:in `sync'
2024-07-13T03:25:20.414 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
2024-07-13T03:25:20.416 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
2024-07-13T03:25:20.419 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each'
2024-07-13T03:25:20.421 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes'
2024-07-13T03:25:20.423 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
2024-07-13T03:25:20.425 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:259:in `apply'
2024-07-13T03:25:20.428 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:279:in `eval_resource'
2024-07-13T03:25:20.430 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `call'
2024-07-13T03:25:20.433 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block (2 levels) in evaluate'
2024-07-13T03:25:20.435 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2024-07-13T03:25:20.437 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2024-07-13T03:25:20.440 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2024-07-13T03:25:20.442 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block in evaluate'
2024-07-13T03:25:20.444 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
2024-07-13T03:25:20.447 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:173:in `evaluate'
2024-07-13T03:25:20.449 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
2024-07-13T03:25:20.451 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2024-07-13T03:25:20.454 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2024-07-13T03:25:20.456 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2024-07-13T03:25:20.459 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
2024-07-13T03:25:20.461 /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
2024-07-13T03:25:20.463 /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
2024-07-13T03:25:20.466 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
2024-07-13T03:25:20.468 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:190:in `block (2 levels) in apply_catalog'
2024-07-13T03:25:20.471 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2024-07-13T03:25:20.473 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2024-07-13T03:25:20.476 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2024-07-13T03:25:20.478 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:189:in `block in apply_catalog'
2024-07-13T03:25:20.480 /usr/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
2024-07-13T03:25:20.483 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2024-07-13T03:25:20.485 /usr/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
2024-07-13T03:25:20.487 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:188:in `apply_catalog'
2024-07-13T03:25:20.490 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:390:in `run_internal'
2024-07-13T03:25:20.492 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:238:in `block in run'
2024-07-13T03:25:20.495 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2024-07-13T03:25:20.497 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2024-07-13T03:25:20.499 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:215:in `run'
2024-07-13T03:25:20.501 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:355:in `apply_catalog'
2024-07-13T03:25:20.503 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
2024-07-13T03:25:20.505 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2024-07-13T03:25:20.507 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2024-07-13T03:25:20.510 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
2024-07-13T03:25:20.512 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2024-07-13T03:25:20.514 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2024-07-13T03:25:20.516 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
2024-07-13T03:25:20.518 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
2024-07-13T03:25:20.520 /usr/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
2024-07-13T03:25:20.523 /usr/lib/ruby/vendor_ruby/puppet/util.rb:710:in `exit_on_fail'
2024-07-13T03:25:20.525 /usr/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
2024-07-13T03:25:20.527 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run'
2024-07-13T03:25:20.529 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
2024-07-13T03:25:20.532 /usr/bin/puppet:11:in `<main>'
2024-07-13T03:25:20.534 Error: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm/Exec[restart lighttpd for helm]/returns: change from 'notrun' to ['0'] failed: 'systemctl restart lighttpd.service' returned 1 instead of one of [0]
2024-07-13T03:25:20.584 Notice: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm::Repositories/Platform::Helm::Repository[stx-platform]/File[/var/www/pages/helm_charts/stx-platform]: Dependency Exec[restart lighttpd for helm] has failures: true
2024-07-13T03:25:20.586 Warning: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm::Repositories/Platform::Helm::Repository[stx-platform]/File[/var/www/pages/helm_charts/stx-platform]: Skipping because of failed dependencies
2024-07-13T03:25:20.589 Warning: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm::Repositories/Platform::Helm::Repository[stx-platform]/Exec[Generate index: /var/www/pages/helm_charts/stx-platform]: Skipping because of failed dependencies
2024-07-13T03:25:20.591 Warning: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm::Repositories/Platform::Helm::Repository[stx-platform]/Exec[Adding StarlingX helm repo: stx-platform]: Skipping because of failed dependencies
2024-07-13T03:25:20.593 Warning: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm::Repositories/Platform::Helm::Repository[starlingx]/File[/var/www/pages/helm_charts/starlingx]: Skipping because of failed dependencies
2024-07-13T03:25:20.595 Warning: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm::Repositories/Platform::Helm::Repository[starlingx]/Exec[Generate index: /var/www/pages/helm_charts/starlingx]: Skipping because of failed dependencies
2024-07-13T03:25:20.598 Warning: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm::Repositories/Platform::Helm::Repository[starlingx]/Exec[Adding StarlingX helm repo: starlingx]: Skipping because of failed dependencies
2024-07-13T03:25:20.600 Warning: 2024-07-13 03:25:20 +0000 /Stage[main]/Platform::Helm::Repositories/Exec[Updating info of available charts locally from chart repo]: Skipping because of failed dependencies
2024-07-13T03:25:20.602 Warning: 2024-07-13 03:25:20 +0000 /Stage[main]/Openstack::Horizon/Exec[Stop lighttpd]: Skipping because of failed dependencies
2024-07-13T03:25:20.605 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Coredump::Reload/Exec[restart-coredump-service]: Skipping because of failed dependencies
2024-07-13T03:25:20.607 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Config::Post/Service[cron]: Skipping because of failed dependencies
2024-07-13T03:25:20.609 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Config::Post/File[/etc/platform/.config_applied]: Skipping because of failed dependencies
2024-07-13T03:25:20.612 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Drbd::Rook::Post/File[/etc/platform/.node_drbd_rook_configured]: Skipping because of failed dependencies
2024-07-13T03:25:20.614 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Ceph::Post/File[/var/run/.ceph_started]: Skipping because of failed dependencies
2024-07-13T03:25:20.617 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Config::Aio::Post/File[/etc/platform/.initial_controller_config_complete]: Skipping because of failed dependencies
2024-07-13T03:25:20.619 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Config::Aio::Post/File[/var/run/.controller_config_complete]: Skipping because of failed dependencies
2024-07-13T03:25:20.621 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Config::Worker::Post/File[/etc/platform/.initial_config_complete]: Skipping because of failed dependencies
2024-07-13T03:25:20.624 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Config::Worker::Post/File[/etc/platform/.bootstrap_completed]: Skipping because of failed dependencies
2024-07-13T03:25:20.626 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Config::Worker::Post/File[/etc/platform/.initial_worker_config_complete]: Skipping because of failed dependencies
2024-07-13T03:25:20.628 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Config::Worker::Post/File[/var/run/.worker_config_complete]: Skipping because of failed dependencies
2024-07-13T03:25:20.631 Warning: 2024-07-13 03:25:20 +0000 /Stage[post]/Platform::Compute::Grub::Audit/File[/var/run/worker_goenabled]: Skipping because of failed dependencies
2024-07-13T03:25:20.660 Notice: 2024-07-13 03:25:20 +0000 Applied catalog in 74.36 seconds
Tried restarting lighttpd.service manually and saw below output
root@controller-1:/var/log/puppet/latest# systemctl restart lighttpd.service
Job for lighttpd.service failed because the control process exited with error code.
See "systemctl status lighttpd.service" and "journalctl -xe" for details.
root@controller-1:/var/log/puppet/latest# systemctl status lighttpd.service
● lighttpd.service - Lighttpd Daemon
Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2024-07-13 06:45:28 UTC; 11s ago
Process: 405489 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=255/EXCEPTION)
CPU: 4ms
While checking the configuration of the Lighttpd web server without starting it
sysadmin@controller-1:~$ sudo /usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf
Password:
2024-07-13 07:06:03: (configfile.c.59) Warning: please add "mod_openssl" to server.modules list in lighttpd.conf. A future release of lighttpd 1.4.x *will not* automatically load mod_openssl and lighttpd *will not* use SSL/TLS where your lighttpd.conf contains ssl.* directives
2024-07-13 07:06:03: (mod_openssl.c.471) SSL: BIO_read_filename('/etc/ssl/private/server-cert.pem') failed
2024-07-13 07:06:03: (server.c.1207) Initialization of plugins failed. Going down.
Severity
--------
Major
Steps to Reproduce
------------------
Install lab which has more than one controller
Expected Behavior
-----------------
Installation should go fine without any issues
Actual Behavior
---------------
Installation failed due to above mentioned error in puppet log
Reproducibility
---------------
5/5 (seen in all lab types - DX, DX+, Standard, Storage, DC system controller (2+1))
System Configuration
--------------------
System Type: AIO-DX
Lab Name: R750_003_004
Load Information
----------------
20240714T060056Z
Logs
----
Collect Logs: /folk/cgts_logs/logs/CGTS-64814
Fix merged: https:/