DC system controller failed with error "Error from server: rpc error: code = Unknown desc = malformed header: missing HTTP content-type" in puppet log

Bug #2073644 reported by Raphael Lima
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Raphael Lima

Bug Description

Brief Description
-----------------

Installation failed with error message "Error from server: rpc error: code = Unknown desc = malformed header: missing HTTP content-type" in puppet log

2024-07-19T11:01:12.197 Notice: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Storage/Exec[apply globalnetworkpolicies controller-storage-if-gnp with /tmp/gnp_controller-storage-if-gnp.yaml]/returns: Error from server: rpc error: code = Unknown desc = malformed header: missing HTTP content-type
2024-07-19T11:01:12.204 Error: 2024-07-19 11:01:12 +0000 'calico_firewall_apply_policy.sh controller-storage-if-gnp /tmp/gnp_controller-storage-if-gnp.yaml /etc/kubernetes/admin.conf' returned 1 instead of one of [0]
2024-07-19T11:01:12.212 /usr/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
2024-07-19T11:01:12.220 /usr/lib/ruby/vendor_ruby/puppet/type/exec.rb:168:in `sync'
2024-07-19T11:01:12.228 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:239:in `sync'
2024-07-19T11:01:12.235 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
2024-07-19T11:01:12.242 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
2024-07-19T11:01:12.250 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `each'
2024-07-19T11:01:12.256 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:87:in `perform_changes'
2024-07-19T11:01:12.261 /usr/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
2024-07-19T11:01:12.271 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:259:in `apply'
2024-07-19T11:01:12.276 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:279:in `eval_resource'
2024-07-19T11:01:12.280 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `call'
2024-07-19T11:01:12.283 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block (2 levels) in evaluate'
2024-07-19T11:01:12.287 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2024-07-19T11:01:12.290 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2024-07-19T11:01:12.293 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2024-07-19T11:01:12.295 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block in evaluate'
2024-07-19T11:01:12.299 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
2024-07-19T11:01:12.302 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:173:in `evaluate'
2024-07-19T11:01:12.305 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
2024-07-19T11:01:12.307 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2024-07-19T11:01:12.309 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2024-07-19T11:01:12.312 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2024-07-19T11:01:12.315 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
2024-07-19T11:01:12.318 /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
2024-07-19T11:01:12.320 /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
2024-07-19T11:01:12.323 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
2024-07-19T11:01:12.325 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:190:in `block (2 levels) in apply_catalog'
2024-07-19T11:01:12.327 /usr/lib/ruby/vendor_ruby/puppet/util.rb:539:in `block in thinmark'
2024-07-19T11:01:12.330 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2024-07-19T11:01:12.333 /usr/lib/ruby/vendor_ruby/puppet/util.rb:538:in `thinmark'
2024-07-19T11:01:12.335 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:189:in `block in apply_catalog'
2024-07-19T11:01:12.338 /usr/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
2024-07-19T11:01:12.340 /usr/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
2024-07-19T11:01:12.342 /usr/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
2024-07-19T11:01:12.344 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:188:in `apply_catalog'
2024-07-19T11:01:12.347 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:390:in `run_internal'
2024-07-19T11:01:12.349 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:238:in `block in run'
2024-07-19T11:01:12.351 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2024-07-19T11:01:12.353 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2024-07-19T11:01:12.355 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:215:in `run'
2024-07-19T11:01:12.358 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:355:in `apply_catalog'
2024-07-19T11:01:12.360 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
2024-07-19T11:01:12.363 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2024-07-19T11:01:12.365 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2024-07-19T11:01:12.367 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
2024-07-19T11:01:12.369 /usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
2024-07-19T11:01:12.373 /usr/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
2024-07-19T11:01:12.376 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
2024-07-19T11:01:12.378 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
2024-07-19T11:01:12.381 /usr/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
2024-07-19T11:01:12.383 /usr/lib/ruby/vendor_ruby/puppet/util.rb:710:in `exit_on_fail'
2024-07-19T11:01:12.386 /usr/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
2024-07-19T11:01:12.388 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run'
2024-07-19T11:01:12.391 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
2024-07-19T11:01:12.393 /usr/bin/puppet:11:in `<main>'
2024-07-19T11:01:12.395 Error: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Storage/Exec[apply globalnetworkpolicies controller-storage-if-gnp with /tmp/gnp_controller-storage-if-gnp.yaml]/returns: change from 'notrun' to ['0'] failed: 'calico_firewall_apply_policy.sh controller-storage-if-gnp /tmp/gnp_controller-storage-if-gnp.yaml /etc/kubernetes/admin.conf' returned 1 instead of one of [0]
2024-07-19T11:01:12.398 Notice: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/File[/tmp/hep_controller-0-cluster0-if-hep.yaml]: Dependency Exec[apply globalnetworkpolicies controller-storage-if-gnp with /tmp/gnp_controller-storage-if-gnp.yaml] has failures: true
2024-07-19T11:01:12.401 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/File[/tmp/hep_controller-0-cluster0-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.404 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/Exec[apply hostendpoints controller-0-cluster0-if-hep with /tmp/hep_controller-0-cluster0-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.406 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/File[/tmp/hep_controller-0-mgmt0-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.408 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/Exec[apply hostendpoints controller-0-mgmt0-if-hep with /tmp/hep_controller-0-mgmt0-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.411 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/File[/tmp/hep_controller-0-oam-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.413 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/Exec[apply hostendpoints controller-0-oam-if-hep with /tmp/hep_controller-0-oam-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.416 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/File[/tmp/hep_controller-0-pxeboot0-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.418 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/Exec[apply hostendpoints controller-0-pxeboot0-if-hep with /tmp/hep_controller-0-pxeboot0-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.420 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/File[/tmp/hep_controller-0-storage0-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.422 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/Exec[apply hostendpoints controller-0-storage0-if-hep with /tmp/hep_controller-0-storage0-if-hep.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.424 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/Exec[get active hostendepoints: [controller-0-cluster0-if-hep, controller-0-mgmt0-if-hep, controller-0-oam-if-hep, controller-0-pxeboot0-if-hep, controller-0-storage0-if-hep]]: Skipping because of failed dependencies
2024-07-19T11:01:12.427 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Calico::Hostendpoint/Exec[remove unused hostendepoints controller-0 /tmp/hep_active.txt]: Skipping because of failed dependencies
2024-07-19T11:01:12.430 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Rbac::Worker/File[/tmp/rbac_worker_permission_for_firewall.yaml]: Skipping because of failed dependencies
2024-07-19T11:01:12.432 Warning: 2024-07-19 11:01:12 +0000 /Stage[main]/Platform::Firewall::Rbac::Worker/Exec[apply permission to worker node firewall configuration]: Skipping because of failed dependencies
2024-07-19T11:06:16.576 Notice: 2024-07-19 11:06:16 +0000 /Stage[main]/Dcdbsync::Keystone::Auth/Keystone::Resource::Service_identity[dcdbsync]/Keystone_endpoint[RegionOne/dcdbsync::dcorch-dbsync]/public_url: public_url changed '' to 'https://[2620:10a:a001:d41::1084]:8219/v1.0'
2024-07-19T11:06:28.428 Notice: 2024-07-19 11:06:28 +0000 /Stage[main]/Dcdbsync::Keystone::Auth/Exec[Delete public endpoint]/returns: executed successfully
2024-07-19T11:06:41.516 Warning: 2024-07-19 11:06:41 +0000 /Stage[post]/Platform::Coredump::Reload/Exec[restart-coredump-service]: Skipping because of failed dependencies
2024-07-19T11:06:41.519 Warning: 2024-07-19 11:06:41 +0000 /Stage[post]/Platform::Config::Dnsmasq/File[/opt/platform/config/24.09//dnsmasq.addn_conf]: Skipping because of failed dependencies
2024-07-19T11:06:41.521 Warning: 2024-07-19 11:06:41 +0000 /Stage[post]/Platform::Config::Post/Service[cron]: Skipping because of failed dependencies
2024-07-19T11:06:41.523 Warning: 2024-07-19 11:06:41 +0000 /Stage[post]/Platform::Config::Post/File[/etc/platform/.config_applied]: Skipping because of failed dependencies
2024-07-19T11:06:41.526 Warning: 2024-07-19 11:06:41 +0000 /Stage[post]/Platform::Nfv::Reload/Platform::Sm::Restart[vim]/Exec[sm-restart-vim]: Skipping because of failed dependencies
2024-07-19T11:06:41.605 Notice: 2024-07-19 11:06:41 +0000 Applied catalog in 467.75 seconds

Severity
-----------------

Major

Steps to Reproduce
-----------------

Install DC

Expected Behavior
-----------------

Installation should go fine without any failures

Actual Behavior
-----------------

Installation failed with error message "Error from server: rpc error: code = Unknown desc = malformed header: missing HTTP content-type" in puppet log

Reproducibility
-----------------

2/2

System Configuration
-----------------
DC system

Raphael Lima (r-lima)
Changed in starlingx:
assignee: nobody → Raphael Lima (r-lima)
Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to distcloud (master)

Reviewed: https://review.opendev.org/c/starlingx/distcloud/+/924542
Committed: https://opendev.org/starlingx/distcloud/commit/e0ed2b09ed8e1fc54ac64ae78450d3f3150559ab
Submitter: "Zuul (22348)"
Branch: master

commit e0ed2b09ed8e1fc54ac64ae78450d3f3150559ab
Author: Raphael Lima <email address hidden>
Date: Fri Jul 19 18:07:32 2024 -0300

    Add __init__.py for validators' folder

    This commit adds the missing __init__.py file inside the validator's
    folder for dcmanager's orchestrator and fix the related issues in unit
    tests.

    Test plan:
    1. PASS: run tox verify that it passes successfully, executing the
       validators't tests.

    Closes-Bug: 2073644

    Change-Id: I97e8edd7c1f1f832606a4af8964ee63cdd2482b6
    Signed-off-by: Raphael Lima <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ghada Khalil (gkhalil)
Changed in starlingx:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to distcloud (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/distcloud/+/924756

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.