[midonet] fails trying to create neutron networks

[Kevin Scheunemann]

Error on the fuel dashboard
Deployment has failed. Method granular_deploy. Failed to execute hook 'midonet-fuel-plugin-2.0.0' Puppet run failed. Check puppet logs for details
---
uids:
- '19'
- '12'
- '17'
parameters:
  puppet_modules: puppet/modules:/etc/puppet/modules
  puppet_manifest: puppet/manifests/midonet-restart-neutron.pp
  timeout: 720
  cwd: "/etc/fuel/plugins/midonet-fuel-plugin-2.0/"
priority: 2500
fail_on_error: true
type: puppet
diagnostic_name: midonet-fuel-plugin-2.0.0
.
Inspect Astute logs for the details

Errors in the puppet run:
2015-08-25 18:24:41 NOTICE
 Finished catalog run in 134.54 seconds
2015-08-25 18:24:40 INFO
 (Stage[main]) Evaluated in 0.00 seconds
2015-08-25 18:24:40 INFO
 (Stage[main]) Starting to evaluate the resource
2015-08-25 18:24:40 INFO
 (Class[Main]) Evaluated in 0.00 seconds
2015-08-25 18:24:40 INFO
 (Class[Main]) Starting to evaluate the resource
2015-08-25 18:24:40 INFO
 (/Schedule[puppet]) Evaluated in 0.00 seconds
2015-08-25 18:24:40 INFO
 (/Schedule[puppet]) Starting to evaluate the resource
2015-08-25 18:24:40 INFO
 (/Schedule[weekly]) Evaluated in 0.00 seconds
2015-08-25 18:24:40 INFO
 (/Schedule[weekly]) Starting to evaluate the resource
2015-08-25 18:24:40 INFO
 (/Stage[main]/Main/Neutron_router_interface[router04:net04__subnet]) Evaluated in 0.00 seconds
2015-08-25 18:24:40 WARNING
 (/Stage[main]/Main/Neutron_router_interface[router04:net04__subnet]) Skipping because of failed dependencies
2015-08-25 18:24:40 NOTICE
 (/Stage[main]/Main/Neutron_router_interface[router04:net04__subnet]) Dependency Neutron_network[net04] has failures: true
2015-08-25 18:24:40 INFO
 (/Stage[main]/Main/Neutron_router_interface[router04:net04__subnet]) Starting to evaluate the resource
2015-08-25 18:24:37 INFO
 (/Stage[main]/Main/Neutron_router[router04]) Evaluated in 0.00 seconds
2015-08-25 18:24:37 WARNING
 (/Stage[main]/Main/Neutron_router[router04]) Skipping because of failed dependencies
2015-08-25 18:24:37 NOTICE
 (/Stage[main]/Main/Neutron_router[router04]) Dependency Neutron_network[net04] has failures: true
2015-08-25 18:24:37 INFO
 (/Stage[main]/Main/Neutron_router[router04]) Starting to evaluate the resource
2015-08-25 18:24:36 INFO
 (/Stage[main]/Main/Neutron_subnet[net04_ext__subnet]) Evaluated in 0.00 seconds
2015-08-25 18:24:36 WARNING
 (/Stage[main]/Main/Neutron_subnet[net04_ext__subnet]) Skipping because of failed dependencies
2015-08-25 18:24:36 NOTICE
 (/Stage[main]/Main/Neutron_subnet[net04_ext__subnet]) Dependency Neutron_network[net04] has failures: true
2015-08-25 18:24:36 INFO
 (/Stage[main]/Main/Neutron_subnet[net04_ext__subnet]) Starting to evaluate the resource
2015-08-25 18:24:36 INFO
 (/Stage[main]/Main/Neutron_network[net04_ext]) Evaluated in 0.00 seconds
2015-08-25 18:24:36 WARNING
 (/Stage[main]/Main/Neutron_network[net04_ext]) Skipping because of failed dependencies
2015-08-25 18:24:36 NOTICE
 (/Stage[main]/Main/Neutron_network[net04_ext]) Dependency Neutron_network[net04] has failures: true
2015-08-25 18:24:36 INFO
 (/Stage[main]/Main/Neutron_network[net04_ext]) Starting to evaluate the resource
2015-08-25 18:24:36 INFO
 (/Stage[main]/Main/Neutron_subnet[net04__subnet]) Evaluated in 0.00 seconds
2015-08-25 18:24:36 WARNING
 (/Stage[main]/Main/Neutron_subnet[net04__subnet]) Skipping because of failed dependencies
2015-08-25 18:24:36 NOTICE
 (/Stage[main]/Main/Neutron_subnet[net04__subnet]) Dependency Neutron_network[net04] has failures: true
2015-08-25 18:24:36 INFO
 (/Stage[main]/Main/Neutron_subnet[net04__subnet]) Starting to evaluate the resource
2015-08-25 18:24:35 INFO
 (/Stage[main]/Main/Neutron_network[net04]) Evaluated in 124.70 seconds
2015-08-25 18:24:35 ERR
 (/Stage[main]/Main/Neutron_network[net04]/ensure) change from absent to present failed: did not get expected message on network creation, got
2015-08-25 18:24:35 ERR
 /usr/bin/puppet:4
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:91:in `execute'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:137:in `run'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:478:in `exit_on_fail'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:470:in `plugin_hook'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:364:in `run'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:146:in `run_command'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:218:in `main'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/application/apply.rb:268:in `apply_catalog'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:192:in `run'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:124:in `apply_catalog'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:160:in `benchmark'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:161:in `benchmark'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/configurer.rb:125:in `apply_catalog'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:163:in `apply'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/report.rb:108:in `as_logging_destination'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/log.rb:149:in `with_destination'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:164:in `apply'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:108:in `evaluate'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/graph/relationship_graph.rb:118:in `traverse'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:326:in `thinmark'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util.rb:327:in `thinmark'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `evaluate'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:117:in `call'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:187:in `eval_resource'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction.rb:174:in `apply'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:18:in `evaluate'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:77:in `perform_changes'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:107:in `manage_via_ensure_if_possible'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:130:in `sync_if_needed'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/transaction/resource_harness.rb:193:in `sync'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:581:in `sync'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:498:in `set'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:197:in `call_valuemethod'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/property.rb:197:in `send'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/property/ensure.rb:16:in `set_present'
2015-08-25 18:24:35 ERR
 /etc/puppet/modules/neutron/lib/puppet/provider/neutron_network/neutron.rb:108:in `create'
2015-08-25 18:24:35 ERR
 /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:97:in `fail'
2015-08-25 18:24:35 ERR
 did not get expected message on network creation, got
2015-08-25 18:24:31 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 0 sec.
2015-08-25 18:24:28 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 3 sec.
2015-08-25 18:24:25 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 6 sec.
2015-08-25 18:24:21 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 10 sec.
2015-08-25 18:24:18 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 13 sec.
2015-08-25 18:24:15 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 16 sec.
2015-08-25 18:24:10 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 21 sec.
2015-08-25 18:24:06 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 25 sec.
2015-08-25 18:24:02 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 29 sec.
2015-08-25 18:23:58 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 33 sec.
2015-08-25 18:23:55 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 36 sec.
2015-08-25 18:23:51 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 40 sec.
2015-08-25 18:23:47 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 44 sec.
2015-08-25 18:23:43 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 48 sec.
2015-08-25 18:23:40 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 51 sec.
2015-08-25 18:23:37 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 54 sec.
2015-08-25 18:23:34 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 57 sec.
2015-08-25 18:23:30 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 61 sec.
2015-08-25 18:23:26 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 65 sec.
2015-08-25 18:23:23 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 68 sec.
2015-08-25 18:23:20 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 71 sec.
2015-08-25 18:23:16 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 75 sec.
2015-08-25 18:23:13 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 78 sec.
2015-08-25 18:23:09 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 82 sec.
2015-08-25 18:23:02 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 89 sec.
2015-08-25 18:22:58 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 93 sec.
2015-08-25 18:22:54 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 97 sec.
2015-08-25 18:22:50 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 101 sec.
2015-08-25 18:22:47 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 104 sec.
2015-08-25 18:22:43 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 108 sec.
2015-08-25 18:22:40 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 111 sec.
2015-08-25 18:22:37 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 114 sec.
2015-08-25 18:22:33 NOTICE
 (Puppet::Type::Neutron_network::ProviderNeutron) Neutron API not avalaible. Wait up to 118 sec.
2015-08-25 18:22:31 NOTICE
 (Neutron_network[net04](provider=neutron)) ***N*** neutron_network::create *** tenant_id='"761dfe3d8972459a92753995f1cbf007"'
2015-08-25 18:22:30 INFO
 (/Stage[main]/Main/Neutron_network[net04]) Starting to evaluate the resource
2015-08-25 18:22:28 INFO
 (/Filebucket[puppet]) Evaluated in 0.00 seconds
2015-08-25 18:22:28 INFO
 (/Filebucket[puppet]) Starting to evaluate the resource
2015-08-25 18:22:28 INFO
 (/Schedule[never]) Evaluated in 0.00 seconds
2015-08-25 18:22:28 INFO
 (/Schedule[never]) Starting to evaluate the resource
2015-08-25 18:22:28 INFO
 (/Stage[main]/Main/Exec[refresh-metadata-agent]) Evaluated in 0.55 seconds
2015-08-25 18:22:28 NOTICE
 (/Stage[main]/Main/Exec[refresh-metadata-agent]/returns) executed successfully
2015-08-25 18:22:27 INFO
 (/Stage[main]/Main/Exec[refresh-metadata-agent]) Starting to evaluate the resource
2015-08-25 18:22:27 INFO
 (/Stage[main]/Main/Exec[refresh-dhcp-agent]) Evaluated in 0.73 seconds
2015-08-25 18:22:27 NOTICE
 (/Stage[main]/Main/Exec[refresh-dhcp-agent]/returns) executed successfully
2015-08-25 18:22:26 INFO
 (/Stage[main]/Main/Exec[refresh-dhcp-agent]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 (Class[Settings]) Evaluated in 0.00 seconds
2015-08-25 18:22:26 INFO
 (Class[Settings]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 (Class[Settings]) Evaluated in 0.01 seconds
2015-08-25 18:22:26 INFO
 (Class[Settings]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 (/Stage[main]/Main/Service[neutron-server]) Evaluated in 0.06 seconds
2015-08-25 18:22:26 INFO
 (/Stage[main]/Main/Service[neutron-server]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 (Class[Main]) Evaluated in 0.00 seconds
2015-08-25 18:22:26 INFO
 (Class[Main]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 (Stage[main]) Evaluated in 0.00 seconds
2015-08-25 18:22:26 INFO
 (Stage[main]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 (/Schedule[hourly]) Evaluated in 0.00 seconds
2015-08-25 18:22:26 INFO
 (/Schedule[hourly]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 (/Schedule[monthly]) Evaluated in 0.00 seconds
2015-08-25 18:22:26 INFO
 (/Schedule[monthly]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 (/Schedule[daily]) Evaluated in 0.00 seconds
2015-08-25 18:22:26 INFO
 (/Schedule[daily]) Starting to evaluate the resource
2015-08-25 18:22:26 INFO
 Applying configuration version '1440526931'

here's the support snapshot with all the logs:
https://drive.google.com/file/d/0B7EJjYvFX-E7b0lhaFZZaktBTDQ/view?usp=sharing

[Jaume Devesa (devvesa)]

Hi Kevin,

I've seen the error comes from the MidoNet API, which seems to not be spawned. Unfortunately, there is no logs on the files you provided me. Could you, by any chance, attach the

`/var/log/tomcat/midonet-api.log` files from all the controller nodes?

Many thanks.

[Jaume Devesa (devvesa)]

Heads up that puppet does not complain about tomcat in any of the controller nodes:

Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): Augeas version 1.0.0 is installed
147914 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): Will attempt to save and only run if files changed
147915 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): sending command 'set' with params ["/files/usr/share/tomcat/conf/server.xml/Server/Service[#attribute/name='Catalina'] /Connector[#attribute/port='8081']/#attribute/port", "8081"]
147916 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): sending command 'set' with params ["/files/usr/share/tomcat/conf/server.xml/Server/Service[#attribute/name='Catalina'] /Connector[#attribute/port='8081']/#attribute/protocol", "HTTP/1.1"]
147917 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): sending command 'set' with params ["/files/usr/share/tomcat/conf/server.xml/Server/Service[#attribute/name='Catalina'] /Connector[#attribute/port='8081']/#attribute/URIEncoding", "UTF-8"]
147918 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): sending command 'set' with params ["/files/usr/share/tomcat/conf/server.xml/Server/Service[#attribute/name='Catalina'] /Connector[#attribute/port='8081']/#attribute/redirectPort", "8443"]
147919 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): sending command 'set' with params ["/files/usr/share/tomcat/conf/server.xml/Server/Service[#attribute/name='Catalina'] /Connector[#attribute/port='8081']/#attribute/maxHttpHeaderSize", "65536"]
147920 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): sending command 'set' with params ["/files/usr/share/tomcat/conf/server.xml/Server/Service[#attribute/name='Catalina'] /Connector[#attribute/port='8081']/#attribute/address", "10.7.214.12"]
147921 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): sending command 'set' with params ["/files/usr/share/tomcat/conf/server.xml/Server/Service[#attribute/name='Catalina'] /Connector[#attribute/port='8081']/#attribute/connectionTimeout", "20000"]
147922 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): Skipping because no files were changed
147923 Tue Aug 25 18:10:42 +0000 2015 Augeas[server-/usr/share/tomcat-Catalina-connector-8081](provider=augeas) (debug): Closed the augeas connection
147924 Tue Aug 25 18:10:42 +0000 2015 /Stage[main]/Midonet::Midonet_api::Run/Tomcat::Config::Server::Connector[HTTP/1.1]/Augeas[server-/usr/share/tomcat-Catalina-connector-8081] (info): Evaluated in 0.08 seconds
147925 Tue Aug 25 18:10:42 +0000 2015 /Stage[main]/Midonet::M...

[Kevin Scheunemann (kvvin)]

Attached are all the tomcat logs from all the controller nodes.
Looks like there's an error with a missing keystone:

2015.08.26 11:25:47.894 WARN KeystoneClient - KeystoneClient: keystone login creds not found auth: { tenantName: admin, passwordCredentials: { username: admin } }
2015.08.26 11:25:47.895 ERROR LoginFilter - LoginFilter: auth error occurred.
com.sun.jersey.api.client.UniformInterfaceException: POST http://10.7.214.2:35357/v2.0/tokens returned a response status of 401 Unauthorized

[Jaume Devesa (devvesa)]

Yes, sure. This has to be the problem. We are logging with a wrong user/password for sure. From the fuel plugin we call the midonet api using the astute config files:

class {'::midonet::midonet_api':
  zk_servers => $zoo_ips,
  keystone_auth => true,
  keystone_host => $::fuel_settings['management_vip'],
  keystone_admin_token => $::fuel_settings['keystone']['admin_token'],
  bind_address => $::ipaddress_br_mgmt,
  api_ip => $::fuel_settings['public_vip'],
  api_port => '8081'
}

so maybe the problem resides on the midonet puppet module.

[Kevin Scheunemann (kvvin)]

Yeah I just traced that down as well,
I verified that /usr/share/midonet-api/WEB-INF/web.xml has the correct keystone information:
The admin token below matches the one in /etc/keystone/keystone.conf
I even tried restarting the tomcat service and re-running the midonet-restart-neutron.pp manifest with the same errors in the logs.

    <!-- Keystone configuration -->
  <context-param>
    <param-name>keystone-service_protocol</param-name>
    <param-value>http</param-value>
  </context-param>
  <context-param>
    <param-name>keystone-service_host</param-name>
    <param-value>10.7.214.2</param-value>
  </context-param>
  <context-param>
    <param-name>keystone-service_port</param-name>
    <param-value>35357</param-value>
  </context-param>
  <context-param>
    <param-name>keystone-admin_token</param-name>
    <param-value>PnlhOhgK</param-value>
  </context-param>
  <!-- This tenant name is used to get the scoped token from Keystone, and
       should be the tenant name of the user that owns the token sent in the
       request -->
  <context-param>
    <param-name>keystone-tenant_name</param-name>
    <param-value>admin</param-value>
  </context-param>

[Jaume Devesa (devvesa)]

Sorry to bother you again, Kevin. Can I see the contents of the file:

/usr/share/midonet-api/WEB-INF/web.xml

please?

[Kevin Scheunemann (kvvin)]

attached is the web.xml file for midonet-api