Deployment fails when Memcached and SwiftProxy run on different nodes

[Marius Cornea]

When swift::proxy::cache is used without also including memcached in the catalog, it fails even if remote memcache server ips are provided.

Deployment fails with the following errors:

   Error: Could not find resource 'Class[Memcached]' for relationship on 'Class[Swift::Proxy::Cache]' on node overcloud-serviceapi-1.localdomain
    Error: Could not find resource 'Class[Memcached]' for relationship on 'Class[Swift::Proxy::Cache]' on node overcloud-serviceapi-1.localdomain

    Error: Could not find resource 'Class[Memcached]' for relationship on 'Class[Swift::Proxy::Cache]' on node overcloud-serviceapi-0.localdomain
    Error: Could not find resource 'Class[Memcached]' for relationship on 'Class[Swift::Proxy::Cache]' on node overcloud-serviceapi-0.localdomain

    Error: Could not find resource 'Class[Memcached]' for relationship on 'Class[Swift::Proxy::Cache]' on node overcloud-serviceapi-2.localdomain
    Error: Could not find resource 'Class[Memcached]' for relationship on 'Class[Swift::Proxy::Cache]' on node overcloud-serviceapi-2.localdomain

[Alex Schultz (alex-schultz)]

This is probably a bug in upstream since this is defined in swift::proxy::cache.

[Alex Schultz (alex-schultz)]

Actually, the upstream only ensures memcache if the server list includes localhost so it might be a bug in the memcache ip calculations.

https://github.com/openstack/puppet-swift/blob/master/manifests/proxy/cache.pp#L27-L30

[Marius Cornea (mcornea)]

Attaching the tht templates that I used. This is the deploy command:

source ~/stackrc
export THT=/home/stack/templates/tripleo-heat-templates
openstack overcloud deploy --templates $THT \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e ~/templates/network-environment.yaml \
-e $THT/environments/storage-environment.yaml \
-e ~/templates/disk-layout.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
--control-scale 3 \
--control-flavor controller \
--compute-scale 1 \
--compute-flavor compute \
--ceph-storage-scale 1 \
--ceph-storage-flavor ceph \
--ntp-server clock.redhat.com

and the network-environment.yaml file:

resource_registry:
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
  OS::TripleO::CephStorage::Net::SoftwareConfig: /home/stack/templates/nic-configs/ceph-storage.yaml

  OS::TripleO::ServiceApi::Net::SoftwareConfig: /home/stack/templates/nic-configs/serviceapi.yaml
  OS::TripleO::ServiceApi: /home/stack/templates/tripleo-heat-templates/puppet/serviceapi.yaml
  OS::TripleO::ServiceApi::Ports::ExternalPort: /home/stack/templates/tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::ServiceApi::Ports::InternalApiPort: /home/stack/templates/tripleo-heat-templates/network/ports/internal_api.yaml
  OS::TripleO::ServiceApi::Ports::StoragePort: /home/stack/templates/tripleo-heat-templates/network/ports/storage.yaml
  OS::TripleO::ServiceApi::Ports::StorageMgmtPort: /home/stack/templates/tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::ServiceApi::Ports::TenantPort: /home/stack/templates/tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::ServiceApi::Ports::ManagementPort: /home/stack/templates/tripleo-heat-templates/network/ports/management.yaml
  OS::TripleO::Tasks::ServiceApiPostConfig: OS::Heat::None
  OS::TripleO::Tasks::ServiceApiPreConfig: OS::Heat::None
  OS::TripleO::ServiceApiConfig: /home/stack/templates/tripleo-heat-templates/puppet/serviceapi-config.yaml

parameter_defaults:
  ServiceApiCount: 3
  OvercloudServiceApiFlavor: serviceapi
  ServiceNetMap:
    ServiceApiHostnameResolveNetwork: internal_api

  InternalApiNetCidr: 10.0.0.0/25
  InternalApiAllocationPools: [{'start': '10.0.0.10', 'end': '10.0.0.100'}]
  InternalApiNetworkVlanID: 200

  StorageNetCidr: 10.0.0.128/25
  StorageAllocationPools: [{'start': '10.0.0.138', 'end': '10.0.0.200'}]
  StorageNetworkVlanID: 300

  StorageMgmtNetCidr: 10.0.1.0/25
  StorageMgmtAllocationPools: [{'start': '10.0.1.10', 'end': '10.0.1.100'}]
  StorageMgmtNetworkVlanID: 301

  TenantNetCidr: 10.0.1.128/25
  TenantAllocationPools: [{'start': '10.0.1.138', 'end': '10.0.1.200'}]

  ManagementNetCidr: 172.16.17.128/25
  ManagementAllocationPools: [{'start': '172.16.17.160', 'end': '172.16.17.200'}]
  ManagementInterfaceDefaultRoute: 172.16.17.254

  ExternalNetCidr: 172.16.18.0/25
  ExternalAllocationPools: [{'start': '172.16.18.25', 'end': '172.16.18.100'}]
  ExternalInterfaceDefaultRoute: 172.16.18.126
  ExternalNetworkVlanID: 100

  ControlPlaneSubnetCidr: "25"
  Cont...

[Alex Schultz (alex-schultz)]

Just to confirm this is an issue with the puppet module because it's not properly determining if 127.0.0.1 is in the server list.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379796

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379796
Committed: https://git.openstack.org/cgit/openstack/puppet-swift/commit/?id=fbbcd892358d47f5734adcefd5406146a6f33d0e
Submitter: Jenkins
Branch: master

commit fbbcd892358d47f5734adcefd5406146a6f33d0e
Author: Alex Schultz <email address hidden>
Date: Thu Sep 29 14:08:07 2016 -0600

    Fixed swift::proxy::cache requirement on memcached

    Fixed the incorrect inclusion of a requirement for the memcached class
    to be in the catalog for swift::proxy::cache due to a bad grep of
    the memcached server list. The grep command will return an array of
    matching values. Unfortunately this includes if nothing is found so the
    if condition was always matching. Now if you are not using a local
    memcache server on 127.0.0.1, the memcached class is not required to
    be in the catalog.

    Change-Id: I79b92dc67e46dfa0b62ba07b00633de6fb17bf1e
    Closes-Bug: #1628967

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/380370

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380370
Committed: https://git.openstack.org/cgit/openstack/puppet-swift/commit/?id=3ae58f377b48668d7c3e113a8cbe269f12fa2fd2
Submitter: Jenkins
Branch: stable/newton

commit 3ae58f377b48668d7c3e113a8cbe269f12fa2fd2
Author: Alex Schultz <email address hidden>
Date: Thu Sep 29 14:08:07 2016 -0600

    Fixed swift::proxy::cache requirement on memcached

    Fixed the incorrect inclusion of a requirement for the memcached class
    to be in the catalog for swift::proxy::cache due to a bad grep of
    the memcached server list. The grep command will return an array of
    matching values. Unfortunately this includes if nothing is found so the
    if condition was always matching. Now if you are not using a local
    memcache server on 127.0.0.1, the memcached class is not required to
    be in the catalog.

    Change-Id: I79b92dc67e46dfa0b62ba07b00633de6fb17bf1e
    Closes-Bug: #1628967
    (cherry picked from commit fbbcd892358d47f5734adcefd5406146a6f33d0e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-swift 9.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-swift 10.0.0 release.