Performed some live testing during deploy time that highlights the race condition.

After, the first gnocchi unit advertises itself as ready to the ceilometer the other units are still in progress. The SSL version of the test shows this quite clearly. Hitting the VIP which then gets sent to HAProxy and an indeterminate gnocchi back end, some back ends are ready and respond and some have not even setup SSL yet:

$ wget --no-check-certificate -O-  https://10.5.150.191:8041                                                                                 
--2018-02-26 10:46:47--  https://10.5.150.191:8041/                             
Connecting to 10.5.150.191:8041... connected.                                   
WARNING: cannot verify 10.5.150.191's certificate, issued by ‘CN=Ubuntu Cloud Intermediate Certificate Authority,OU=Ubuntu Cloud,O=Ubuntu’:                      
  Unable to locally verify the issuer's authority.                              
HTTP request sent, awaiting response... 200 OK                                  
Length: 144 [application/json]          
Unable to establish SSL connection.     

$ wget --no-check-certificate -O-  https://10.5.150.191:8041                                                                                 
--2018-02-26 10:49:03--  https://10.5.150.191:8041/                             
Connecting to 10.5.150.191:8041... connected.                                   
Saving to: ‘STDOUT’

$ wget --no-check-certificate -O-  https://10.5.150.191:8041  
--2018-02-26 10:49:22--  https://10.5.150.191:8041/                             
Connecting to 10.5.150.191:8041... connected.                                   
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol    
Unable to establish SSL connection.     

$ wget --no-check-certificate -O-  https://10.5.150.191:8041                                                                                 
--2018-02-26 10:49:24--  https://10.5.150.191:8041/                             
Connecting to 10.5.150.191:8041... connected.                                   
Saving to: ‘STDOUT’                     
                     
$ wget --no-check-certificate -O-  https://10.5.150.191:8041  
--2018-02-26 10:49:44--  https://10.5.150.191:8041/                             
Connecting to 10.5.150.191:8041... connected.                                   
WARNING: cannot verify 10.5.150.191's certificate, issued by ‘CN=Ubuntu Cloud Intermediate Certificate Authority,OU=Ubuntu Cloud,O=Ubuntu’:                      
  Unable to locally verify the issuer's authority.                              
HTTP request sent, awaiting response... 200 OK                                  
Length: 144 [application/json]          
Saving to: ‘STDOUT’                     

2018-02-26 10:49:46 (11.4 MB/s) - written to stdout [144/144]                   

$ wget --no-check-certificate -O-  https://10.5.150.191:8041  
--2018-02-26 10:49:48--  https://10.5.150.191:8041/                             
Connecting to 10.5.150.191:8041... connected.                                   
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol    
Unable to establish SSL connection.     

$ wget --no-check-certificate -O-  https://10.5.150.191:8041                                                                                 
--2018-02-26 10:50:06--  https://10.5.150.191:8041/                             
Connecting to 10.5.150.191:8041... connected.                                   
WARNING: cannot verify 10.5.150.191's certificate, issued by ‘CN=Ubuntu Cloud Intermediate Certificate Authority,OU=Ubuntu Cloud,O=Ubuntu’:                      
  Unable to locally verify the issuer's authority.                              
HTTP request sent, awaiting response... 200 OK                                  
Length: 144 [application/json]          
Saving to: ‘STDOUT’                     

2018-02-26 10:50:06 (15.5 MB/s) - written to stdout [144/144]


This seems to indicate we are in a losing fight against race conditions during deploy time. Options as I see them.

1) Run ceilometer-upgrade as an action post deploy
2) Update the gnocchi interface on the provides side and charm such that the gnocchi_url is not advertised on the metric-service relation until all units are ready.
3) Update the gnocchi interface on the requires side (as well as classic charms like ceilometer) to not set metric-service.available until all units have advertised their readiness.