i/o timeout errors can cause non-atomic service deploys

[Данило Шеган]

We've recently started seeing "i/o timeout errors" when issuing serviceDeploy API calls from Landscape:

EDIT
##########
To clarify:

landscape ---------> |17070:state server -> mongo:37017|

The timeout is happening inside the juju state server box, between the api and mongo.

It's NOT happening between landscape and juju. It's inside juju.
##########

Aug 7 20:45:06 job-handler-1 INFO Traceback (failure with no frames): <class 'canonical.juju.errors.RequestError'>: cannot add service "mongodb": read tcp 127.0.0.1:37017: i/o timeout
...
Aug 7 20:45:06 job-handler-1 INFO Traceback (failure with no frames): <class 'canonical.juju.errors.RequestError'>: cannot add service "hacluster-mysql": read tcp 192.168.216.21:37017: i/o timeout
...

On retry, we hit the "service already exists" error, so the service has actually been deployed.

However, recently, we've also hit a case where service was only partially deployed:

Aug 17 16:22:18 job-handler-1 INFO Traceback (failure with no frames): <class 'canonical.juju.errors.RequestError'>: cannot add
service "ceph-radosgw": read tcp 127.0.0.1:37017: i/o timeout

We emit a service deploy call together with the full service configuration, but no service configuration was done in this case. It seems as if service deploy and configuration are not atomic: we'd expect the call to either fail (so we don't hit service-already-exists error later) or to succeed *fully* (including configuration) even if it fails to report back to the client due to network issues.

See bug 1482791 for reference.

[Dimiter Naydenov (dimitern)]

It is not atomic indeed - it can fail mid-way for various reasons, like "charm not found" (which leaves the service "empty" but existing).

[Curtis Hovey (sinzui)]

1.22.8 and 1.22.6 was tested and failed

[Dimiter Naydenov (dimitern)]

Dropped the milestore as it was not set correctly: there are no plans (that I know of) to fix it by the 1.26.0 release.
Consider that to fix the issue properly we need to have a (cooperating) persistency layer with a reliable support for transactions across multiple collections, including the ability to rollback such transactions.

[Free Ekanayaka (free.ekanayaka)]

@Dimiter: out of curiosity, I guess the "cooperating persistency layer" you are alluding to is the juju/state package?

Because the underlying technologies (mongodb + mgo/txn) do seem to have reliable support for transactions across multiple collections, afaiu. The problem seems a "factoring" one in juju/state which afaics basically ends up doing:

RunTransaction(<ADD SERVICE OPS>)
RunTransaction(<ATTACH FETCHED CHARM TO SERVICE OPS>
RunTransaction(<SET SERVICE CONFIG OPS>)

as opposed to:

RunTransaction(
    <ADD SERVICE OPS> + <ATTACH FETCHED CHARM TO SERVICE OPS> + <SET SERVICE CONFIG OPS>)

Is this the case?

[Nate Finch (natefinch)]

I'm looking into this one now.

[Nate Finch (natefinch)]

Note that users have sometimes experienced the timeout causing the service's settings set at deploy time to not actually get set.

[Menno Finlay-Smits (menno.smits)]

It's worth noting that MongoDB i/o timeout errors always happen after the ensure-availability has been issued. As each new member of the replicaset comes up MongoDB breaks all connections to it, causing these errors. Juju should handle this better.

Could that be what's happening here? Was the replicaset coming up when the deploy was requested?

[David Britton (davidpbritton)]

On Sunday, September 6, 2015, Menno Smits <email address hidden> wrote:
>
> Could that be what's happening here? Was the replicaset coming up when
> the deploy was requested?
>
>
In this case, no. Non HA juju.

--
David Britton <email address hidden>

[Nate Finch (natefinch)]

FYI, I have a change that adds service settings to the same transaction as the service creation, so there's no possible failure in between the two. Now working on getting unit creation into the same transaction. Assigning units to machines is going to be done asynchronously by a worker to ensure retries in the case of timeouts etc.

[Free Ekanayaka (free.ekanayaka)]

Great to see service creation and config atomic. FWIW Landscape sets NumUnits to 0 when deploying a service (and the add units later with a separate API call), so atomicity of the unit creation part is not critical for our use case.

[Nate Finch (natefinch)]

The first half of this bug is landing in 1.24 today (the part about making sure the settings get applied to the service atomically with service deployment).

[Nate Finch (natefinch)]

And this has now landed in 1.24.

[Nate Finch (natefinch)]

The second half of this bug - making unit assignment atomic(ish) is being tracked here: https://bugs.launchpad.net/juju-core/+bug/1497312

[Cheryl Jennings (cherylj)]

Nate - has this landed in master?