Comment 31 for bug 1597601

Hey Menno,

Sorry, I was confused on which bug this was - we won't be able to take non
released builds for this. We hit this running in our general pipeline - it
takes many deployments to ever reproduce it - and we only run released
builds in that.

Jason

On Mon, Sep 19, 2016 at 5:33 AM, Menno Smits <email address hidden>
wrote:

> I can now reproduce the problem more accurately now, if still somewhat
> artificially.
>
> In the latest log tarball I had noticed that mongodb logs in
> /var/log/syslog stopped for the period between 06:28:15 and 06:28:40.
> It's during this period where the machine-0 jujud logged tons of i/o
> timeout errors. It's almost like mongodb got stuck for a while or was
> heavily loaded for some reason (although I see no reason why this should
> be so).
>
> This gave me the idea to try pausing the mongod process using SIGSTOP.
> By doing this I'm able to cause jujud to experience the same i/o timeout
> errors. By pausing the mongod process at the right place inside a Juju's
> API handler I'm able to get the i/o timeouts errors at the client as
> reported. I suspect that for some reason mongod is occasionally getting
> stuck/busy for a while and this is what's cause the i/o timeout errors
> in OIL. MongoDB's leader election (which typically happens soon after
> bootstrapping the controller) are certainly another cause, but that's
> not happened in the the latest logs attached to this ticket.
>
> Being able to do this will speed up the development of a fix/workaround.
> I'd still like it to be tested within the OIL setup once I have it ready
> though.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1597601
>
> Title:
> ERROR cannot deploy bundle: cannot deploy application: i/o timeout
>
> Status in juju:
> In Progress
>
> Bug description:
> Running this command:
> juju-2.0 deploy -m maas:default artifacts/oil_deployment
>
> I got this output:
> added charm cs:trusty/cinder-253
> ERROR cannot deploy bundle: cannot deploy application "cinder": cannot
> add application "cinder": read tcp 127.0.0.1:54084->127.0.0.1:37017: i/o
> timeout
>
> I've attached the machine-0 log.
>
> From this build in OIL:
> http://10.245.162.43:8080/job/pipeline_deploy/6818/
>
> This is with 2.0-beta10 and 2.0beta11.
>
> We hit this a couple of times a day in OIL.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1597601/+subscriptions
>