jenkins master sometimes fails to add slave node
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
jenkins (Juju Charms Collection) |
New
|
Undecided
|
Unassigned |
Bug Description
The addnode code often executes before the jenkins master service is fully started, resulting in:
hook failed: "master-
Check & wait logic is needed there.
#### deployer fail
2014-10-01 14:44:47 [DEBUG] deployer.env: Connected to environment
2014-10-01 14:44:47 [INFO] deployer.import: Adding relations...
2014-10-01 14:44:47 [INFO] deployer.import: Adding relation osci:master <-> osci-slave:slave
2014-10-01 14:44:48 [DEBUG] deployer.import: Waiting for relation convergence 60s
2014-10-01 14:45:48 [ERROR] deployer.env: The following units had errors:
unit: osci/0: machine: 1 agent-state: error details: hook failed: "master-
2014-10-01 14:45:48 [INFO] deployer.cli: Deployment stopped. run time: 539.98
Traceback (most recent call last):
File "./deploy.py", line 632, in <module>
main()
File "./deploy.py", line 541, in main
'-c', deployer_bundle])
File "/usr/lib/
raise CalledProcessEr
#### juju log
unit-osci-0: 2014-10-01 14:44:53 INFO master-
unit-osci-0: 2014-10-01 14:44:53 INFO master-
unit-osci-0: 2014-10-01 14:44:53 INFO master-
unit-osci-0: 2014-10-01 14:44:53 INFO master-
unit-osci-0: 2014-10-01 14:44:53 INFO master-
unit-osci-0: 2014-10-01 14:44:53 INFO master-
unit-osci-0: 2014-10-01 14:44:53 INFO master-
unit-osci-0: 2014-10-01 14:44:53 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1
.
Digging a little deeper, simulating and re-creating: I restarted the master jenkins service, then immediately ran these commands, which expose 2 race scenarios:
#### jenkins master service is not started, no socket
ubuntu@
<jenkins.Jenkins object at 0x7fa96719c7d0>
Traceback (most recent call last):
File "./addnode", line 16, in <module>
if l_jenkins.
File "/usr/lib/
self.
File "/usr/lib/
response = self.jenkins_
File "/usr/lib/
return urllib2.
File "/usr/lib/
return _opener.open(url, data, timeout)
File "/usr/lib/
response = self._open(req, data)
File "/usr/lib/
'_open', req)
File "/usr/lib/
result = func(*args)
File "/usr/lib/
return self.do_
File "/usr/lib/
r = h.getresponse(
File "/usr/lib/
response.
File "/usr/lib/
version, status, reason = self._read_status()
File "/usr/lib/
line = self.fp.readline()
File "/usr/lib/
data = self._sock.
socket.error: [Errno 104] Connection reset by peer
#### jenkins master service is started, but not ready
ubuntu@
<jenkins.Jenkins object at 0x7f7b640587d0>
Adding node to Jenkins master
Traceback (most recent call last):
File "./addnode", line 20, in <module>
l_jenkins.
File "/usr/lib/
raise JenkinsExceptio
jenkins.
The linked branch adds wait/check loop logic.