ppc64el - jujud: Syntax error: word unexpected (expecting ")")
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | juju-core |
High
|
Andrew Wilkins | ||
| | 1.22 |
High
|
Andrew Wilkins | ||
Bug Description
jujud: Syntax error: word unexpected (expecting ")")
When deploying to ppc64el-hosted lxc containers, with the bootstrap node being amd64, all lxc units timeout in a pending state.
Inspection of juju unit logs in each lxc container show only the following in each:
/var/lib/
See http://
The use case is:
We have one service which needs to be on amd64, one service which needs to be on ppc64el natively, and several services which need to be in lxc containers on the ppc64el host. Namely, using ppc64el as the nova-compute and OpenStack api services node, and the amd64 node for bootstrap + neutron-gateway.
The same bundle and charms deploy successfully when both hosts are amd64.
See attached additional info.
This scenario work in 1.20.x
| Ryan Beisner (1chb1n) wrote : | #1 |
| Ryan Beisner (1chb1n) wrote : | #2 |
| Ryan Beisner (1chb1n) wrote : | #3 |
FYI, for continuity from #juju-dev conversation:
<davecheney> [18:30:16] best I can tell the error is coming from upstart
<davecheney> [18:34:44] can you make an issue and attach the raw file from upstart
<davecheney> [18:34:46] the one you pasted
<davecheney> [18:34:50] it has to be the raw file
<davecheney> [18:35:06] i suspect there is some control characters or other whitespace crap in there that is throwing off upstart
| Ryan Beisner (1chb1n) wrote : | #5 |
Oh sorry, forgot that important detail. This is from the perspective of the admin machine which performs the juju bootstrap and deploy.
ubuntu@
juju:
Installed: 1.21.1-
Candidate: 1.21.1-
Version table:
*** 1.21.1-
500 http://
100 /var/lib/
1.
500 http://
1.
500 http://
| Changed in juju-core: | |
| status: | New → Triaged |
| importance: | Undecided → High |
| milestone: | none → 1.23 |
| tags: | added: regression |
| summary: |
- jujud: Syntax error: word unexpected (expecting ")") + ppc64el - jujud: Syntax error: word unexpected (expecting ")") |
| Ryan Beisner (1chb1n) wrote : | #6 |
Re-tested with juju/devel. The same error occurs in the ppc64el-hosted lxc container unit logs.
ubuntu@
juju:
Installed: 1.22-beta2-
Candidate: 1.22-beta2-
Version table:
*** 1.22-beta2-
500 http://
100 /var/lib/
1.
500 http://
1.
500 http://
1.
500 http://
| description: | updated |
| Tim Penhey (thumper) wrote : | #7 |
On the machine that is hosting the lxc containers, there is a directory "/var/lib/
There will be a directory for each container. That directory contains the cloud-init file and the logging from the lxc commands. Can you please attach those files?
Thanks.
| Ryan Beisner (1chb1n) wrote : | #8 |
@thumper The environment is torn down at the moment, will re-deploy and collect tomorrow.
| Ryan Beisner (1chb1n) wrote : | #9 |
Redeployed with juju/stable, collected /var/lib/
| Ryan Beisner (1chb1n) wrote : | #10 |
Also attaching /var/log/juju. Thanks again.
| Ryan Beisner (1chb1n) wrote : | #11 |
^ Both from the ppc64-el host of course.
| Tim Penhey (thumper) wrote : | #12 |
OK, looking at those logs it is clear that something is screwed up but I can't quit tell what.
cloud-init is fine, except that the juju machine agent is repeatedly failing
AFAICT the upstart script is correct, but I'd like try some things.
Can you ping me on IRC around start of day NZ time? I'm on from about 2000UTC
| Tim Penhey (thumper) wrote : | #13 |
Actually it looks like the LXC container is being started with amd64 tools, not power ones. This is an issue.
| Dave Cheney (dave-cheney) wrote : Re: [Bug 1420049] Re: ppc64el - jujud: Syntax error: word unexpected (expecting ")") | #14 |
That is because machine-0 is amd64, tools selection must be fixated on
that version even though constraints force the provider to create
ppc64le machines.
On Thu, Feb 12, 2015 at 12:22 PM, Tim Penhey <email address hidden> wrote:
> Actually it looks like the LXC container is being started with amd64
> tools, not power ones. This is an issue.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> Matching subscriptions: MOAR JUJU SPAM!
> https:/
>
> Title:
> ppc64el - jujud: Syntax error: word unexpected (expecting ")")
>
> Status in juju-core:
> Triaged
> Status in juju-core 1.21 series:
> Triaged
> Status in juju-core 1.22 series:
> Triaged
>
> Bug description:
> jujud: Syntax error: word unexpected (expecting ")")
>
> When deploying to ppc64el-hosted lxc containers, with the bootstrap
> node being amd64, all lxc units timeout in a pending state.
>
> Inspection of juju unit logs in each lxc container show only the following in each:
> /var/lib/
>
> See http://
>
> The use case is:
> We have one service which needs to be on amd64, one service which needs to be on ppc64el natively, and several services which need to be in lxc containers on the ppc64el host. Namely, using ppc64el as the nova-compute and OpenStack api services node, and the amd64 node for bootstrap + neutron-gateway.
>
> The same bundle and charms deploy successfully when both hosts are
> amd64.
>
> See attached additional info.
>
> To manage notifications about this bug go to:
> https:/
| no longer affects: | juju-core/1.21 |
| Curtis Hovey (sinzui) wrote : | #15 |
Does this scenario work with 1.20.x?
| Ryan Beisner (1chb1n) wrote : | #16 |
This use case does indeed succeed with juju 1.20.x. See attached txt. Will also attach new tarballs.
| Ryan Beisner (1chb1n) wrote : | #17 |
| Ryan Beisner (1chb1n) wrote : | #18 |
| description: | updated |
| Dimiter Naydenov (dimitern) wrote : | #19 |
An important log seems to be missing here - machine-0.log from the amd64 machine showing provisioning failures and what tools are selected, but also it will help to set logging-config: <root>=TRACE in environments.yaml and reproduce the issue, then the logs will be much more useful (now machine-1.log seems to be mostly empty, because logging-config is set to <root>=WARNING as soon as it starts).
Regarding the second case (1.20.x) - your juju client binary is 1.20.11, but your agent versions all show 1.20.14, which most likely means you've used --metadata-source or --upload-tools or something like that as arguments to juju bootstrap. If you're using custom images and metadata, then it's likely there's a issue with them which lead to selection of the wrong tools.
| Curtis Hovey (sinzui) wrote : | #20 |
I think the mismatched 1.20.x versions was caused by juju client's optimistic behaviour to select the highest micro version of jujud. Since 1.20.14 is the last of the 1.20's, anyone bootstrapping with any 1.20 will get it when working with public streams.
| Ryan Beisner (1chb1n) wrote : | #21 |
FYI - The commands for reproducing are in the attachment https:/
Should we redeploy and collect machine-0.log? If so, which juju version should we use?
| Dimiter Naydenov (dimitern) wrote : | #22 |
The issue should be fixed in 1.22 using https:/
| Ryan Beisner (1chb1n) wrote : | #23 |
Tested with juju-core 1.22-beta4-
Units no longer get stuck in 'pending' state; they are now getting stuck in an 'allocating' state.
The unit log for lxc machines now shows:
ubuntu@
/var/lib/
/var/lib/
/var/lib/
Let me know if we should provide any other info. The environment is still up as shown.
Thanks again!
| Andrew Wilkins (axwalk) wrote : | #24 |
Ryan, it's not enough to use the updated client; Juju is still locating the old beta3 tools when bootstrapping and adding machines. You can check this by looking at the "agent-version" in the latest "juju status" output.
I guess CI hasn't published beta4 to the devel stream yet because of the test failure that Dimiter mentioned above.
| Changed in juju-core: | |
| assignee: | nobody → James Tunnicliffe (dooferlad) |
| Changed in juju-core: | |
| status: | Triaged → In Progress |
| Changed in juju-core: | |
| status: | In Progress → Fix Committed |
| status: | Fix Committed → In Progress |
| Dimiter Naydenov (dimitern) wrote : | #25 |
Landed on trunk with https:/
| Changed in juju-core: | |
| status: | In Progress → Fix Committed |
| Ryan Beisner (1chb1n) wrote : | #26 |
The issue persists in 1.22beta4:
juju-core:
Installed: 1.22-beta4-
Candidate: 1.22-beta4-
Version table:
*** 1.22-beta4-
500 http://
100 /var/lib/
1.
500 http://
1.
500 http://
1.
500 http://
environment: maas
machines:
"0":
agent-state: started
agent-version: 1.22-beta3
dns-name: innocent-
instance-id: /MAAS/api/
series: trusty
containers:
0/lxc/0:
dns-name: 10.245.172.254
series: trusty
hardware: arch=amd64
hardware: arch=amd64 cpu-cores=8 mem=8192M
state-
"1":
agent-state: started
agent-version: 1.22-beta3
dns-name: gregory-
instance-id: /MAAS/api/
series: trusty
containers:
1/lxc/0:
series: trusty
hardware: arch=ppc64el
1/lxc/1:
series: trusty
hardware: arch=ppc64el
1/lxc/2:
series: trusty
hardware: arch=ppc64el
1/lxc/3:
series: trusty
hardware: arch=ppc64el
hardware: arch=ppc64el cpu-cores=152 mem=130552M
services:
keystone-
charm: cs:trusty/
exposed: false
relations:
cluster:
- keystone-
units:
keystone-
machine: 1/lxc/2
mongodb:
charm: cs:trusty/
exposed: false
relations:
replica-set:
- mongodb
units:
mongodb/0:
machine: "1"
open-ports:
- 27017/tcp
- 27019/tcp
- 27021/tcp
- 28017/tcp
mysql:
charm: cs:trusty/mysql-20
exposed: false
relations:
cluster:
- mysql
units:
mysql/0:
machine: "0"
mysql-
charm: cs:trusty/mysql-20
exposed: false
relations:
cluster:
- mysql-lxc-ppc64el
units:
mysql-
machine: 1/lxc/1
nfs-lxc-amd64:
charm: cs:trusty/nfs-0
exposed: ...
| Ryan Beisner (1chb1n) wrote : | #27 |
Ahh bugger. I see the agent is still 1.22beta3. Why is that?
| Andrew Wilkins (axwalk) wrote : | #28 |
Did you try setting "agent-stream: devel" in your environ config? I believe that is necessary to pick up the right agent version. Not sure how you'd get beta3 without it though.
| Ryan Beisner (1chb1n) wrote : | #29 |
Yes, the enviro definition is as follows:
maas:
type: maas
maas-server: 'http://
maas-oauth: XXXXXX
admin-secret: XXXXXX
agent-stream: devel
On Thu, Feb 26, 2015 at 7:14 PM, Andrew Wilkins <
<email address hidden>> wrote:
> Did you try setting "agent-stream: devel" in your environ config? I
> believe that is necessary to pick up the right agent version. Not sure
> how you'd get beta3 without it though.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> ppc64el - jujud: Syntax error: word unexpected (expecting ")")
>
> Status in juju-core:
> Fix Committed
> Status in juju-core 1.22 series:
> Fix Released
>
> Bug description:
> jujud: Syntax error: word unexpected (expecting ")")
>
> When deploying to ppc64el-hosted lxc containers, with the bootstrap
> node being amd64, all lxc units timeout in a pending state.
>
> Inspection of juju unit logs in each lxc container show only the
> following in each:
> /var/lib/
> /var/lib/
> (expecting ")")
>
> See http://
>
> The use case is:
> We have one service which needs to be on amd64, one service which needs
> to be on ppc64el natively, and several services which need to be in lxc
> containers on the ppc64el host. Namely, using ppc64el as the nova-compute
> and OpenStack api services node, and the amd64 node for bootstrap +
> neutron-gateway.
>
> The same bundle and charms deploy successfully when both hosts are
> amd64.
>
> See attached additional info.
>
> This scenario work in 1.20.x
>
> To manage notifications about this bug go to:
> https:/
>
| Ryan Beisner (1chb1n) wrote : | #30 |
FWIW, during bootstrap, I'm seeing this:
WARNING failed to find 1.22-beta4 tools, will attempt to use 1.22-beta3
| Andrew Wilkins (axwalk) wrote : | #31 |
I saw on the juju mailing list that the beta4 images were missing from streams, and that's been rectified now. Can you please retry and let us know how it goes?
| Ryan Beisner (1chb1n) wrote : | #32 |
Behavior has changed. This is what I'm seeing currently:
* 1.22beta4 tools are indeed being used now.
* amd64 lxc deploy succeeds as expected; ppc64el lxc deploy fails in a different way.
* ppc64el lxc unit agent-states are stuck in 'allocating' status.
* Machine 1 (ppc64el) logs show that the 1.22beta4 ppc64el tools were used to deploy a charm successfully (not in lxc).
* Machine 1 logs also show that juju thinks there are no ppc64el tools available for the charm deployed to lxc on the ppc64el host. However, tree output of /var/lib/juju shows the ppc64el 1.22beta4 tools available.
* ppc64el machine 1 has no /var/lib/
* See new attachments.
* General info:
## 1.22beta4 tools are now used:
2015-03-03 04:36:57 INFO juju.environs.
2015-03-03 04:36:57 INFO juju.environs.
units:
mongodb/0:
## package versions
juju:
Installed: 1.22-beta4-
--
juju-core:
Installed: 1.22-beta4-
--
juju-deployer:
Installed: 0.4.3-0ubuntu1~
--
juju-local:
Installed: 1.22-beta4-
--
juju-mongodb:
Installed: 2.4.9-0ubuntu3
--
juju-quickstart:
Installed: 1.6.0+bzr115+
--
python-jujuclient:
Installed: 0.50.1-2
| Ryan Beisner (1chb1n) wrote : | #33 |
| Ryan Beisner (1chb1n) wrote : | #34 |
| Ryan Beisner (1chb1n) wrote : | #35 |
| Ryan Beisner (1chb1n) wrote : | #36 |
| Ryan Beisner (1chb1n) wrote : | #37 |
| Ryan Beisner (1chb1n) wrote : | #38 |
| Ryan Beisner (1chb1n) wrote : | #39 |
| Ryan Beisner (1chb1n) wrote : | #40 |
FYI, reproducer for the latest iteration ^
#!/bin/bash -e
juju switch maas
juju bootstrap --debug --constraints arch=amd64
juju --debug deploy --constraints arch=ppc64el mongodb
juju --debug deploy --to lxc:0 nfs nfs-lxc-amd64
juju --debug deploy --to lxc:1 nfs nfs-lxc-ppc64el
| Ryan Beisner (1chb1n) wrote : | #41 |
PS, May I add... the new tabular juju stat output format is quite nice!
| Andrew Wilkins (axwalk) wrote : | #42 |
Thanks for showing the commands, that highlights the problem quite clearly. You're setting an arch constraint for the environment; we should be ignoring this for LXC containers.
You can work around this for the moment by issuing "juju set-constraints" after bootstrapping to clear the constraints. Or set arch=ppc64el in constraints when deploying nfs-lxc-ppc64el.
| Andrew Wilkins (axwalk) wrote : | #43 |
So thinking about that a bit more, I don't think we should ignore the arch constraint. If you've set a constraint, we should honour it and either produce a machine with that architecture, or fail. We should just be failing earlier.
| Andrew Wilkins (axwalk) wrote : | #44 |
wallyworld convinced me it's okay to ignore the arch. I'll get onto it.
| Ryan Beisner (1chb1n) wrote : | #45 |
FYI: We have to set the amd64 arch constraint at the time of bootstrap so that a ppc64el host isn't arbitrarily chosen as the bootstrap node by MAAS.
The expected behavior in this scenario is for juju lxc deployments to mirror the behavior of non-lxc deployments. Since it is working in this case for non-lxc units, it should also be made to work for lxc units.
PS the reproducer script has been in the bug since comment #1, though I've cut it down for simplification.
| Changed in juju-core: | |
| status: | Fix Committed → Triaged |
| assignee: | James Tunnicliffe (dooferlad) → Andrew Wilkins (axwalk) |
| Changed in juju-core: | |
| status: | Triaged → In Progress |
| Changed in juju-core: | |
| status: | In Progress → Fix Committed |
| Changed in juju-core: | |
| status: | Fix Committed → Fix Released |
| Changed in juju-core: | |
| milestone: | 1.23 → 1.23-beta1 |
| Changed in juju-core: | |
| status: | Fix Released → In Progress |
| Curtis Hovey (sinzui) wrote : | #46 |
beisner: sinzui, things are looking up with those 1.23-alpha1-
| Andrew Wilkins (axwalk) wrote : | #47 |
It's not clear from that comment that the issue is still present. I read that as Ryan testing the fix with the latest code (I assume that deb version meant to read "beta1" and not "alpha1"). Ryan, would you please clarify if the problem persists with beta1?
| Ian Booth (wallyworld) wrote : | #48 |
This was marked from fix committed to in progress for 1.23; I checked the code and both 1.22 and 1.23 branches have had the fix applied. If it works for one, it will also work for the other.
| Ryan Beisner (1chb1n) wrote : | #49 |
This appears to be resolved with 1.22beta6 (looks like it picked up 1.22beta5 tools). See attachment.
| Changed in juju-core: | |
| status: | In Progress → Fix Committed |
| Sean Feole (sfeole) wrote : | #50 |
Confirmed , fixed in 1.22beta5
"1":
agent-state: started
agent-version: 1.22-beta5
dns-name: 10.228.12.125
instance-id: manual:
series: vivid
containers:
1/lxc/0:
dns-name: 10.0.3.134
series: vivid
hardware: arch=arm64
1/lxc/1:
dns-name: 10.0.3.76
series: vivid
hardware: arch=arm64
hardware: arch=arm64 cpu-cores=1 mem=64516M
services: {}
| Changed in juju-core: | |
| status: | Fix Committed → Fix Released |
Raw upstart init file tarball attached for review.