OpenStack Snap

cannot bootstrap, juju asks for credentials

Bug #2019536 reported by Marian Gasparovic on 2023-05-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Snap	Fix Released	High	Unassigned

Bug Description

Freshly deployed Jammy machine, sunbeam bootstrap fails, I am not sure what is different in my environment

ubuntu@node6:~$ ll .local # no juju leftovers
ls: cannot access '.local': No such file or directory

ubuntu@node6:~$ sudo snap install openstack --channel yoga/edge --revision=50
openstack yoga from Canonical✓ installed

ubuntu@node6:~$ sudo addgroup $USER snap_daemon
Adding user `ubuntu' to group `snap_daemon' ...
Adding user ubuntu to group snap_daemon
Done.

logout/login

ubuntu@node6:~$ groups
ubuntu adm dialout cdrom floppy sudo audio dip video plugdev netdev lxd snap_daemon

ubuntu@node6:~$ openstack.sunbeam prepare-node-script | bash -x
++ whoami
+ USER=ubuntu
+ sudo snap connect openstack:ssh-keys
+ sudo addgroup ubuntu snap_daemon
The user `ubuntu' is already a member of `snap_daemon'.
+ newgrp snap_daemon
Generating public/private rsa key pair.
Your identification has been saved in /home/ubuntu/.ssh/id_rsa
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:9B3t6VB3wQAHgLVFpzacIbOryUOrO/ycFi+XvnVpK/8 ubuntu@node6
The key's randomart image is:
+---[RSA 4096]----+
| o=o*o+o. |
| . B *. ..|
| .o *. o o|
| . .o..+ o.|
| .S.. o o |
| o.+ o. |
| . *o .. +. |
| oooo+..o . |
| o=++o. oo.E |
+----[SHA256]-----+
# 10.244.40.223:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 10.244.40.223:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 10.244.40.223:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 10.244.40.223:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 10.244.40.223:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 10.244.8.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 10.244.8.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 10.244.8.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 10.244.8.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.33.35:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.33.35:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.33.35:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.33.35:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.33.35:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.35.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.35.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.35.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.35.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
192.168.35.7: Connection closed by remote host
# 192.168.36.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.36.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
192.168.36.7: Connection closed by remote host
# 10.244.8.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.36.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
# 192.168.36.7:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
juju (3.2/beta) 3.2-beta3 from Canonical✓ installed

ubuntu@node6:~$ time openstack.sunbeam cluster bootstrap
Checking for presence of Juju ... done
Checking for presence of ssh-keys interface ... done
Checking if user ubuntu is member of group snap_daemon ... done
Checking for ~/.local/share directory for Juju ... done
Sunbeam Cluster not initialized
Bootstrapping Sunbeam cluster ... done
Bootstrapping Juju onto machine ... done
Creating user for machine in Juju ... done
Adding Juju controller to cluster database ... done
Adding Juju user to cluster database ... done
Saving bootstrap user credentials ... done
Save Juju user for local usage ... done
Error registering user node6.silo1.lab0.solutionsqa in Juju
Traceback (most recent call last):
  File "/snap/openstack/50/lib/python3.10/site-packages/sunbeam/commands/juju.py", line 451, in run
    index = child.expect(expect_list, PEXPECT_TIMEOUT)
  File "/snap/openstack/50/lib/python3.10/site-packages/pexpect/spawnbase.py", line 343, in expect
    return self.expect_list(compiled_pattern_list,
  File "/snap/openstack/50/lib/python3.10/site-packages/pexpect/spawnbase.py", line 372, in expect_list
    return exp.expect_loop(timeout)
  File "/snap/openstack/50/lib/python3.10/site-packages/pexpect/expect.py", line 181, in expect_loop
    return self.timeout(e)
  File "/snap/openstack/50/lib/python3.10/site-packages/pexpect/expect.py", line 144, in timeout
    raise exc
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x7f5393323190>
command: /snap/openstack/50/juju/bin/juju
args: ['/snap/openstack/50/juju/bin/juju', 'register',
'MIG8Exxub2RlNi5zaWxvMS5sYWIwLnNvbHV0aW9uc3FhMGQTEjE5Mi4xNjguMzYuNzoxNzA3MBMTMTAuMjQ0LjQwLjIyMzoxNzA3MBMQMTAuMjQ0LjguNzoxNzA3MBMTMTkyLjE2OC4zMy4zNToxNzA3MBMSMTkyLjE2OC4zNS43OjE3MDcwBCDjOs8LAUNa_ZxJMbRjRtnpQocHJtsmhi8xokG7JsRX7RMSc3
VuYmVhbS1jb250cm9sbGVyEwAA', '--replace']
buffer (last 100 chars): b'o1.lab0.solutionsqa.\r\nplease enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller: '
before (last 100 chars): b'o1.lab0.solutionsqa.\r\nplease enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller: '
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 20318
child_fd: 11
closed: False
timeout: 60
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: re.compile(b'Enter a new password')
    1: re.compile(b'Confirm password')
    2: re.compile(b'Enter a name for this controller')
    3: EOF
Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x7f5393323190>
command: /snap/openstack/50/juju/bin/juju
args: ['/snap/openstack/50/juju/bin/juju', 'register',
'MIG8Exxub2RlNi5zaWxvMS5sYWIwLnNvbHV0aW9uc3FhMGQTEjE5Mi4xNjguMzYuNzoxNzA3MBMTMTAuMjQ0LjQwLjIyMzoxNzA3MBMQMTAuMjQ0LjguNzoxNzA3MBMTMTkyLjE2OC4zMy4zNToxNzA3MBMSMTkyLjE2OC4zNS43OjE3MDcwBCDjOs8LAUNa_ZxJMbRjRtnpQocHJtsmhi8xokG7JsRX7RMSc3
VuYmVhbS1jb250cm9sbGVyEwAA', '--replace']
buffer (last 100 chars): b'o1.lab0.solutionsqa.\r\nplease enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller: '
before (last 100 chars): b'o1.lab0.solutionsqa.\r\nplease enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller: '
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 20318
child_fd: 11
closed: False
timeout: 60
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: re.compile(b'Enter a new password')
    1: re.compile(b'Confirm password')
    2: re.compile(b'Enter a name for this controller')
    3: EOF
Registering Juju user using token ... failed
Error: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x7f5393323190>
command: /snap/openstack/50/juju/bin/juju
args: ['/snap/openstack/50/juju/bin/juju', 'register', 'MIG8Exxub2RlNi5zaWxvMS5sYWIwLnNvbHV0aW9uc3FhMGQTEjE5Mi4xNjguMzYuNzoxNzA3MBMTMTAuMjQ0LjQwLjIyMzoxNzA3MBMQMTAuMjQ0LjguNzoxNzA3MBMTMTkyLjE2OC4zMy4zNToxNzA3MBMSMTkyLjE2OC4zNS43OjE3MDcwBCDjOs8LAUNa_ZxJMbRjRtnpQocHJtsmhi8xokG7JsRX7RMSc3VuYmVhbS1jb250cm9sbGVyEwAA', '--replace']
buffer (last 100 chars): b'o1.lab0.solutionsqa.\r\nplease enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller: '
before (last 100 chars): b'o1.lab0.solutionsqa.\r\nplease enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller: '
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 20318
child_fd: 11
closed: False
timeout: 60
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: re.compile(b'Enter a new password')
    1: re.compile(b'Confirm password')
    2: re.compile(b'Enter a name for this controller')
    3: EOF

real 3m57.105s
user 0m15.270s
sys 0m7.214s

# juju dirs exist now

ubuntu@node6:~$ ll .local/
total 12
drwxrwxr-x 3 ubuntu snap_daemon 4096 May 15 06:46 ./
drwxr-x--- 6 ubuntu ubuntu 4096 May 15 06:46 ../
drwxrwxr-x 3 ubuntu snap_daemon 4096 May 15 06:46 share/
ubuntu@node6:~$ ll .local/share/juju/
total 40
drwx------ 4 ubuntu ubuntu 4096 May 15 06:49 ./
drwxrwxr-x 3 ubuntu snap_daemon 4096 May 15 06:46 ../
-rw------- 1 ubuntu ubuntu 103 May 15 06:49 accounts.yaml
-rw------- 1 ubuntu ubuntu 4742 May 15 06:46 bootstrap-config.yaml
-rw------- 1 ubuntu ubuntu 95 May 15 06:46 clouds.yaml
-rw------- 1 ubuntu ubuntu 2137 May 15 06:49 controllers.yaml
drwx------ 2 ubuntu ubuntu 4096 May 15 06:50 cookies/
-rw------- 1 ubuntu ubuntu 160 May 15 06:46 models.yaml
drwx------ 2 ubuntu ubuntu 4096 May 15 06:46 ssh/

Tags:

Revision history for this message

Marian Gasparovic (marosg) wrote on 2023-05-15:

jujudir.tgz Edit (6.7 KiB, application/x-tar)

Revision history for this message

Marian Gasparovic (marosg) wrote on 2023-05-15:

ubuntu@node6:~$ juju clouds
please enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller:
ubuntu@node6:~$ juju controllers
Use --refresh option with this command to see the latest information.

Controller Model User Access Cloud/Region Models Nodes HA Version
sunbeam-controller* - node6.silo1.lab0.solutionsqa login 1 - - (unknown)

Revision history for this message

Marian Gasparovic (marosg) wrote on 2023-05-15:

I have not hit this issue with revision 64 so far.

Revision history for this message

Marian Gasparovic (marosg) wrote on 2023-05-15:

Download full text (7.2 KiB)

I hit this issue with revision 65 during `sunbeam cluster join` call

I hit this issue with revision 65 during `sunbeam cluster join` call

Checking for presence of Juju ... done                                                                                                                                                                                                 
Checking for presence of ssh-keys interface ... done                                                                                                                                                                                   
Checking if user ubuntu is member of group snap_daemon ... done                                                                                                                                                                        
Checking for ~/.local/share directory for Juju ... done                                                                                                                                                                                
Sunbeam Cluster not initialized                                                                                                                                                                                                        
Adding node to Sunbeam cluster ... done                                                                                                                                                                                                
Save Juju user for local usage ... done                                                                                                                                                                                                
Error registering user node6.silo1.lab0.solutionsqa in Juju                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                                     
  File "/snap/openstack/64/lib/python3.10/site-packages/sunbeam/commands/juju.py", line 451, in run                                                                                                                                    
    index = child.expect(expect_list, PEXPECT_TIMEOUT)                                                                                                                                                                                 
  File "/snap/openstack/64/lib/python3.10/site-packages/pexpect/spawnbase.py", line 343, in expect                                                                                                                                     
    return self.expect_list(compiled_pattern_list,                                                                                                                                                                                     
  File "/snap/openstack/64/lib/python3.10/site-packages/pexpect/spawnbase.py", line 372, in expect_list                                                                                                                                
    return exp.expect_loop(timeout)                                                                                                                                                                                                    
  File "/snap/openstack/64/lib/python3.10/site-packages/pexpect/expect.py", line 181, in expect_loop                                                                                                                                   
    return self.timeout(e)                                                                                                                                                                                                             
  File "/snap/openstack/64/lib/python3.10/site-packages/pexpect/expect.py", line 144, in timeout                                                                                                                                       
    raise exc                                                                                                                                                                                                                          
pexpect.exceptions.TIMEOUT: Timeout exceeded.                                                                                                                                                                                          
<pexpect.pty_spawn.spawn object at 0x7fe5a53e0c10>                                                                                                                                                                                     
command: /snap/openstack/64/juju/bin/juju                                                                                                                                                                                              
args: ['/snap/openstack/64/juju/bin/juju', 'register',                                                                                                                                                                                 
'MIG8Exxub2RlNi5zaWxvMS5sYWIwLnNvbHV0aW9uc3FhMGQTEzEwLjI0NC40MC4yMjQ6MTcwNzATEDEwLjI0NC44LjI6MTcwNzATEzE5Mi4xNjguMzMuMzA6MTcwNzATEjE5Mi4xNjguMzUuMjoxNzA3MBMSMTkyLjE2OC4zNi4yOjE3MDcwBCCwSUnO1KhWzdlAW8yWg8SBL9C-5EeJ9yBrbz1EHSfcyhMSc3
VuYmVhbS1jb250cm9sbGVyEwAA']                                                                                                                                                                                                           
buffer (last 100 chars): b'o1.lab0.solutionsqa.\r\nplease enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller: '                                                                                                     
before (last 100 chars): b'o1.lab0.solutionsqa.\r\nplease enter password for node6.silo1.lab0.solutionsqa on sunbeam-controller: '

sunbeam cluster list shows the node as online

$ sunbeam cluster list                                                                                                                                                                                                   
Checking if user ubuntu is member of group snap_daemon ... done                                                                                                                                                                        
Listing nodes in Sunbeam cluster ... done                                                                          
Sunbeam Cluster Node List:                                                                                                                                                                                                             
[{'name': 'node1.silo1.lab0.solutionsqa', 'address': '10.244.40.224:7000', 'status': 'ONLINE'}, {'name': 'node2.silo1.lab0.solutionsqa', 'address': '10.244.40.225:7000', 'status': 'ONLINE'}, {'name': 'node6.silo1.lab0.solutionsqa',
 'address': '10.244.40.226:7000', 'status': 'ONLINE'}]

Revision history for this message

Marian Gasparovic (marosg) wrote on 2023-05-15:

^^^ revision 64

Marian Gasparovic (marosg) on 2023-05-16

tags:

added: cdo-qa

Revision history for this message

James Page (james-page) wrote on 2023-05-22:

Link to juju bug 2020360

Changed in snap-openstack:
status:	New → Confirmed
importance:	Undecided → High

Revision history for this message

James Page (james-page) wrote on 2023-05-30:

The register call now handles the prompt for the password just set.

Changed in snap-openstack:
status:	Confirmed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

jujudir.tgz Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.