Snappy

Update doesn't work when upgrading from devel r145 to r383 devel-proposed

Bug #1444500 reported by Kick In on 2015-04-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Snappy	Fix Released	Critical	James Hunt

Bug Description

kick@kick-Lenovo-Yoga-2-13:~/work/sprint/base_work$ ./start_it.sh dev
Les ports de cette machine seront les suivants:
SSH: 8022
HTTP: 8080
HTTPS: 8443
Un rappel des ports est disponible: dev.ports
kick@kick-Lenovo-Yoga-2-13:~/work/sprint/base_work$ ./ssh_it dev
ssh_exchange_identification: read: Connection reset by peer
kick@kick-Lenovo-Yoga-2-13:~/work/sprint/base_work$ ./ssh_it dev
Warning: Permanently added '[127.0.0.1]:8022' (ECDSA) to the list of known hosts.
Welcome to Ubuntu Vivid Vervet (development branch) (GNU/Linux 3.19.0-12-generic x86_64)

* Documentation: https://help.ubuntu.com/
Welcome to the Ubuntu Core rolling development release.

* See https://ubuntu.com/snappy

It's a brave new world here in snappy Ubuntu Core! This machine
does not use apt-get or deb packages. Please see 'snappy --help'
for app installation and transactional updates.

Last login: Tue Apr 14 15:46:36 2015
ubuntu@localhost:~$ cat /etc/system-image/channel.ini | grep devel
channel: ubuntu-core/devel
ubuntu@localhost:~$ cat /writable/cache/system/etc/system-image/channel.ini | grep devel
channel: ubuntu-core/devel
ubuntu@localhost:~$ sudo mount -o remount rw /
ubuntu@localhost:~$ sudo mount -o remount rw /writable/cache/system
ubuntu@localhost:~$ sudo sed -i s/devel/devel-proposed/ /etc/system-image/channel.ini
ubuntu@localhost:~$ cat /etc/system-image/channel.ini | grep devel
channel: ubuntu-core/devel-proposed
ubuntu@localhost:~$ sudo sed -i s/devel/devel-proposed/ /writable/cache/system/etc/system-image/channel.ini
ubuntu@localhost:~$ cat /writable/cache/system/etc/system-image/channel.ini | grep devel
channel: ubuntu-core/devel-proposed
ubuntu@localhost:~$ sudo mount -o remount ro /
ubuntu@localhost:~$ sudo mount -o remount ro /writable/cache/system
ubuntu@localhost:~$ sudo reboot
Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.
kick@kick-Lenovo-Yoga-2-13:~/work/sprint/base_work$ ./ssh_it dev
Warning: Permanently added '[127.0.0.1]:8022' (ECDSA) to the list of known hosts.
Welcome to Ubuntu Vivid Vervet (development branch) (GNU/Linux 3.19.0-12-generic x86_64)

* Documentation: https://help.ubuntu.com/
Welcome to the Ubuntu Core rolling development release.

* See https://ubuntu.com/snappy

It's a brave new world here in snappy Ubuntu Core! This machine
does not use apt-get or deb packages. Please see 'snappy --help'
for app installation and transactional updates.

Last login: Wed Apr 15 14:08:18 2015 from 10.0.2.2
ubuntu@localhost:~$ cat /writable/cache/system/etc/system-image/channel.ini | grep devel
channel: ubuntu-core/devel-proposed
ubuntu@localhost:~$ cat /writable/cache/system/etc/system-image/channel.ini
[service]
base: system-image.ubuntu.com
http_port: 80
https_port: 443
channel: ubuntu-core/devel-proposed
device: generic_amd64
build_number: 145
version_detail: ubuntu=20150223.2,raw-device=20150223.2,version=145
ubuntu@localhost:~$ cat /etc/system-image/channel.ini | grep devel
channel: ubuntu-core/devel-proposed
ubuntu@localhost:~$ cat /etc/system-image/channel.ini
[service]
base: system-image.ubuntu.com
http_port: 80
https_port: 443
channel: ubuntu-core/devel-proposed
device: generic_amd64
build_number: 146
version_detail: ubuntu=20150410.3,raw-device=20150410.3,version=146
ubuntu@localhost:~$ sudo snappy update
Installing ubuntu-core (384)
Starting download of ubuntu-core
135.00 MB / 135.00 MB [=============================================================================================================================================================================================] 100.00 % 1.55 MB/s
Done
error from system-image-cli: ('%s exited with %s:\n%s', ['ubuntu-core-upgrade'], 1, 'I: preparing root filesystem\nTraceback (most recent call last):\n File "/usr/bin/ubuntu-core-upgrade", line 182, in <module>\n main()\n File "/usr/bin/ubuntu-core-upgrade", line 155, in main\n prepare_upgrade(options)\n File "/usr/bin/ubu')

Journalctl is not very verbose:

Apr 15 14:11:06 localhost.localdomain systemd[731]: Starting Default.
Apr 15 14:11:36 localhost.localdomain sudo[766]: ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/snappy update
Apr 15 14:11:36 localhost.localdomain sudo[766]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
Apr 15 14:12:35 localhost.localdomain kernel: EXT4-fs (sda3): re-mounted. Opts: (null)
Apr 15 14:12:35 localhost.localdomain kernel: EXT4-fs (sda4): mounted filesystem with ordered data mode. Opts: (null)
Apr 15 14:12:35 localhost.localdomain kernel: EXT4-fs (sda4): mounted filesystem with ordered data mode. Opts: (null)
Apr 15 14:13:04 localhost.localdomain sudo[766]: pam_unix(sudo:session): session closed for user root
Apr 15 14:17:01 localhost.localdomain CRON[908]: pam_unix(cron:session): session opened for user root by (uid=0)
Apr 15 14:17:01 localhost.localdomain CRON[909]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Apr 15 14:17:01 localhost.localdomain CRON[908]: pam_unix(cron:session): session closed for user root
Apr 15 14:17:13 localhost.localdomain sudo[911]: ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -l
Apr 15 14:17:13 localhost.localdomain sudo[911]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)

Related branches

lp:~jamesodhunt/ubuntu/vivid/ubuntu-core-upgrader/bug-1444500

Merged into lp:ubuntu/vivid/ubuntu-core-upgrader at revision 28

Ubuntu branches: Pending requested 2015-04-15

lp:ubuntu/vivid-proposed/ubuntu-core-upgrader

Kick In (kick-d) on 2015-04-15

Changed in snappy-ubuntu:
importance:	Undecided → High

Revision history for this message

Kick In (kick-d) wrote on 2015-04-15:

Additional traces:

root@localhost:~# ubuntu-core-upgrade
ERROR: need command file
root@localhost:~# ubuntu-core-upgrade /writable/
cache/ lost+found/ system-data/ user-data/
root@localhost:~# ubuntu-core-upgrade /writable/cache/
assets/ system/
device-dd02f728677fa1ccdf2646ca5bbdb77d747a1c180a10d0e80e0c9419d2eea53d.tar.xz system-ro-788/
device-dd02f728677fa1ccdf2646ca5bbdb77d747a1c180a10d0e80e0c9419d2eea53d.tar.xz.asc ubuntu-ff327cc343626daccb831f8a87034ed975c7be73a02cc3d264a738f8ff5791e0.tar.xz
image-master.tar.xz ubuntu-ff327cc343626daccb831f8a87034ed975c7be73a02cc3d264a738f8ff5791e0.tar.xz.asc
image-master.tar.xz.asc ubuntu_command.applying
image-signing.tar.xz version-384.tar.xz
image-signing.tar.xz.asc version-384.tar.xz.asc
root@localhost:~# ubuntu-core-upgrade /writable/cache/ubuntu_command.applying
I: preparing root filesystem
Traceback (most recent call last):
  File "/usr/bin/ubuntu-core-upgrade", line 182, in <module>
    main()
  File "/usr/bin/ubuntu-core-upgrade", line 155, in main
    prepare_upgrade(options)
  File "/usr/bin/ubuntu-core-upgrade", line 98, in prepare_upgrade
    upgrade(options, commands, remove_list)
  File "/usr/bin/ubuntu-core-upgrade", line 75, in upgrade
    upgrader.run()
  File "/usr/lib/python3/dist-packages/ubuntucoreupgrader/upgrader.py", line 579, in run
    self.dispatcher[cmd](args)
  File "/usr/lib/python3/dist-packages/ubuntucoreupgrader/upgrader.py", line 879, in _cmd_update
    self.options.root_dir))
  File "/usr/lib/python3.4/tarfile.py", line 1973, in extractall
    for tarinfo in members:
  File "/usr/lib/python3/dist-packages/ubuntucoreupgrader/upgrader.py", line 453, in tar_generator
    os.unlink(path)
IsADirectoryError: [Errno 21] Is a directory: '/writable/cache/system/usr/share/apparmor/easyprof/policygroups/ubuntu-snappy'
root@localhost:~# ls -l /writable/cache/system/usr/share/apparmor/easyprof/policygroups/ubuntu-snappy
total 4
drwxr-xr-x 2 root root 4096 Apr 10 16:54 1.3
root@localhost:~#

Additional traces:

root@localhost:~# ubuntu-core-upgrade 
ERROR: need command file
root@localhost:~# ubuntu-core-upgrade /writable/
cache/       lost+found/  system-data/ user-data/   
root@localhost:~# ubuntu-core-upgrade /writable/cache/
assets/                                                                             system/
device-dd02f728677fa1ccdf2646ca5bbdb77d747a1c180a10d0e80e0c9419d2eea53d.tar.xz      system-ro-788/
device-dd02f728677fa1ccdf2646ca5bbdb77d747a1c180a10d0e80e0c9419d2eea53d.tar.xz.asc  ubuntu-ff327cc343626daccb831f8a87034ed975c7be73a02cc3d264a738f8ff5791e0.tar.xz
image-master.tar.xz                                                                 ubuntu-ff327cc343626daccb831f8a87034ed975c7be73a02cc3d264a738f8ff5791e0.tar.xz.asc
image-master.tar.xz.asc                                                             ubuntu_command.applying
image-signing.tar.xz                                                                version-384.tar.xz
image-signing.tar.xz.asc                                                            version-384.tar.xz.asc
root@localhost:~# ubuntu-core-upgrade /writable/cache/ubuntu_command.applying 
I: preparing root filesystem
Traceback (most recent call last):
  File "/usr/bin/ubuntu-core-upgrade", line 182, in <module>
    main()
  File "/usr/bin/ubuntu-core-upgrade", line 155, in main
    prepare_upgrade(options)
  File "/usr/bin/ubuntu-core-upgrade", line 98, in prepare_upgrade
    upgrade(options, commands, remove_list)
  File "/usr/bin/ubuntu-core-upgrade", line 75, in upgrade
    upgrader.run()
  File "/usr/lib/python3/dist-packages/ubuntucoreupgrader/upgrader.py", line 579, in run
    self.dispatcher[cmd](args)
  File "/usr/lib/python3/dist-packages/ubuntucoreupgrader/upgrader.py", line 879, in _cmd_update
    self.options.root_dir))
  File "/usr/lib/python3.4/tarfile.py", line 1973, in extractall
    for tarinfo in members:
  File "/usr/lib/python3/dist-packages/ubuntucoreupgrader/upgrader.py", line 453, in tar_generator
    os.unlink(path)
IsADirectoryError: [Errno 21] Is a directory: '/writable/cache/system/usr/share/apparmor/easyprof/policygroups/ubuntu-snappy'
root@localhost:~# ls -l /writable/cache/system/usr/share/apparmor/easyprof/policygroups/ubuntu-snappy
total 4
drwxr-xr-x 2 root root 4096 Apr 10 16:54 1.3
root@localhost:~#

Michael Vogt (mvo) on 2015-04-15

Changed in snappy-ubuntu:
importance:	High → Critical

James Hunt (jamesodhunt) on 2015-04-15

Changed in snappy-ubuntu:
assignee:	nobody → James Hunt (jamesodhunt)

Steve Langasek (vorlon) on 2015-04-15

summary:

- Update doesn't work when upgrading from devel r146 to r383 devel-
+ Update doesn't work when upgrading from devel r145 to r383 devel-
proposed

Revision history for this message

James Hunt (jamesodhunt) wrote on 2015-04-15:

This problem was caused by the partition sync code being called even when a full image was being applied to "other".

I'll add further tests to the branch.

Thought: maybe we should add an option to snappy to request s-i perform a full image upgrade since (once this bug is fixed), that would allow another route in case of further upgrade issues. All we'd need to do is pass '-b 0' to system-image-cli via some means (see bug 1410370). We could even semi-automate this - if a delta update fails, retry by performing a full image update.

James Hunt (jamesodhunt) on 2015-04-16

Changed in snappy-ubuntu:
status:	New → Fix Committed

Revision history for this message

Kick In (kick-d) wrote on 2015-04-16:

Thanks James, but it seems to be crashing again, do I make a mistake in my commands ?

http://pastebin.ubuntu.com/10835049/

Revision history for this message

Kick In (kick-d) wrote on 2015-04-16:

Or maybe I need a newer image than r397 ?

Revision history for this message

James Hunt (jamesodhunt) wrote on 2015-04-16:

The fix is in ubuntu-core-upgrader 0.7.10 so if you are starting from a revision with an older version of the upgrader (unclear from your pb), it will still crash of course.

Revision history for this message

Kick In (kick-d) wrote on 2015-04-16: Re: [Bug 1444500] Re: Update doesn't work when upgrading from devel r145 to r383 devel-proposed

Download full text (6.3 KiB)

Sorry, I missed this obvious reason :( .

2015-04-16 15:57 GMT-05:00 James Hunt <email address hidden>:

Sorry, I missed this obvious reason :( .

2015-04-16 15:57 GMT-05:00 James Hunt <james.hunt@canonical.com>:

> The fix is in ubuntu-core-upgrader 0.7.10 so if you are starting from a
> revision with an older version of the upgrader (unclear from your pb),
> it will still crash of course.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1444500
>
> Title:
>   Update doesn't work when upgrading from devel r145 to r383 devel-
>   proposed
>
> Status in Snappy Ubuntu:
>   Fix Committed
>
> Bug description:
>   kick@kick-Lenovo-Yoga-2-13:~/work/sprint/base_work$ ./start_it.sh dev
>   Les ports de cette machine seront les suivants:
>   SSH: 8022
>   HTTP: 8080
>   HTTPS: 8443
>   Un rappel des ports est disponible: dev.ports
>   kick@kick-Lenovo-Yoga-2-13:~/work/sprint/base_work$ ./ssh_it dev
>   ssh_exchange_identification: read: Connection reset by peer
>   kick@kick-Lenovo-Yoga-2-13:~/work/sprint/base_work$ ./ssh_it dev
>   Warning: Permanently added '[127.0.0.1]:8022' (ECDSA) to the list of
> known hosts.
>   Welcome to Ubuntu Vivid Vervet (development branch) (GNU/Linux
> 3.19.0-12-generic x86_64)
>
>    * Documentation:  https://help.ubuntu.com/
>   Welcome to the Ubuntu Core rolling development release.
>
>    * See https://ubuntu.com/snappy
>
>   It's a brave new world here in snappy Ubuntu Core! This machine
>   does not use apt-get or deb packages. Please see 'snappy --help'
>   for app installation and transactional updates.
>
>   Last login: Tue Apr 14 15:46:36 2015
>   ubuntu@localhost:~$ cat /etc/system-image/channel.ini | grep devel
>   channel: ubuntu-core/devel
>   ubuntu@localhost:~$ cat
> /writable/cache/system/etc/system-image/channel.ini | grep devel
>   channel: ubuntu-core/devel
>   ubuntu@localhost:~$ sudo mount -o remount rw /
>   ubuntu@localhost:~$ sudo mount -o remount rw /writable/cache/system
>   ubuntu@localhost:~$ sudo sed -i s/devel/devel-proposed/
> /etc/system-image/channel.ini
>   ubuntu@localhost:~$ cat /etc/system-image/channel.ini | grep devel
>   channel: ubuntu-core/devel-proposed
>   ubuntu@localhost:~$ sudo sed -i s/devel/devel-proposed/
> /writable/cache/system/etc/system-image/channel.ini
>   ubuntu@localhost:~$ cat
> /writable/cache/system/etc/system-image/channel.ini | grep devel
>   channel: ubuntu-core/devel-proposed
>   ubuntu@localhost:~$ sudo mount -o remount ro /
>   ubuntu@localhost:~$ sudo mount -o remount ro /writable/cache/system
>   ubuntu@localhost:~$ sudo reboot
>   Connection to 127.0.0.1 closed by remote host.
>   Connection to 127.0.0.1 closed.
>   kick@kick-Lenovo-Yoga-2-13:~/work/sprint/base_work$ ./ssh_it dev
>   Warning: Permanently added '[127.0.0.1]:8022' (ECDSA) to the list of
> known hosts.
>   Welcome to Ubuntu Vivid Vervet (development branch) (GNU/Linux
> 3.19.0-12-generic x86_64)
>
>    * Documentation:  https://help.ubuntu.com/
>   Welcome to the Ubuntu Core rolling development release.
>
>    * See https://ubuntu.com/snappy
>
>   It's a brave new world here in snappy Ubuntu Core! This machine
>   does not use apt-get or deb packages. Please see 'snappy --help'
>   for app installation and transactional updates.
>
>   Last login: Wed Apr 15 14:08:18 2015 from 10.0.2.2
>   ubuntu@localhost:~$ cat
> /writable/cache/system/etc/system-image/channel.ini | grep devel
>   channel: ubuntu-core/devel-proposed
>   ubuntu@localhost:~$ cat
> /writable/cache/system/etc/system-image/channel.ini
>   [service]
>   base: system-image.ubuntu.com
>   http_port: 80
>   https_port: 443
>   channel: ubuntu-core/devel-proposed
>   device: generic_amd64
>   build_number: 145
>   version_detail: ubuntu=20150223.2,raw-device=20150223.2,version=145
>   ubuntu@localhost:~$ cat /etc/system-image/channel.ini | grep devel
>   channel: ubuntu-core/devel-proposed
>   ubuntu@localhost:~$ cat /etc/system-image/channel.ini
>   [service]
>   base: system-image.ubuntu.com
>   http_port: 80
>   https_port: 443
>   channel: ubuntu-core/devel-proposed
>   device: generic_amd64
>   build_number: 146
>   version_detail: ubuntu=20150410.3,raw-device=20150410.3,version=146
>   ubuntu@localhost:~$ sudo snappy update
>   Installing ubuntu-core (384)
>   Starting download of ubuntu-core
>   135.00 MB / 135.00 MB
> [=============================================================================================================================================================================================]
> 100.00 % 1.55 MB/s
>   Done
>   error from system-image-cli: ('%s exited with %s:\n%s',
> ['ubuntu-core-upgrade'], 1, 'I: preparing root filesystem\nTraceback (most
> recent call last):\n  File "/usr/bin/ubuntu-core-upgrade", line 182, in
> <module>\n    main()\n  File "/usr/bin/ubuntu-core-upgrade", line 155, in
> main\n    prepare_upgrade(options)\n  File "/usr/bin/ubu')
>
>   Journalctl is not very verbose:
>
>   Apr 15 14:11:06 localhost.localdomain systemd[731]: Starting Default.
>   Apr 15 14:11:36 localhost.localdomain sudo[766]: ubuntu : TTY=pts/0 ;
> PWD=/home/ubuntu ; USER=root ; COMMAND=/usr/bin/snappy update
>   Apr 15 14:11:36 localhost.localdomain sudo[766]: pam_unix(sudo:session):
> session opened for user root by ubuntu(uid=0)
>   Apr 15 14:12:35 localhost.localdomain kernel: EXT4-fs (sda3):
> re-mounted. Opts: (null)
>   Apr 15 14:12:35 localhost.localdomain kernel: EXT4-fs (sda4): mounted
> filesystem with ordered data mode. Opts: (null)
>   Apr 15 14:12:35 localhost.localdomain kernel: EXT4-fs (sda4): mounted
> filesystem with ordered data mode. Opts: (null)
>   Apr 15 14:13:04 localhost.localdomain sudo[766]: pam_unix(sudo:session):
> session closed for user root
>   Apr 15 14:17:01 localhost.localdomain CRON[908]: pam_unix(cron:session):
> session opened for user root by (uid=0)
>   Apr 15 14:17:01 localhost.localdomain CRON[909]: (root) CMD (   cd / &&
> run-parts --report /etc/cron.hourly)
>   Apr 15 14:17:01 localhost.localdomain CRON[908]: pam_unix(cron:session):
> session closed for user root
>   Apr 15 14:17:13 localhost.localdomain sudo[911]: ubuntu : TTY=pts/0 ;
> PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -l
>   Apr 15 14:17:13 localhost.localdomain sudo[911]: pam_unix(sudo:session):
> session opened for user root by ubuntu(uid=0)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snappy-ubuntu/+bug/1444500/+subscriptions
>

Revision history for this message

Steve Langasek (vorlon) wrote on 2015-04-16:

does this bug also explain a failure I'm seeing trying to update on devel-proposed from image 373?

Revision history for this message

Steve Langasek (vorlon) wrote on 2015-04-16:

(also with an ubuntu-core-upgrader backtrace, to be clear)

Revision history for this message

James Hunt (jamesodhunt) wrote on 2015-04-17:

Hi Steve - looks like r373 was one of the 2-3 images that somehow lost the ubuntu-core-upgrader package entirely prior to the promotion.

James Hunt (jamesodhunt) on 2015-04-17

Changed in snappy-ubuntu:
status:	Fix Committed → Won't Fix
status:	Won't Fix → Fix Released

Michael Terry (mterry) on 2015-05-18

affects:

snappy-ubuntu → snappy

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.