Wesnoth save game file format in jaunty is incompatible (1.4 -> 1.6) and saves just vanish

[Gerv]

Binary package hint: wesnoth

I have completed 23 out of 25 scenarios in the Wesnoth campaign "Heir to the Throne". I then upgraded to the Jaunty beta, which replaces Wesnoth 1.4 with 1.6. Now, none of my saved games are available in the GUI, and so I can't load them!

This campaign took rather a long time to play. If this were my only computer, I would cry. Fortunately, I have another one and can use rsync to move the saves across and finish the campaign. But other people will not be so lucky.

There needs to be some sort of warning, or a conversion script, or the ability to keep 1.4 installed as well... or something! Just "losing" all the saved games (although the data remains in the Wesnoth directory, at least at first) and not telling the user anything is unacceptable IMO.

[Rhonda D'Vine (rhonda)]

* Gerv <email address hidden> [2009-04-04 13:36:40 CEST]:
> I have completed 23 out of 25 scenarios in the Wesnoth campaign "Heir to
> the Throne". I then upgraded to the Jaunty beta, which replaces Wesnoth
> 1.4 with 1.6. Now, none of my saved games are available in the GUI, and
> so I can't load them!

 There is a NEWS.Debian entry about that - and it was the same thing
with the upgrade from 1.2 to 1.4. For your convenience I am citing it
here:

wesnoth (1:1.6-1) unstable; urgency=low

  * Please be aware what a new stable upstream release of wesnoth means for
    your save games: You won't be able to continue playing from them. The
    reasons for those are several, not limited to these two bigger points:
    - Most of all, balancing changes: Units might have different
      characteristics and thus fights would end differently, or your units
      could move to places they have never seen before (and weren't meant to).
    - Most obvious, changes in the campaigns themself. A unit you kept from a
      former scenario might not be in the campaign anymore, or a whole
      scenario might have been added or removed somewhere in between. Map
      changes also affect multiplayer games.
    Again: You won't be able to continue playing your old save games from
    older stable releases (e.g. 1.4.x). Further updates to 1.6.x will though
    be compatible with this version.

 -- Gerfried Fuchs <email address hidden> Wed, 18 Mar 2009 17:27:36 +0100

 Unfortunately there is no way to support 1.4 savegames in the 1.6
release. Given that it's already quite big workload to support wesnoth
the way it is currently given pretty regular releases, bug tracking and
long build times I unfortunately don't have the spare time, the
diskspace and the hardware power to support both 1.4 and 1.6
side-by-side, so this is the only path that I can offer at the moment.

> This campaign took rather a long time to play. If this were my only
> computer, I would cry. Fortunately, I have another one and can use rsync
> to move the saves across and finish the campaign. But other people will
> not be so lucky.

 Shouldn't entries in the NEWS.Debian be displayed to people upgrading
their systems? At least, if apt-listchanges is installed, which I hope
most people have nowadays - but unfortunately I can't do much more than
offer documentation about it without abusing other means.

> There needs to be some sort of warning, or a conversion script, or the
> ability to keep 1.4 installed as well... or something!

 See above, a conversion script won't be possible for these reasons.

> Just "losing" all the saved games (although the data remains in the
> Wesnoth directory, at least at first) and not telling the user
> anything is unacceptable IMO.

 I used the means that are there to tell the users about it, and I'm
sorry about the inconvenience, but I fear there's not much more that I
can offer.

 Hope you can enjoy it nevertheless,
Rhonda

[Gerv (gerv)]

I realise conversion is hard/impossible, and I agree you can't be expected to support both versions. But I would happily have delayed my upgrade to Jaunty by a week if I had known that this was coming. So I guess what I'm really looking for is a warning at the start of the upgrade process. Is there any facility for that sort of thing?

Also, I'm looking for Wesnoth, when I start it with old saved games, to say "You have old saved games. Sorry you can't play them. You need to downgrade to 1.4" or another helpful message, rather than just pretend that they aren't there. I would hope that's not too big a patch. I agree that's a bug for upstream rather than the Debian or Ubuntu maintainer, though.

Gerv

[Gergely Máté (sportember)]

I have the same problem here; but I don't think that it is an upstream problem, in fact, I think it is an Ubuntu special problem. I had this kind of problem before, when updating Ubuntu from 8.04 to 8.10, and for now I see it to be not a Wesnoth specific problem. It affects the whole Ubuntu lifecycle, and probably many applications.

In those times I was upgrading to Intrepid, I was learning using the Gimp from it's help files. I was learning day by day, always working myself through more and more chapters from Gimp-help. During the update to Intrepid, Gimp version has changed, a rather nice upgrade: from 2.4 to 2.6. But the help package was not ready, and offline Gimp-help stopped to work at all. So after the distro update, I was not able to continue my learning experience the way I was doing it before. I filed a bug-report, thus.

These days I played Wesnoth. I enjoyed it and was involved in a campaign (like Gerfried). After the distro upgrade, the experience is lost: I cannot continue along the path I was going. Version incompatibility. I played some other games, such as SuperTuxKart, which works terribly after the distro upgrade - the problem is with Intel video performance, a well documented bug, although. I knew I would have problems after the upgrade. But I am a free software enthusiast, and I was very interested in the work my other enthusiastic fellows did worldwide. So I upgraded.

There may be always pains and losses as the world I live in change. Now I try to do my best to cure those problems that arose from the change - not only to myself, but to the whole Ubuntu community. In my "spare time", I probably will start another Wesnoth campaign, or simply try playing online - a thing I was not brave enough to do before.

Meanwhile, thinking about how to make distribution upgrades less a pain and more a joy seems an important task if Ubuntu seriously wanted to conquer desktop market majority ;-) !

[Rhonda D'Vine (rhonda)]

* Gerv <email address hidden> [2009-04-14 19:45:23 CEST]:
> I realise conversion is hard/impossible, and I agree you can't be
> expected to support both versions. But I would happily have delayed my
> upgrade to Jaunty by a week if I had known that this was coming. So I
> guess what I'm really looking for is a warning at the start of the
> upgrade process. Is there any facility for that sort of thing?

 apt-listchanges is the tool that allows you this. It will display
entries from the NEWS.Debian file to you and also the changelogs, if
wanted. It's conveniently configured through debconf and by default only
displays the NEWS entries. Having installed that package would have
given you that warning at the start of the upgrade process, the message
I quoted before.

> Also, I'm looking for Wesnoth, when I start it with old saved games, to
> say "You have old saved games. Sorry you can't play them. You need to
> downgrade to 1.4" or another helpful message, rather than just pretend
> that they aren't there. I would hope that's not too big a patch. I agree
> that's a bug for upstream rather than the Debian or Ubuntu maintainer,
> though.

 That's because historically it was stored in ~/.wesnoth and is now
stored in ~/.wesnoth1.6. I am not exactly sure what the need for the
switch was but I expect that upstream does have a reason for this
approach - I rather not like to patch the configdir path to be different
without knowing the rationale behind that and divert from how its done
everywhere else.

 Thanks. :)
Rhonda

[Pauli (paniemin)]

>> Also, I'm looking for Wesnoth, when I start it with old saved games, to
>> say "You have old saved games. Sorry you can't play them. You need to
>> downgrade to 1.4" or another helpful message, rather than just pretend
>> that they aren't there. I would hope that's not too big a patch. I agree
>> that's a bug for upstream rather than the Debian or Ubuntu maintainer,
>> though.
>
>That's because historically it was stored in ~/.wesnoth and is now
>stored in ~/.wesnoth1.6. I am not exactly sure what the need for the
>switch was but I expect that upstream does have a reason for this
>approach - I rather not like to patch the configdir path to be different
>without knowing the rationale behind that and divert from how its done
>everywhere else.

This was just to solved the problem that average user would have old userdata/saves after upgrade which would then cause many problems in the new stable release. So directory name change was made to prevent users from shooting their own leg because they aren't excepted to know all the details about userdata directory.

@rhonda: How about making wesnoth packages install wesnothX.Y directory so users could have multiple stable versions installed same time. This would help transition period when the new stable version hits distros. Then you would need wesnoth meta packages that depend on the latest wesnoth packages with version in the package names.

I think that wouls just require a few path changes and extra packages build. But of course larger question is if debian or ubuntu would accept that kind of change.

[Rhonda D'Vine (rhonda)]

* Pauli <email address hidden> [2009-05-04 12:57:48 CEST]:
> >That's because historically it was stored in ~/.wesnoth and is now
> >stored in ~/.wesnoth1.6. I am not exactly sure what the need for the
> >switch was but I expect that upstream does have a reason for this
> >approach - I rather not like to patch the configdir path to be different
> >without knowing the rationale behind that and divert from how its done
> >everywhere else.
>
> This was just to solved the problem that average user would have old
> userdata/saves after upgrade which would then cause many problems in the
> new stable release. So directory name change was made to prevent users
> from shooting their own leg because they aren't excepted to know all the
> details about userdata directory.

 Well, as was raised in here it also raises confusion about claiming
that there aren't the old savegames anymore. And the dialogue about old
save games was already there, so that part of the reasoning is hard to
follow for me, to be honest.

> @rhonda: How about making wesnoth packages install wesnothX.Y directory
> so users could have multiple stable versions installed same time. This
> would help transition period when the new stable version hits distros.
> Then you would need wesnoth meta packages that depend on the latest
> wesnoth packages with version in the package names.

 See above about the required time/effort that would require (from me),
for IMHO not too much of a benefit (for everyone). It would require
major changes to how the game is packaged, for a start having to rename
every single binary package to contain the X.Y part in its package name
and having every single file carry the X.Y part either in the path or in
its own name.

 And I don't plan to continue to support the 1.4 branch when it was
abandoned upstream already. If you remember the security issue there
wasn't any further release that fixed it and I have no interest to carry
on upstream development for abandoned branches.

 ... for a start, the package is considered minor by the
Ubuntu/Canonical crew anyway, so from that area it would mean even more
burden on my part. Please don't forget that I do this all out of my
spare time - and from the 30something packages I maintain wesnoth is now
already taking up most of the time. I also need to do things to get my
bills payed.

> I think that wouls just require a few path changes and extra packages
> build. But of course larger question is if debian or ubuntu would accept
> that kind of change.

 I don't ask that question because I myself don't see a way to do that
in a sane way that doesn't explode in needed time investment on my part,
so it already fails at myself.

 Thanks for your understanding.
Rhonda

[Gerv (gerv)]

I agree with Rhonda - changing the directory name seems very shortsighted. a) What happens when we get to Wesnoth 1.8? b) Is it really impossible to create saved game files so that they have versioning info, and the game can say "sorry, we can't open that file now"? It seems to be it does this anyway - so why on earth change the directory so that code becomes useless?

Rhonda: I realise you have already spent time on this, but might it be worth checking with upstream to see if it would be OK to switch the data directory back? Then people would get a sensible error message instead of just thinking all their games had vanished.

Also, while I'm here, if there's any way of changing the packaging so a small update to the main game doesn't require re-downloading all the music, that would be very cool too :-)

Gerv

[Rhonda D'Vine (rhonda)]

* Gerv <email address hidden> [2009-05-04 13:53:08 CEST]:
> I agree with Rhonda - changing the directory name seems very
> shortsighted.

 Erm, I didn't claim that, you must have misread me. I said that there
propably are valid reasons behind the decision, I just don't know them
offhand. E.g. incompatible changes to the preferences file or similar,
or additional downloaded data, no idea.

> a) What happens when we get to Wesnoth 1.8?

 It most propably will be a ~/.wesnoth1.8 directory then.

> b) Is it really impossible to create saved game files so that they
> have versioning info, and the game can say "sorry, we can't open that
> file now"? It seems to be it does this anyway - so why on earth change
> the directory so that code becomes useless?

 Because there aren't only save games stored in there.

> Rhonda: I realise you have already spent time on this, but might it be
> worth checking with upstream to see if it would be OK to switch the data
> directory back? Then people would get a sensible error message instead
> of just thinking all their games had vanished.

 See above, it's not only about save games. If it only were about save
games I'm pretty sure that upstream wouldn't have done the change
because they are extremely sensible and cautious with their approaches.

> Also, while I'm here, if there's any way of changing the packaging so a
> small update to the main game doesn't require re-downloading all the
> music, that would be very cool too :-)

 It doesn't require re-downloading all the music. Noone forces you to
download the music package along with the others. In fact, you don't
even need to download the campaigns when you only want to update the
main binaries because the Dependencies are set appropriately. It's just
that they are distributed from the same source package and when you have
an updated core binary package you also get an updates music package and
campaign packages with that. It's not possible to avoid that without
splitting the source and investing quite a lot of time for checks along
those lines, which would explode needed time to invest, too.

 So long. :)
Rhonda

[Emmet Hikory (persia)]

Providing notification of this external to NEWS.Debian would mean adding it to the Release Notes. Typically, the Release Notes for each flavour of Ubuntu is restricted to packages shipped by default in that flavour. As wesnoth is not included by default in any flavour, this package is typically not considered for release notes. Significant changes for most packages are also reported in NEWS.Debian: if you are performing a distribution upgrade, and need to ensure data compatibility with new upstream versions of software, you may do best to inspect at least the NEWS.Debian changes for those packages of particular concern, if not the full changelog.

    Of course, if you want to write a data file conversion tool, I'm sure that would be appreciated upstream. In the presence of such a tool, it becomes feasible to attempt to migrate user data.

[Gergely Máté (sportember)]

OK, so saved games did not really vanish, but are unusable with Jaunty's version of Wesnoth - and you want to continue the campaign you already played a lot. If this is the situation, you have some resources at your hand, and you are not afraid of the Terminal, keep reading. ;-)

If you want to play Wesnoth the way you did played in Intrepid, you need Intrepid. Fortunately, you can have Intrepid and Jaunty on your computer at the same time. The solution is called virtualization, and the tool I just used continue my campaign is VirtualBox Open Source Edition.

These are the steps to play :-) :
1. install virtualbox-ose
2. Start VirtualBox and configure a new virtual machine. I gave it a hard drive of 8 Gigs, and an Intrepid LiveCD iso as CD-ROM. Also don't forget to enable audio among the settings (my choice was PulseAudio).
3. Start the new virtual machine. The livecd boots, and you can install Intrepid onto the newly created virtual hard disk.
4. Than, in your Intrepid virtual machine:
 1. Install roughly the following packages :
  wesnoth
  dkms
It took me about 90 minutes to reach this state. In theory, here you can start Wesnoth, but the display resolution may be to low to enjoy it. So here comes the harder stuff - and the Terminal. You will have to tell your virtual Intrepid that it is running in a virtualized environment to be able to change display resolution. dkms was required to make that easier.

 2. From the tools menu of your VirtualBox window that displays your Intrepid, select "Install Guest Additions". This will download a 18megs ISO file and mount it. Still in your Intrepid virtual machine, start up a terminal, and navigate to the mount point of the downloaded ISO file (it was /media/cdrom0 for me). There enter:
> ./VBoxLinuxAdditions-x86.run
After this, you have to configure your virtual Intrepid's X server. So edit /etc/X11/xorg.conf like this:
in Section "Device" set:
     Driver "vboxvideo"
in Section "InputDevice" set
     Driver "vboxmouse"
There should be NO "alwayscore" or "sendcoreevents" anywhere in your virtualized xorg.conf file.
After the change, you have to restart the X server in your Intrepid. Simply logging out and back again probably solves this. Here, if you try to resize your VirtualBox window, and the Intrepid desktop resizes with it, than you are almost done. In case of problems, you still can ask for help :-).

      3. Now you have Intrepid, Wesnoth, and can resize your display. The last thing to do is to copy your old, well played .wesnoth directory to this newly created virtual system. Simply copy .wesnoth from your non-virtualized home directory to the home directory on Intrepid - overwrite the .wesnoth dir that came with installing Wesnoth to the new system. Every setup information and every save game for Wesnoth is in the .wesnoth directory.
In fact, I was unable to transfer data directly to the virtual machine. So I sent it to the Internet, and downloaded it from inside my Intrepid VM.

With the .wesnoth directory in place, the last thing to do is go fullscreen with the VM, and launch Wesnoth. Voilá! It works. It is quite playable on my 1.6 Ghz Celeron (I have 1GB ram, that's needed, as I g...

[Pauli (paniemin)]

install:

wget http://downloads.sourceforge.net/wesnoth/wesnoth-1.4.7.tar.bz2?download
tar xf wesnoth-1.4.7.tar.bz2
cd wesnoth-1.4.7
sudo apt-get build-dep wesnoth
./autogen.sh --program-suffix=_old
make
sudo make install
wesnoth_old

remove:
cd wesnoth-1.4.7
sudo make uninstall
cd ..
rm -r wesnoth-1.4.7

[Emmet Hikory (persia)]

No need to recompile. Just grab the intrepid wesnoth from launchpad, and install that (unless you want to play both versions at once). Or just don't upgrade wesnoth at the time of upgrade, until you've finished the campaign.