Locations using bom.gov.au for forecast data no longer can no longer retrieve forecast data

[menthurae]

[Impact]
libgweather is unable to parse the HTML data previously returned from the Australian Bureau of Meteorology

[Development Fix]
Fix committed upstream as a80552f51c641754ef8410eebea6debc23a25561, 73829e640d2ebd7f926563145ca81702c17d0abb. Picked up in 3.2.1-1ubuntu1

[Stable Fix]
Debdiffs attached for Lucid (https://bugs.launchpad.net/ubuntu/+source/libgweather/+bug/629646/+attachment/2610674/+files/libgweather_2.30.0-0ubuntu1.1.debdiff), Maverick (https://bugs.launchpad.net/ubuntu/+source/libgweather/+bug/629646/+attachment/2610675/+files/libgweather_2.30.3-0ubuntu1.1.debdiff), Natty (https://bugs.launchpad.net/ubuntu/+source/libgweather/+bug/629646/+attachment/2610676/+files/libgweather_2.30.3-1ubuntu1.1.debdiff), and Oneiric (https://bugs.launchpad.net/ubuntu/+source/libgweather/+bug/629646/+attachment/2610677/+files/libgweather_3.2.0-0ubuntu1.1.debdiff)

[Test Case]
1. Login using a session with the gnome-panel (Ubuntu for Lucid, Maverick; Ubuntu Classic for Natty; GNOME Classic on Oneiric)
2. Right-click on the panel (Alt-right click for Oneiric) and select Add to Panel...
3. Choose the Weather Report applet and add it to the panel
4. Right click on the weather applet and choose preferences
5. Go to the Location tab
6. Search for and select "Sydney Airport", then close the preferences window
7. Left-click on the weather applet
8. Click "Update" from the details tab
9. Switch to the "Forecast" tab, and observe that the forecast is not available (or is, with the patch)

[Regression Potential]
Minimal. The patch is isolated, only changes parsing for the affected region (AU), and cherry-picked from upstream

===================
Original bug description:
-----------------------------------
Issue Description
-----------------------------------

In relation to "weather-applet" and "libgweather" - over the past several months, the Australian Bureau of Meteorology (BOM) has updated its website to deliver a HTML forecast page for a number of locations around Australia.

Currently - the <zone> tag from "Locations.xml" is used within the "weather-bom.c" to retrieve information from the appropriate bom.gov.au URL which if successful is then parsed in the "bom_parse" function.

With the new HTML forecast pages for various locations however, the data can not be successfully parsed and instead the message "Forecast not currently available for this location." shows up on the Forecast tab of weather-applet.

-----------------------------------
Area of Interest
-----------------------------------

Within "weather-bom.c" - the following line of code points to URLs (depending on location) that now have HTML forecast data rather than plain text...

    url = g_strdup_printf ("http://www.bom.gov.au/cgi-bin/wrap_fwo.pl?%s.txt",
      loc->zone + 1);

-----------------------------------
Suggested Fix
-----------------------------------

I suggest/propose that the URL above be changed to point to plain text documents on the BOM FTP server (ftp://ftp2.bom.gov.au/anon/gen/fwo/) with the appropriate weather station zone.

e.g. for Sydney (Zone IDN10064) - currently the code would point to:

http://www.bom.gov.au/cgi-bin/wrap_fwo.pl?IDN10064.txt

However this is now a HTML forecast, which causes the issue noted above... by pointing the code to:

ftp://ftp2.bom.gov.au/anon/gen/fwo/IDN10064.txt

The plain text forecast can be used without any modification to the other functions/aspects of the code.

-----------------------------------
Patches - Untested!
-----------------------------------

Please see the attached "weather-bom.c" which includes the suggested fix (i.e. patched version). Note it will need to be/should be tested by someone who knows how to recompile the libgweather source.

---------------------------------------------------------------------------------------------------------
System information
---------------------------------------------------------------------------------------------------------
Description: Ubuntu 10.04.1 LTS
Release: 10.04

libgweather-common:
  Installed: 2.30.0-0ubuntu1
  Candidate: 2.30.0-0ubuntu1
  Version table:
 *** 2.30.0-0ubuntu1 0
        500 http://ftp.iinet.net.au/linux/ubuntu/ lucid/main Packages
        100 /var/lib/dpkg/status
---------------------------------------------------------------------------------------------------------

[ooze (zoe-gauthier)]

This bug is an upstream one and it would be quite helpful if you could send the bug the to the people writing the software by following the instructions at https://wiki.ubuntu.com/Bugs/Upstream/GNOME . If you have done so, please tell us the number of the upstream bug (or the link), so we can add a bugwatch that will inform us about its status. Thanks in advance!

[menthurae (menthurae)]

@Philippe: I have submitted the bug to GNOME.

https://bugzilla.gnome.org/show_bug.cgi?id=628750

Thanks for the advice/suggestion!

[John Garland (johnnybg)]

I've added the attached patch upstream so hopefully it will get fixed soon.

[Glenn Moloney (g-moloney)]

Is it possible for someone to post step by step instructions on how to recompile the libgweather source to test this weather-bom.c patch for lucid? It's a very handy applet to find out the weather forecast without using your web browser. Missed it greatly :(.
Thanks in advance if anyone can help. Cheers.

[Javier Moreno (elpasmo)]

Confirmed in Maverick 10.10. libgweather1: 2.30.3-0ubuntu1

[Javier Moreno (elpasmo)]

Confirmed in Natty 11.04. libgweather1: 2.30.3-1ubuntu1

[David Clayton (dcstar)]

While I can understand something like this being considered fairly trivial, I would like to point out to the developers that in my very location (Victoria, Australia) less than 2 years ago hundreds of people died here in bushfires, and some of that was because of a lack of information regarding weather conditions - so anything that provides up to date weather info is considered by us to have a bit more importance than perhaps it is in other locales.

As to those who want to update their own system with a patched file, go to the upstream bug report and follow the posts on how it was done:

https://bugzilla.gnome.org/show_bug.cgi?id=628750

[Glenn Moloney (g-moloney)]

Just confirming. After recompiling the libgweather source with the patch (weather-bom.c.patch), installed the deb package then moving the lib files to the correct directory (From /usr/local/lib to /usr/lib), this resolves the issue :). It's just a shame that a fix (patch) was created back in September why does it take so long to upstream it?? Most new user to Ubuntu would have difficulty in installing it.

[David Clayton (dcstar)]

And to add to my previous whinge about how this patch could be of use to people in Australia, I certainly hope that none of the hundreds of thousands of people affected by the recent flash floods along the east coast here Ubuntu users trying to use this weather applet.

With the Australian BOM using every available means to try and warn the population about these devastating events is seems a shame that the method someone using a Linux computer would have of possibly seeing these warnings was not available to them.

[David Clayton (dcstar)]

Over 12 months now and a bug THAT HAS A WORKING PATCH AVAILABLE has still not been fixed.

I just did a test install of Ubuntu 11.10 and the bug is still there, yet a dopey user like myself was previously able to use the patch files to fix the Australian only file which causes the problem - but the maintainers of this package cannot make this seemingly trivial change.

Looks good when I show a Windows user how "great" Linux is and they ask why this bit doesn't work and I tell them that it was reported over 12 months ago - not!

Wassup?

[Glenn Moloney (g-moloney)]

Excuse my noobness but I didn't think you could use this because Ubuntu 11.10 uses the Unity panel which doesn't have / can't add the weather applet. I thought you could only test it with the traditional gnome panel.
David could you please advise how you tested libgweather?

[John Garland (johnnybg)]

I'm not sure if/how David tested it, but you can clearly see from the source code that it hasn't been fixed: http://git.gnome.org/browse/libgweather/tree/libgweather/weather-bom.c
Alternatively you could see from the logs that the file in particular hasn't been touched since 2008: http://git.gnome.org/browse/libgweather/log/libgweather/weather-bom.c

[John Garland (johnnybg)]

I should mention that the above links are for the latest code i.e. what's shipped in Gnome 3.2, which currently has no support for applets so I don't think libgweather is being used at all. There is work being done on a gnome-shell weather extension, so hopefully when that gets integrated this patch (or something similar) will get committed as well.

[David Clayton (dcstar)]

Some of use are using the gnome-session-fallback option because the Unity interface no longer has useful tools like the Weather applet, and it is still broken using that shell.

[Evan Broder (broder)]

This bug was fixed with 3.2.1-1

[Evan Broder (broder)]

Debdiff for Lucid SRU attached. (Will be updating the bug description shortly)

[Evan Broder (broder)]

Debdiff for Maverick SRU attached.

[Evan Broder (broder)]

Debdiff for Natty SRU

[Evan Broder (broder)]

Debdiff for Oneiric SRU

[Martin Pitt (pitti)]

All uploaded, thanks Evan! Now in ubuntu-sru queue, unsubscribing sponsors.

[Martin Pitt (pitti)]

Hello menthurae, or anyone else affected,

Accepted libgweather into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[David Clayton (dcstar)]

New package works fine on my 11.10 test system with Melbourne Australia location. Correct forecast info now visible in Forecast tab.

Thanks to everyone who contributed to this fix.

[Glenn Moloney (g-moloney)]

Would love to test but running Lucid. Are you preparing the package for lucid-proposed?

[Clint Byrum (clint-fewbar)]

Hello menthurae, or anyone else affected,

Accepted libgweather into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Clint Byrum (clint-fewbar)]

Hello menthurae, or anyone else affected,

Accepted libgweather into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Clint Byrum (clint-fewbar)]

Hello menthurae, or anyone else affected,

Accepted libgweather into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Glenn Moloney (g-moloney)]

Just tested the new package for Lucid & the Melbourne forecast is now correctly displayed (Had remove & reinstall applet for forecast to be displayed) :). Thank you to all concerned who worked on the issue. Moving forward my only wish is for Ubuntu 12.04 LTS with Unity to have the option of installing this applet as the weather forecast is far more informative than what's currently available with Unity.

[David Clayton (dcstar)]

Can confirm working ok in 10.10 & 11.04.

[Clint Byrum (clint-fewbar)]

Thanks testers, all -proposed versions are verified. This will need to wait 6 more days then it will proceed to *-updates.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libgweather - 3.2.0-0ubuntu1.1

---------------
libgweather (3.2.0-0ubuntu1.1) oneiric-proposed; urgency=low

  * debian/patches/50_fix_bom.gov.au_part1.patch,
    debian/patches/51_fix_bom.gov.au_part2.patch:
    - Cherry-pick upstream commits a80552f5 and 73829e64 to fix fetching
      weather data from bom.gov.au (LP: #629646)
 -- Evan Broder <email address hidden> Sun, 27 Nov 2011 13:16:30 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package libgweather - 2.30.3-1ubuntu1.1

---------------
libgweather (2.30.3-1ubuntu1.1) natty-proposed; urgency=low

  * debian/patches/50_fix_bom.gov.au_part1.patch,
    debian/patches/51_fix_bom.gov.au_part2.patch:
    - Cherry-pick upstream commits a80552f5 and 73829e64 to fix fetching
      weather data from bom.gov.au (LP: #629646)
 -- Evan Broder <email address hidden> Sun, 27 Nov 2011 12:24:25 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package libgweather - 2.30.3-0ubuntu1.1

---------------
libgweather (2.30.3-0ubuntu1.1) maverick-proposed; urgency=low

  * debian/patches/50_fix_bom.gov.au_part1.patch,
    debian/patches/51_fix_bom.gov.au_part2.patch:
    - Cherry-pick upstream commits a80552f5 and 73829e64 to fix fetching
      weather data from bom.gov.au (LP: #629646)
 -- Evan Broder <email address hidden> Sun, 27 Nov 2011 12:24:25 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package libgweather - 2.30.0-0ubuntu1.1

---------------
libgweather (2.30.0-0ubuntu1.1) lucid-proposed; urgency=low

  * debian/patches/50_fix_bom.gov.au_part1.patch,
    debian/patches/51_fix_bom.gov.au_part2.patch:
    - Cherry-pick upstream commits a80552f5 and 73829e64 to fix fetching
      weather data from bom.gov.au (LP: #629646)
 -- Evan Broder <email address hidden> Sun, 27 Nov 2011 12:13:10 -0800

[Alistair Fitchie (alistair-fitchie)]

great!.
Now can someone please do the same for the U.K?
It's been 4 years since I took up Linux, and not once has the forecast worked.
A previous suggestion to allow the user to be able to change the web site would have been the way to go, but obviously, you know different.
Try this page for size then...
http://www.metoffice.gov.uk/public/weather/forecast/livingston-fc#?tab=regionalForecast&fcTime=1370818800

There are another 499 sites similar to above, to change to get the forecasts for the whole of the UK to work.

I wish you the best of British...

[Alistair Fitchie (alistair-fitchie)]

Actually, I tell a lie..

"Text forecasts are provided for the each of the nations of England, Scotland, Wales and Northern Ireland along with six regional forecasts for Scotland and eight regions of England."

Let me know when you've done them all...

[David Clayton (dcstar)]

The bug is back in Ubuntu 15.10 in libgweather 3.16.1-1ubuntu1

I just installed this release in preparation for the next Ubuntu LTS in 2016 and have seen that the earlier fixes of the Australian BOM forecasts have not propagated through to this version.

I would add Wily to the list of affected versions but I cannot seem to do it!