If-Modfied-Since undhandled case causes apt lists corruption with https repositories

Bug #1179781 reported by Paul Wise (Debian) on 2013-05-14
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
apt (Debian)
Fix Released
Unknown
apt (Ubuntu)
Medium
Dave Chiluk
Precise
Medium
Unassigned
Quantal
Medium
Unassigned
Raring
Medium
Unassigned
curl (Debian)
Fix Released
Unknown
curl (Ubuntu)
Medium
Dave Chiluk
Precise
Medium
Unassigned
Quantal
Medium
Unassigned
Raring
Medium
Unassigned

Bug Description

[Impact]

Users of apt repositories that use https will occasionally recieve an
error message and not be able to update properly:

W: Size of file /var/lib/apt/lists/partial/archive.example.com_dists_precise_main_binary-amd64_Packages is not what the server reported 0 25118

This should be backported to Ubuntu precise because some companies may
be delivering their internal apt repositories over SSL for security.

[Test Case]

Install apt-transport-https and setup an apt repository delivered over
https. Do some updates and eventually the above error will happen.

[Regression Potential]

I've patched the Ubuntu precise version of apt and found no issues.
Curl - Exceedingly minimal as a variable needed to be reset to zero on structure re-use.Apt - Possible regression if curl is not updated as well, as this adds a dependency on curl's CURLINFO_CONDITION_UNMET which is currently broken in 0.8.16~exp12ubuntu10.12.

[Other Info]

APT
* fix needs to be applied in Precise, quantal and raring
* Apt fix is already in Saucy since at least version 0.9.9.1~ubuntu1

Curl
* fix needs to be applied in Precise, quantal and raring
* fix is in saucy as of version 7.31.0-1ubuntu1

This was reported to Debian first and is fixed in Debian unstable:

http://bugs.debian.org/705648

The maintainers of apt in Ubuntu have not yet synced this to saucy.

Fixes for both ubuntu/apt and ubuntu/curl need to be included for this to be properly fixed.

--
bye,
pabs

http://wiki.debian.org/PaulWise

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apt (Ubuntu):
status: New → Confirmed
OoberMick (oobermick) wrote :

I'm effected by this in precise, any chance of it being fixed there?

Dave Chiluk (chiluk) wrote :

According to
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705783
this looks to also be dependent on a required fix in curl. In order to avoid breaking apt.

http://curl.haxx.se/mail/lib-2013-04/0311.html

Dave Chiluk (chiluk) on 2013-07-19
Changed in curl (Ubuntu):
status: New → Confirmed
Dave Chiluk (chiluk) wrote :

I put together a ppa with fixes for precise for this issue.

Can someone who is experiencing this issue please test this ppa, and let me know if it fixes the issue. Please report back so I can continue with an SRU for this fix.

https://launchpad.net/~chiluk/+archive/lp1179781/

I have attached the debdiffs to curl and apt as well. Both are required.

Dave Chiluk (chiluk) wrote :

Also it should be noted that whoever applies the SRU needs to be careful to maintain the order of patches in the series file for curl, as applying this patch later down the stack will result in a failed build.

The attachment "curl debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
OoberMick (oobermick) wrote :

I've just downloaded the new versions of apt and curl and the issue seems to be resolved.

Dave Chiluk (chiluk) on 2013-07-22
Changed in apt (Ubuntu):
assignee: nobody → Dave Chiluk (chiluk)
Changed in curl (Ubuntu):
assignee: nobody → Dave Chiluk (chiluk)
Dave Chiluk (chiluk) on 2013-07-25
Changed in apt (Ubuntu):
status: Confirmed → In Progress
Changed in curl (Ubuntu):
status: Confirmed → In Progress
Dave Chiluk (chiluk) wrote :

Thanks oobermick, by new versions I assume you are referring to the ones in my ppa, and not the ones in updates right?

Anyhow I'm filling out the SRU now.

Yeah the ones in your PPA.

On 25 July 2013 11:42, Dave Chiluk <email address hidden> wrote:
> Thanks oobermick, by new versions I assume you are referring to the ones
> in my ppa, and not the ones in updates right?
>
> Anyhow I'm filling out the SRU now.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1179781
>
> Title:
> If-Modfied-Since undhandled case causes apt lists corruption with
> https repositories
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1179781/+subscriptions

--
Michael Graham <email address hidden>

Dave Chiluk (chiluk) on 2013-07-25
description: updated
Marc Deslauriers (mdeslaur) wrote :

Thanks for the debdiffs!

Here are a few comments:
1- Could you please fix your two typos in the apt changelog
2- Please change apt version to 0.8.16~exp12ubuntu10.13
3- Please change curl version to 7.22.0-3ubuntu4.3
4- The upstream Debian bug seems to indicate updating apt without updating curl actually causes a regression. Could you please add a specific dependency to the updated curl version in the apt package?

Thanks!

Changed in apt (Ubuntu):
status: In Progress → Incomplete
Changed in curl (Ubuntu):
status: In Progress → Incomplete
Dave Chiluk (chiluk) wrote :

Here's updated patches for curl, for precise, quantal and raring. These will need to be uploaded before apt, as apt has a build-dep on the updated curl.

Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) wrote :

@mdeslaur

Updated the debdiffs, and attached debdiffs for quantal and raring as well. They are pretty much identical with the exception of a few line changes.

Changed in apt (Ubuntu):
status: Incomplete → In Progress
Changed in curl (Ubuntu):
status: Incomplete → In Progress
tags: added: precise quantal raring
Brian Murray (brian-murray) wrote :

I'll take care of sponsoring this.

description: updated
Changed in apt (Ubuntu):
assignee: Dave Chiluk (chiluk) → nobody
importance: Undecided → Medium
status: In Progress → Fix Released
Changed in apt (Ubuntu Precise):
importance: Undecided → Medium
status: New → Triaged
Changed in apt (Ubuntu Quantal):
importance: Undecided → Medium
status: New → Triaged
Changed in apt (Ubuntu Raring):
importance: Undecided → Medium
status: New → Triaged
Changed in curl (Ubuntu Precise):
importance: Undecided → Medium
status: New → Triaged
Changed in curl (Ubuntu Quantal):
importance: Undecided → Medium
status: New → Triaged
Changed in curl (Ubuntu Raring):
importance: Undecided → Medium
status: New → Triaged
Changed in curl (Ubuntu):
assignee: Dave Chiluk (chiluk) → nobody
importance: Undecided → Medium
status: In Progress → Fix Released
Changed in apt (Debian):
status: Unknown → Fix Released
Changed in curl (Debian):
status: Unknown → Fix Released
Dave Chiluk (chiluk) on 2013-08-26
Changed in apt (Ubuntu):
assignee: nobody → Dave Chiluk (chiluk)
Changed in curl (Ubuntu):
assignee: nobody → Dave Chiluk (chiluk)

Hello Paul, or anyone else affected,

Accepted curl into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/curl/7.22.0-3ubuntu4.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in curl (Ubuntu Precise):
status: Triaged → Fix Committed
tags: added: verification-needed
Steve Langasek (vorlon) wrote :

Hello Paul, or anyone else affected,

Accepted curl into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/curl/7.27.0-1ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in curl (Ubuntu Quantal):
status: Triaged → Fix Committed
Changed in curl (Ubuntu Raring):
status: Triaged → Fix Committed
Steve Langasek (vorlon) wrote :

Hello Paul, or anyone else affected,

Accepted curl into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/curl/7.29.0-1ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Dave Chiluk (chiluk) wrote :

Updated apt so that
apt-transport-https depends on libcurl3-gnutls (appropriate curl version) instead of a builddep for the whole package.

Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) wrote :

@slangasek

Verification for curl can not really be completed, until the updated apt gets pushed. That's because the issue that exists in curl only gets exacerbated when using the updated apt.

Steve Langasek (vorlon) wrote :

Hello Paul, or anyone else affected,

Accepted apt into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.9.7.5ubuntu5.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in apt (Ubuntu Quantal):
status: Triaged → Fix Committed
Changed in apt (Ubuntu Raring):
status: Triaged → Fix Committed
Steve Langasek (vorlon) wrote :

Hello Paul, or anyone else affected,

Accepted apt into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.9.7.7ubuntu5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Steve Langasek (vorlon) on 2013-09-04
Changed in apt (Ubuntu Precise):
status: Triaged → Fix Committed
Dave Chiluk (chiluk) wrote :

I got the below e-mails after the upload and build today.

"
Hello Dave Chiluk,

On 2013-09-04 23:30z (2 hours 46 minutes ago), you uploaded a file with
translation templates for apt in Ubuntu Raring package "apt" to
Launchpad.

We were unable to import the file because of errors in its format:

No header found in this pofile

If you use gettext, you can check your file for correct formatting with
the 'msgfmt -c' command. Please fix any errors raised by msgfmt and
upload the file again. If you check the file and you don't find any
error in it, please look for an answer or file a question at
https://answers.launchpad.net/rosetta/

For your convenience, you can get the file you uploaded at:
http://launchpadlibrarian.net/149368746/apt.pot

Thank you,

The Launchpad team
"

So I ran the below, but I'm afraid I'm still not sure what to do.

$ msgfmt -c apt.pot
msgfmt: apt.pot: warning: PO file header missing or invalid
                 warning: charset conversion will not work
msgfmt: found 1 fatal error

I rechecked my uploads, and didn't touch the translation files, so I'm not quite sure what's going on here.

Dave Chiluk (chiluk) wrote :

Infinity, just let me know on irc to ignore this, as a known issue.

Precise-verified

tags: added: verification-done-precise
removed: verification-needed
Paul Wise (Debian) (pabs) wrote :

I've tested the updates on Ubuntu precise and they look fine.

Steve Langasek (vorlon) wrote :

Hello Paul, or anyone else affected,

Accepted apt into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt/0.8.16~exp12ubuntu10.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Dave Chiluk (chiluk) on 2013-09-08
tags: added: verification-needed-quantal
removed: verification-needed
tags: added: verification-needed-raring
Brian Murray (brian-murray) wrote :

The apt package was only added to precise-proposed on 9/7 after the verification-done-precise tag was added. I've now removed the tag, could someone verify with the version of apt and curl from -proposed?

tags: added: verification-need-precise verification-needed
removed: verification-done-precise
Dave Chiluk (chiluk) on 2013-09-13
tags: added: verification-done-precise verification-done-raring
removed: verification-need-precise verification-needed-raring
Dave Chiluk (chiluk) wrote :

Spun up a few VMs, and did verification, everything seems to be going swimingly. Marked as verified.

tags: added: verification-done verification-done-quantal
removed: verification-needed verification-needed-quantal
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 0.8.16~exp12ubuntu10.14

---------------
apt (0.8.16~exp12ubuntu10.14) precise; urgency=low

  * Fix unhandled If-Modified-Since case that causes apt lists corruption.
    LP: #1179781
 -- Dave Chiluk <email address hidden> Tue, 20 Aug 2013 09:56:40 -0500

Changed in apt (Ubuntu Precise):
status: Fix Committed → Fix Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package curl - 7.22.0-3ubuntu4.3

---------------
curl (7.22.0-3ubuntu4.3) precise; urgency=low

  * Reset timecond when clearing session-info variables (LP: #1179781)
    This fixes CURLINFO_CONDITION_UNMET incorrectly reporting "1"
 -- Dave Chiluk <email address hidden> Fri, 23 Aug 2013 16:05:09 -0700

Changed in curl (Ubuntu Precise):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 0.9.7.7ubuntu5

---------------
apt (0.9.7.7ubuntu5) raring; urgency=low

  * Fix unhandled If-Modified-Since case that causes apt lists corruption.
    LP: #1179781
 -- Dave Chiluk <email address hidden> Wed, 21 Aug 2013 13:12:14 -0500

Changed in apt (Ubuntu Raring):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package curl - 7.29.0-1ubuntu3.2

---------------
curl (7.29.0-1ubuntu3.2) raring; urgency=low

  * Reset timecond when clearing session-info variables (LP: #1179781)
    This fixes CURLINFO_CONDITION_UNMET incorrectly reporting "1"
 -- Dave Chiluk <email address hidden> Wed, 21 Aug 2013 13:09:13 -0500

Changed in curl (Ubuntu Raring):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package curl - 7.27.0-1ubuntu1.4

---------------
curl (7.27.0-1ubuntu1.4) quantal; urgency=low

  * Reset timecond when clearing session-info variables (LP: #1179781)
    This fixes CURLINFO_CONDITION_UNMET incorrectly reporting "1"
 -- Dave Chiluk <email address hidden> Fri, 23 Aug 2013 14:58:40 -0700

Changed in curl (Ubuntu Quantal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt - 0.9.7.5ubuntu5.5

---------------
apt (0.9.7.5ubuntu5.5) quantal; urgency=low

  * Backport kernel auto-removal/retention policy from raring (LP: #923876)
    - debian/apt.auto-removal.sh, debian/rules, debian/apt.dirs: Add new
      script to /etc/kernel/postinst.d/ that ensures we always retain the
      currently-running, being-installed, and newest-installed kernels.
    - debian/apt.conf.autoremove: don't include linux-restricted-modules*,
      linux-image*, and linux-ubuntu-modules* in the never-removed list.
  * Fix unhandled If-Modified-Since case that causes apt lists corruption.
    LP: #1179781
 -- Dave Chiluk <email address hidden> Wed, 21 Aug 2013 13:14:06 -0500

Changed in apt (Ubuntu Quantal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.