[apt-utils] Changing "Section", "Maintainer" or "Depends" (and probably some others) fields in a control file makes apt-ftparchive miss the new package in "Packages" file.

[jean-christophe manciot]

On Linux 4.2.0-30-generic #36-Ubuntu SMP Fri Feb 26 00:58:07 UTC 2016 x86_64 GNU/Linux
with apt-utils 1.0.10.2 corresponding to previously named issue "apt-ftparchive reports wrong file size and checksums in Packages"
Cf. post #2 which matches the current bug description with apt-utils 1.2.4.

Generating Packages with (apt-ftparchive.conf is attached below.):
apt-ftparchive generate --contents=off ./apt-ftparchive.conf
Generated Packages contains:
#################################################################
Package: nemo
...
Installed-Size: 9508
...
Version: 2.8.6-11
...
Filename: dists/wily/system-tools/binary-amd64/nemo_2.8.6-11_amd64.deb
Size: 1392338
MD5sum: de162d539df28402f54761f671259278
SHA1: fb5ebacfc032071d3715dce18640deb649732048
SHA256: d0c52eb9bd8224e3d876dc6426ccc1ed49f975038152c53faacd5e71e0a19a33
SHA512: e1cc7eb9865c9f3f5a8d4a8f4406862617e285a080bf6c894b3295088a4406a149b53e27ae379c43335a4231e60966f7068412e93173a6e9bfaa8c48b43ac616
#################################################################

However, on disk, we have very different file size & checksums:
ll nemo*
-rw-r--r-- 1 actionmystique actionmystique 1082324 Feb 27 15:01 nemo_2.8.6-11_amd64.deb

md5sum nemo_2.8.6-11_amd64.deb
68d5c945d9ccd9d3d2807ba46e33cc42 nemo_2.8.6-11_amd64.deb

If we generate Packages differently with:
apt-ftparchive packages . > Packages
Generated Packages contains this time correct file size & checksums:
#################################################################
Package: nemo
...
Installed-Size: 5284
...
Version: 2.8.6-11
...
Filename: ./nemo_2.8.6-11_amd64.deb
Size: 1082324
MD5sum: 68d5c945d9ccd9d3d2807ba46e33cc42
SHA1: eecee7a24c20843c15997adcddcd3dad38c9c831
SHA256: 83451ab9c9975dc7e18d43f2a7d223000c4ee84c9383e82a6461bffef641760a
SHA512: ce715c1d480607e347511562f6d29d10eddb900d2ced3a82d1db08e2bc1a9c24974d3c59cc1e1e83b19ba5dac90d0cc80c7288606e6e3ac06d3551df3550ad09
#################################################################

What is very strange is that all other packages in the same folder are correctly described by Packages when using the first method for generating Packages.

I have the same issue with another package in another folder.

I have also uploaded the deb.
Any suggestion?

N.B: I can upload the second package if necessary.

[jean-christophe manciot (manciot-jeanchristophe)]

With apt-utils 1.2.4, the behavior has changed: 5 packages are not even listed in Packages anymore, including nemo_2.8.6-11_amd64.deb package:

Here, 24 packages are located in the folder:

Ubuntu/dists/wily/system-tools/binary-amd64# ll *.deb
-rw-r--r-- 1 root root 1116256 Mar 5 12:19 apt_1.2.4_amd64.deb
-rw-r--r-- 1 root root 342418 Mar 5 12:19 apt-doc_1.2.4_all.deb
-rw-r--r-- 1 root root 151352 Mar 5 12:19 apt-transport-https_1.2.4_amd64.deb
-rw-r--r-- 1 root root 387240 Mar 5 12:19 apt-utils_1.2.4_amd64.deb
-rw-r--r-- 1 actionmystique actionmystique 12244162 Feb 28 16:02 gimp_2.9.2-11_amd64.deb
-rw-rw-r-- 1 actionmystique actionmystique 3663042 Feb 23 11:21 git_2.6.0-11_amd64.deb
-rw-rw-r-- 1 actionmystique actionmystique 3647806 Feb 23 11:21 git_2.6.3-11_amd64.deb
-rw-r--r-- 1 actionmystique actionmystique 3684722 Feb 28 12:03 git_2.7.2-11_amd64.deb
-rw-rw-r-- 1 actionmystique actionmystique 1646294 Feb 23 11:21 gparted_0.24.0-11_amd64.deb
-rw-rw-r-- 1 actionmystique actionmystique 1653508 Feb 28 15:32 gparted_0.25.0-11_amd64.deb
-rw-r--r-- 1 root root 180902 Mar 5 12:19 libapt-inst2.0_1.2.4_amd64.deb
-rw-r--r-- 1 root root 813750 Mar 5 12:19 libapt-pkg5.0_1.2.4_amd64.deb
-rw-r--r-- 1 root root 221594 Mar 5 12:19 libapt-pkg-dev_1.2.4_amd64.deb
-rw-r--r-- 1 root root 994680 Mar 5 12:20 libapt-pkg-doc_1.2.4_all.deb
-rw-r--r-- 1 actionmystique actionmystique 408538 Feb 28 14:21 libbabl-0.1-0_0.1.16-11_amd64.deb
-rw-r--r-- 1 actionmystique actionmystique 21538808 Feb 28 14:00 libgegl-0.3-0_0.3.4-11_amd64.deb
-rw-r--r-- 1 actionmystique actionmystique 1072388 Mar 5 12:46 nemo_2.8.6-11_amd64.deb
-rw-rw-r-- 1 actionmystique actionmystique 25388 Feb 23 11:21 nemo-fileroller_2.8.1-11_amd64.deb
-rw-r--r-- 1 actionmystique actionmystique 129268 Feb 23 11:21 nemo-gtkhash_0.7.0-11_amd64.deb
-rw-r--r-- 1 actionmystique actionmystique 779684 Mar 2 11:45 realtek-wifi-drivers_4.2.6-10_amd64.deb
-rw-rw-r-- 1 actionmystique actionmystique 781530 Feb 23 11:21 realtek-wifi-drivers_4.2-rc1-10_amd64.deb
-rw-rw-r-- 1 actionmystique actionmystique 533770 Feb 23 11:21 redis-server_3.0.4-10_amd64.deb
-rw-r--r-- 1 actionmystique actionmystique 538628 Feb 23 11:21 redis-server_3.0.7-10_amd64.deb
-rw-rw-r-- 1 actionmystique actionmystique 1071418 Feb 23 11:21 xca_1.3.2-10_amd64.deb

Only 19 packages are described in Packages (cf. attached file).
I have also uploaded the config file I use for apt-ftparchive.

I can't find what the 5 missing packages have in common to explain this issue.

[jean-christophe manciot (manciot-jeanchristophe)]

Packages has been built the same way with:
apt-ftparchive generate --contents=off ./apt-ftparchive.conf

[jean-christophe manciot (manciot-jeanchristophe)]

With latest release 1.2.6, the issue remains.

[jean-christophe manciot (manciot-jeanchristophe)]

If I remove all sections but one in the tree and in the configuration file (apt-ftparchive.conf" here), the apt-ftparchive-generated Packages contains the exact number of packages.
I won't check whether there is a lower number of sections which triggers that issue (<8) or not, but it's fair to say it has been localized.

[jean-christophe manciot (manciot-jeanchristophe)]

Actually, the story is not over: it is not definitive to have only one section to workaround this issue.
All the packages which are missing from "Packages" have a common point: all of them have at least a different value in one of the fields in their "control" file than their previous release.

For example, one of gns3-gui_1.4.5 dependencies has changed as compared to gns3-gui_1.4.4: gns3-net-converter instead of gns3-converter; here are both control files:

Package: gns3-gui
Priority: extra
Section: net
Installed-Size: 23396
Maintainer: Jean-Christophe Manciot <email address hidden>
Architecture: amd64
Version: 1.4.4-11
Depends: gns3-converter,python3,python3-setuptools,python3-pyqt5,python3-pyqt5.qtsvg,python3-pyqt5.qtwebkit,python3-libcloud,python3-requests,python3-paramiko,python3-configobj
Provides: gns3-gui
Description: GNS3 GUI controls the server using a HTTP REST API.

Package: gns3-gui
Priority: extra
Section: net
Installed-Size: 23344
Maintainer: Jean-Christophe Manciot <email address hidden>
Architecture: amd64
Version: 1.4.5-11
Depends: gns3-net-converter,python3,python3-setuptools,python3-pyqt5,python3-pyqt5.qtsvg,python3-pyqt5.qtwebkit,python3-libcloud,python3-requests,python3-paramiko,python3-configobj
Provides: gns3-gui
Description: GNS3 GUI controls the server using a HTTP REST API.

In that case, the last package - gns3-gui_1.4.5 - is missing from 'Packages".

Another example: I have changed the section and maintainer values between virt-viewer_3.0 and virt-viewer_3.1:

Package: virt-viewer
Priority: extra
Section: checkinstall
Installed-Size: 6488
Maintainer: <email address hidden>
Architecture: amd64
Version: 3.0-11
Depends: openssh-client,desktop-file-utils,spice-client-gtk,libcelt051
Provides: virt-viewer
Description: The virt-viewer application is a lightweight interface for interacting with the graphical display of virtualized guest OS.
 It can display VNC or SPICE, and uses libvirt to lookup the graphical connection details.

Package: virt-viewer
Priority: extra
Section: utils
Installed-Size: 6620
Maintainer: Jean-Christophe Manciot <email address hidden>
Architecture: amd64
Version: 3.1-11
Depends: openssh-client,desktop-file-utils,spice-client-gtk,libcelt051
Provides: virt-viewer
Description: The virt-viewer application is a lightweight interface for interacting with the graphical display of virtualized guest OS.
 It can display VNC or SPICE, and uses libvirt to lookup the graphical connection details.

Again, the last version - virt-viewer_3.1 is missing from "Packages".

Obviously, not all fields are affected by this issue; apt-ftparchive accepts that some of them may change (Installed-Size, Version,...), but others cannot when they could.

I will change the title of this thread to reflect that.

[jean-christophe manciot (manciot-jeanchristophe)]

I've just added apt 1.2.7 to the repository and the following packages are not included in "Packages":
- apt-utils_1.2.7; I've noticed that only "Version" and apt version in "Depends" have changed, and the "Installed Size" has not changed.
- libapt-inst2.0_1.2.7
- libapt-pkg-doc_1.2.7_all.deb

However, I've just found a workaround: if I delete packages-amd64.db and rerun apt-ftparchive, all packages are present in "Packages".