The problem here is that S3 is broken regarding HTTP pipeline responses. What is happening is that the client is that apt is requesting files A, B, C, D in the same requests and S3 appears to be responding with files B, C, D, A (or some simularly arbitrary order). The result is that apt saves the data under the wrong file name.
Here is a summary of a packet trace (edited for readability). S3 uses the "ETag" header to store the MD5 of the file. So using the the Etag, you can conclude what file contents are.
GET /ubuntu/pool/main/libx/libxdamage/libxdamage1_1.1.3-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxfixes/libxfixes3_5.0-4ubuntu1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxxf86vm/libxxf86vm1_1.1.1-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/m/mesa/libgl1-mesa-glx_8.0.1-0ubuntu2_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/g/gtk+2.0/libgtk2.0-common_2.24.10-0ubuntu5_all.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxcomposite/libxcomposite1_0.4.3-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxcursor/libxcursor1_1.1.12-1_amd64.deb HTT
GET /ubuntu/pool/main/libx/libxi/libxi6_1.5.99.3-0ubuntu2_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxinerama/libxinerama1_1.1.1-3build1_amd64.deb HTTP/1.1 6e773d7e05527fa35a201209660b517c - libxfixes3 x-amz-id-2: EgVy5vizxmUOYRxfyjFXJzt7k30EVGtn2gyZ1q0P/jN3ROZS+3KjUEwSzX3JFJKE x-amz-request-id: 8CD6DF060D72CFDA c06221dc919029cf8482198e724fa111 - libxxf86vm1 x-amz-id-2: xr56YqhmQzov1YP+Oev9lnpKQtOYX98DkrJD7aWkSwQq8Ke77+4PD4yHiRvcBo6w x-amz-request-id: 67B263265E35A521
GET /ubuntu/pool/main/g/gtk+2.0/libgtk2.0-0_2.24.10-0ubuntu5_amd64.deb HTTP/1.1 a47eb17218d742a4ceae59cc08e80d8c - libgl1-mesa-glx x-amz-id-2: Ou92TrRA7dfHyF97UEfO0TvQ9XI9vOxYj0LgWCR3GXFHMhAZ7nzQ71cGUqMQHI1X x-amz-request-id: C9393E43C43C6471 90787d373defad09a6dd99db3c3e7614 - libgtk2.0-common x-amz-id-2: ze4hg1y4hT8+ZXXzY7JHRclypybdYj+HIQEG6r2eulAt+twaZn/GG+y6ifi9MnL8 x-amz-request-id: 5DDC52A0301600FC 77440ea3c635a3243cc3f6bc98305f22 - libxcomposite1 x-amz-id-2: +hzkmvonwW+1nh4gcsQ/qXsmXUq2CUj4hAQ69EjqAEb6MVOpxHvPnh8l7Qh2Wc5Q x-amz-request-id: 54FDE3289C1021F3 f5754520b029296a6f0a5a14a1dd4b5a - libxcursor1 x-amz-id-2: WvhZ9o1R3kMvsFEmERvI2V/eOY+0ksHnvs1suVCX1Es8aa/XxyU7drgvj0Bj9zHj x-amz-request-id: 9F408ACCBFFA66E5 728ef8819d6f3ec4b827d887951a86ce - libxi6 x-amz-id-2: Tgqse2GgWGu+OEVzv/atXhjP6FdszvjVeFGic3gq5LxGUXDFVS5YdF+CGFtF7dkC x-amz-request-id: C57783745F1192CE 90b84d263ad2b7f76868ce02967c2107 - libxinerama1 x-amz-id-2: 1Mz7B1r13S1NnNIingOxj2iOlQq8a5arP7K/Wt/zgd+NL78Q73/MRDzFhPf4kXFL x-amz-request-id: DB666BE3004C6CD9 652e1f2550726fc72fa432413e6e287b - libxdamage1 x-amz-id-2: X5TQqCJey0xtne7jKZYlX+8c5AqRM2aTVD1cvEHWPIx4qcc8L9rvyqlG0FJcPT4g x-amz-request-id: 74A2EB9093B4F00A c8f474321079048af432d47d66a4afb4 - libgtk2.0-0 x-amz-id-2: Fgkgv/JY7WnSojk+pBeQ430gNCvjRvXG1KZzrbQfEG2j/jqPvvcFVRS07ZBMsFa9 x-amz-request-id: 57FF62A675D21685
GET /ubuntu/pool/main/g/gtk+2.0/libgtk2.0-bin_2.24.10-0ubuntu5_amd64.deb HTTP/1.1 e03a09b33f955e8da797a2f33fbf3423 - libgtk2.0-bin x-amz-id-2: RaJe83jyFhTwQdvnp6ZjYJQMwKOIengoH2U//VnJZxgagvTPmGYMsAwKGkjGh+Mz x-amz-request-id: B9D4754504E9CA3E
The problem here is that S3 is broken regarding HTTP pipeline responses. What is happening is that the client is that apt is requesting files A, B, C, D in the same requests and S3 appears to be responding with files B, C, D, A (or some simularly arbitrary order). The result is that apt saves the data under the wrong file name.
Here is a summary of a packet trace (edited for readability). S3 uses the "ETag" header to store the MD5 of the file. So using the the Etag, you can conclude what file contents are.
GET /ubuntu/ pool/main/ libx/libxdamage /libxdamage1_ 1.1.3-2build1_ amd64.deb HTTP/1.1 pool/main/ libx/libxfixes/ libxfixes3_ 5.0-4ubuntu1_ amd64.deb HTTP/1.1 pool/main/ libx/libxxf86vm /libxxf86vm1_ 1.1.1-2build1_ amd64.deb HTTP/1.1 pool/main/ m/mesa/ libgl1- mesa-glx_ 8.0.1-0ubuntu2_ amd64.deb HTTP/1.1 pool/main/ g/gtk+2. 0/libgtk2. 0-common_ 2.24.10- 0ubuntu5_ all.deb HTTP/1.1 pool/main/ libx/libxcompos ite/libxcomposi te1_0.4. 3-2build1_ amd64.deb HTTP/1.1 pool/main/ libx/libxcursor /libxcursor1_ 1.1.12- 1_amd64. deb HTT pool/main/ libx/libxi/ libxi6_ 1.5.99. 3-0ubuntu2_ amd64.deb HTTP/1.1 pool/main/ libx/libxineram a/libxinerama1_ 1.1.1-3build1_ amd64.deb HTTP/1.1
6e773d7e05527f a35a201209660b5 17c - libxfixes3
x- amz-id- 2: EgVy5vizxmUOYRx fyjFXJzt7k30EVG tn2gyZ1q0P/ jN3ROZS+ 3KjUEwSzX3JFJKE
x- amz-request- id: 8CD6DF060D72CFDA
c06221dc919029 cf8482198e724fa 111 - libxxf86vm1
x- amz-id- 2: xr56YqhmQzov1YP +Oev9lnpKQtOYX9 8DkrJD7aWkSwQq8 Ke77+4PD4yHiRvc Bo6w
x- amz-request- id: 67B263265E35A521 pool/main/ g/gtk+2. 0/libgtk2. 0-0_2.24. 10-0ubuntu5_ amd64.deb HTTP/1.1
a47eb17218d742 a4ceae59cc08e80 d8c - libgl1-mesa-glx
x- amz-id- 2: Ou92TrRA7dfHyF9 7UEfO0TvQ9XI9vO xYj0LgWCR3GXFHM hAZ7nzQ71cGUqMQ HI1X
x- amz-request- id: C9393E43C43C6471
90787d373defad 09a6dd99db3c3e7 614 - libgtk2.0-common
x- amz-id- 2: ze4hg1y4hT8+ ZXXzY7JHRclypyb dYj+HIQEG6r2eul At+twaZn/ GG+y6ifi9MnL8
x- amz-request- id: 5DDC52A0301600FC
77440ea3c635a3 243cc3f6bc98305 f22 - libxcomposite1
x- amz-id- 2: +hzkmvonwW+ 1nh4gcsQ/ qXsmXUq2CUj4hAQ 69EjqAEb6MVOpxH vPnh8l7Qh2Wc5Q
x- amz-request- id: 54FDE3289C1021F3
f5754520b02929 6a6f0a5a14a1dd4 b5a - libxcursor1
x- amz-id- 2: WvhZ9o1R3kMvsFE mERvI2V/ eOY+0ksHnvs1suV CX1Es8aa/ XxyU7drgvj0Bj9z Hj
x- amz-request- id: 9F408ACCBFFA66E5
728ef8819d6f3e c4b827d887951a8 6ce - libxi6
x- amz-id- 2: Tgqse2GgWGu+ OEVzv/atXhjP6Fd szvjVeFGic3gq5L xGUXDFVS5YdF+ CGFtF7dkC
x- amz-request- id: C57783745F1192CE
90b84d263ad2b7 f76868ce02967c2 107 - libxinerama1
x- amz-id- 2: 1Mz7B1r13S1NnNI ingOxj2iOlQq8a5 arP7K/Wt/ zgd+NL78Q73/ MRDzFhPf4kXFL
x- amz-request- id: DB666BE3004C6CD9
652e1f2550726f c72fa432413e6e2 87b - libxdamage1
x- amz-id- 2: X5TQqCJey0xtne7 jKZYlX+ 8c5AqRM2aTVD1cv EHWPIx4qcc8L9rv yqlG0FJcPT4g
x- amz-request- id: 74A2EB9093B4F00A
c8f47432107904 8af432d47d66a4a fb4 - libgtk2.0-0
x- amz-id- 2: Fgkgv/JY7WnSojk +pBeQ430gNCvjRv XG1KZzrbQfEG2j/ jqPvvcFVRS07ZBM sFa9
x- amz-request- id: 57FF62A675D21685 pool/main/ g/gtk+2. 0/libgtk2. 0-bin_2. 24.10-0ubuntu5_ amd64.deb HTTP/1.1
e03a09b33f955e 8da797a2f33fbf3 423 - libgtk2.0-bin
x- amz-id- 2: RaJe83jyFhTwQdv np6ZjYJQMwKOIen goH2U// VnJZxgagvTPmGYM sAwKGkjGh+ Mz
x- amz-request- id: B9D4754504E9CA3E
GET /ubuntu/
GET /ubuntu/
GET /ubuntu/
GET /ubuntu/
GET /ubuntu/
GET /ubuntu/
GET /ubuntu/
GET /ubuntu/
GET /ubuntu/
GET /ubuntu/