poldek: internal fetcher doesn't support chunked encoding as required by HTTP 1.1

[Arkadiusz Miśkiewicz]

+ poldek -q --st=metadata --source http://dl.google.com/linux/chrome/rpm/stable/x86_64/ --update
błąd: vfff: chunked: unimplemented HTTP transfer encoding

[Elan Ruusamäe (glen666)]

with http proxy set, it gives even more bizarre error

# poldek -vv --st=metadata --source http://dl.google.com/linux/chrome/rpm/stable/x86_64/ --update
Setting temporary cache directory path to /root/.poldek-cache
Setting cache directory path to /root/.poldek-cache
Enabling multilib mode, transaction color = 3
Retrieving http://dl.google.com/linux/chrome/rpm/[...]/repomd.xml...
Connecting to dl.google.com:80...
< GET /linux/chrome/rpm/stable/x86_64/repodata/repomd.xml HTTP/1.1
< Host: dl.google.com
< User-Agent: poldek-vhttp/0.30
> HTTP/1.0 200 OK
> Last-Modified: Wed, 18 Jul 2012 23:20:00 GMT
> Accept-Ranges: bytes
> Content-Type: application/xml
> ETag: 2f22e
> Vary: *
> Date: Fri, 20 Jul 2012 08:40:32 GMT
> Server: downloads
> Cache-Control: private
> X-XSS-Protection: 1; mode=block
> X-Frame-Options: SAMEORIGIN
> X-Cache: MISS from proxy.local
> X-Cache-Lookup: MISS from proxy.local:3128
> Via: 1.0 proxy.local:3128 (squid/2.7.STABLE9)
> Connection: close
error: vfff: Content-Length parse error ((null))
Retrying...(#2)
Connecting to dl.google.com:80...
< GET /linux/chrome/rpm/stable/x86_64/repodata/repomd.xml HTTP/1.1
< Host: dl.google.com
< User-Agent: poldek-vhttp/0.30
> HTTP/1.0 200 OK
> Last-Modified: Wed, 18 Jul 2012 23:20:00 GMT
> Accept-Ranges: bytes
> Content-Type: application/xml
> ETag: 2f22e
> Vary: *
> Date: Fri, 20 Jul 2012 08:40:33 GMT
> Server: downloads
> Cache-Control: private
> X-XSS-Protection: 1; mode=block
> X-Frame-Options: SAMEORIGIN
> X-Cache: MISS from proxy.localproxy.local
> X-Cache-Lookup: MISS from proxy.local:3128
> Via: 1.0 proxy.local:3128 (squid/2.7.STABLE9)
> Connection: close
error: vfff: Content-Length parse error ((null))
Retrying...(#3)
^C

[Elan Ruusamäe (glen666)]

and enforcing external fetcher also fails:

# poldek -vv -O 'default fetcher = http: wget' --st=metadata --source http://dl.google.com/linux/chrome/rpm/stable/x86_64/ --update
Setting temporary cache directory path to /root/.poldek-cache
Setting cache directory path to /root/.poldek-cache
Enabling multilib mode, transaction color = 3
error: vfile: wget: no such external fetcher found
Retrieving http://dl.google.com/linux/chrome/rpm/[...]/repomd.xml...
...

# cat /etc/poldek/fetch.conf
...

[fetcher]
name = wget
proto = http,ftp,https
cmd = wget -N --dot-style=binary -P %d %Pn

[Cactus (uzi18)]

But aria2 works:
poldek -vv -O 'default fetcher = http: aria' --st=metadata --source http://dl.google.com/linux/chrome/rpm/stable/x86_64/ --update

[Cactus (uzi18)]

[fetcher]
name = aria
proto = http,ftp,https
cmd = aria2c --log-level=info --summary-interval=0 --truncate-console-readout=true -c -k1M -j5 -s5 -x5 -Z -P -d %d %Pn

[Cactus (uzi18)]

wget have no support for transfer-encoding, but curl have.

[Arkadiusz Miśkiewicz (arekm)]

Well, wget works anyway, just doesn't know the size:

$ LC_ALL=C wget -S http://dl.google.com/linux/chrome/rpm/stable/x86_64/repodata/repomd.xml
--2012-07-21 14:45:01-- http://dl.google.com/linux/chrome/rpm/stable/x86_64/repodata/repomd.xml
Resolving dl.google.com (dl.google.com)... 2a00:1450:4001:c01::5d, 173.194.70.190, 173.194.70.91, ...
Connecting to dl.google.com (dl.google.com)|2a00:1450:4001:c01::5d|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Last-Modified: Wed, 18 Jul 2012 23:20:00 GMT
  Accept-Ranges: bytes
  Content-Type: application/xml
  ETag: 2f22e
  Vary: *
  Date: Sat, 21 Jul 2012 12:45:01 GMT
  Server: downloads
  Cache-Control: private
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Transfer-Encoding: chunked
Length: unspecified [application/xml]
Saving to: `repomd.xml'

    [ <=> ] 951 --.-K/s in 0s

2012-07-21 14:45:01 (118 MB/s) - `repomd.xml' saved [951]

[Cactus (uzi18)]

But it is imposible to interpret repomd.xml file because no chunks support in wget.

[Elan Ruusamäe (glen666)]

this did not work, because %include fetch.conf was not enabled in poldek.conf:

# poldek -vv -O 'default fetcher = http: wget'

imho fetch.conf should be loaded by default (to have fetchers defined), but it should not complain on missing commands and enable any of the fetchers by default (unless default = yes or similar is also specified in [fetcher])

[Cactus (uzi18)]

Glen here is the patch propose for chunked transfers.
Please patch Your sources and test if it works for You i will apply it to git.

Thanks

[Elan Ruusamäe (glen666)]

just apply it, lets assume it works, i gave pointers how to test here, if you don't bother testing with that url, then just apply it