IBP: HTTP pipelining is turned on by default for apt

[Alexander Gordeev]

HTTP pipelining is broken due to a bug in recent gateway firmware https://tools.cisco.com/quickview/bug/CSCei34455

It causes to slow down http connection speed extremely (eg. to 12kbps even on 300M uplink), thus making impossible:
1) to build ubuntu image and to fit within 1h timeout (it could be worked around by creating local mirror via fuel-createmirror script)
2) to deploy cluster. As granular tasks are using apt tools too. They were failed by timeout too.

In order to disable pipelining, we should add the following settings to apt.conf.d

echo "Acquire::http::No-Cache true;" >> /etc/apt/apt.conf.d/80http

echo "Acquire::http::Pipeline-Depth 0;" >> /etc/apt/apt.conf.d/80http

[Alexander Gordeev (a-gordeev)]

according to https://en.wikipedia.org/wiki/HTTP_pipelining

> The pipelining of requests results in a dramatic improvement in the loading times of HTML pages, especially over high latency connections such as satellite Internet connections. The speedup is less apparent on broadband connections, as the limitation of HTTP 1.1 still applies

> Out of all the major browsers, only Opera based on Presto layout engine had a fully working implementation that was enabled by default. In all other browsers HTTP pipelining is disabled or not implemented.

It sounds like, HTTP pipelining could be disabled without any harm as nobody uses it actually. Only small amount of HTTP related software has enabled it by default.

[Alexander Gordeev (a-gordeev)]

Unfortunately, the adding of custom settings for apt is not implemented yet. Placing the suggested settings regardless of user's wishes doesn't sound right.

Bug won't be fixed, but work around will be documented.

Only 2 steps required:

1. It is recommended to use 'fuel-createmirror' script in order to get images built. More details could be found at the link https://docs.mirantis.com/openstack/fuel/fuel-6.1/operations.html#setting-up-local-mirrors NOTE: it may take a while.
2. Then, one should start to deploy a cluster. Once deployment fails with task timeout error right after the provisioning of nodes has been done, add a file with suggested settings to all nodes. NOTE: step could be performed via ssh or rsync from fuel-master node.

Create a new file with the following content:
Acquire::http::No-Cache true;
Acquire::http::Pipeline-Depth 0;

Save it somewhere. Copy to every node of your cluster and store it under /etc/apt/apt.conf.d/80http
After that one should proceed to re-deploy the cluster in order to apply new settings for apt. Deployment should succeed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Aleksandr Gordeev (<email address hidden>) on branch: master
Review: https://review.openstack.org/197603
Reason: Decided not to fix it.

[Alexander Adamov (aadamov)]

Add to Knowledge Base.

[Michele Fagan (michelefagan)]

Review for inclusion as KB for 9.0.

[Bug Checker Bot (esikachev-l)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Michele Fagan (michelefagan)]

Review for inclusion as KB for 9.0.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/352922
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=426d2a39f421773fbe682ebab5297c20a587e3cc
Submitter: Jenkins
Branch: master

commit 426d2a39f421773fbe682ebab5297c20a587e3cc
Author: Dmitry Teselkin <email address hidden>
Date: Tue Aug 9 16:58:43 2016 +0300

    Fix bootstrap apt-get update issues

    Sometimes 'apt-get update' fail with strange error:

      Get:22 http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0 mos10.0-updates Release.gpg [473 B]
      Err:22 http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0 mos10.0-updates Release.gpg
        Writing more data than expected (473 > 20) [IP: 208.78.244.194 80]

    This is only reproduced during running bootstrap, and never reproduced manually.

    APT::http::Pipeline-Depth=0 fixes the problem.

    By default pipeline depth is 10 in apt 1.2.10 (Ubuntu Xenial) [1], [2].

    [1] https://github.com/Debian/apt/blob/1.2.10/doc/apt.conf.5.xml#L402-L407
    [2] https://github.com/Debian/apt/blob/1.2.10/methods/server.cc#L754-L756

    Related-bug: #1612731
    Related-bug: #1470468

    Change-Id: Iac8c22d87f18718ae9a318dda56266f8ed57f866