IBP: HTTP pipelining is turned on by default for apt

Bug #1470468 reported by Alexander Gordeev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Won't Fix
High
Evgeny Konstantinov
6.1.x
Won't Fix
High
Fuel Documentation Team
7.0.x
Won't Fix
Undecided
Fuel Documentation Team
8.0.x
Won't Fix
Medium
Fuel Documentation Team
Future
Won't Fix
Undecided
Fuel Documentation Team
Mitaka
Won't Fix
High
Evgeny Konstantinov

Bug Description

HTTP pipelining is broken due to a bug in recent gateway firmware https://tools.cisco.com/quickview/bug/CSCei34455

It causes to slow down http connection speed extremely (eg. to 12kbps even on 300M uplink), thus making impossible:
1) to build ubuntu image and to fit within 1h timeout (it could be worked around by creating local mirror via fuel-createmirror script)
2) to deploy cluster. As granular tasks are using apt tools too. They were failed by timeout too.

In order to disable pipelining, we should add the following settings to apt.conf.d

echo "Acquire::http::No-Cache true;" >> /etc/apt/apt.conf.d/80http

echo "Acquire::http::Pipeline-Depth 0;" >> /etc/apt/apt.conf.d/80http

Revision history for this message
Alexander Gordeev (a-gordeev) wrote :
affects: mos → fuel
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
milestone: none → 7.0
Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

according to https://en.wikipedia.org/wiki/HTTP_pipelining

> The pipelining of requests results in a dramatic improvement in the loading times of HTML pages, especially over high latency connections such as satellite Internet connections. The speedup is less apparent on broadband connections, as the limitation of HTTP 1.1 still applies

> Out of all the major browsers, only Opera based on Presto layout engine had a fully working implementation that was enabled by default. In all other browsers HTTP pipelining is disabled or not implemented.

It sounds like, HTTP pipelining could be disabled without any harm as nobody uses it actually. Only small amount of HTTP related software has enabled it by default.

Changed in fuel:
assignee: Fuel provisioning team (fuel-provisioning) → Aleksandr Gordeev (a-gordeev)
Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Alexander Gordeev (a-gordeev) wrote :

Unfortunately, the adding of custom settings for apt is not implemented yet. Placing the suggested settings regardless of user's wishes doesn't sound right.

Bug won't be fixed, but work around will be documented.

Only 2 steps required:

1. It is recommended to use 'fuel-createmirror' script in order to get images built. More details could be found at the link https://docs.mirantis.com/openstack/fuel/fuel-6.1/operations.html#setting-up-local-mirrors NOTE: it may take a while.
2. Then, one should start to deploy a cluster. Once deployment fails with task timeout error right after the provisioning of nodes has been done, add a file with suggested settings to all nodes. NOTE: step could be performed via ssh or rsync from fuel-master node.

Create a new file with the following content:
Acquire::http::No-Cache true;
Acquire::http::Pipeline-Depth 0;

Save it somewhere. Copy to every node of your cluster and store it under /etc/apt/apt.conf.d/80http
After that one should proceed to re-deploy the cluster in order to apply new settings for apt. Deployment should succeed.

Changed in fuel:
assignee: Aleksandr Gordeev (a-gordeev) → Fuel Documentation Team (fuel-docs)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-web (master)

Change abandoned by Aleksandr Gordeev (<email address hidden>) on branch: master
Review: https://review.openstack.org/197603
Reason: Decided not to fix it.

Changed in fuel:
status: In Progress → New
Igor Shishkin (teran)
no longer affects: fuel/7.0.x
Changed in fuel:
milestone: 7.0 → 8.0
Dmitry Pyzhov (dpyzhov)
tags: added: docs
Dmitry Pyzhov (dpyzhov)
tags: added: area-docs
Revision history for this message
Alexander Adamov (aadamov) wrote :

Add to Knowledge Base.

Changed in fuel:
status: New → Confirmed
Changed in fuel:
milestone: 8.0 → 9.0
status: Confirmed → New
Revision history for this message
Michele Fagan (michelefagan) wrote :

Review for inclusion as KB for 9.0.

Changed in fuel:
status: New → Confirmed
Revision history for this message
Bug Checker Bot (esikachev-l) wrote : Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
Revision history for this message
Michele Fagan (michelefagan) wrote :

Review for inclusion as KB for 9.0.

Changed in fuel:
assignee: Fuel Documentation Team (fuel-docs) → Evgeny Konstantinov (evkonstantinov)
tags: removed: need-info
Changed in fuel:
milestone: 9.0 → 10.0
Changed in fuel:
status: Confirmed → Won't Fix
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-agent (master)

Reviewed: https://review.openstack.org/352922
Committed: https://git.openstack.org/cgit/openstack/fuel-agent/commit/?id=426d2a39f421773fbe682ebab5297c20a587e3cc
Submitter: Jenkins
Branch: master

commit 426d2a39f421773fbe682ebab5297c20a587e3cc
Author: Dmitry Teselkin <email address hidden>
Date: Tue Aug 9 16:58:43 2016 +0300

    Fix bootstrap apt-get update issues

    Sometimes 'apt-get update' fail with strange error:

      Get:22 http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0 mos10.0-updates Release.gpg [473 B]
      Err:22 http://mirror.fuel-infra.org/mos-repos/ubuntu/10.0 mos10.0-updates Release.gpg
        Writing more data than expected (473 > 20) [IP: 208.78.244.194 80]

    This is only reproduced during running bootstrap, and never reproduced manually.

    APT::http::Pipeline-Depth=0 fixes the problem.

    By default pipeline depth is 10 in apt 1.2.10 (Ubuntu Xenial) [1], [2].

    [1] https://github.com/Debian/apt/blob/1.2.10/doc/apt.conf.5.xml#L402-L407
    [2] https://github.com/Debian/apt/blob/1.2.10/methods/server.cc#L754-L756

    Related-bug: #1612731
    Related-bug: #1470468

    Change-Id: Iac8c22d87f18718ae9a318dda56266f8ed57f866

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.