Failed to fetch http://10.109.0.2:8080/2014.2-6.1/ubuntu/x86_64/dists/mos6.1/main/binary-i386/Packages 404 Not Found

[Stanislav Makar]

priority: 1800
type: shell
uids:
- '1'
- '2'
parameters:
  retries: 3
  cmd: apt-get update
  cwd: "/"
  timeout: 180
  interval: 1

2015-03-25T14:52:50 debug: [657] c182cd0e-44f6-41bd-a234-e3e4ce73473e: MC agent 'execute_shell_command', method

stderr: W: Failed to fetch http://10.109.0.2:8080/2014.2-6.1/ubuntu/x86_64/dists/mos6.1/main/binary-i386/Packages 404 Not Found

W: Failed to fetch http://10.109.0.2:8080/2014.2-6.1/ubuntu/x86_64/dists/mos6.1/restricted/binary-i386/Packages 404 Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.

exit code: 100

ISO
http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_6.1_iso/830/

ubuntu , classic provisioning type

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/167930

[Kyrylo Romanenko (kromanenko)]

Confirm on
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "233"
  build_id: "2015-03-26_21-32-43"
  nailgun_sha: "b163f6fc77d6639aaffd9dd992e1ad96951c3bbf"
  python-fuelclient_sha: "e5e8389d8d481561a4d7107a99daae07c6ec5177"
  astute_sha: "3f1ece0318e5e93eaf48802fefabf512ca1dce40"
  fuellib_sha: "9c7716bc2ce6075065d7d9dcf96f4c94662c0b56"
  ostf_sha: "a4cf5f218c6aea98105b10c97a4aed8115c15867"
  fuelmain_sha: "320b5f46fc1b2798f9e86ed7df51d3bda1686c10"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/167930
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=77002e7df9ca0e59dbe701a38d09a96c8b109b9a
Submitter: Jenkins
Branch: master

commit 77002e7df9ca0e59dbe701a38d09a96c8b109b9a
Author: Igor Kalnitsky <email address hidden>
Date: Thu Mar 26 11:46:26 2015 +0200

    Set amd64 architecture for dpkg

    Since bp/separate-mos-from-linux the repo on the master node doesn't have
    a package index for the i386 architecture. Meantime, by default Ubuntu
    installer configures dpkg on slaves to use both amd64 and i386, which is
    wrong because it's not intended.

    This commit explicitly sets the architecture to use - 'amd64'.

    DocImpact
    Closes-Bug: #1436459
    Change-Id: If8a0225c51cec8862b90f9cc62e660b08b41edf8

[Kyrylo Romanenko (kromanenko)]

I have reproduction of this bug on build
{"build_id": "2015-03-31_22-54-44", "ostf_sha": "1c7f799cb0e159737d1f910b13e4073b16d1eb8c", "build_number": "260", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-31_22-54-44", "ostf_sha": "1c7f799cb0e159737d1f910b13e4073b16d1eb8c", "build_number": "260", "api": "1.0", "nailgun_sha": "51ddbe0a2e379e82af531b1c6d45fd386d2f894e", "production": "docker", "python-fuelclient_sha": "9b0b6ab2f09ff75445550b7210104e21df5d8cd6", "astute_sha": "7292fc2a673cb1c32a688a46fd4836ca0500a957", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "178812b1971a900c49a8afc1688afd7475a6ffbb", "fuellib_sha": "27459a89cdac078ec4046a442a11d750c5eccc2e"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "51ddbe0a2e379e82af531b1c6d45fd386d2f894e", "production": "docker", "python-fuelclient_sha": "9b0b6ab2f09ff75445550b7210104e21df5d8cd6", "astute_sha": "7292fc2a673cb1c32a688a46fd4836ca0500a957", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "178812b1971a900c49a8afc1688afd7475a6ffbb", "fuellib_sha": "27459a89cdac078ec4046a442a11d750c5eccc2e"}

Attempted to create cluster of following configuration:
1 controller, 1 compute, 1 cinder
QEMU, Cinder-LVM, Neutron VLAN, no additional components.

Preceding to failure there was 4 error messages detected in web back-end log:
ERROR
[7fad023c0740] (helpers) create_action_log failed: 'ThreadedDict' object has no attribute 'env'

[Ihor Kalnytskyi (ikalnytskyi)]

Kyrylo, which bug? This bug? If so, why did you pos error about ThreadedDict?

[Kyrylo Romanenko (kromanenko)]

I have just mentioned those error messages because they was occurred too. But it looks they are not connected to the current bug.

[Kyrylo Romanenko (kromanenko)]

One more reproduction
Deployment has failed. Method granular_deploy. Failed to execute hook .
---
priority: 1800
type: shell
uids:
- '1'
- '3'
- '2'
parameters:
  retries: 3
  cmd: apt-get update
  cwd: "/"
  timeout: 180
  interval: 1

Fuel master Astute log:
[608] Error running RPC method granular_deploy: Failed to execute hook .
---
priority: 1800
type: shell
uids:
- '1'
- '3'
- '2'
parameters:
  retries: 3
  cmd: apt-get update
  cwd: "/"
  timeout: 180
  interval: 1
, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:54:in `block in process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/nailgun_hooks.rb:26:in `process'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine/granular_deployment.rb:203:in `pre_deployment_actions'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/deployment_engine.rb:32:in `deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:133:in `deploy_cluster'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/orchestrator.rb:56:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/dispatcher.rb:111:in `granular_deploy'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.0.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

Other nodes. Source: messages.

2015-04-06 09:21:38 ERR
udevd-work[5827]: error opening ATTR{/sys/devices/virtual/block/md0/queue/iosched/quantum} for writing: No such file or directory
2015-04-06 09:21:38 ERR
udevd-work[5827]: error opening ATTR{/sys/devices/virtual/block/md0/queue/iosched/slice_idle} for writing: No such file or directory

1. Controller+ceph messages:
http://pastebin.com/SSusBgSG

2. Compute+ceph messages
http://pastebin.com/mntVHYJU

3. CephOSD node messages:
http://pastebin.com/hyJKZSfp

Logs diagnostic snapshot.

[Kyrylo Romanenko (kromanenko)]

cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  api: "1.0"
  build_number: "281"
  build_id: "2015-04-06_03-15-13"
  nailgun_sha: "a2dc04af9c5838a990adeab81789d4c090943701"
  python-fuelclient_sha: "9ad42671130deb2cc81135fb453ef0793883be2c"
  astute_sha: "240bfefe0a0b5af21b79854d0c96d8a3ee0fd885"
  fuellib_sha: "601cc8e1dccbdcf328f83f071270609e824d6c5a"
  ostf_sha: "b978badb43243d3f1b85dde15b8e301a21f12254"
  fuelmain_sha: "f90d7d40ceff39ee6d7587b183c63bd00ed0743f"