Detailed bug description:
Provision has failed. Failed to execute hook 'shell' Failed to run command cd / && fa_build_image --image_build_dir /var/lib/fuel/ibp --log-file /var/log/fuel-agent-env-1.log --data_driver nailgun_build_image --input_data '{"image_data": {"/boot": {"container": "gzip", "uri": "http://10.20.0.2:8080/targetimages/env_1_ubuntu_1404_amd64-boot.img.gz", "format": "ext2"}, "/": {"container": "gzip", "uri": "http://10.20.0.2:8080/targetimages/env_1_ubuntu_1404_amd64.img.gz", "format": "ext4"}}, "output": "/var/www/nailgun/targetimages", "repos": [{"name": "ubuntu", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty", "type": "deb"}, {"name": "ubuntu-updates", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty-updates", "type": "deb"}, {"name": "ubuntu-security", "section": "main universe multiverse", "uri": "http://archive.ubuntu.com/ubuntu/", "priority": null, "suite": "trusty-security", "type": "deb"}, {"name": "mos", "section": "main restricted", "uri": "http://10.20.0.2:8080/mitaka-9.0/ubuntu/x86_64", "priority": 1050, "suite": "mos9.0", "type": "deb"}, {"name": "mos-updates", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/", "priority": 1050, "suite": "mos9.0-updates", "type": "deb"}, {"name": "mos-security", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/", "priority": 1050, "suite": "mos9.0-security", "type": "deb"}, {"name": "mos-holdback", "section": "main restricted", "uri": "http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/", "priority": 1100, "suite": "mos9.0-holdback", "type": "deb"}, {"name": "Auxiliary", "section": "main restricted", "uri": "http://10.20.0.2:8080/mitaka-9.0/ubuntu/auxiliary", "priority": 1150, "suite": "auxiliary", "type": "deb"}], "packages": ["acl", "anacron", "bash-completion", "bridge-utils", "bsdmainutils", "build-essential", "cloud-init", "curl", "daemonize", "debconf-utils", "gdisk", "grub-pc", "hpsa-dkms", "hwloc", "i40e-dkms", "linux-firmware", "linux-firmware-nonfree", "linux-headers-generic-lts-trusty", "linux-image-generic-lts-trusty", "lvm2", "mcollective", "mdadm", "multipath-tools", "multipath-tools-boot", "nailgun-agent", "nailgun-mcagents", "network-checker", "ntp", "openssh-client", "openssh-server", "puppet", "python-amqp", "ruby-augeas", "ruby-ipaddress", "ruby-json", "ruby-netaddr", "ruby-openstack", "ruby-shadow", "ruby-stomp", "telnet", "ubuntu-minimal", "ubuntu-standard", "uuid-runtime", "vim", "virt-what", "vlan"], "codename": "trusty"}'
from Astute logs:
[18638] Error running provisioning: Image build task failed. Please check build log here for details: /var/log/fuel-agent-env-1.log. Hint: restart deployment can help if no error in build log was found, trace:
From fuel-agent-logs:
2016-04-20 13:45:48.686 6096 ERROR fuel_agent.cmd.agent [-] 404 Client Error: Not Found for url: http://10.20.0.2:8080/mitaka-9.0/ubuntu/x86_64/dists/mos9.0/Release
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent Traceback (most recent call last):
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent File "/usr/lib/python2.7/site-packages/fuel_agent/cmd/agent.py", line 121, in main
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent getattr(mgr, action)()
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent File "/usr/lib/python2.7/site-packages/fuel_agent/manager.py", line 1046, in do_build_image
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent direct_repo_addrs=driver_os.proxies.direct_repo_addr_list)
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent File "/usr/lib/python2.7/site-packages/fuel_agent/manager.py", line 491, in _set_apt_repos
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent proxies=proxies, direct_repo_addrs=direct_repo_addrs)
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent File "/usr/lib/python2.7/site-packages/fuel_agent/utils/build.py", line 469, in add_apt_preference
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent direct_repo_addrs=direct_repo_addrs))
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent File "/usr/lib/python2.7/site-packages/fuel_agent/utils/build.py", line 394, in get_release_file
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent noproxy_addrs=direct_repo_addrs).text
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent File "/usr/lib/python2.7/site-packages/fuel_agent/utils/utils.py", line 265, in init_http_request
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent response_obj.raise_for_status()
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent File "/usr/lib/python2.7/site-packages/requests/models.py", line 837, in raise_for_status
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent raise HTTPError(http_error_msg, response=self)
2016-04-20 13:45:48.686 6096 TRACE fuel_agent.cmd.agent HTTPError: 404 Client Error: Not Found for url: http://10.20.0.2:8080/mitaka-9.0/ubuntu/x86_64/dists/mos9.0/Release
ls /var/www/nailgun/mitaka-9.0/ubuntu/x86_64/
images
Steps to reproduce:
1) Create VM - 2VCPU, 4096 RAM 50HDD, Centos Minimal Install, 3 network interfaces
2) download http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7/os/x86_64/Packages/fuel-release-9.0.0-1.mos6341.noarch.rpm
3) yum install fuel-release
4) yum install fuel-setup
5) bootstrap_admin_node.sh
6) check fuel-services
7) bootstrap 2 slaves
8) create env with controller and compute
9) Deploy changes
10 Verify networks
Expected results:
Deployment successfully finished
Actual result:
Failed to provision with 404 error
Reproducibility:
100% on KVM, VBOX
Workaround:
Change mos and Auxiliary repos from local to online
Impact:
Separate master node deployment feature
Description of the environment:
cat /etc/fuel_release:
9.0
cat /etc/fuel_openstack_version:
mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
fuel-misc-9.0.0-1.mos8309.noarch
fuel-bootstrap-cli-9.0.0-1.mos273.noarch
python-fuelclient-9.0.0-1.mos312.noarch
fuel-9.0.0-1.mos6341.noarch
fuel-ui-9.0.0-1.mos2668.noarch
fuel-notify-9.0.0-1.mos8309.noarch
fuel-release-9.0.0-1.mos6341.noarch
fuel-provisioning-scripts-9.0.0-1.mos8653.noarch
python-packetary-9.0.0-1.mos128.noarch
fuelmenu-9.0.0-1.mos269.noarch
fuel-utils-9.0.0-1.mos8309.noarch
fuel-migrate-9.0.0-1.mos8309.noarch
nailgun-mcagents-9.0.0-1.mos724.noarch
shotgun-9.0.0-1.mos88.noarch
network-checker-9.0.0-1.mos72.x86_64
fuel-openstack-metadata-9.0.0-1.mos8653.noarch
fuel-setup-9.0.0-1.mos6341.noarch
fuel-library9.0-9.0.0-1.mos8309.noarch
fuel-mirror-9.0.0-1.mos128.noarch
fuel-agent-9.0.0-1.mos273.noarch
fuel-nailgun-9.0.0-1.mos8653.noarch
fuel-ostf-9.0.0-1.mos918.noarch
rubygem-astute-9.0.0-1.mos724.noarch
I don't think it should be considered as bug. In case of deploying master on a pre-provisioned server, one should use fuel-mirror tool to create the local MOS Ubuntu packages mirror, and deploy environments from it. This should be reflected in the documentation.