ext-ports accumulate after mojo tests: IpAddressGenerationFailureClient: No more IP addresses available on network
Bug #1656306 reported by
Ryan Beisner
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Charm Test Infra |
Fix Released
|
High
|
Ryan Beisner |
Bug Description
Following many iterations of mojo spec full deploy jobs, OSCI slaves are clogged with residual ext-port ports and IP address assignments. This leads to depletion of the the DHCP pools.
neutronclient.
Neutron server returns request_ids: ['req-6bb32e95-
ubuntu@
549
Related branches
lp:~1chb1n/ubuntu-openstack-ci/use-port-cleanup-tool
- Andrew McLeod (community): Approve
-
Diff: 836 lines (+196/-227)19 files modified.bzrignore (+1/-0)
common/osci_utils.py (+4/-4)
job-parts/build_deploy_test.sh (+1/-1)
job-parts/build_deployer_quickcheck.sh (+1/-1)
job-parts/build_git_deploy_test.sh (+1/-1)
job-parts/build_juju_deploy_single_charm.sh (+2/-2)
job-parts/build_mojo_runner.sh (+9/-5)
job-parts/build_osi.sh (+1/-1)
job-parts/deprecated/Z_DEPRECATE_build_mojo_runner_baremetal.sh (+0/-69)
job-parts/deprecated/part_juju_cleanup.sh (+0/-22)
job-parts/juju1_bighammer_destroy.sh (+32/-27)
job-parts/juju1_deploy_ok.sh (+45/-35)
job-parts/osci_job_common.sh (+33/-16)
job-parts/osci_mojo_common.sh (+21/-35)
job-parts/trigger_by_mojo_group.sh (+6/-0)
populate/mappings.yaml (+9/-7)
tools/juju_status_health_check.py (+1/-1)
tools/requirements.txt (+10/-0)
tools/tox.ini (+19/-0)
Changed in charm-test-infra: | |
status: | New → Confirmed |
assignee: | nobody → Ryan Beisner (1chb1n) |
importance: | Undecided → High |
Changed in charm-test-infra: | |
status: | In Progress → Fix Committed |
Changed in charm-test-infra: | |
milestone: | none → 17.02 |
Changed in charm-test-infra: | |
status: | Fix Committed → Fix Released |
To post a comment you must log in.
Finally circling back to this issue. We've have a number of jenkins slaves offline due to this.
⟫ neutron port-list | grep "juju-osci- .*-machine. *ext-port" | grep "172\.17\." | wc -l
818
I have a WIP to add ext-port cleanup on all jobs (pre and post), to ensure these are always reaped.