Tech Debt: Convert bash scripts in contrib into python code and add tests

Bug #1302552 reported by Julien Vey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Solum
Fix Released
Medium
James Li

Bug Description

Our bash scripts in contrib should be integrated in the python tree, and we should add proper tests for this new code

Tags: tech-debt
Julien Vey (vey-julien)
description: updated
Changed in solum:
importance: Undecided → Medium
Changed in solum:
assignee: nobody → Ed Cranford (ed--cranford)
Revision history for this message
Angus Salkeld (asalkeld) wrote :

In the case we are using a job runner like jenkins, it's handy to have them as a bash script.

Adrian Otto (aotto)
summary: - Convert bash scripts in contrib into python code and add tests
+ Tech Debt: Convert bash scripts in contrib into python code and add
+ tests
Changed in solum:
status: New → Triaged
Revision history for this message
Julien Vey (vey-julien) wrote :

Ed, are you currently working on this or can I take the issue ?

Revision history for this message
Ed Cranford (ed--cranford) wrote :

I've been nose-to-grindstone on our M2 Demo, please take this from me.

Changed in solum:
assignee: Ed Cranford (ed--cranford) → nobody
Adrian Otto (aotto)
tags: added: tech-debt
Changed in solum:
milestone: none → juno-1
Julien Vey (vey-julien)
Changed in solum:
assignee: nobody → Julien Vey (vey-julien)
Adrian Otto (aotto)
Changed in solum:
milestone: juno-1 → juno-2
Revision history for this message
Ravi Sankar Penta (ravips) wrote :

I did some work to convert shell code to python, mostly code around app workflow (clone/build/test/deploy). While doing this, I tried to:
- Abstract clear interfaces between solum worker and the language pack handlers.
- Wrap common functionality (environment setup for app build, etc.) in the base language pack plugin.
   Woker loads language pack plugins selected by the operator.
- Simplify passing application info from assembly hander to worker.

Relevant patches(not merged):
https://review.openstack.org/#/c/132381/
https://review.openstack.org/#/c/132718/
https://review.openstack.org/#/c/132933/
https://review.openstack.org/#/c/133111/

At some point, I had working version of these patches with cedarish-docker plugin. A lot has changed after that and I don't have enough cycles to finish this task. Anyone who picks this task needs to do:
- Rebase latest code and resolve conflicts
- Incorporate the feedback got from the team for the above patches
- Add/Update tests to validate the functionality
- Fix any issues

As this task touches a lot of functionality activitly improved by other team meembers, we have few options to get this into master:
Option1: Make the new code co-exist with the existing shell code and switch one language pack plugin at a time to use the python code. Once all LPs are switched to python code, we can safely delete the shell code.
Option2: Replace shell code with these pathces ensuring at least one working LP plugin (mostly dockerfile LP). Gradually add support/fix other LP plugins to use python.

Changed in solum:
assignee: Julien Vey (vey-julien) → nobody
Changed in solum:
assignee: nobody → Akshay Chhajed (akshaychhajed)
Revision history for this message
Adrian Otto (aotto) wrote :

 Akshay Chhajed, feel free to re-claim this bug if you'd like to work on it.

Changed in solum:
milestone: juno-2 → 2015.1.1
assignee: Akshay Chhajed (akshaychhajed) → nobody
Revision history for this message
Akshay Chhajed (akshaychhajed) wrote :

OK. Thanks Adrian!

Adrian Otto (aotto)
Changed in solum:
assignee: nobody → James Li (james-li-3)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to solum (master)

Reviewed: https://review.openstack.org/181181
Committed: https://git.openstack.org/cgit/openstack/solum/commit/?id=24649a7e24c22821b18c77a425e57a9f0ba73a86
Submitter: Jenkins
Branch: master

commit 24649a7e24c22821b18c77a425e57a9f0ba73a86
Author: James Li <email address hidden>
Date: Thu May 7 22:29:58 2015 +0000

    Convert bash scripts to Python

    This patch sets up a LP handler framework to accommodate
    build-lp, unittest-app and build-app functionalities.

    Additionally, the patch also implements isolation improvements in
    building an application container on worker node.
    Specifically, the git clone command for cloning application code
    is run inside a docker container. This helps in constraining
    the resources of memory, CPU, disk and network bandwidth
    to prevent Solum from the DoS attacks of a malicious git server.

    Partially-fixes-bug: #1302552
    Change-Id: Ie0f700c0d4bd74e24ac1a9727f4b17907f88cca1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/192928
Committed: https://git.openstack.org/cgit/openstack/solum/commit/?id=962857598b0010e065ef2fc9fd39dfcfffda1bc7
Submitter: Jenkins
Branch: master

commit 962857598b0010e065ef2fc9fd39dfcfffda1bc7
Author: James Li <email address hidden>
Date: Wed Jun 17 21:28:54 2015 +0000

    Convert unittest-app bash script to python

    1. Added timeout for running user's arbitrary unittest script;
    2. Limit memory usage of a container in which user's unittest
    script runs.

    Partially-fixes-bug: #1302552
    Change-Id: Ibd0fe3b09295839e7f8279cddabb56442c1d87c8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on solum (master)

Change abandoned by James Li (<email address hidden>) on branch: master
Review: https://review.openstack.org/196498

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to solum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/283370

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to solum (master)

Reviewed: https://review.openstack.org/283370
Committed: https://git.openstack.org/cgit/openstack/solum/commit/?id=fc58f9422ff3a205fb3f7a37caf49342f4cc57f4
Submitter: Jenkins
Branch: master

commit fc58f9422ff3a205fb3f7a37caf49342f4cc57f4
Author: James Li <email address hidden>
Date: Tue Feb 23 03:30:33 2016 +0000

    Convert build-app and a new worker handler

    Change-Id: Ief90ef40e2f21db6345316e3125f198131d2b034
    Partially-fixes-bug: #1302552

Changed in solum:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.