[Arale / feature requirement] repack factory images from OTA channels.

[Alex Tu]

Per-talked with Caio on mail, Arale need a new feature that can repack factory images (flashable images) from the tarball downloaded from system-image.u.c.

[Description]
Currently we have a script "mkflashable.sh" can do that, and this is the usage of that script (it can be run without root privilege currently):
http://paste.ubuntu.com/11768175/

For example, generate factory images from the latest tarballs on channel "ubuntu-touch/rc/meizu.en"
$ ./mkflashable.sh --autobuild --channel ubuntu-touch/rc/meizu.en

The scenario will be like this:
1. Some one need to get a flashable image which map to latest release on system-image.u.c for flash-tool using (ex. make a factory image release).
2. That one can execute my script to download ubuntu, device, customer, version tarballs from latest release on the channel he want from system-image.u.c. Then unpack those tarballs and create images to include the content of unpacked tarballs.

And we also need a input box when execute this script on capomastro for parameters of mkflashable.sh.
ex. "--autobuild --channel ubuntu-touch/rc/meizu.en"

This is an example console output when executing "mkflashable.sh":
http://paste.ubuntu.com/11768243/

Finally user should can get the generated images in out/target/product/arale/flash-parts-full
or get its tarball "arale-flash-20150608-6e66f3c.tar.gz", "arale-flash-20150608-6e66f3c.tar.gz.md5sum" as well.

This is the generated images and its tarball list for reference:
http://paste.ubuntu.com/11768233/

BTW, we also need capamastro to do this on the code tree which based on branch "m75_ubuntu-daily" for manifest.
https://bugs.launchpad.net/capomastro/+bug/1386713/comments/3

This feature also can be used on other project in the future, just need some change in script.

[Conclusion]
So, the expect scenario with capomastro will like:
1. user input parameter in input box on capomastro. ex. "--autobuild --channel ubuntu-touch/rc/meizu.en" then kick it run.
2. capomastro checkout code based "m75_ubuntu-daily" on manifest.
ex. $ repo init -u ssh://git.tangxi.canonical.com:29418/m75/platform/manifest -b m75_ubuntu-daily ; repo sync
3. capomastro execute "mkflashable.sh" with user imputed parameters:
ex. $ ./mkflashable.sh --autobuild --channel ubuntu-touch/rc/meizu.en
4. capomastro let user can download generated tarball like "arale-flash-20150608-6e66f3c.tar.gz", "arale-flash-20150608-6e66f3c.tar.gz.md5sum" or files in out/target/product/arale/flash-parts-full

[Caio Begotti (caio1982)]

Alex, the revision fix is done and I'll let you test it soon but just so I get this bug here right:

1) you just want to use the --autobuild feature of your script now, correct? I mean, to repack existing images on system-image servers instead of using local ones from the build

2) both Tangxi and Barajas will use the very same script? Or will they differ in some aspect?

I'm asking that because it would be better to have this decoupled from the phone builds. So in Capomastro you could build the images for testing or development and also have a separate way to generate the flashable images at any time.

That's my idea at least, so we don't have to re-clone and checkout everything in the repos just to get your shell script.

Would the following be okay for you in this case?

git clone -n ssh://<email address hidden>:29418/m75/ubuntu/tools --depth 1
cd tools/
git checkout HEAD mkflashable.sh
./mkflashable.sh --autobuild <all its parameters here>

If you also need to run mkflashable.sh without --autobuild then my plan will not work as you will need the local files and we'll need to clone, checkout and compile everything using the current scripts we have.

[Caio Begotti (caio1982)]

Btw, you said we'd need to init and sync the repo but that would be only to have some necessary files available or because you'd need to build anything? I'm trying to avoid dummy checkouts here that consumes too much space so please let me know if I got this wrong.

[Caio Begotti (caio1982)]

Alex, please take a look at https://capomastro.staging.canonical.com whenever you can to confirm. I have set up there what you needed based on the talks we had.

You can pass extra options to mkflashable's --autobuild via the AUTOBUILD='' parameter in the UI, as usual, for dependencies using the 'factory-image' new jobtype . You can take a look at a successful run https://capomastro.staging.canonical.com/dependencies/4/ and its log https://capomastro.staging.canonical.com/jenkins/builds/2/console/ and its artifacts (the actual factory files) at https://sis.capomastro.staging.canonical.com/android-tangxi-factory-image/current/

Since my questions from past comments still need some commenting I'll leave it here and will wait for your feedback on this feature! Let us know please :-)

[Alex Tu (alextu)]

@Caio,
Sorry, I missed some comments.

For #3, I can not access capomastro staging server, guessing you need to open my permission, if convenient please also open yc and vicamo's permission for testing.

For #1, 1), Yes, so far, I think --autobuild is enough instead of using local tarballs. But we still need the parameters follows --autobuild.
  ex. --autobuild --channel ubuntu-touch/rc/meizu.en
  ex. --autobuild --channel ubuntu-touch/rc-proposed/meizu.en
And I also plan to have one more parameter to get different build version on system-image.u.c
  ex. --autobuild --channel ubuntu-touch/rc-proposed/meizu.en -r 32

For #1, 2), And I think this script can be used by Krillin as well.
But because mkflashable.sh used make_ext4fs which built from tree, so still need to confirm if Krillin have same in-tree tool.

@ John McAleely, Maybe you have some comment for this question if you are interested in making mkflashable.sh support Krillin as well.

For #2) mkflashable.sh assumed some fixed path exist, so it need some prepared before executing when trying to execute it without checkout whole Tangxi project:
1. MAKE_EXT4FS=ubuntu/tools/make_ext4fs
2. origin/m75_ubuntu-daily branch of manifests module exist in ".repo/manifests"

Hope these can help :)

[Caio Begotti (caio1982)]

Alex, I had already shared the permissions to staging with you last night via LastPass. Can you confirm if you haven't really received an automatic e-mail about this? I just shared it again with YC and Vicamo.

About make_ext4fs, it's available in package android-tools-fsutils but I wonder if it's too old and that's why you guys have a binary of it in the tree. Maybe it's worth updating the package upstream so Krillin can easily benefit from this too?

[John McAleely (john.mcaleely)]

I think the version in the tree has been hacked/supported by Mediatek. We
need to use the in-tree tool in this case.

On 30 June 2015 at 12:25, Caio Begotti <email address hidden> wrote:

> Alex, I had already shared the permissions to staging with you last
> night via LastPass. Can you confirm if you haven't really received an
> automatic e-mail about this? I just shared it again with YC and Vicamo.
>
> About make_ext4fs, it's available in package android-tools-fsutils but I
> wonder if it's too old and that's why you guys have a binary of it in
> the tree. Maybe it's worth updating the package upstream so Krillin can
> easily benefit from this too?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1468410
>
> Title:
> [Arale / feature requirement] repack factory images from OTA channels.
>
> Status in Capomastro:
> In Progress
>
> Bug description:
> Per-talked with Caio on mail, Arale need a new feature that can repack
> factory images (flashable images) from the tarball downloaded from
> system-image.u.c.
>
> [Description]
> Currently we have a script "mkflashable.sh" can do that, and this is the
> usage of that script (it can be run without root privilege currently):
> http://paste.ubuntu.com/11768175/
>
> For example, generate factory images from the latest tarballs on channel
> "ubuntu-touch/rc/meizu.en"
> $ ./mkflashable.sh --autobuild --channel ubuntu-touch/rc/meizu.en
>
> The scenario will be like this:
> 1. Some one need to get a flashable image which map to latest release on
> system-image.u.c for flash-tool using (ex. make a factory image release).
> 2. That one can execute my script to download ubuntu, device, customer,
> version tarballs from latest release on the channel he want from
> system-image.u.c. Then unpack those tarballs and create images to include
> the content of unpacked tarballs.
>
> And we also need a input box when execute this script on capomastro for
> parameters of mkflashable.sh.
> ex. "--autobuild --channel ubuntu-touch/rc/meizu.en"
>
> This is an example console output when executing "mkflashable.sh":
> http://paste.ubuntu.com/11768243/
>
> Finally user should can get the generated images in
> out/target/product/arale/flash-parts-full
> or get its tarball "arale-flash-20150608-6e66f3c.tar.gz",
> "arale-flash-20150608-6e66f3c.tar.gz.md5sum" as well.
>
> This is the generated images and its tarball list for reference:
> http://paste.ubuntu.com/11768233/
>
> BTW, we also need capamastro to do this on the code tree which based on
> branch "m75_ubuntu-daily" for manifest.
> https://bugs.launchpad.net/capomastro/+bug/1386713/comments/3
>
> This feature also can be used on other project in the future, just
> need some change in script.
>
> [Conclusion]
> So, the expect scenario with capomastro will like:
> 1. user input parameter in input box on capomastro. ex. "--autobuild
> --channel ubuntu-touch/rc/meizu.en" then kick it run.
> 2. capomastro checkout code based "m75_ubuntu-daily" on manifest.
> ex. $ repo init -u ssh://
> git.tangxi.canonical.com:29418/m75/platform/manif...

[Alex Tu (alextu)]

@ Caio
Because I have collected the built binary "make_ext4fs" to ubuntu/tools,
So, it convenient to get both "mkflashable.sh" and "make_ext4fs" after you cloned ubuntu/tools.

And I have the same idea with John, say that we should use the in-tree tool to make sure the output image always as we expect.

BTW, I also flashed the image in https://sis.capomastro.staging.canonical.com/android-tangxi-factory-image/current/
It works well.

And I also created two project to build rc and rc-proposed will test that tomorrow in office (cause the downloading is too slow now).

[Caio Begotti (caio1982)]

Thanks, Alex! That sounds very promising :-)