Request: Ability to add a whole folder directly into preseed

Bug #1853914 reported by Pico
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cubic
Status tracked in Release
Release
Fix Released
Low
Cubic PPA

Bug Description

I would like to be able to import a whole folder full of files into preseed instead of having to create each file manually and then copy and paste the contents in.

I would also like to be able to add binary files which my preseed scripts rely on as well as add folders which contain files and sub-folders full of files. One example of a useful folder like this would be including the java jre in preseed to be able to run a .jar during preseed.

While this feature would allow a whole lot of complex capability, I hope it wouldn't be too technically difficult on the back-end other than needing to rework the GUI for the preseed section (which I know would probably not be trivial).

Thanks so much for your consideration!

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Pico,

I assume the copied files need to go into the current "preseed" folder?

Revision history for this message
Pico (picomitchell) wrote :

Yes, I'd like to be able select a folder and have all of the contents of that folder (while preserving sub-folder structures and files) copied into the "pressed" folder for the current iso.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :
Download full text (5.1 KiB)

Here are some thoughts / considerations that must be resolved to implement this.

Note that your desire to preserve the folder structure will require additional work, as well.

I've listed some questions below, in order to document things that a solution would need to take into account. You don't have to answer those questions.

However, I do have one question that I would like you to answer...

Would it be OK for a new file that is being copied to overwrite an existing file, without warning to the user?

(If a warning is required, the solution could become even more complex, and it could be annoying to the user, if they have to make a decision for each duplicate file. We currently do not do this on the Copy Files page that is invoked from the Terminal Page).

CONSIDERATIONS
==============

Handling Non-text Files
-----------------------
- Show all files in the "preseed" directory left pane of Preseed tab.
- Include text (preseed) files.
- Include non-text (binary) files.
- Clicking on a non-text file will open a text editor in the right pane of the Preseed tab.
- Clicking on a non-text file will open a message in the right pane of the Preseed tab letting the user know that the file is binary and can not be edited.

Copying Files
-------------
Drag-and-drop files onto left pane of Preseed tab.
- Should the new files:
  - Approach A: Simply add to the list of files in the left pane of the Preseed tab?
  - Approach B: Present the user with a list of files that are about to be copied, and allow the user to accept or cancel the operation.
- If Approach B is used, it would be consistent with what we do on the Terminal page, when we drag files onto it.
- If Approach B is used, should the copy files page be superimposed on top of the Options page?
- There are two ways to implement Approach B:
  - Approach B1: Replace the Options Page with a list of files about to be copied, and allow the user to accept or cancel.
  - Approach B2: Continue to show the Options Page and the Preseed tab, but show the list of files to be copied in the right pane of the Preseed tab.
- If Approach B1 is used, it would be consistent with what we do on the Terminal page, when we drag files onto it.
- If Approach B2 is used, it would be consistent with what we do on the Preseed tab, when we create a new file or delete a preseed file.
  - We probably don't even need a cancel button, because the user can simply click in the Left pane to cancel the action (as is done for creating or deleting preseed files).

Behavior
--------
- If a file already exists in the preseed directory, and a file with the same name is copied into the directory, the existing file will be overwritten without warning.
  - Is this reasonable?
- If a directory is selected as the source of preseed files, do we?...
  - Approach C: Copy the entire directory structure into the preseed folder?
    - If so, the left pane of the Preseed tab only handles files, and can not handle directory trees.
    - This would have to be written if we need to support files in sub-directories.
  - Approach D: Copy files from all sub directories only, into a flattened preseed directory on the ISO.
    - The left pane of the Presee...

Read more...

Changed in cubic:
assignee: nobody → Cubic PPA (cubic-wizard)
Revision history for this message
Pico (picomitchell) wrote :

Thanks so much for the thoughtful analysis of this feature idea!

I think it would be fine to overwrite without warning, at least for my uses I know that the folder I am adding to the iso will always be my master source I will never be editing directly in Cubic so I will never be worried about losing the versions that Cubic has.

When I have a bit more time, I plan to reply some of the other considerations that you've mentioned. But, the main point for my use case is that if the folder contents are flattened and not preserved, it will not work for what I'm hoping to achieve. Namely being able to include Java JRE in the preseed folder.

Revision history for this message
Pico (picomitchell) wrote :

Related to this feature request... I'm currently trying to work around this issue by creating a bash script that I can include in the preseed folder that will download a .zip of the executables I want and then expand that .zip before the rest of the bash script runs.

But, I've run into an issue being able to run the bash script at all because it is not executable.

Would it be possible to have Cubic run "chmod +x" on any ".sh" files in the preseed folder? Or is there some way that I can achieve this on my own?

This is something to keep in mind for this future feature as well though. There should be some way to selectively make the included binary and .sh files (among other types) executable. I'm not sure it would hurt to actually just make every file in preseed executable, but I leave that decision to you.

Thanks!

Revision history for this message
Pico (picomitchell) wrote :

I've been able to workaround these issues by manually editing the contents of the "custom-live-iso/preseed" folder. I was able to add my desired folder as well as make the scripts and binaries executable.

Only thing is that Cubic does not show the contents of the preseed folder in the GUI (it seems to stop listing files after it runs into the folder in alphabetical order), but that isn't an issue for me at this point.

In the future, it would be nice to be able to do all of this directly through Cubic though.

Cubic PPA (cubic-wizard)
Changed in cubic:
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
Cubic PPA (cubic-wizard) wrote :

The new version of Cubic has just been released.

Although your requested feature is not a part of this release, one benefit of the new release is that you no longer need to use sudo on the command-line or run a file manager with admin privileges to edit the preseed folder.

You can simply change the files in `.../custom-disk/preseed` with regular user permissions; this should make things a little easier.

Revision history for this message
Pico (picomitchell) wrote :

Awesome, that is a nice change :-)

Thank you for letting me know!

Revision history for this message
Pico (picomitchell) wrote :

I've finally gotten around to checking out the latest version of Cubic and updating my scripts for this change you mentioned. Firstly, I want to say that I really like the new GUI and it's great to see that you're making Cubic compatible with Wayland and making other great improvements across the board.

In regards to my particular use-case here though, the new update has broken my current workaround. When I copy a folder into "[CUBIC-PROJECT]/custom-disk/preseed/" it now shows as if it's an empty file in the Preseed files list, if I proceed to the "generating the customized disk image" window, the process stalls forever on the "save configuration and boot files" step and I can see that the folder in question is the last item listed that was being processed.

By the way, I've always been copying this folder into the preseed folder when Cubic is on its Terminal window. I do this because I run a script on my live OS that copies this folder, other preseed files, and custom boot menus into the "custom-disk" folder (previously "custom-live-iso" folder), as well as copying another script into the "custom-root" folder (previously "squashfs-root" folder) which I then run from within the Cubic Terminal to run updates and installations for my iso image. This way I just need to run one script on my OS, and then one script in Cubic Terminal all during the same step in Cubic without needing to do any extra manual copying. Up until this new version, this process has been working great.

It looks like in the meantime, with this new version of Cubic, I can do this same process except for copying that preseed folder. It appears that I can then copy the preseed folder later when Cubic is on the new Compression level window. This seems to basically trick Cubic into not trying to do extra processing to the preseed folder, but does still includes it in the iso.

But, until Cubic fully supports folder in preseed, would it be possible for Cubic to hide folders from the GUI but then process them correctly (just including them as-is) during the generating iso phase? That way, I could continue using my easier script process as described above without breaking Cubic.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Pico,

Thanks for the comments.
I am working on adding this feature into Cubic.
(It is taking a little time because it is surprisingly complex).

Regarding your current workflow, if you copy files into the preseed folder while still on the Terminal page, Cubic ~should~ pick up the files and show them on the Preseed tab. I reviewed the code, and it was not immediately apparent to me why this is no working; I'll do a test to see if I can pinpoint the issue.

In the mean time, would you please answer this question:

What is the ownership/permissions of the copied files?

Revision history for this message
Pico (picomitchell) wrote :

If I copy only the files (without the folder), everything works and displays perfectly in Cubic. The display in the Pressed tab only get's messed up when I copy the folder during the Terminal phase.

As I mentioned, I've got an effective workaround by copying all files (without the folder) during the Terminal phase and then copying the folder on the Compressing phase.

As for ownership, the files and folder are all owned by my user (should I change them to be owned by "cubic"?)

The permissions for my folders are all 755, the files are all 644 except when I've set a script or binary to be executable, then they are 755.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

File permissions in custom-disk should be writable and readable by your user account, since Cubic is running under your user permissions.

I'll try this out with a folder.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

(Note: the only exception is ../custom-disk/casper/filesystem.squashfs, which is owned by root)

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Pico,

The particular issue you are facing is that Cubic can not handle preseed files in sub-folders.
Would you please open a separate bug report for this issue, and I will try to fix it under that bug report.

Revision history for this message
Pico (picomitchell) wrote :
Revision history for this message
Pico (picomitchell) wrote :

Just thinking after using your latest version with the sub-folder fix (https://bugs.launchpad.net/cubic/+bug/1883801) that a reasonable temporary solution for this feature could be to simply add a button to Cubic's Preseed page to open the Preseed folder and suggest that users copy their desired files and folders into there. Also, in Cubic's Preseed page there could be another button to reload the Preseed folders contents after a user has modified it. That's just a workaround, but I figured it could be a more quick and convenient solution for users wanting to do more advanced Preseed customization since the latest version is already halfway there. Also takes the burden off your code when handling the possibility of overwriting duplicate files, etc.

As it is now though, with my existed scripted solution, Cubic is now 100% functionality and working great for my use-case. Thanks for that!

Revision history for this message
Cubic PPA (cubic-wizard) wrote : Re: [Bug 1853914] Re: Request: Ability to add a whole folder directly into preseed

The new solution will be a new code-base than what we have now.
It will be more like a file browser in the left pane.
Refreshing files can be achieved by collapsing the document
tree, and expanding it again. The buttons in the header bar will be
dynamic, if you are on a folder or viewing a file.

I've added some prototype screenshots to the bug report.
https://bugs.launchpad.net/cubic/+bug/1883801

On Tue, Jun 16, 2020 at 10:25 PM Pico <email address hidden> wrote:

> Just thinking after using your latest version with the sub-folder fix
> (https://bugs.launchpad.net/cubic/+bug/1883801) that a reasonable
> temporary solution for this feature could be to simply add a button to
> Cubic's Preseed page to open the Preseed folder and suggest that users
> copy their desired files and folders into there. Also, in Cubic's
> Preseed page there could be another button to reload the Preseed folders
> contents after a user has modified it. That's just a workaround, but I
> figured it could be a more quick and convenient solution for users
> wanting to do more advanced Preseed customization since the latest
> version is already halfway there. Also takes the burden off your code
> when handling the possibility of overwriting duplicate files, etc.
>
> As it is now though, with my existed scripted solution, Cubic is now
> 100% functionality and working great for my use-case. Thanks for that!
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1853914
>
> Title:
> Request: Ability to add a whole folder directly into preseed
>
> Status in Cubic:
> In Progress
> Status in Cubic release series:
> In Progress
>
> Bug description:
> I would like to be able to import a whole folder full of files into
> preseed instead of having to create each file manually and then copy
> and paste the contents in.
>
> I would also like to be able to add binary files which my preseed
> scripts rely on as well as add folders which contain files and sub-
> folders full of files. One example of a useful folder like this would
> be including the java jre in preseed to be able to run a .jar during
> preseed.
>
> While this feature would allow a whole lot of complex capability, I
> hope it wouldn't be too technically difficult on the back-end other
> than needing to rework the GUI for the preseed section (which I know
> would probably not be trivial).
>
> Thanks so much for your consideration!
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cubic/+bug/1853914/+subscriptions
>

Revision history for this message
Pico (picomitchell) wrote :

That new file browser looks like it will be great!

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

@Johnny,

I'm curious why you marked this Bug as "Fix Released"?

I'm still working on it.

Revision history for this message
Johnny (johnnybee) wrote :

Sorry I think it's a mistake I don't know how I could have done that.

I'm following the bug evolution

I apologize again for this mistake
Best Regards

-------- Message d'origine --------
Le 30 oct. 2020 à 20:09, Cubic PPA < <email address hidden> > a écrit :
@Johnny,
I'm curious why you marked this Bug as "Fix Released"?
I'm still working on it.
** Changed in: cubic/release
Status: Fix Released => In Progress
--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1853914
Title:
Request: Ability to add a whole folder directly into preseed
Status in Cubic:
In Progress
Status in Cubic release series:
In Progress
Bug description:
I would like to be able to import a whole folder full of files into
preseed instead of having to create each file manually and then copy
and paste the contents in.
I would also like to be able to add binary files which my preseed
scripts rely on as well as add folders which contain files and sub-
folders full of files. One example of a useful folder like this would
be including the java jre in preseed to be able to run a .jar during
preseed.
While this feature would allow a whole lot of complex capability, I
hope it wouldn't be too technically difficult on the back-end other
than needing to rework the GUI for the preseed section (which I know
would probably not be trivial).
Thanks so much for your consideration!
To manage notifications about this bug go to:
https://bugs.launchpad.net/cubic/+bug/1853914/+subscriptions

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

No problem!
I've added you as a subscriber.

This is taking me a while to implement (especially considering that I've had to stop to tackle more urgent bugs that have come up) but rest-assured, I am working on it.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :
Download full text (5.7 KiB)

The following new features and changes have been implemented...

(1)
Removed the following steps from the Prepare page:

‣ Identify preseed files.
‣ Identify ISO boot configuration files.

(2)
Updated the ISO Kernel tab on the Options page.

‣ Slightly reduced the height of the rows.
‣ Automatically update the Preseed files with the correct vmlinux and
  initrd file names when kernels are changed [existing].
‣ Automatically update the ISO Boot configuration files with the correct
  vmlinux and initrd file names when kernels are changed [existing].
‣ For efficiency, updates to file are now only applied when the user
  leaves the ISO Kernels tab.
‣ TBD: The updates should be stored in the file's undo buffer.

(3)
Updated the Preseed tab on the Options page.

‣ Files are listed in a tree in the left pane.
‣ File icons in the tree reflect the file type [to be improved].
‣ Directories can be expanded or collapsed.
‣ Files are synchronized with the file system.
  • Changes made to the file system are immediately reflected in the
    tree:
    ◦ New files copied into the preseed directory
    ◦ Files moved within the preseed directory
    ◦ Files renamed within the preseed directory
    ◦ Files deleted from the preseed directory
  • Changes made to preseed files in Cubic are immediately reflected in
    the file system:
    ◦ New files copied into the preseed directory
    ◦ Files renamed within the preseed directory
    ◦ Files deleted from the preseed directory
‣ Text files are displayed for editing in the right pane when selected
  [existing].
‣ Text files are immediately saved if changes are made.
‣ Ability to undo and redo edits using <CTRL><Z> and <CTRL><Shift><Z>
  [existing].
‣ Image files are displayed for viewing in the right pane, when selected.
‣ Other files and directories are displayed in the right pane as an icon,
  when selected.
‣ Added dynamic header bar toggle buttons:
  • When a directory is selected, the following header bar buttons are
    available:
    ◦ Copy files
    ◦ Create a new file
    ◦ Create a new directory
    ◦ Rename directory
    ◦ Delete directory
  • When a file is selected, the following header bar buttons are
    available:
    ◦ Rename file
    ◦ Delete file
‣ Revised layouts for the right pane, for selected header bar toggle
  buttons:
  • New layout of the Create a new file panel.
  • New layout of the Rename file panel.
  • New layout of the Delete file panel.
  • New layout of the Create a new directory panel.
  • New layout of the Rename directory panel.
  • New layout of the Delete directory panel.

(4)
Updated the ISO Boot tab on the Options page.

‣ Files are listed in a tree in the left pane.
‣ Supports multiple ISO Boot directories (boot/grub and isolinux).
‣ Ability to view ~all~ ISO Boot files (see dynamic header bar toggle
  buttons below).
‣ File icons in the tree reflect the file type [to be improved].
‣ Directories can be expanded or collapsed.
‣ Files are synchronized with the file system.
  • Changes made to the file system are immediately reflected in the
    tree:
    ◦ New files copied into the ISO Boot directory
    ◦ Files moved within the ISO Boot directory
    ◦ Files renamed within...

Read more...

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

The code is currently not in the "release" branch yet.

You can test it using the "experimental" branch...

    $ sudo apt autoremove --purge cubic
    $ sudo apt-add-repository --remove ppa:cubic-wizard/release

    $ sudo apt-add-repository ppa:cubic-wizard/experimental
    $ sudo apt update
    $ sudo apt install cubic

Although the new features seem to work well and are very stable, there are some minor bugs to fix, and through testing is necessary before it can be officially released.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Fix released in version 2020.12-36.

For a detailed list of the changes in this release, reference the change log for revision 19 of the experimental branch (https://bazaar.launchpad.net/~cubic-wizard/cubic/experimental/view/head:/debian/changelog).

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

Note: Comment number 5 (https://bugs.launchpad.net/cubic/+bug/1853914/comments/5) mentions needing files to be executable. This was not implemented in version 2020.12-36, and I'd like to understand a little more about this before incorporating such a feature into Cubic.

Revision history for this message
Pico (picomitchell) wrote :

This is awesome! Thanks so much for all the work you've done on this feature!

I'm sorry I didn't have time to check out the beta before release. I will take some time this week to dig into the new version of Cubic and also re-investigate my request about executable scripts and binaries with this new feature.

Revision history for this message
Pico (picomitchell) wrote :

The new folder navigation as well as ability to add multiple files and whole folders is awesome and works great! Thank you so much for doing this!

As for making files executable, the reason that I originally requested that was because of my use case of having my preseed bash scripts (.sh files) saved on my computer without being set as executable. This makes it easier to open and edit these files since I never actually want to run them on my computer.

But, I can see the danger in making files with particular extensions executable without a user being explicitly informed or realizing that's going to happen. The fact that Cubic preserves the permissions for copied files is probably enough. If a user wants a file to be executable, they can set that before copying into Cubic (or by doing more advanced scripting by copying files and setting permissions in the correct Cubic folders without using the Cubic GUI to add them).

Although, I could see the value in having a little permissions view/edit bar above or below the file content display. This way a user could easily see the current permissions of their file and also edit them after being copied into Cubic if needed. But, I understand if this too much extra work for probably a very small use case.

Revision history for this message
Cubic PPA (cubic-wizard) wrote :

OK, thanks for the explanation.
I'll keep this in mind for a future enhancement. (I may incorporate it whenever I add cut/paste/copy capabilities for files edited in Cubic).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.