Copying files in and out of VMs is inconvenient

[khb]

I find myself often copying things into or out of the uvt-kvm created VM. This would be facilitated by

1) adding scp to ssh as a subcommand
2) adding the IP address of the HOST to the environment so from inside the VM one could use
     scp $USER@$HOST:$MYFILES .

uvtool:
  Installed: 0~git140-0ubuntu1

Distributor ID: Ubuntu
Description: Ubuntu 18.10
Release: 18.10
Codename: cosmic

[khb (khbkhb)]

A little context. uvt is great for taking existing cloud projects local. The use-case(s) I'm evaluating it's use for is the other direction. Legacy apps (the most pessimal I've dealt with was primos->vms->unix(es)->linux ...) such projects tend to iterative layers of archeological discovery. Having a bunch of persistent tools can be accommodated by mounting a second volume ... but often results in prolonging the effort as dependencies on that volume creep in. Also, it's possible to use uvt-kvm create && wait && virsh shutdown && virt-copyin && virsh start ... or etc.

Along these lines, having an explicit template cloud-init that uvt-kvm is using (or allow it to be generated) could be a useful stepping stone.

[Robie Basak (racb)]

Thank you for the report.

Yes, I agree it would certainly be useful to give the user easier access to scp with built-in IP address lookup and so forth. Right now I usually do something like "scp ubuntu@`uvt-kvm ip foo`:... ..." which is tedious.

Unfortunately I'm not sure there's a really clean way to do this. The devil is in the (implementation) detail and in the proposed UX.

"uvt-kvm <foo> ssh <...>" currently passes <...> to ssh exactly, so we can't add subcommands to "uvt-kvm ssh". "uvt-kvm scp" at the top level would be fine. However, how is that command supposed to work and be implemented? What would a typical invocation look like, and what algorithm would uvt-kvm use to determine what hosts' IP addresses to look up and what arguments to use to call the scp command?

If you want an environment variable to set in the guest, what mechanism would that use to get there? How would the guest have an ssh key acceptable to the host?

Since I'm not sure there's any good answer, I'm marking this as Incomplete since we don't know what to implement to fix this request. To make progress, we should work towards consensus on a full specification of the change. I wouldn't attempt an entire spec in one go - I suggest incrementally discussing ideas here in this bug.

[khb (khbkhb)]

Let's start with what I conjecture is the least difficult case.

uvt-kvm scp $FILELIST $DOM

This would copy $FILELIST from the host into the uvt-kvm managed $DOM (using pretty much whatever the wrapper for ssh is ;>). The trialing ":" could be optional or required (the former allows for a little less stress on aged wrists ... but trivially so ...).

$FILELIST could "trivially" be extended to the full scp syntax (just pass it down) so $HOST@FILEIST. If the keys are appropriately aligned, it "just works" if not, the usual scp fallbacks (potentially password, etc.). Since uvt-kvm ssh works, we must have at least the local/host keys already properly embedded in $DOM ;>

IF we were to attempt to tackle the more problematic direction (inside->outside) adding a $ENV which the user could prepopulate as desired. Scoping the request down to the "outside-in" paralleling the uvt-kvm ssh command should suffice for most things (or at least provide a handy bootstrap ...uvt-kvm scp $DOFIRST $DOM && uvt-kvm ssh $DOM ... ./$DOFIRST (once inside the machine).

[Robie Basak (racb)]

On Mon, Apr 29, 2019 at 04:01:36PM -0000, khb wrote:
> Let's start with what I conjecture is the least difficult case.
>
> uvt-kvm scp $FILELIST $DOM
>
> This would copy $FILELIST from the host into the uvt-kvm managed $DOM
> (using pretty much whatever the wrapper for ssh is ;>). The trialing
> ":" could be optional or required (the former allows for a little less
> stress on aged wrists ... but trivially so ...).

"scp foo bar" copies the local file called "foo" into the local file
called "bar", as I expect. "uvt-kvm scp" implies, by using "scp" rather
than anything else, that the command works like scp. So I would expect
"uvt-kvm scp a b" without any colons to be the equivalent of "scp a b",
which in your description without colons would not work as expected, so
I must rule that option out.

> $FILELIST could "trivially" be extended to the full scp syntax (just
> pass it down) so $HOST@FILEIST. If the keys are appropriately aligned,
> it "just works" if not, the usual scp fallbacks (potentially password,
> etc.). Since uvt-kvm ssh works, we must have at least the local/host
> keys already properly embedded in $DOM ;>

What about glob expansion? And more generally, how are we going to
implement in uvt-kvm the exact equivalent of scp syntax including all
its edge cases? We can't just pass it down since we need special
behaviour to make it work in the first place. And then: how does scp
escape colons, and how are we going to make that work as expected by
those familiar with scp escaping?

> IF we were to attempt to tackle the more problematic direction
> (inside->outside) adding a $ENV which the user could prepopulate as
> desired. Scoping the request down to the "outside-in" paralleling the
> uvt-kvm ssh command should suffice for most things (or at least provide
> a handy bootstrap ...uvt-kvm scp $DOFIRST $DOM && uvt-kvm ssh $DOM ...
> ./$DOFIRST (once inside the machine).

I'm not sure about adding this to uvt-kvm itself. I think it'll open up
too many failed edge case expectations by users. I think it's the sort
of thing that would be better implemented indivdually by users (aliases
or local scripts, etc). I'm still open to putting something into
uvt-kvm, but it needs to be designed well such that an experienced
sysadmin will be able to remember how it functions without difficulty,
including in edge cases. I'm not sure an entirely new pattern will fit
this.

[khb (khbkhb)]

so make the $DOM: mandatory ... then just what complexity of the "shim script" am I missing? All it would appear to have to do is translate $DOM into $IP. globbing and any other shell dependent behavior remains shell (and scp) dependent behavior.

That is,

uvt-kvm scp $FILES $DOM expands to

scp $FILES ubuntu@$(uvt-kvm ip $DOM):

(which, while it goes against your argument that the : should be mandatory, is slightly easier to script without the : ... but clearly not a big deal either way). And yes, it's certainly easy enough for the user to script; but that's true for all of the uvt-kvm functionality ... (building atop libvirt) ... and just as it really is handy to not have to script the create, destroy, ssh ... it seems to me that it's not a big step to improve the copyin facility.

[Robie Basak (racb)]

On Mon, Apr 29, 2019 at 04:43:59PM -0000, khb wrote:
> so make the $DOM: mandatory ... then just what complexity of the "shim
> script" am I missing? All it would appear to have to do is translate
> $DOM into $IP. globbing and any other shell dependent behavior remains
> shell (and scp) dependent behavior.

You don't know which is $DOM and which is a local file then though. scp
works both ways round (to copy to and from a guest). Users are likely to
have files named after their own VMs. We'd be trying to parse what scp
would do, adjust how we call scp (to expand the IP and username), and
likely get the parsing wrong.

> That is,
>
> uvt-kvm scp $FILES $DOM expands to
>
> scp $FILES ubuntu@$(uvt-kvm ip $DOM):

Then a user will try "uvt-kvm scp $FILES otheruser@$DOM" and then that
will break, for example.

If we're going to "pretend to wrap" scp, then we really need to behave
the same as scp for the edge cases. I think this is far more complicated
than you think, and the edge case in my previous paragraph is a perfect
example.

It might be better to drop any pretence of emulating scp and just
implement a "uvt-kvm copy" that does something much simpler and clearly
defined. But then we're going to end up getting wishlist requests to
build it back up and find ourselves reimplementing scp again.

To be clear, I appreciate the need for something like this. I just don't
think it's as trivial to get right as you're making out.

[khb (khbkhb)]

that it's not entirely trivial is true; I regret characterizing it as such. If it were entirely trivial, it wouldn't be worth adding to uvt-kvm!

If having it become uvt-kvm copyin [-r] $name $files (and optionally copyout) would reduce it to implementable in a near to medium term, I'd say it was fine result for this wishlist item.

In the interim, I'll limp along with my even more limited local function:

uscp () { . # scp "copyin" to a uvt-kvm "domain"
 OPT=$1
 if [[ $OPT == '-r' ]]
 then
  shift
 else
  OPT=""
 fi
 DOM=$1
 shift
 MHOST="$(uvt-kvm ip $DOM):"
 scp $OPT "$@" ubuntu@$MHOST
}

which lacking all forms of error checking is near malpractice, and certainly not a template for production ... but suffices for my modest needs for now.

Sample usage:
     uscp foobar Downloads/*
     uscp -r foobar Documents

[Launchpad Janitor (janitor)]

[Expired for uvtool because there has been no activity for 60 days.]