Include IPs in apt-get output

[Haw Loeung]

Hi,

When running apt-get update/dist-upgrade, output looks like this:

| Hit:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease
| Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [972 kB]
| Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 c-n-f Metadata [29.1 kB]
| ...
| Need to get 101 MB of archives.
| After this operation, 10.5 MB of additional disk space will be used.
| Do you want to continue? [Y/n] y
| Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 dpkg amd64 1.19.7ubuntu3 [1128 kB]
| Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 login amd64 1:4.8.1-1ubuntu4 [221 kB]
| Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 libc6-dbg amd64 2.31-0ubuntu7 [5673 kB]

With various online CI/CD services such as GitHub Actions, it looks like this:

| Get:14 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [833 kB]
| Get:16 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [292 kB]
| Get:17 http://azure.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1044 kB]
| ...
| Get:31 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 cmake amd64 3.10.2-1ubuntu2.18.04.1 [3152 kB]
| Get:32 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic/main amd64 gdb amd64 8.2-0ubuntu1~18.04 [3024 kB]
| Get:33 http://azure.archive.ubuntu.com/ubuntu bionic/universe amd64 epstool amd64 3.08+repack-7 [108 kB]

archive.u.c resolves to multiple IPs. azure.archive.u.c resolves to a load balancer (Azure's Traffic Manager). It would be nice if we also had the IPs of the servers/hosts in the output and would really help us in determining which servers or regions users may be experiencing issues with.

[Julian Andres Klode (juliank)]

All error messages and progress messages (e.g. connecting to ...) include ip addresses. Hence if connecting failed, you get all addresses it tried, for example.

Now. You will likely not see those messages. If it connects fast enough, you'll never see the IP. In non-interactive mode, I don't think the progress messages are shown at all.

So I'm not sure what you're after, given that you get addresses for errors. Investigating slowness? Surely there are better ways to figure that out?

[Julian Andres Klode (juliank)]

Oh I guess error messages post-connect do not include the IP address, as we don't know it anymore.

Given the structure of the non-interactive log, I slightly fail to see how we could include connection messages. UX-wise, it does not make much sense to give you details you're not going to need 99.9% of the time.

[Julian Andres Klode (juliank)]

FWIW, these are all status messages. They are not persistently logged, but are in the [...] per item things in the last line of an interactive session.

I guess we could display the "Connected to %s" messages somehow, but I don't want to display the "Connecting to ..." messages, as there might be a ton of them and that gets really distracting (worst case it's more than the number of IP addresses per address, it tries an additional IP address every 250ms).

So, that at least means that status messages would get priorities, and we'd log the higher priority ones.

methods/http.cc
317: Owner->Status(_("Connecting to %s (%s)"), "HTTP proxy", URI::SiteOnly(Proxy).c_str());

methods/rsh.cc
457: Status(_("Connecting to %s"), Get.Host.c_str());

methods/aptmethod.h
387: PrintStatus("104 Warning", Format, args);

methods/basehttp.cc
53: Owner->Status(_("Waiting for headers"));

methods/connect.cc
150: Owner->Status(_("Connected to %s (%s)"), Host.c_str(), Name);
167: Owner->Status(_("Connecting to %s (%s)"),Host.c_str(),Name);
364: Owner->Status(_("Connecting to %s"),Host.c_str());
576: Owner->Status(_("Connecting to %s (%s)"), "SOCKS5h proxy", ProxyInfo.c_str());

methods/ftp.cc
174: Owner->Status(_("Logging in"));
1071: Status(_("Query"));

[Junien F (axino)]

Can we maybe get IP and download speed displayed if a "verbose" or "debug" or something flag is specified on the command line ?

Something like this maybe :

 | Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [972 kB] 91.189.88.142 12kB/s

What do you think ?

[Julian Andres Klode (juliank)]

Neither of those is available when the message is logged, the message is logged _before_ the item is being fetched, and multiple items may be in progress at the same time, you can't go back and edit the previous line.

So you could display double the amount of entries, but that seems very distracting. I think you could have

Con:<number> Connected to foo (...)

where <number> is the first package for which we need to (re-, in case of host closing connection)connect to a host.

Speed measurements I'm not sure. We only record total speed now. It might make sense to record per-host speed. Then you could display a message per host every $N seconds (N=30?) showing the current speed [well, total average, really].

Note that we just had an ABI/API break, and not intend to break it again for the next 2-4 years, which might prevent this from happening, or at least make it harder.

[Haw Loeung (hloeung)]

Thanks for taking the time to look into this and see if this was currently possible.

The original intent was so that we could check back at user's CI/CD jobs (in particular GitHub's Actions) and see what archive server or region was used.

[Junien F (axino)]

@juliank : could we at least get a debug flag of some sort that would just log everything ? With timings ? So that we could map, for each file, the IP of the server it got downloaded from, time when the download started (maybe even time when the HTTP request got sent, time when we start receiving data), time at the end of transfer. And also, very important, proxy information as well, if any.

And so whenever someones reports "slowness" or misbehaviour, we can give him this flag and ask for output and debug on our end.

Would that be possible ?

Thanks !