subiquity download progress-bar on s390x swamps console messages and buffer

[Frank Heimes]

The challenge here is that due to the limited functionality in the IBM Z and LinuxONE console ("Operating System Messages" task)
a progress-bar is not replacing it's current line with an updated progress (consuming just a single line), but just instead prints a new line on every update.
Hence one line per update is printed and in case of high update intervalls that swamps the console buffer and leads to partly unresponsive (browser-based) console window (probably due to the high amount of data).

As an example we have here 145 lines, and with that updates, for the "8%" download status - that makes up about 1500 lines in total for the entire download (100%).

"57950K .......... .......... .......... .......... 8% 93.1M 6s"
"58000K .......... .......... .......... .......... 8% 142M 6s"
"58050K .......... .......... .......... .......... 8% 107M 6s"
"58100K .......... .......... .......... .......... 8% 110M 6s"
"58150K .......... .......... .......... .......... 8% 129M 6s"
"58200K .......... .......... .......... .......... 8% 98.8M 6s"
"58250K .......... .......... .......... .......... 8% 111M 6s"
"58300K .......... .......... .......... .......... 8% 91.6M 6s"
"58350K .......... .......... .......... .......... 8% 119M 6s"
"58400K .......... .......... .......... .......... 8% 166M 6s"
"58450K .......... .......... .......... .......... 8% 81.2M 6s"
"58500K .......... .......... .......... .......... 8% 139M 6s"
"58550K .......... .......... .......... .......... 8% 95.8M 6s"
"58600K .......... .......... .......... .......... 8% 149M 6s"
"58650K .......... .......... .......... .......... 8% 85.6M 6s"
"58700K .......... .......... .......... .......... 8% 144M 6s"
"58750K .......... .......... .......... .......... 8% 105M 6s"
"58800K .......... .......... .......... .......... 8% 136M 6s"
"58850K .......... .......... .......... .......... 8% 81.4M 6s"
"58900K .......... .......... .......... .......... 8% 71.7M 6s"
"58950K .......... .......... .......... .......... 8% 6.59M 6s"
"59000K .......... .......... .......... .......... 8% 9.40M 6s"
"59050K .......... .......... .......... .......... 8% 103M 6s"
"59100K .......... .......... .......... .......... 8% 138M 6s"
"59150K .......... .......... .......... .......... 8% 85.4M 6s"
"59200K .......... .......... .......... .......... 8% 133M 6s"
"59250K .......... .......... .......... .......... 8% 131M 6s"
"59300K .......... .......... .......... .......... 8% 107M 6s"
"59350K .......... .......... .......... .......... 8% 115M 6s"
"59400K .......... .......... .......... .......... 8% 107M 6s"
"59450K .......... .......... .......... .......... 8% 100M 6s"
"59500K .......... .......... .......... .......... 8% 111M 6s"
"59550K .......... .......... .......... .......... 8% 125M 6s"
"59600K .......... .......... .......... .......... 8% 107M 6s"
"59650K .......... .......... .......... .......... 8% 116M 6s"
"59700K .......... .......... .......... .......... 8% 85.2M 6s"
"59750K .......... .......... .......... .......... 8% 151M 6s"
"59800K .......... .......... .......... .......... 8% 108M 6s"
"59850K .......... .......... .......... .......... 8% 114M 6s"
"59900K .......... .......... .......... .......... 8% 100M 6s"
"59950K .......... .......... .......... .......... 8% 133M 6s"
"60000K .......... .......... .......... .......... 8% 111M 6s"
"60050K .......... .......... .......... .......... 8% 125M 6s"
"60100K .......... .......... .......... .......... 8% 83.9M 6s"
"60150K .......... .......... .......... .......... 8% 108M 6s"
"60200K .......... .......... .......... .......... 8% 97.6M 6s"
"60250K .......... .......... .......... .......... 8% 131M 6s"
"60300K .......... .......... .......... .......... 8% 110M 6s"
"60350K .......... .......... .......... .......... 8% 96.1M 6s"
"60400K .......... .......... .......... .......... 8% 166M 6s"
"60450K .......... .......... .......... .......... 8% 98.7M 6s"
"60500K .......... .......... .......... .......... 8% 130M 6s"
"60550K .......... .......... .......... .......... 8% 94.0M 6s"
"60600K .......... .......... .......... .......... 8% 100M 6s"
"60650K .......... .......... .......... .......... 8% 154M 6s"
"60700K .......... .......... .......... .......... 8% 99.0M 6s"
"60750K .......... .......... .......... .......... 8% 153M 6s"
"60800K .......... .......... .......... .......... 8% 84.8M 6s"
"60850K .......... .......... .......... .......... 8% 107M 6s"
"60900K .......... .......... .......... .......... 8% 152M 6s"
"60950K .......... .......... .......... .......... 8% 96.1M 6s"
"61000K .......... .......... .......... .......... 8% 156M 6s"
"61050K .......... .......... .......... .......... 8% 105M 6s"
"61100K .......... .......... .......... .......... 8% 76.0M 6s"
"61150K .......... .......... .......... .......... 8% 178M 6s"
"61200K .......... .......... .......... .......... 8% 125M 6s"
"61250K .......... .......... .......... .......... 8% 95.2M 6s"
"61300K .......... .......... .......... .......... 8% 144M 6s"
"61350K .......... .......... .......... .......... 8% 76.9M 6s"
"61400K .......... .......... .......... .......... 8% 136M 6s"
"61450K .......... .......... .......... .......... 8% 125M 6s"
"61500K .......... .......... .......... .......... 8% 135M 6s"
"61550K .......... .......... .......... .......... 8% 85.7M 6s"
"61600K .......... .......... .......... .......... 8% 112M 6s"
"61650K .......... .......... .......... .......... 8% 143M 6s"
"61700K .......... .......... .......... .......... 8% 105M 6s"
"61750K .......... .......... .......... .......... 8% 92.0M 6s"
"61800K .......... .......... .......... .......... 8% 141M 6s"
"61850K .......... .......... .......... .......... 8% 105M 6s"
"61900K .......... .......... .......... .......... 8% 98.5M 6s"
"61950K .......... .......... .......... .......... 8% 127M 6s"
"62000K .......... .......... .......... .......... 8% 96.2M 6s"
"62050K .......... .......... .......... .......... 8% 117M 6s"
"62100K .......... .......... .......... .......... 8% 106M 6s"
"62150K .......... .......... .......... .......... 8% 116M 6s"
"62200K .......... .......... .......... .......... 8% 102M 6s"
"62250K .......... .......... .......... .......... 8% 142M 6s"
"62300K .......... .......... .......... .......... 8% 112M 6s"
"62350K .......... .......... .......... .......... 8% 110M 6s"
"62400K .......... .......... .......... .......... 8% 134M 6s"
"62450K .......... .......... .......... .......... 8% 77.5M 6s"
"62500K .......... .......... .......... .......... 8% 63.3M 6s"
"62550K .......... .......... .......... .......... 8% 168M 6s"
"62600K .......... .......... .......... .......... 8% 97.4M 6s"
"62650K .......... .......... .......... .......... 8% 151M 6s"
"62700K .......... .......... .......... .......... 8% 90.1M 6s"
"62750K .......... .......... .......... .......... 8% 104M 6s"
"62800K .......... .......... .......... .......... 8% 111M 6s"
"62850K .......... .......... .......... .......... 8% 133M 6s"
"62900K .......... .......... .......... .......... 8% 115M 6s"
"62950K .......... .......... .......... .......... 8% 75.4M 6s"
"63000K .......... .......... .......... .......... 8% 169M 6s"
"63050K .......... .......... .......... .......... 8% 108M 6s"
"63100K .......... .......... .......... .......... 8% 127M 6s"
"63150K .......... .......... .......... .......... 8% 121M 6s"
"63200K .......... .......... .......... .......... 8% 86.6M 6s"
"63250K .......... .......... .......... .......... 8% 121M 6s"
"63300K .......... .......... .......... .......... 8% 126M 6s"
"63350K .......... .......... .......... .......... 8% 95.9M 6s"
"63400K .......... .......... .......... .......... 8% 95.5M 6s"
"63450K .......... .......... .......... .......... 8% 128M 6s"
"63500K .......... .......... .......... .......... 8% 94.6M 6s"
"63550K .......... .......... .......... .......... 8% 170M 6s"
"63600K .......... .......... .......... .......... 8% 123M 6s"
"63650K .......... .......... .......... .......... 8% 91.6M 6s"
"63700K .......... .......... .......... .......... 8% 94.3M 6s"
"63750K .......... .......... .......... .......... 8% 151M 6s"
"63800K .......... .......... .......... .......... 8% 118M 6s"
"63850K .......... .......... .......... .......... 8% 116M 6s"
"63900K .......... .......... .......... .......... 8% 79.4M 6s"
"63950K .......... .......... .......... .......... 8% 135M 6s"
"64000K .......... .......... .......... .......... 8% 129M 6s"
"64050K .......... .......... .......... .......... 8% 111M 6s"
"64100K .......... .......... .......... .......... 8% 93.3M 6s"
"64150K .......... .......... .......... .......... 8% 143M 6s"
"64200K .......... .......... .......... .......... 8% 104M 6s"
"64250K .......... .......... .......... .......... 8% 110M 6s"
"64300K .......... .......... .......... .......... 8% 142M 6s"
"64350K .......... .......... .......... .......... 8% 73.3M 6s"
"64400K .......... .......... .......... .......... 8% 127M 6s"
"64450K .......... .......... .......... .......... 8% 105M 6s"
"64500K .......... .......... .......... .......... 8% 148M 6s"
"64550K .......... .......... .......... .......... 8% 122M 6s"
"64600K .......... .......... .......... .......... 8% 75.7M 6s"
"64650K .......... .......... .......... .......... 8% 186M 6s"
"64700K .......... .......... .......... .......... 8% 102M 6s"
"64750K .......... .......... .......... .......... 8% 120M 6s"
"64800K .......... .......... .......... .......... 8% 122M 6s"
"64850K .......... .......... .......... .......... 8% 82.4M 6s"
"64900K .......... .......... .......... .......... 8% 98.9M 6s"
"64950K .......... .......... .......... .......... 8% 19.0M 6s"
"65000K .......... .......... .......... .......... 8% 3.04M 6s"
"65050K .......... .......... .......... .......... 8% 177M 6s"
"65100K .......... .......... .......... .......... 8% 84.0M 6s"
"65150K .......... .......... .......... .......... 8% 150M 6s"

And since more lines got produced than the entire console buffer can store, it results in the fact that the initial/first lines can not be reached while scrolling up.
(In fact the 'oldest' msg one can get back to is roughly:
 22250K .......... .......... .......... .......... 99% 79.2M 0s
)

A quick solution would be to reduce the update interval of that download progress-bar - e.g. by 150:1 (if possible) .

In the early boot phase it lokks nicer - just a few lines are displayed:

"host : s1lp14"
"rootserver: 0.0.0.0 rootpath:"
"filename :"
"Connecting to 10.13.0.2:21 (10.13.0.2:21)"
"focal-live-server-s3 4% |* | 28.7M 0:00:23 ETA"
"focal-live-server-s3 16% |***** | 115M 0:00:10 ETA"
"focal-live-server-s3 27% |******** | 196M 0:00:07 ETA"
"focal-live-server-s3 38% |************ | 275M 0:00:06 ETA"
"focal-live-server-s3 50% |**************** | 360M 0:00:04 ETA"
"focal-live-server-s3 63% |******************** | 445M 0:00:03 ETA"
"focal-live-server-s3 75% |************************ | 533M 0:00:02 ETA"
"focal-live-server-s3 91% |***************************** | 645M 0:00:00 ETA"
"focal-live-server-s3 100% |********************************| 707M 0:00:00 ETA"

See attachment for an almost entire console boot-up log ...

[Frank Heimes (fheimes)]

A screenshot that shows the browser responsiveness ...

[Michael Hudson-Doyle (mwhudson)]

If we had full wget in the initrd we could pass --progress=dot:giga or something but we don't.

[Frank Heimes (fheimes)]

I agree "wget --progress=dot:giga" or "wget --progress=bar:force:noscroll" could have been an option, Both can be used in d-i shell - but looks like we are in this case limited to the stripped down (busybox) wget.

[Dimitri John Ledkov (xnox)]

hm.... I'm not sure where the dot progress bar is coming from.

busybox wget progress is with stars. It prints update every second, and manages to download the whole thing in like 9 seconds.

[Dimitri John Ledkov (xnox)]

"Error parsing preconfiguration file: Syntax error: unable to determine template"
owner

[Dimitri John Ledkov (xnox)]

I think this is the same bug as "failing to preseed ubiquity .iso"

url= is both a netboot/url= preseed key and a casper iso download key.

We need to stop downloading it the second time as a preseed, when clearly it cannot be a preseed.....

[Frank Heimes (fheimes)]

I tried the initial installer steps today on z/VM and the problem with the massive updates on the download progress is even worse there.
The amount of lines for 1% progress image download status is on z/VM in the 3270 console 144.
With the default 3270 screen (80x"43") it's a bit more than 3 page scrolls.
That are in total more than 300 page scrolls!
And page scrolls are unfortunately not really fast in a 3270 env, hence this takes several minutes.

[Frank Heimes (fheimes)]

I just updated the installation environment to the latest daily image (timestamp Dec 15th 2019):
"Ubuntu-Server 20.04 LTS _Focal Fossa_ - Alpha s390x (20191215)"
that includes subiquity 19.12.1 and can confirm that the issue reported here (with the huge amount of download progress update lines) does not exist anymore.
Thx for solving this!

I'll change the project entry to Fix Released.
The Focal entry can be changed to Fix Released, too (in my opinion).
And I think we can set Eoan to Won't Fix, since the default installer for Eoan (on s390x) was still d-i, and the live image for Eoan (s390x) was always considered as experimental.

[Brian Murray (brian-murray)]

eoan has reached end of life, so this bug will not be fixed for that release

[Frank Heimes (fheimes)]

This is btw. already fix released - and I only marked the project entry as fix released, since I did not wanted to mess with the package entries cared by the maintainers.
I guess the update was just forgotten - now exceptionally updating the casper entry to fix released, too (to get the entire LP bug closed and marked as done).