unused memory filled with 0x00 instead of 0xFF

[TC1988]

Qemu, ever since it was made (so, since 2003), has this problem in DOS (either PC-DOS or MS-DOS and partly Windows 9x) not recognizing the memory available when the memory is filled with 0x00 but when it is filled with 0xFF it gets recognized properly, where should I patch qemu to solve this memory problem?

[TC1988 (panie)]

well, actually, memory gets filled with the wrong byte, pardon my original post.

[Battler (oubattler)]

Yeah, Qemu fills all bytes of the unused memory with 0x00 whereas it should fill them with 0xFF. This causes DOS to incorrectly interpret available memory as used by Option RAM, breaking the operation of EMM386.EXE which then fails to find available upper memory. This also causes problems with running Windows 3.x.

[Battler (oubattler)]

The problem is specifically about the first 1 MB of RAM. Also, another related problem is that in all Qemu 1.x versions, segments E000-EFFF, possibly even D000-DFFF, get used by an Option RAM that can't be removed. This drastically reduces the amount of available upper memory.

[Stefan Hajnoczi (stefanha)]

On Thu, May 16, 2013 at 9:20 PM, Paolo Bonzini <email address hidden> wrote:
> Il 16/05/2013 19:34, TC1988 ha scritto:
>> Public bug reported:
>>
>> Qemu, ever since it was made (so, since 2003), has this problem in DOS
>> (either PC-DOS or MS-DOS and partly Windows 9x) not recognizing the
>> memory available when the memory is filled with 0x00 but when it is
>> filled with 0xFF it gets recognized properly, where should I patch qemu
>> to solve this memory problem?
>
> Can you provide reproduction instructions for this bug?

Sounds like (risky) memory probing. On a PC the memory regions that
are unpopulated produce 0xff.

But you're supposed to use e820 or other mechanisms to retrieve the
proper memory layout from the firmware.

Stefan

[Peter Maydell (pmaydell)]

On 16 May 2013 20:52, Stefan Hajnoczi <email address hidden> wrote:
> Sounds like (risky) memory probing. On a PC the memory regions that
> are unpopulated produce 0xff.

Presumably you could fix the PC model to do that by putting a big
background (overlappable) MemoryRegion across the whole of the
system address space which returned -1 for every access?

-- PMM

[Battler (oubattler)]

>>But you're supposed to use e820 or other mechanisms to retrieve the proper memory layout from the firmware.
Well go back to the early 1990's and tell Microsoft and IBM that. :p
DOS as it is, refuses to recognize memory not filled with 0xFF's as free. It instead thinks such memory is used by Option RAM.
On Qemu 0.9.1, I got around the issues by padding the video BIOS with 0xFF's to the 64 kB boundary, and using the relevant option to load a a 64 kB .BIN file full of 0xFF's as an option ROM, as well as using the BOCHS legacy BIOS which has the first 64 kB filled with 0xFF's, thus getting D000-EFFF + from about CA00 or so to CFFF filled with 0xFF's, making DOS work correctly.
But on Qemu 1.x and later, this no longer works. Well, the video BIOS padding does work, but I am unable to load an option ROM like in 0.9.1.
Also, for some reason, all Qemu VGA BIOS'es are bigger than 32 kB, even the ones for CL-GD 5446 and VMWare SVGA II., even though the real CL-GD 5446 and VMWare SVGA II. BIOS'es are both 32 kB. This messes with the working of Windows 3.0 which expects segments C800-CFFF to be free.

[Battler (oubattler)]

Also, as for reproduction instruction:
Start MS-DOS and make sure to bypass CONFIG.SYS and AUTOEXEC.BAT.
Then run Microsoft Diagnostics (MSD) and press M for Memory. Look at the Memory Map: areas that are available, get marked as either "potentially available" (which means EMM386 will treat them with caution and not address them unless told to) or "Option RAM", well as anything but "available".
Or just load EMM386.EXE from CONFIG.SYS with a simple DEVICE=EMM386.EXE (preceded by DEVICE=HIMEM.SYS obviously) and reboot. You'll get an error message telling you that EMM386 was unable to find a page frame for EMS, and MEM will report 0 upper memory.

[Anthony Liguori (anthony-codemonkey)]

Paolo Bonzini <email address hidden> writes:

> Il 16/05/2013 22:00, Peter Maydell ha scritto:
>>> > Sounds like (risky) memory probing. On a PC the memory regions that
>>> > are unpopulated produce 0xff.
>> Presumably you could fix the PC model to do that by putting a big
>> background (overlappable) MemoryRegion across the whole of the
>> system address space which returned -1 for every access?
>
> On a modern PC, there is really no non-present memory region where UMBs
> used to be. Memory between 0xc0000 and 0x100000 will be mostly unused,
> but it is not mentioned in the firmware ("e820") memory map.

It's not memory (at least not by default).

> For example on KVM the memory map is:
>
> [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
> [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
> [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
> [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007ffdfff] usable
> [ 0.000000] BIOS-e820: [mem 0x0000000007ffe000-0x0000000007ffffff] reserved
> [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
> [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
>
> but SeaBIOS says:
>
> Copying option rom (size 36352) from 0xfebc0000 to 0x000c0000
> Copying option rom (size 65536) from 0xfebd0000 to 0x000c9000
> Copying option rom (size 1024) from genroms/linuxboot.bin to 0x000ca000
> Copying option rom (size 9216) from genroms/kvmvapic.bin to 0x000ca800
> Space available for UMB: cd000-ee800, f0000-f1c20

There are a couple things happening here.

Long before the PCI roms are moved to the option rom space, that space
is made writable via PAM. PAM is a more modern feature that was
introduced to make firmware execution much faster.

Without PAM, this whole range is mapped to ROM memory which means it's
going to get put out on a bus. If there isn't a ROM mapped, the result
is going to be 0xFF.

We don't emulate this correctly because we don't emulate execute from
ROM which means that we always treat the option rom space as RAM even
though is it not at start of day.

> So you can see it in three ways:
>
> 1) It's not a QEMU problem, but a firmware problem. You could initialize
> the UMBs with 0xFF in SeaBIOS for example.

SeaBIOS is not supposed to do any initialization. Once it's RAM,
there's no guarantee of what the contents will be. The problem is that
it's supposed to be ROM.

I doubt it's still there, but BIOSes often had an option to disable
ROM shadowing expressly because it breaks applications that assume that
this space is ROM after BIOS loads.

Perhaps SeaBIOS could support such an option but we can't support it in
KVM at the moment.

> However, it is probably
> necessary to round the UMBs somehow, because emm386 probably probes
> memory with a larger granularity than 32 bytes. For example, in the case
> above you may want to leave 0x00 outside 0xD0000-0xEC000 (16kb granularity).
> You also need to ensure the UMBs are write-protected (I do not remember if
> that's already the case).
>
> 2) It's not a QEMU prablem, but an EMM386 problem. Simply, EM...

[Battler (oubattler)]

Well about it being an EMM386 problem - no, it's DOS itself that maps the memory incorrectly if it's not filled the way it expects it. EMM386 just asks DOS for a memory map and tries to find the first free segment. But in this case, DOS hasn't mapped any segment as free, so EMM386 is unable to do anything and errors out.
Also, I've used a real PCI machine from the late 1990's - a Celeron 333 MHz, and EMM386 worked fine on it. At least segments D000-DFFF were free. That's not the case on Qemu for some reason - I guess Qemu emulates a 21st century PCI machine instead, with no option to make it emulate one from the late 1990's.
As for using UMBPCI - I already tried it and it's not compatible with Windows 3.x and earlier. And it wouldn't change much anyway since it's not the EMM manager that probes the memory but DOS itself. This is why MSD is able to show a memory map of the upper memory even when neither HIMEM nor EMM386 are loaded.

[Thomas Huth (th-huth)]

Looking through old bug tickets... is this still an issue with the latest version of QEMU? Or could we close this ticket nowadays?

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]

[David Glover (me-davidglover)]

I know this is expired but it's still a problem in qemu 7.0.0. For example, when running MS-DOS 6.22, "mem" reports 0K of upper memory, and memmaker fails to run, complaining that it could not allocate any. I'd love to know if there's a workaround.

[Yonggang Luo (luoyonggang)]

I create a issue at https://gitlab.com/qemu-project/qemu/-/issues/1133 to
tracking this

On Mon, Aug 1, 2022 at 5:06 AM David Glover <email address hidden>
wrote:

> I know this is expired but it's still a problem in qemu 7.0.0. For
> example, when running MS-DOS 6.22, "mem" reports 0K of upper memory, and
> memmaker fails to run, complaining that it could not allocate any. I'd
> love to know if there's a workaround.
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1180923
>
> Title:
> unused memory filled with 0x00 instead of 0xFF
>
> Status in QEMU:
> Expired
>
> Bug description:
> Qemu, ever since it was made (so, since 2003), has this problem in DOS
> (either PC-DOS or MS-DOS and partly Windows 9x) not recognizing the
> memory available when the memory is filled with 0x00 but when it is
> filled with 0xFF it gets recognized properly, where should I patch
> qemu to solve this memory problem?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1180923/+subscriptions
>
>
>

--
         此致
礼
罗勇刚
Yours
    sincerely,
Yonggang Luo

[Evren (evren320)]

I am using Qemu 8.0.0 and this issue happens only when kvm is enabled, tcg initialized the unused bytes to 0xFF as a real machine and works normally, so a question in my mind, does kvm use those umb area for increased performance?
If emm386 M5 is used for using area D000-D7FF as frame page, emm386 gives a warning that there is an option rom in this location but it still uses this area, seems like a workaround, i do not know if it will have any side effect