LIMSPEC, HIMEM.sys and EMM386.sys. So that this history is not lost, here is more information on expanded memory managers than you ever wanted to know.
What is EMM386?
EMM386 is the DOS 5 implemented device driver to implement an EXPANDED memory manager in DOS using the services of the 80386 processor instead of dedicated hardware. Your next question: What is an EXPANDED memory manager and how does it differ from an EXTENDED memory manager? Good questions. This will take some text to answer, and this was all very important stuff in the late 1980s!
Nobody will ever need more than 640KB
With the Intel 8088 and the original IBM PC, there was addressability for 1MB of memory. People describe the original PC has having a memory limit of 640KB, but that is not really accurate. The machine had 20 bits of memory addressing on the ISA bus and (2**20) -1 is 1MB. By convention, the bottom 640K is dedicated to software and the high 360KB is dedicated to hardware – such as screen memory. Much of the hardware space was unused; reserved for possible new hardware, but things were scattered holes, monochrome screen memory being 4KB at B000:0000 for example and the graphics memory adapter living at B8000:0000. Write bits to those spaces and characters show up on the screen! Hardware devices are above 640KB and BIOS, DOS and applications are below. This became not absolute.
When Lotus 123 ruled the world – the 640KB RAM limit was exhausted with large spreadsheets as well as with other large applications. Application implemented paging storage to and from disk was a common work around – but it is slow, and hard to program. Think diskettes. Even spinning hard disk media of the day was also dreadfully slow.
More RAM
The first solution was to buy memory cards that plugged in the ISA bus and filled memory space above 640KB inside the holes where hardware memory was not used. On some systems, it was possible to free the first 64KB of hardware space and add RAM memory to fill that hole and you could end up with a DOS computer with 704KB of usable RAM. This was interesting, but it was not expanded memory.
What was needed was an ability to get “more RAM” and page that RAM in when needed.
640KB is the DOS limit and there were a few interim 8088 usable expanded memory solutions that never took hold and diverged the industry. Each implemented “more memory” in an ISA bus card and provided some hardware means to copy memory to and from 8088 visible memory.
Each application that wanted to use this additional memory, had to code to the hardware of the individual hardware card. Applications coding to multiple hardware specs! This is untenable.
LIMSPEC – Lotus Intel Microsoft Specification
Eventually, Lotus, Intel and Microsoft got together and invented “LIM Spec” (Lotus Intel Microsoft SPEC) to standardize this behavior, providing an API that implemented the memory paging to and from memory outside of DOS visible space and this was officially titled “Expanded Memory Specification“, though the programmers continued to call it LIM Spec so they understood which API people were discussing.
Some of the other ones included “XMS from IBM”?? These were not the same.
Hardware
The LIM spec hardware worked via hardware paging. The LIM spec memory adapter hardware could have LOTS of memory on it – Like 512KB or 1 MB or even more! I think 16MB was the theoretical limit, but nobody could afford that much memory.
By whacking a few registers on the hardware memory card via I/O bus INs and OUTs, the backing store behind the memory at particular “block” could be changed (was actually called a “page”, but it was a 16KB or 64KB page). So, if you have a large spreadsheet, instead of paging it to and from disk, you could page it to and from “invisible memory” by touching a couple I/O ports on the ISA bus, which meant you could page this memory in very fast! Only a few milliseconds!
This was blindly fast compared to other stuff like trying to use the 8088 CPU to copy 4 KB in and 4KB out of some other memory solution. LIM spec was used primarily for DATA. One could theoretically move program code to LIM Spec memory, but this wasn’t the normal need.
The important thing was that it was a standard that an application could code against, and then the hardware makers could produce compatible hardware and know it would work with the important applications of the day.
Standards
Even better, a device driver was written to do the talking to the hardware stuff, once implemented, expanded memory aware applications could code to the device driver LIMSPEC API and the applications were now insulated from how the expanded memory managers implemented paging.
LIM spec was good stuff – but you had a purchase a very expensive card to make it do things.
Intel releases the 80286
Now, there is 16MB of genuine addressable memory (24 bits) – but DOS still knows how to use only 1MB. RAM is also becoming cheaper but still nobody could afford 16MB, it was common at the time for machines to come with 2MB of RAM. If only DOS could see higher than 640KB, this memory could do something useful and there is lots of it. 2MB is a huge increase!
What is EXTENDED memory?
EXTENDED memory is the genuine 80286 visible system memory that is present in the machine, but isn’t usable because DOS can’t see that high. It is the stuff above 1MB. It is just memory, but it is called “Extended” so that you know it isn’t DOS normal memory.
Note that “extended memory” as a concept is incorrect, its just memory and all the memory present in the machine is addressable by the processor, the fact that DOS and applications under DOS understand only the memory below 1MB is just a statement to the architecture at the time.
Hardware ISA bus implemented LIM spec solutions still selling well at this time and the standard API exists, it now became possible to use the 80286 addressable extended memory to implement LIM Spec! In concept, use a couple GDT selectors of the 80286 and you can page memory to and from EXTENDED memory to accomplish LIM Spec, and usable by DOS and most importantly, cheap! No longer need a hardware memory card. But, the 80286 was never an important chip, so these techniques never took off, the hardware solutions for LIMSPEC kept selling.
VCPI – Virtual Control Program Interface
Virtual Control Program Interface (VCPI) became a standard for running applications in the protected memory space of the 80286. Here you could load PROGRAMS into the 80286 high memory and run that application code! Quarterdeck with QEMM were the players here, but I’m going from a long ago memory.
You had to “come back to DOS” for OS functions or I/O with device drivers, but you could put gobs of program and data up high. I once wrote application programs under VCPI; it was a small part of the DOS programming world that most people did not understand. Getting I/O to work to external hardware required writing code to “switch” to and from protected mode. The 80286 was a stepping-stone to the right solution. Yes, it would be possible to write device drivers in the high memory space, but interrupt delivery was space of the VCPI and this was not a common thing.
DPMI (80386) – DOS Protect Mode Interface
The market progresses and the Intel 80386 CPU comes out, a CPU with all the things one expects in the modern world. It has selectors rather than segments, just like the 80286, but it also adds page-table based memory addressing and MVDM (multiple virtual DOS machine) support. To be fair, the 80286 also had selector-based addressing, but it was the 80386 processor addition of page tables that really made it possible for solutions to high memory addressing.
The execution of applications in high memory under DOS on the 80386 was governed by DOS Protect Mode Interface.
80386 and DPMI
PharLap was another player on the high memory execution front, and the definition of DOS Protected Mode Interface. This provided a standard means to run programs in the high memory space, but it also implemented LIMSPEC in software which performed every bit as efficiently as LIMSPEC in hardware memory expansion cards. More efficiently actually because the modern 80386 machines and their RAM were hugely fast compared to ISA bus implemented RAM and the page tables of the processor could be quickly programmed to change backing store for LIM Spec paging.
Running application up high required a memory manager to accomplish, but the same techniques needed to manage the high memory, could also be used to implement LIMSPEC. During boot, you could load “DOS” device drivers which was really the extended memory manager, which put the machine into protected mode, and implemented DPMI and LIMSPEC.
DPMI provided a standardized method to get programs and data into high memory spaces.
The biggest runtime for executing applications in this high space was DOS4GW.exe distributed by Watcom. PharLap was the main player in the memory manager space, with the PharLap DOS extender. Exactly the relationship between PharLap and Watcom, I do not know.
DPMI was quite successful.
Microsoft releases DOS 5
DOS 5 included HIMEM.sys and EMM386! The functions of LIMSPEC and DPMI were brought to the market with DOS, and no need to purchase an expanded memory manager, and within 6 months, PharLap folded.
- HIMEM.sys puts the CPU into “protect mode” — Implements DPMI
- EMM386.sys uses the services of HIMEM.sys to implement “Expanded Memory Management (LIM Spec) using the 80386 CPU
I am partially vague on what is implemented in HIMEM.sys and what is implemented in EMM386.sys. The functions varied over time or at least my understanding of them and there were other drivers involved.
Who provides the DPMI manager?
One of the PRIMARY THINGS of DPMI is that all DPMI applications can INCLUDE a DPMI memory manager in their binary. The rule is that if a DPMI manager has not yet been loaded – you BECOME the DPMI manager. If one is already there, you use its services and do not load your memory manager. This also became not absolute over time.
If the DPMI manager already running is “old” – you can perform a coup.
You can also package the executable separate from the memory manager.
Thankfully for the market, once DOS 5 included the DPMI manager, the need for application programs to carry one along went away and the world became a more stable place.
Windows 3.1 to Windows 95
The most successful DPMI application ever was Windows. Starting with Windows 3.0 Enhanced mode along with Windows 3.1 and eventually Windows for Workgroups 3.11. Over time, more and more OS code was moved from DOS space to high memory space, and eventually Windows 95 was born. Windows 95 while celebrated as a great new operating system, was really a very large, DPMI application running on top of DOS. Andrew Schulman documented this well in his book Undocumented Windows.
Eventually, Windows NT took hold as a “real OS” and once the world got to Windows XP, the days of DOS were behind us. Except, its 2022, and yes, DOS is still running.
DOS 5, 6, 7
As DOS matured, the management of the hardware memory gaps above 640KB became well managed. By this time, much hardware allowed its hardware memory address to be configured in software, making it easier to produce large “gaps” in the memory above 640KB and below 1MB. Careful placement of device drivers during boot into these holes allowed maximizing of DOS memory below 640KB and the memory managers of DOS made this organization possible. MEM.exe included with DOS made it possible to get a clear map of all the memory being used and a bit of careful adjustment could produce a very cleanly running machine with RAM available for DOS and applications maximized.
Conclusion
There are places in this blog where the information is “good”, but also likely incomplete. If you have more details, please add in the comments – and send me an email at joe at this domain. The blog system is set to not post comments unless approved, and I’ll need a little kick to move them to published.
Joe Nord