I have been kicking around a few ideas in the Lisp OS direction, and wanted to capture this one.
The problem I see with a classic Lisp OS is that the architecture is either completely new, and therefore will take a lot of work to get to near parity with Lisp-on-current-OS, possibly revealing serious deficiencies before it gets there (e.g., I believe Genera would require substantial clean-up work to port to a typical SMP system), or is roughly similar to a current OS, and therefore, what real difference does it make to write it in Lisp, except fun and NIH?
Instead, I think it makes sense to try to address any performance issues a current OS kernel causes for Lisp, and possibly to incorporate the basic “Lisp core” as a kernel personality, meaning simple Lisp executables could be lightweight. Meanwhile, you get all the driver work the usual kernel guys provide, with SMP, optimized virtual memory and file systems, robust networking stacks, etc. Some people have worked on kernel patches that help Java virtual machines give hints to the Linux virtual memory system.
Lisp guys get a set of kernel patches to make their life somewhat nicer if they own their box, with 100% Linux compatibility for all of their other needs, and in the real world, as opposed to dreams of Lisp paradise that never materialize. And if the Lisp personality is really nice, one could look at more kernel extensions (some kind of very Lispy inter-process communication, for instance).
I really just want a way for the core Lisp library and system Lisp globals to be provided by the kernel, protected against modification by the user process, except through syscalls, try to figure out a simple but nice memory map to support this, looking at linking GC to the scheduler and/or virtual memory in some way if it would help performance, possibly handling the memory protection barriers without a context switch, or providing a “shadow” GC thread for every Lisp process.
Most of the Lisp stuff would come from an implementation like SBCL (or at least compatible calling conventions and memory layout, so that the compiler work could eventually be used), perhaps with some surgery being done on the process memory map and “purification” to separate the Lisp into chunks such as
- Basic Lisp library (car, cdr, NIL, aref, …, basically the CL package), non-volatile, shared between Kernel Lisp and user Lisp processes.
- “Lispux” package that wraps the OS networking stacks, multithreading support, etc., non-volatile, provided to user Lisp processes. Kernel Lisp wouldn’t use these.
- A memory space holding Lisp system globals (glimpse of the underlying OS environment; maybe some interprocess communication stuff)
- A memory space holding Lisp-specific kernel structures; for instance, GC bookkeeping. The Kernel Lisp would presumably include the code to handle garbage collection events. The user process would not be allowed to muck with these. There would also probably be a Kernel Lisp heap unless it could be written to be totally non-consing.
- The Kernel Lisp FASL; this is the portion of the Kernel Lisp implementation written in Lisp.
- A memory space holding the user process’s “purified” executable image. Possibly partitioned into non-volatile portions and usual dynamic portions.
- Usual process heap, process stack
- Some kind of support for shared pages of tagged data, that can only be usefully shared with other Lisp-aware processes.
One of the main issues I am trying to get my head around is what Lisp tagged data looks like when the context switch is made to kernel space. In order for GC to work, the pointer values which make sense under the user process memory map must still make sense in the GC context. This makes me think that the Kernel Lisp needs to have basically the same memory layout, with the current Lisp process mapped into it at the standard location, but with different permissions, and with certain parts visible that are invisible in the user context. I believe this is quite different from Linux; from what I can tell, the Linux kernel simply sees the skeleton of the process (the kernel memory map structure, and which pages of physical memory the process has in core), but has to do special work to follow a pointer, for instance. It seems to me doing GC like this would require more overhead than it could save.
I’m wondering whether there is a better alternative to Linux; the main criteria are source hackability, acceptance in the Lisp community, acceptable driver support both current and in the future (including SMP, because that is where the future is going), and relatively good existing applications support. The only alternatives I see is are the BSD variants. I don’t know which of these makes the most sense. I’m guessing SBCL/x86/Linux folks represent the bulk of people who would be interested in something like this. It would be nice to have an explicit mechanism for different “personalities” so that launching an executable FASL would invoke this personaity, but launching a native binary would not see anything much different (except perhaps for a few new syscalls that provide hooks to some of the Lispy goodness, or maybe just an ability to inspect and debug Kernel Lisp.)
RSS Feed
BrassTacks (wpc) 