Archive for the ‘chaosnet’ Category

ChaosNET implementations for PDP-10

February 10, 2010

This is not particularly interesting, but I wanted to record a couple of links to ChaosNET code for the PDP-10. The ChaosNET memo describes an ITS (Incompatible Time-Sharing System) interface, for which the code is somewhere in the ITS sources, and a Lisp Machine implementation, for which code is in the CADR sources and Symbolics code base, but also a TOPS-20/TENEX implementation, a VMS/VAX implementation, and a UNIX implementation. The UNIX implementation was documented by James E. Kulp for Symbolics, and the code seems to have survived. The TOPS-20/TENEX is harder to find, but it looks like BitSavers’ Foonly archives have some files.


I have no idea whether this is at all related to the implementation the AI MEMO mentions.

A Google search turns up some interesting code (might be for the PDP-11 front end, for ITS)


CH10.defs1 “PDP-10 chaosnet interface definitions”
CH11.defs1 “CH11 Unibus chaosnet interface definitions”
CHSDEF.15 “Chaos net packet definition (for PDP-10s)”
KSNET.1 “Chaosnet interface on Unibus” (presumably for the KS10)
IOELEV.432 “i/o eleven”
NET.32 General ITS networking code

MacIvory speaks!

August 8, 2007

I’ve only had a bit of time to play with my MacIvory, but already things are getting interesting.

One is getting used to the system administration style: typical UNIX system configuration resides in text files, and the boot process reads them. Genera is different. The state of the system is contained in the memory image (“world”); you change it through various operations—some interactive, some by loading Lisp files—and preserve it by saving a new world as a file, which can be defined incrementally based on an existing world. The boot process involves loading that saved file back into the living Lisp machine. I haven’t yet developed the sense to complete a block of tasks before saving the world, and how much change pushes one from incremental to a complete world.

Another was getting the network to speak, and part of that is the timewarp of working in Mac OS 7, an era in which Mac networking was going through the introduction of Open Transport, and an AppleTalk era which Mac OS X refuses to acknowledge. Luckily, I’ve got a PowerBook G3 that runs Mac OS 9. My first ethernet card didn’t work, either because I screwed up some software setting or because the hardware was bad. (More embarrassing disclosure later.) DKS sent me a replacement, which worked immediately. I’ve only got ChaosNet configured on the MacIvory side at this point, and my ChaosNet work isn’t up-to-date on my Powerbook, so I’ve only seen raw ethernet packets in tcpdump, but my archaeology is already progressing—documentation to follow soon, as well as getting some Python software (with libpcap or Mac OS X kernel packet filters?) to respond to Chaos-formatted Ethernet packets.

As for the embarrasing disclosure, one of the first things I did was try to set the video card to a mode not supported by my Sony Multisync 15sf monitor. Instead of waiting for the change to timeout and revert, I hit Esc or Command-. or some other key which confirmed the change, leaving me with a blank screen. I tried various reboot, PRAM zapping, or magic key sequences to force the Radius Precision Color 24X to a good mode, but none of that worked, and at some point I screwed up the system. Once I hooked up the Dell 2007fp that understood 1152×870, I saw the floppy-with-blinking-question-mark. Recovery without a working network card involved the ancient hassles of multiple-floppy installs.

I have a Kensington ADB multi-button mouse, but all the drivers I have downloaded are either for Mac OS 8 or cause serious bugginess: system freezes or error -192 when launching the configuration application. I hope I don’t have to roll my own software to get multi-button goodness to work with the Ivory. (At this date, few vendors pay much attention to 68k Macintosh support.)

UPDATE: I’ve captured a bit of the tcpdump results in my ChaosNET information page.

Ordering a Lisp machine

July 11, 2007

Just sent off the Paypal payment for a MacIvory II system with 2.6 MW of memory. I’m hoping also to get a Symbolics keyboard, although Symbolics is apparently out of the ADB box that allows the keyboard to be used with the vintage Macs.

It was a tough call whether to splurge for this or, for about twice the money, a new MacBook Pro. I’m hoping the Symbolics machine will be much more entertaining.

My main idea of a project is to use the Lisp Machine as a Chaos FILE host, in order to allow development on the CADR simulator with a versioned file system, without having to simulate and learn ITS or TOPS-20.  To get that to work, I probably will need to do a little bit of investigation to understand how the Symbolics box gets configured to talk on the network, what Chaos packets look like on the Ethernet, how to get those to my Python/Lisp code on the modern machine, and, for data security what would be needed to talk to a CD-ROM drive on a remote computer, or to use a remote computer to do backups (to a simulated tape drive?).  In order to get 2.6 MW of NuBus memory, I had to choose a Mac config without a CD-ROM drive, and I’m not too excited about trying to get an external SCSI drive for this old machine. Backing up to floppies is another thing I don’t want to relive.

Some other things I’m looking forward to are Zmail, Concordia (to generate HTML output, anyone?) and Document Examiner,  using CLIM, and grokking the Symbolics compiler(s).

ChaosNET and Python

February 5, 2007

My blogging has been much less frequent of late, mostly because of an unfortunate abundance of non-computer issues. Despite those, I believe I have made some progress in my Chaosnet-on-OpenMCL work, but I’ve switched most of my hobby-development to Python, mostly to get in touch with trends outside the Lisp Universe.

My current main project circulates around ChaosNET emulation; the main thing going for Python here is a socket interface that is roughly compatible between UNIX-like and Windows operating systems. Lisp can support that as well, but not for my favorite and familiar implementations. (One thing I hope to try before I get through this is to get an asyncore-like framework set up in CLISP.)

So far, I have implemented a `chaosd’ process in Python that simulates multiple ChaosNET `ethers’ in the form of localhost TCP streams (on Windows or UNIX-like) or UNIX-domain sockets (on UNIX-like); each stream corresponds to a subnet. The idea has manifold goals:

  • Allow the usim CADR simulator to communicate on Windows with minimal alterations using Winsock in place of BSD sockets (Done) (Brad Parker’s server, however, works with forking and other UNIXisms that would require more work)
  • Allow separation between local activity and activity exposed or connected to the wider Internet
  • Provide ‘masquerade’ and ‘network translation’ bridges, to flexibly map external resources, such as an ITS or TOPS-20 simulator into something that appears to be a local network. (The translation here is necessary because ChaosNET is a LAN architecture; outside resources might have their own incompatible scheme of subnetting and addressing. These translating gateways would allow me to place external UDP-responding resources into arbitrary Chaos addresses in my local scheme, without requiring any change on the external resource.)
  • Eventually, provide small modular `gateway’ simulators that appear to be ChaosNET resources on a local subnet, but translate to external services such as AOL Instant Messenger or IRC.
  • Recreate `historically accurate’ network organizations for amusement
  • If I ever break down and get a real Symbolics machine, I expect it to be able to speak Chaos over IP or raw Ethernet; I would like some way to connect that to multiple simulators on a single Mac or PC

As a step in this direction, I have implemented a stream-to-UDP masquerade, that automatically takes traffic on a local `ether’ for a particular address and feeds it over UDP using Bjorn Victor’s protocol. Running this on two machines, I am able (mostly) to allow a Lisp machine simulator on my Linux box and another on my Windows box to both believe they are on a single Chaosnet LAN, and to communicate with one another. The next step is to get something like a local ITS or TOPS-20 simulator running in KLH-10 on Linux supporting Bjorn’s UDP protocol (or possibly Bjorn’s machine) to provide FILE service and e-mail service to my Lisp machine simulators on either machine. I have been learning to recompile ITS, with mixed success. My initial impressions on Python in another post.

More adventures in multithreading

July 31, 2006

Just wanted to capture a few more thoughts; over the weekend, I experienced what I think was my first case of deadlock. My chaosnet-on-OpenMCL implementation had worked just fine when the two sides of a connection were careful to exchange only one packet at a time, taking turns. When I tried to listen to a SEND (the Lisp machine analogue to an instant message), the CADR emulator sent two packets in quick succession, and my OpenMCL process would freeze trying to receive it.

As an aside, I get the feeling that I don’t really know how to debug this kind of thing in OpenMCL: looking at backtraces does not always make it completely obvious which semaphore or lock is being waited for in the various threads, and I don’t think it is possible to look at semaphores to see which thread(s) are waiting.

In any case, I concluded that my somewhat haphazard use of a “count lock” to protect access to packet number information in a connection was either being used recursively, or violating a lock heirarchy. So, after some time trying to apply my powers of deduction, I decided to look at the ChaosNET problem from a high level to determine what locking is actually necessary.

The main complexity has to do with the transmission window. In theory, it seems like an ideal case for a counting semaphore: the send window is a limited resource, and sending processes must wait until a spot within the window is available. (Generally, one would wish to preserve FIFO order, but strictly speaking, multiple threads trying to transmit on the same channel might not deserve any guarantees that their packets will happen in any particular order.)

However, there are several things unusual about the window: the thread handling incoming packets can get both acknowledgements in ordinary packets, as the receiving user process consumes packets from the window, and in STS packets that could, at any time resize the window. In the usual semaphore model, it is not easy to deterministically reduce the maximum level of the semaphore. I coded an unsatisfactory approach which does a trial-wait-on-semaphore for the number of window packets that have been taken away. But these might not succeed, if the window is filled above the new capacity. In that case, the window must presumably be allowed to drain below the new level before any waiting process gets allowed to put packets in. It might be possible to have a separate integer variable tracking the “excess window consumption,” and for processes to loop back and wait again for the semaphore if the window has been “tightened” since they first waited. I don’t know if it is possible for threads to maintain their original FIFO order in this case, or whether it matters. (I’m not sure it is reasonable for Chaos processes to reduce the window, once it is declared in the initial STS negotiating the connection, as opposed to widening it to allow for better throughput. The AIM memo is silent on this.)

Interestingly, the draft chaosnet memo does discuss enlarging and shrinking the window, and mentions WIN and WTS packets.

In terms of avoiding deadlock, there are many participants: the user reading process looks at the window to see if the connection might be getting “stymied”, and needs an STS to be transmitted. When STS or acknowledgements come in, the retransmission of the receipted or acknowledged packets should be suppressed, and the acknowledgements (but not receipts) open up space in the window. In the Lisp Machine design, the receive-sts et al. use the RECEIPT function to trim the send-pkts list. I felt that contending with the sending/retransmitting process was asking for trouble. Therefore, I chose to have the retransmission process look at the receipt and acknowledgement status to clean up the send-pkts while it needs exclusive access to the send-pkts list anyway. The retransmission process needs access to the send-pkts list to retransmit, and to the counters if it is required to do clean up. Furthermore, updates of the acknowledgement and receipt states should rachet only forward, i.e., a stale packet coming through the pipe should not erase the acknowledgement or receipting indicated by the logically latest packet received up to this point. That requires locking to protect the read-modify-write update of those status numbers.

One interesting gap in the ChaosNET documentation is whether retransmitted packets can or should update their acknowledgement and receipt information to reflect what has happened to the connection in the meantime. The Lisp Machine implementation seems to retransmit the packet byte-for-byte. An alternative would be to update those packets. In practice, I suspect retransmission should be conservative; it is necessary when the receiving process has fallen behind, or the channel is experiencing loss, and the bias should be toward more retransmission than strictly necessary, instead of hoping to “make up for lost time” by advancing the state of the return channel.

I have created enough of a system to work with the SEND transaction, except for one detail: in a Chaos “stream”, my functions indicate an EOF condition only when the request for a byte has gone “off the end.” I felt the EOF detection should predict whether the next byte request should succeed; but I could not remember how this works elsewhere.

More thinking about multithreaded queueing

July 19, 2006

Looking back on my previous post about multithreaded queues, and based on a little hacking I was able to do last night on my multithreaded-Lisp-user-mode-Chaos-client, and having read a bit more in the Little Book of Semaphores, (must stop typing ‘sempahores’!), in particular the turnstile pattern, and the barbershop problem, there are a few other ways to slice the onion.

  1. A “waiting room”, i.e., the privilege of blocking on the queue is exclusive. That actually makes a certain amount of sense when, for instance, Chaos packets are already sorted by the destination connection. Does it really make sense for multiple consumers to be concurrently transacting on a single connection? What would make sense, in a C10k kind of way, is rapidly generating server connections in response to RFCs, and handing these connections to waiting server threads. Or, if a connection is known to have independent incoming packets, or is deliberately recycled, connections can queue through a turnstile, each take one incoming packet, then go off to chew on it. I think the mechanism could be a simple lock.
  2. “Signal enough” can be expanded to include a “turnstile” approach, where a disappointed consumer signals the semaphore on his way “out” so that the next waiting consumer can be disappointed as well. The idea that the reason for the disappointment should be visible to incoming consumers before they block is really optional. The main reason to prefer it is that, if we believe the “disappointment” is permanent, that we will eventually want to discard the queue, but we presumably cannot do so if consumers can continue to block. That is, if the disposal process must race against the arrival of new consumers. Discarding a queue when the semaphore still indicates the presence of (non-existent) packets and no threads are waiting is OK. Discarding a queue when the semaphore indicates no packets, but threads are blocking requires a well-defined mechanism for blocking threads to cleanly deal with a “killed” semaphore. I.e. the “abort” mechanism.

I really like the Little Book of Semaphores. However, I find it hard to be sure that I truly grasp the concepts. I can read through the solutions to the problems, and understand the discussion, but that doesn’t give me a huge amount of confidence in my own solutions to slight variations on the problem. Maybe this kind of programming is simply hard.

I’ve tried casting some of the solutions in OpenMCL Common Lisp. It’s hard to verify that things don’t work by accident. One thing that wasn’t particularly clear to me is whether it is kosher to do things like

(let ((semaphore (ccl:make-semaphore)))
   (labels ((client-function () #|...|#) 
            (server-function () #|...|#))
      (let ((thread1 (ccl:process-run-function #'client-function)) 
            (thread2 (ccl:process-run-function #'server-function)))
         (#|...some way to wait for both threads to terminate..|#))))

It seems to work, except I haven’t yet tried to make a clean “wait for both threads to terminate”, and have just waited for the length of a global list to get to a certain point, or similar kludges. The idea, however, that the let can create “global” variables, but the lambdas can have “thread-local” storage, seems not to be formally specified.

link dump for 22 Feb 2006

February 23, 2006

A bunch of stuff I was reading back then

Weak Data Structures write-up by Bruno Haible, courtesy of Gary King.

The classic UNIX HATERS Handbook.

Panda’s TOPS-20 System.

Spare Time Gizmos store front to buy actual hardware emulators of the PDP-8.
Rainer Joswig e-mail on feel of old Symbolics Lisp Machines

An e-mail discussion about CHAOS vs. TCP in port collisions, a list of Symbolics CHAOS services

Google cache of mahalito
Symbolics Y2K statement
ITS site HACK directory

Google Groups discussion on Basic Chaosnet information
Google Groups discussion on Chaosnet and ARP
Google Groups discussion on Transmitting Chaosnet over Ethernet

Google Groups discussion on Why Chaosnet (nostalgia)
Google groups search for chaosnet

ITS build information build.doc.txt
Retro PDP-10

Some ITS links

February 16, 2006

Wanted to collect a few links on ITS, the Incompatible Time-Sharing System

Some ChaosNET thoughts

February 14, 2006

I looked a little more closely at Bjorn Victor’s (TODO: add diacritic to o in Bjorn) work on Chaos emulation for ITS emulators, particularly the CHUDP protocol for passing Chaos packets through UDP.

Apparently, it uses a simple header (different from the user-mode UNIX header), and dynamically adds to its routing table when it receives incoming packets. This seems particularly handy.

Also, I noticed that the chaosd/server code for the user-mode UNIX implementation does not typically check the hardware destination, but jumps right to the software header for the destination information. This seems wrong.

I’m trying to figure out the most useful way to improve the Chaos support. Some of the use cases would be

  • use Internet hosts as repositories for CADR Lisp Machine microcode and “world” bands
  • use Internet hosts as file servers for Lisp machine source code
  • provide Internet-based early-21st-century substitutes for 1980’s functionality, such as e-mail and instant messaging (e.g. get Zmail to use Google’s Gmail service, use Jabber or AIM to support instant SEND messages, use NTP to provide TIME service)

Some of these would simply be a local process posing as a Chaos server which could respond to various contact names with useful functionality, configured by various users to, e.g., use the instant messaging protocol or e-mail provider of choice. Some of this would require or allow collaboration between nostalgists: someone volunteers, similar to the ITS community, to host the equivalent of AI, and provide ITS-like FILE service on an IP address & port, with a corresponding Chaos subnet/host address.

I suppose part of my problem is I don’t understand all the issues involved in internetworking. Chaos addresses were supposed to be roughly “institution-wide” with specific gateways to other networks, advertised by response to a particular Chaos contact name (such as ARPA). How does this translate to isolated hosts on the Internet posing as institutions with Chaos servers? Can we replace it with a DNS-mediated free-for-all where various IP-based users advertise Chaos DNS entries? How would subnets be allocated? Or will we depend on strictly local, handmade routing tables, where each chaos address gets mapped to an IP host & port? In this case, how do the Chaos source and destination addresses get fixed up when they make it to the IP host & port? Can the sender and receiver basically hide their respective internal Chaos subnetting schemes from each other?

What is this about

January 29, 2006

This blog is a simple experiment for now, to understand what is available for a “scratchpad on the Web.”

I have a picture which is something more like a lab notebook for my various hacking projects. The requirements are roughly

  • access from various places where hacking might happen; an idea for a project can happen at work, but I don’t want my personal laptop with me at all times.
  • some kind of permanence, as suits a lab notebook. Various things this might mean:
    • version control with timestamping
    • no revisions allowed at all (this seems extreme: one wants to separate a polished presentation view from the underlying archival lab notebook. One can revise slides for a talk without altering the historical record of the lab notebook)
    • ability to archive into a big .tgz for burning to disk as necessary
  • flexible project documentation
    • easy way to archive associated files; diffs, error messages, screen shots, sketches, links, data sheets downloaded for the web.
    • perhaps I need to wait for my copy of Kanare’s Writing the Laboratory Notebook (as seen on Lispmeister) to arrive to understand what should be required here.

Some projects to comment on here

  • work on CADR emulator and MIT-released Lisp machine code
    • make it work on Mac OS X/PPC for my laptop
    • make Chaos emulation work on Mac OS X
    • fix some Y2K-type issues, site configuration
    • create Chaos layer in OpenMCL or more portable Common Lisp, to develop gateways to things like
      • AIM/Jabber
      • regular e-mail
      • other retro people running Chaos emulation
      • support versioning file system semantics (e.g. blah.type~version~) on a UNIX FILE host.
      • build a world from the MIT-provided code, alllowing bootstrap
  • Apple II file transfer (extend ADT = Apple Disk Transfer) to include 140k, 800k ProDOS and Macintosh HFS floppy formats
  • Possibly, an Apple II compatible USB peripheral interface, to allow easy transfers to a Mac/PC with USB connection
  • Mac OS X support for foot-pedal shift & mouse-click operations

Interesting that Netscape 7 on Windows gives me little editing widgets on my post that Safari on Mac OS 10.4 didn’t give me (as I recall.) Perhaps I should check, try to add that capability.