Lisp system development

I’m trying to capture a series of quasi-random thoughts that I had when considering the problems of getting a Linux SBCL installation to use clim-desktop. (Apart from the apparent bugginess of the apps themselves)

  • not everything is ASDF-installable
  • not everything has stable “releases” with well-defined “version numbers”
  • several interdependent things either just don’t work, or have version skew between whatever the core developer was using at the time and what asdf-install happens to pull down, and there is no automatic way to detect this, causing breakage
  • sometimes, and I’m not sure why, SBCL can complain that “compile-file fails” but ASDF keeps on loading the system
  • Most ASDF systems don’t have a version number; sometimes the source directory has one, but when you pull from CVS, all you can do is hope you remembered what time you pulled.
  • Sometimes, I need to make slight edits (e.g. clim-desktop “0.2” uses symbols from packages like “CLIMACS-COMMANDS” which climacs “20060504” doesn’t have) to Lisp source files, so even if I ask the system what version I have, it isn’t exactly right. I would have to remember to do an RCS diff on the source tree(s) to determine what changes I made.
  • Oh, and another thing, Closure seems to have a lot of functions in it which, if the names are to be believed, might better be broken off into or inherited from smaller library packages (e.g., graphics, XML/DOM, SGML, FTP, compression, LALR parsing, URL parsing, MIME)

That triggered the following laundry list of features one might want in a “Lisp system management environment”

  • “Properties” of systems/packages/releases which could be automatically introspected, browsed, and reported; these are filled in by the person(s) “responsible for” the library
  • canonical name of the system (should be redundant)
  • human readable headline description
  • longer “press-release” style description
  • list of author(s)
  • provenance/URL of home-page/source code repository including machine-readable description of source-code control tool used
  • license (with standard machine-readable choices)
  • URL/file name in source directory of human-readable documentation, including machine-interpretable description of the format (INFO file, HTML, plain text, Texinfo, LaTeX, etc.)
  • mailing list addresses for support, development, announcement
  • URL/mailing list for bug reporting (ideally, one of a few identifiable methods that an automatic “report bug” feature could use)—Lisp Machine programs seem to have had features built in that would compose an e-mail to the bug-reporting address containing an automatically extracted description of the version numbers of the system, the OS, and the microcode, and allow the user to describe the bug
  • keywords or terms for indexing/search purposes (e.g. RFC numbers, third-party products, relevant standards, buzzwords, comparable libraries in other languages, etc.)
  • Properties which are semi-automatically generated, as part of the build/source-code management process
    • version identifier
    • “point-style” version number (if it is an actual release)
    • timestamp (if it is simply a snapshot)
    • this would presumably be a universal time based on the RCS server’s clock; the epoch would have some clock skew between packages, perhaps this is not fatal
  • timestamp and version number of previous dot-release (i.e. a “stable” reference point that the snapshot could be compared to)
  • timestamp and version number of the next dot-release, if known (i.e. you extracted what might be a stale snapshot, and you and your maintenance tools should be able to recognize that a newer dot release is available.)
  • unique (UUID?) identifier of this combination of package and timestamp/version; possibly in a machine-interpretable form, but perhaps simply a unique “token” for comparison
  • human-readable “herald” summary “foobar subversion 23 Aug 2006 18:11:03 UTC (release 2.13 + 13 days)”
  • Indication that the test suite passed with compiler X on platform Y kernel release Z and dependencies Baz, etc., as described below
  • List of “issues resolved” by this release, probably including machine-parseable references to issues in the bug tracking tool, and the corresponding test suite entries
  • List of “known issues”, similarly referenced to the issues in the bug tracking tool, and the corresponding (failing or expected) test suite entries
  • Properties having to do with dependencies
    • a tree of the dependencies, linking together the instances with the above-listed properties (“dependencies” should probably include the Lisp compiler/implementation and host OS, identified according to a standard heirarchy)
    • also, encoded information by the coder(s)/tester(s) of the parent package indicating various “versions of interest”, e.g.
    • “this version of foobar was developed using library Baz from the CVS snapshot of 10 Apr 2003 10:02 UTC” meaning it should work as the developer intended if the installed version matches exactly
    • “requires later than” or “this version of foobar relies on features of Baz as of Baz release 3.14=3 March 2003 9:45 UTC” or “as of Baz CVS tag ‘March Madness’=… UTC” meaning foobar will be guaranteed not to work, or at least will be missing major functionality, if the installed version of Baz is earlier
    • “verified with Baz releases X, Y, Z…” meaning “somebody ran the foobar test suite with Baz release X and was satisfied with the results”
  • a notice if “foobar includes code that ‘patches’ Baz” with some kind of UUID of the corresponding diffs/changes, so that the Baz bug reporting tool can indicate “hey, foobar mucked with me so the bug might not manifest itself on an ‘original’ Baz even with the same version timestamp”, and a way for the Baz developers to ask for/automatically retrieve the relevant diffs from the foobar developers.
  • all this information is meant to automate identification of “version skew” issues or possible stale or incompatible or subtly customized or altered dependencies

    • Some information one might want to have about the “issues” (perhaps existing bug tracking tools are a better place to look for ideas here)

    • a UUID of some kind, allowing the bug-tracking database to be queried for the issue entry
    • human-readable headline
    • current bug status, and reported severity
    • system(s) known to be affected (including dependency information allowing human and machine comparison to what is installed)
    • ID of a test suite entry that will identify the presence of the issue
    • earliest release in which issue known to exist
    • earliest “fixed” release, if known

    Some “management” tools

    • Automatic ways to make comparisons to the source releases/source trees (e.g., if I patch Climacs to work-around a clim-desktop issue, an inspection of the Climacs “description” should indicate it has been changed, and allow me to recall automatically what the changes were, including generating a UUID that identifies that set of changes, packaged with a diff file for submitting a patch or to document a bug report)
    • Automatic ways to query on the “known issues” to see if the bug tracking database shows the issues have been resolved, presumably by a newer release
    • Automatic method to download the corresponding patch(es) that was/were checked in to resolve a specified issue
    • Automatic way to download e-mail discussions from the developer discussion list that mention the issue’s ID/are linked to the issue
    • Automatic ways to query the bug tracking database to see if any “formerly unknown” issues have been identified for the installed release, updating the (persistent) local “known issue” data
    • Automatic ways to show diffs between a user-specified snapshot (e.g. “current CVS”) and the installed version, to see what source changes the developers have been up to.

    Leave a comment