Cyrus' New Completely Useless Blog

sbcl on x86/darwin Lisp

Ok, everything seems to be working now. I'll make a patch later today or tomorrow and hopefully this will hit the tree sometime in the next week. Thanks to Alistair Bridgewater, Juho Snellman, Reaper, Christophe Rhodes, Andreas Fuchs, et al., for remedial x86 assembly instruction, telepathic debugging help and moral support.

SBCL on darwin/x86 (not yet working) Lisp

I've been working on trying to get SBCL running on Darwin/X86. Unfortunately, it doesn't work yet. But if anyone else wants to play along, here's a patch of the latest stuff.

Whoops. First pass at this didn't include the new files. New diff posted.

Currently, it builds and starts up, but dies in !COLD-INIT.

Comments and suggestions greatly appreciated.

Generational Garbage Collector for PPC/SBCL Lisp

Well, it took quite some work, but the generational garbage collector now works on PPC for both MacOS and Linux. The latest and greatest patch can be found here. The patch is to 0.9.9.29, but it should work on any of the very recent CVS versions.

I thought this was basically done a few days ago, but there was a really nasty and hard-to-find bug in the PPC assembly language routines where we were loading a 32-bit constant into a register with LIS and ADDI. The problem is that the ADDI instruction was sign-extending it's argument if the high-bit of the low-word was set. The solution is to use ORI which does not sign-extend. The problem manifested itself in a hosed LRA register, which, thankfully, Christophe Rhodes was able to hunt down.

This was not an easy task, but it was a great way to learn a lot about the internals of SBCL in a hurry. Thanks to everyone who listened to my griping and who chipped in with ideas and debugging help, especially Raymond Toy, Christophe Rhodes, Juho Snellman. And thanks to KingNato and Raymond Toy getting the ball rolling with the initial SBCL work and the CMUCL port, respectively.

Coming to a CVS repository near you soon. Hopefully.

Now, on to locking and threads...

ASDF URIs Lisp

So it turns out ASDF is very useful for packaging all kinds of documents in a form that can be easily distributed. With the help of some code that walks the ASDF system definition, I can automatically make tarball releases with all of the source code and other files, configuration files, shell scripts, documentation, images, etc... and have these be available to the user who downloads this distribution without worrying about where the package gets installed, relative paths, environment variables, etc...

This has been great, but I find myself writing a lot of code that looks like this:

  (let ((images-component
         (asdf:find-component
          (asdf:find-component
           (asdf:find-system "ch-imageio-test") "test")
          "images")))
    (let ((inputfile
           (asdf:component-pathname
            (asdf:find-component
             images-component "sunset-lzw")))
          (imagedir (asdf:component-pathname images-component)))
           ...

Well, it's a lot easier to do something like this:


(ch-asdf::asdf-lookup "asdf:/ch-imageio-test/test/images/sunset-lzw")

This is a URI of scheme asdf with host nil and the path (:absolute "ch-imageio-test" "test" "images" "sunset-lzw"). Notice that this isn't a path in the filesystem, but rather a path in asdf space. The first element of the path (after the :abosulte) corresponds to the asdf system and the other elements correspond to ASDF components so we can find them with the following code:


(defun asdf-lookup (path)
  (cond ((and path (listp path))
         (reduce #'asdf:find-component (cdr path)
                 :initial-value (asdf:find-system (car path))))
        ((stringp path)
         (let ((uri (puri:parse-uri path)))
           (when uri
             (let ((scheme (puri:uri-scheme uri)))
               (when (and (or (null scheme)
                              (eql scheme :asdf))
                          (puri:uri-parsed-path uri))
                 (asdf-lookup (cdr (puri:uri-parsed-path uri))))))))))

Again, note that the elements of the path are not file names, but rather then names of ASDF components from the following asdf file:


(defsystem :ch-imageio-test
  :version "0.1.2-20050724"
  :depends-on (ch-util ch-imageio tiff-ffi)
  :components
  ((:module
    :test
    :components
    ((:ch-imageio-test-cl-source-file "defpackage")
     (:ch-imageio-test-cl-source-file "test-ch-imageio" :depends-on ("defpackage"))
     (:module
      :images
      :components
      ((:tiff-file "euc-tiff" :pathname #p"euc.tiff")
       (:tiff-file "eucgray-tiff" :pathname #p"eucgray.tiff")
       (:jpeg-file "euc-jpeg" :pathname #p"euc.jpeg")
       (:jpeg-file "eucgray-jpeg" :pathname #p"eucgray.jpeg")
       (:tiff-file "sunset-lzw" :pathname #p"sunset-lzw.tiff")
       (:jpeg-file "sanfran" :pathname #p"sanfran.jpeg")
       ))))))

It's a trivial little hack, but it saves me 6 lines of code all over the place.

Cyrus Lisp

CLEM

Work continues on CLEM (Common-Lisp Egregious Matrix), my matrix math package for common-lisp. CLEM now has a bunch of macros which generate type-specific methods for various matrix operations so that I can do fast, non-consing matrix math operations like addition, multiplication, etc... CLEM uses the MOP to define a standard-matrix-class which serves as the meta-class for matrix classes. This allows one to define typed matrix classes as subclasses of matrix, with class attributes stored in the instance of the metaclass. So now we have matrices for {u,s}b-{8,16,32}, fixnum, integer, single-float, double-float, float, real, complex, number, and even t, although this last one is probably suspect.

In addition to the matrix types and the standard matrix operations (add, subtr, scalar multiply, matrix multiply, hadamard product, abs, log, etc...), aggregate operations (min, max, variance, sum), CLEM supports discrete convolution, affine transformation and a couple different types of interpolation, morphological operations (dilate, and erode), and thresholding. Most of these are relatively non-consing, although there are probably a few cases that need to be re-written in the new macro scheme.

Overall, it seems to work, but it is rather slow to compile and results in large fasl files. this shouldn't be too big of a problem, as I'd rather have a fast matrix package that took a while to compile than a slow package that compiles quickly.

Speeds are decent. I approach naive C algorithms and get to within a factor of 10 for highly-optimized matrix multiplication hand-coded in assembly. More benchmarking and further attention to things like the size of the blocks for the blocked matrix multiply would probably be a good thing.

ch-image

Using CLEM, I've developed a trivial image representation/manipulation package. I should probably follow the lisp tradition of calling this trivial-image except that this isn't really trivial, as it requires CLEM. I've thought about trying to abstract away the core matrix stuff from CLEM into a package that works on arrays, and then trivial-image could use that (trivial-matrix?), but I'm getting sidetracked. ch-image supports images of a few types including ub8, rgb8 and argb8. Adding additional types should be straightforward and I hope to work on this soon.

ch-imageio

ch-image is nice, but rather useless if you can't get images into and out of it. This is where ch-imageio comes in. ch-imageio does the conversion between either files in various formats, or from the results of other file reading packages that load the image files into their own format. Currently, ch-imageio supports the following: 1) reading and writing JPEG via the cl-jpeg library, 2) reading and writing TIFF files via libtiff using interfaces defined by gcc-xml-ffi, and 3) writing PNGs using Zach Beane's SALZA library.

gcc-xml-ffi

gcc-xml-ffi generates FFI definitions from C (and C++ code, sort of) using gcc-xml-ffi. Currently, it spits out sb-alien definitions, although the original incarnation did UFFI. In theory, other backends, like CMUCL's alien interface and CFFI should be fairly trivial, but I haven't gotten around to it yet. UFFI isn't really adequate here as it doesn't support callbacks.

ch-asdf

To facilitate generating xml files from gcc-xml and for generating common-lisp FFI declaration files from gcc-xml-ffi, I've made some extensions to asdf. I've also aped the SBCL asdf extensions for dealing with unix-dsos. This is a straightforward thing that everybody seems to do, but the less-obvious part, for me, was how to package them up in such a way that these extensions can be used by other asdf systems. ch-asdf was my attempt at solving this and allows me to add :unix-dso components without having to redefine what a unix-dso is and what is load and compile methods are in every asd file. Also, I can declare a component as a gcc-xml-c-source-file and the right things happen. Dependencies are sort of tracked, but there are a couple places where things break down. In any event, it makes writing asdf systems for packages that use gcc-xml-ffi and unix-dsos much easier.

tiff-ffi

tiff-ffi uses gcc-xml-ffi (and ch-asdf) to wrap the libtiff library. There are also some rather trivial glue functions that help facilitate things a bit. ch-imageio uses this to read and write TIFF files.

carbon-ffi

carbon-ffi allows one, in theory, to develop native Mac OS applications using SBCL. This sort of works and I have some screenshots of simple apps. I've even built "package"-style apps that allow for double-click launching. The negatives are that this is Carbon only (no Cocoa) and that in order to make this work one needs to use some undocumented Apple API functions. This proved to be a huge pain as getting this to work properly exposed some issues with SBCLs stack alignment where we were not properly aligning the stack on a 16-byte boundary, which was making AltiVec rather unhappy.

quicktime-ffi

Similar to the carbon-ffi, this is an FFI wrapper for quicktime. This also suffered from the stack alignment issue, but these are now fixed. Now one can use the QuickTime API from SBCL directly to make and read movies, etc... I havent't tried the GUI stuff (QT movie playing functionality, e.g., but it should work).

congeal

I have implemented a version of the congealing algorithm in common-lisp. This uses clem and the various image stuff to learn a set of transforms that bring a stack of images into registration and can learn the shape of the item represented in the stack of images.

clsr

I have begun to connect SBCL up with R so that I can evaluate R expressions from common-lisp using the R C API and can get the results back to lisp. Next steps are to have a representation of lisp data objects in R and vice versa so that I can, for instance, call lisp functions from R and to connect up the R plotting stuff so that I can make nice plots from SBCL. This is an area where a clean common-lisp API that wraps the R graphing APIs might be a nice thing.

ch-photo

ch-photo is a library based around FFI definitions to the dcraw package to read NEF and DNG files (come to think of it, this should be rolled into ch-imageio, but that hasn't happened yet). In addition, ch-photo contains some scripts for importing RAW files and for organizing them into file system heirarchies based on date of import and file type.

fftw-ffi

fftw-ffi is a wrapper for the fftw (Fastest Fourier Transform in the West) library. In addition to the FFI stuff, there are routines to translate data between ch-image and fftw compatible representations so that one can use fftw to do ffts of images in ch-image.

SBCL stack alignment issues

In order to get carbon-ffi and quicktime-ffi working properly, I had to fix some bugs with stack alignment on ppc. Thanks to Gary Byers for pointing out the bug after I was at wits end with bizarre results coming back from quicktime due to misaligned stack data being munged by altivec (without complaints). These have no been fixed and everything seems to be OK here.

callback fixes

The SBCL developers have been working on adding callbacks to SBCL. The initial ppc port had a number of bugs that caused problems with non-32-bit arguments, long longs, mixing arguments of different sizes, etc... Raymond Toy fixed a bunch of these problems for CMUCL and I ported these over to SBCL. After the initial round of fixes, there were some more problems with how the arguments got pulled off of the stack in the lisp trampoline, but these have been fixed as well. These patches have not yet hit the tree, but hopefully this will happen after the 0.9.8 release.

SBCL sb-alien field alignment issues

Finally, in order to make the carbon-ffi and quicktime-ffi packages work properly, I had to deal with the fact that some of the MacOS toolbox data structures use a bizarre alignment scheme that is a holdover from the m68k days. The bad news is that a lot of these are core structures for things like graphics and IO. In order to support these weird alignments, Apple's hacked up version of GCC has some pragma directives that take care of the alignment issue. Fortunately for me, someone else must have been having similar problems as the CVS versions of gcc-xml started dumping out the offset of structure members. Now that the quasi-compiler was giving me this information, I had to hack up SBCL's alien interface to allow me to use it to specify non-standard alignment of struct elements. This works and now we can properly use these funky MacOS structures.

Ok, that's the overview. One of the main missing pieces is documentation. I need to go back in and document all of this stuff soon. Perhaps that will be my first New Year's Resolution: I well document as much or more code than I write this year.

Happy New Year,

Cyrus

asdf as a Makefile replacement? SBCL

(I've posted this to the cclan list, but I thought I'd put a copy of this here for posterity's sake.) So I'm trying to package up some C code to use as a test for gcc-xml-ffi. I have some C code that I'd like to use to generate a shared library, which will then be loaded by SBCL at runtime. To do this I can 1) hand-code the Makefile rules with the approrpriate platform-specific compiler/linker incantations to get the shared library, 2) use autotools, or 3) try to rig up some sort of asdf-based system to do this. I've done 1 for darwin and got 2 to work, but I wasn't super happy with distributing this giant mess of m4 files just to compile some shared libraries, so I decided to explore 3. Actually, as I was considering this, I decided that I also wanted to use asdf to invoke gccxml to generate the xml for gcc-xml-ffi.

So far so good. SBCL has examples of how to do this in sb-posix (and somewhere else, IIRC this is a violation of OAOO, but let's put that aside for the moment.

But I've run into a couple issues:

  • asdf perform methods specialize on operation and component, but not on the system. This means that if you define a method to compile C files for compile-op and c-source-file it will be applied to all c-source-files. This might not be desired. Clearly one can work around this by subclassing c-source-file, but that seems a bit silly. having a way to provide per-module or per-system operations seems like a good thing. Perhaps there is a way, but I'm missing on obviously good way to do this.
  • cross-module dependencies. I'd like to declare multiple module with subcomponents like module foo with some c-header-files and c-source-files. Then I'd like for components in module bar to depend on, say, one of the c-header-files in module foo. I can't seem to figure out how to do this. The :depends-on arg as it currently exists seems 1) very lisp specific and 2) limited to dependencies within a particular component. I can't, for instance, say that a particular file depends on another system. I can only do system level dependencies for a system. Or at least I can't figure out how to do them. This isn't actually what I want to do, but I think it illustrates an example of the problem. In my case, what I want is for a source file in module bar to depend on a header file in module foo. This works as is in the sense that if a header file in module foo is changed, foo itself is recompiled, but in this case I want to trigger a recmopile of bar as well.
  • Clearly, asdf is an extensible system and through the right extensions I can make this work, but if anyone has any suggestions or comments on how to address these issues, I'd appreciate it.

    LTU on JStatSoft Issue on the demise of Lisp-Stat Lisp

    LTU has a piece on a special issue of JStatSoft on the demise of LispStat. I haven't read through all of the articles yet, but it does seem clear that lisp-stat isn't much of a player these days and that R is the lingua franca of the statistics world. This isssue is timely for me as I have been working in R on and off for the past few months and while I have come to appreciate some of the features of R, I definitely miss many, many nice features of common lisp when using R. S and R are shining examples of Greenspun's Tenth Law in practice. But at least the R guys were smart enough to build R in C by implementing a lisp-like intermediate language. Nevertheless, there are a ton of things in common lisp that are either just starting to make their way into R or are way off on the horizon that the R community will need to grapple with eventually. Then there's the whole issue of completeness, standardization, maturity of the language, etc... But, in any event, it does seem to be the case that R has taken over the, academic at least, statisticl computing landscape.

    But, I think it may be too early to write lisp off for statistics off just yet. I'm just delving into the land of lisp-stat, but it doesn't seem to me that lisp-stat is the end-all, be-all of statistics in common lisp. Just as the R and S guys no doubt learned from lisp-stat, I'm sure there are lessons from R that could be applied to a lisp-based statistical package. Witness matlisp as an example of a lisp-based package inspired by matlab. Speaking of matrices and linear algebra, and granted I need to catch up with the past 15 years of progress in lisp-stat, but it seems that lisp-stat wasn't designed with what I view as modern statistic in mind. When I think of modern statistics I think of big matrices and methods to operate on them efficiently. The main focus of lisp-stat seems to be on dynamic graphing tools and matrices and linear algebra get 5 or 6 pages in the orginal book. I'm sure this may have changed, but in R, as in matlab, the focus seems to be on efficient matrix math from the get go. As for the dynamic graphing stuff, yes, I can see this being nice, but I see this as more of an "add-on" thing than a core feature. A core feature (or a damn important library) of any statistics package, on the other hand, needs to be high-quality print graphics. It's great to be able to spin and zoom and all that, but ultimately scientists want to present their data for publication. R has 1) a good interface to BLAS/LAPACK (with decent notation), 2) great graphing facilities (even if support for png, jpeg, etc... are a bit janky and require either X or ghostscript), 3) a ton of in-depth (if not complete) library packages for doing a little bit of everything.

    I think one of the major things holding lisp-stat back was the lack of good freely available common lisp implementations. The lisp-stat book claims that (at least in 1990) common lisps were expensive and that the free (subset of) common lisp available didn't have a compiler. Obviously things have changed a great deal since then. I haven't been around long enough to know the history well enough to do all the previous implementations justice, but I can say that first CMUCL and later SBCL have done an amazing job of bringing a fantastic common lisp implementation to the masses. All that aside, I still think that lisp-based systems have a deployment problem. If you look at successful lisp based systems, I'd argue that most of them include their own (perhaps half-assed or limited) lisp implementations, emacs and autocad being cases in point. One of the advantages of emacs and R is that they both just build and run out of the box. There isn't this whole build the language, get the libraries, build them, get the application, build it, get the app. libraries, build them, etc... rigamorole. Granted R libraries aren't necessarily the cleanest thing in the world, but the whole R CMD INSTALL thing is easier, for some reason, for new users to grasp than the asdf install process.

    Nevertheless, I'm still hopeful that the emergence of high quality, freely available lisp systems, and the success of at-least-partially-lisp-inspired (at least in the guts) stats packages such as R suggest that there's still hope for doing linear algebra and statistics in lisp. I'm looking forward to trying to do some of what I currently do in R in common lisp in the future.

    more on compact-info-entries-index SBCL

    Now that Xach has set up planet.sbcl.org, I'll move my gripes, helfpful (yeah, right...) comments, opinions, etc... about SBCL to the new SBCL category of my blog and maybe Xach will be kind enough ot pick up the feed.

    As a first entry, let me continue to try to convince the SBCL gurus on #lisp that changing the size of compact-info-entries-index is a good thing.

    As I have mentioned before, I've been trying to generate FFI definitions for the MacOS Carbon APIs and have run into a problem attempting to run purify, which gets called automatically by save-lisp-and-die. (I'm ignoring the fact that without threads and callbacks on the PPC, the carbon integration is going to be lacking but Brian Mastenbrook has been working on callbacks and hopefully I, or better yet, somebody who knows what they're doing, will have a chance to take a look at threads and the required gencgc stuff one of these days. Now back to the original thread).

    The call to purify fails, giving an error saying that 65536 not being an (integer 0 65535) or something similar. Purify (and perhaps other things?) attempts to create a compact environment that contains the same information as an existing environment in a more compact representation. There is a variable compact-info-env-entries-bits that is used to determine the size (in bits) of the compact-info-env-entries-index type, which, in turn, is used as the type of the compact-info-inv/cache-index and as the type of the simple-array compact-info-env/index.

    As an environment gets more than 65535 entries, it becomes impossible to compact it do tue the size of the index into the underlying data structures. So the obvious thing to try is to boost the size of the index. I made compact-info-env-entries-bits 32 and have been using this for a few weeks (on sbcl 0.8.20.xx on PPC) with no problems.

    Enough background. Now for some specifics:

    Why is this necessary? Without this patch one can only have 65535 functions in an environment.

    Here's a testcase that illustrates the problem:

    (defparameter l nil)                                                                                             
    (dotimes (i 65537)                                                                                               
      (setf l (cons (let ((g (gensym)) (z i))                                                                        
                      (setf (symbol-function g) #'(lambda () z)) g) l)))                                             
    (purify)                                                                                                         
    

    And the output is:

    The value 65536 is not of type (UNSIGNED-BYTE 16).                                                               
       [Condition of type TYPE-ERROR]                                                                                
                                                                                                                    

    What are the impacts of the change? The compact-info-entries-index type itself is only used by the index and cache-index fields of the compact-info-env struct. These data structures hold indices into a table of the particular things being stored in this compact-info-env. Note that this change doesn't change the size of the things in the table, just the size of the index and the cached index into the table, AFAICT.

    There is one potential problem. At some point a table-size gets calculated based on the number of names in the environment. We call primify to get a prime number size for the table. primify declares its argument x as an (unsigned-byte x) and then iterates over the odd numbers >= x until it finds a prime number. But positive-prime declares its argument to be a fixnum, so it's possible that we could have a table size that is > most-positive-fixnum which cause positive-primep to fail. I haven't run into this in practice, but we might want to watch it for this and/or come up with a better approach for getting the size of the table. There's a comment about using almost-primify from hash-table.lisp, but this must be an out-of-date comment, as I can't find this function in hash-table.lisp. But failing when we get larger than most-positive-fixnum is much better than failing when we hit 65536 entries in a compact info env.

    I'm getting tired of flogging this horse, but I think it's important that this get fixed. If anyone disagrees with what I've proposed, I'd love to hear it.

    Index: globaldb.lisp
    ===================================================================
    RCS file: /cvsroot/sbcl/sbcl/src/compiler/globaldb.lisp,v
    retrieving revision 1.36
    diff -u -r1.36 globaldb.lisp
    --- globaldb.lisp       12 Jun 2004 13:55:49 -0000      1.36
    +++ globaldb.lisp       26 Mar 2005 15:03:13 -0000
    @@ -476,7 +476,7 @@
     ;;;; compact info environments
    
     ;;; The upper limit on the size of the ENTRIES vector in a COMPACT-INFO-ENV.
    -(def!constant compact-info-env-entries-bits 16)
    +(def!constant compact-info-env-entries-bits 32)
     (deftype compact-info-entries-index () `(unsigned-byte ,compact-info-env-entries-bits))
    
     ;;; the type of the values in COMPACT-INFO-ENTRIES-INFO
    
    meta-ffi Lisp

    So in my continued adventures in ffi-land, I decided it would be an interesting exercise to make a MOP metaclass that provided for transparent access to FFI structs. I took a first stab at this a while back using OpenMCL and it kinda worked. As part of my recent efforts to switch over to SBCL, I decided to pick this up again and see if I could make this work with UFFI. Sure enough, it works! And I've cleaned up the code a bit to create a package called meta-ffi which provides the MOP infrastructure and another package called vimage which uses the GCC-XML stuff and my GCC-XML-UFFI declarations to automatically parse the vImage (a MacOS framework for doing HW accelerated image manipulation) headers, generate an XML file of the appropriate declarations, generate a CL file with UFFI declarations and finally define a CL class which provides for transparent access to underlying structs. It's an awful lot of machinery, but it should be reusable in other contexts and might help write CL code that uses foreign libraries. Clearly this can be done without all of this machinery, but the point was to try to automate as much as possible. Now I can do something like this:

    (defclass vimage-affine-transform ()
      ((a :accessor a :initarg :a :foreign-field '|a|)
       (b :accessor b :initarg :b :foreign-field '|b|)
       (c :accessor c :initarg :c :foreign-field '|c|)
       (d :accessor d :initarg :d :foreign-field '|d|)
       (tx :accessor tx :initarg :tx :foreign-field '|tx|)
       (ty :accessor ty :initarg :ty :foreign-field '|ty|))
      (:metaclass uffi-foreign-struct-class)
      (:foreign-type |vImage_AffineTransform|))
    
     (make-instance 'vimage::vimage-affine-transform
                     :a 1.0 :b 0.0 :c 0.0 :d 1.0 :tx 0.0 :ty 0.0)
    

    The MOP stuff under the covers takes care of the foreign struct allocation and making sure that slot-value and setf (slot-value... do the right thing.

    I think this is kinda cool, but I can't decide if it's really useful or just Brucio-cool. Time to do some more hacking to see if this really helps. I think it could prove useful for the development of Carbon apps in SBCL.

    walking nested data structures Lisp
    Dan Barlow has an interesting piece on walking (reading and writing) nested data structures, motivated by the ease of which such things are done in Perl. Being a good (great?) lisp hacker, dan_b wants to make this kind of thing easy to do in lisp suggests a nice setf expander that works with plists to allow for setting values in nested data structures as follows:
    (setf (ref l :foo :ban :barry) 17)
    
    This is very cool but dan_b is partial to plists and so he is implementation works on property lists. This is fine for small lists (even deeply nested ones, as long as there aren't too many keys for a given hash-table), but once there's a bunch of key/val pairs in one of the hashtables, this will be slow to search this. One of dan_b's motivating factors for using plists is that hash-tables aren't read-printable, which is true, as hash-tables contain things that are difficult to externalize, such as the :key. I'd suggest that to externalize a set of nested hash-tables, one could just make a plist from it as follows:
    (defun hash-table-to-plist (h &aux l)
      (if (hash-table-p h)
          (progn (maphash
                  #'(lambda (key val)
                      (setf l (cons (hash-table-to-plist val)
                                    (cons key l)))) h)
                 (nreverse l))
          h))
    
    And do the inverse of this on the way back. Of course, one has to make some limiting assumptions about the nature of the hash-table, such as the the test has to be equals if we want this to work with sting keys (not just keywords) as most things that could use this sort of machinery would want string keys. Of course the setf-expander has to be rewritten to work with hash-tables, but if one were to do this and pair it with a way to externalize the data as a plist, I think you could get the performance of hash-tables (this is both good and bad as there is most likely a performance penalty for small hash-tables, but a big win for large hashtables) and get at least some externalizability that is no more limited than what you would get with just a plist approach. Think of this as hash-tables under the covers, with the canonical external representation still being plists (with the further caveat that the order will most likely get munged, but who cares, the assumption is that this is for unordered data sets). Here's the hash-ref function, for more of the easy part:
    (defun hash-ref (h &rest keys)                                                                                  
      (reduce #'(lambda (h k) (gethash k h)) keys :initial-value h))                                                
    
    The tricky part is writing the setf-expander. Well, turns out it's not so tricky after all, as dan_b has already done all the hard work for us! Here's the modification of dan_b's setf-exapnder to work for hash-ref:
    (defun %put-hash-ref (new-value h key &rest more-keys)
      ;; not quite Perl-style autovivification, but we do create
     ;; appropriate list structure for intermediate keys that
    can't be found
      (unless (hash-table-p h) (setf h (make-hash-table :test 'equal)))
      (let* ((sub (gethash key h))
             (val (if more-keys
                      (apply #'%put-hash-ref new-value sub more-keys)
                      new-value)))
        (progn (setf (gethash key h) val) h)))
    
    (define-setf-expander hash-ref (place &rest props
                                      &environment env)                                                             
      ;; %put-ref may cons new structure or mutate its argument.
      ;; all this magic is just so that we can
      ;; (let ((l nil)) (setf (ref l :foo :bar) t))
      (multiple-value-bind (temps values stores set get)
          (get-setf-expansion place env)
        (let ((newval (gensym))
              (ptemps (loop for i in props collect (gensym))))
          (values `(,@temps ,@ptemps )
                  `(,@values ,@props )
                  `(,newval)
                  `(let ((,(car stores) (%put-hash-ref ,newval ,get ,@ptemps)))
                     ,set
                     ,newval)
                  `(hash-ref ,get ,@ptemps)))))
    
    And here's an example that combines all of the above:
    CL-USER> (let ((l nil)) (setf (hash-ref l :foo :bar) 47) (hash-table-to-plist l))
    (:FOO (:BAR 47))                                                                     
    
    Previous 1 2 3 4 5 6 7