Cyrus' New Completely Useless Blog

Fun with R scoping Lisp

R's default variable scoping behavior is weird, at least coming from the Common Lisp world.

I want the equivalent of:

(defparameter *foo* "moose")

(defun doit ()
  (print *foo*))

(doit)

(let ((*foo* "bogus"))
  (doit))

foo is a dynamically scoped (special) variable in Common Lisp parlance. I can rebind foo in the let form, set its value to something and call a function that expects to see foo and have the called function see the new value of foo while but only while executing code in the dynamic scope of the LET form. Simplifying things a bit, if we call doit from code that is not called (directly or indirectly) from inside the LET form, doit will see the value of foo as "moose", whereas any code called from inside the LET form (or from code that calls it inside the LET block) the value of foo will be "bogus". This may seem confusing at first but becomes quite natural over time and is how I think about variable binding and scope.

R has its own ideas.

foo  <- "moose"

doit  <- function() {
    dput(foo)
}

doit()

local ({
    foo  <- "bogus"
    doit()
})

This gives:

{{markdown.cl|blockquote|bq-0}}
    foo  <- "bogus"
    doit()
})
"moose"

Which is obviously not what I want. Fortunately, R gives you lots of rope. There are probably many ways to achieve dynamic scoping in R, but what I've found is:

doit.dynamic  <- function() {
    dput(evalq(foo, parent.frame()))
}

doit.dynamic()

local ({
    foo  <- "bogus"
    doit.dynamic()
})

And running that gives

{{markdown.cl|blockquote|bq-1}}
"moose"
{{markdown.cl|blockquote|bq-2}}
+     foo  <- "bogus"
+     doit.dynamic()
+ })
"bogus"
>

Is this bad R juju?

Cleaning up after upgrading quicklisp Lisp

After a new quicklisp release comes out, I usually spend a few minutes manually cleaning up ~/quicklisp/dists/quicklisp/software/ by deleting newly out of date source directories.

Turns out there's a (much) easier way:

(map nil #'ql-dist:clean (ql-dist:all-dists))

Does the job for me. Nice. Should have found this a long time ago.

Making new Java classes with ABCL Lisp

Armed Bear Common Lisp (ABCL) is an implementation of Common Lisp that runs on top of the JVM. One nice feature of this is that ABCL supports interoperability with java code so one can use java libraries from Common Lisp code.

One neat feature of ABCL is that you can create java classes directly in Common Lisp, in addition to just defining functions, lisp (CLOS) classes, generic functions, methods, etc...

Let's look at a toy example of creating a new java class:

;; let's define a new class
(defparameter *forty-two-class*
  (java:jnew-runtime-class
   "FortyTwo"
   :interfaces (list "java.lang.Comparable")
   :methods `(("compareTo" ,"java.lang.Integer" (,"java.lang.Integer")
               (lambda (this x)
                 (declare (ignore this))
                 (cond ((> 42 x) -1)
                       ((= 42 x) 0)
                       ((< 42 x) 1)
                       (t (error "bogus!"))))
               :modifiers (:public)))
   :access-flags '(:public :static :final)))

Now, from lisp, we can make a new instance of this java class as follows:

;; make an instance of it
(defparameter *forty-two* (java:jnew *forty-two-class*))

and we can call a method on it like so:

CL-USER> (java:jcall "compareTo" *forty-two* 43)
1

A trivial example, but I'll show some real world-examples of how this is useful in an upcoming post.

SBCL 1.4.7 released Lisp

And the train keeps on rollin'. Another SBCL release is out, SBCL 1.4.7. You can find it here, as usual.

Some of the new goodies in SBCL 1.4.7:

  • The core has shrunk by another ~300 Kb (38,149,792 bytes vs 38,444,800 on x86-64 in SBCL 1.4.6). Well done!

  • Various sb-cover (code coverage contrib module) fixes

  • Various sb-sprof (the statistical code profiler contrib module) fixes

  • Lots of bug fixes

  • Various compiler speedups

  • Better support for stack allocation (dynamic extent) of, e.g., functions passed to higher order functions

  • More simplification/cleanup of the host/target code split

As usual, thanks to Stas Boukarev, Douglas Katzman, Jan Moringen, for countless contirbutions, and to Christophe Rhodes for his ceaseless release engineering efforts.

SBCL 1.4.6 Lisp

Like clockwork, another month, another SBCL release. This time it's SBCL 1.4.6. You can find it here, as usual.

Some of the new goodies in SBCL 1.4.6:

  • The core has shrunk by another ~60 Kb (38,444,800 bytes vs 38,510,336 on x86-64 in SBCL 1.4.5).

  • for immobile code, the assembly routines have gotten smaller and more efficient.

  • (write-sequence vector broadcast-stream) has been optimized.

  • the usual batch of compiler fixes to bugs uncovered by pfdietz's random tester (thanks to Stas Boukarev for whacking these bugs nearly as quickly as Paul's tester finds them).

  • the compiler now handles multi-argument /= (that's the not equals function) better.

  • we now stop the finalizer-thread before calling fork.

  • sb-dynamic-core is now supported on arm64.

  • Douglas Katzman's shrinkwrapped binary creation support been improved.

  • on x86-64 thread-base-tn now uses r12 instead of r13, allowing for a more compact encoding of instructions, and a 96K reduction in size of the lisp image.

  • Jan Moringen has optimized the structures and routines used to deal with the compiler's various storage classes.

  • DEF!CONSTANT has been removed.

  • MAP-STACK_REFERENCES (used by ROOM) is now more efficient and less CONSy.

  • The build order of various bits of compiler functionality has once again been improved.

  • The statistical profile (sb-sprof) has seen a number of improvements.

  • More GC optimizations and cleanups.

  • A precise allocation profiler has been added, although it is not yet advertised as being ready for public use, other than by those who wish to live on the bleeding edge.

  • getaddrinfo is now enabled on x86-64. Apparently my comment about this being broken back in 2007 is no longer operative. That's good.

  • test code has been cleaned up.

  • multiple-value-call has been optimized on x86-64.

  • the compiler has seen some fixes for declaring functions not-inline.

As usual, thanks to Stas Boukarev, Douglas Katzman, Jan Moringen, for countless contirbutions, to Guillaume LE VAILLANT for the the stream patches, and to Christophe Rhodes for his tireless release engineering efforts.

SBCL 1.4.5 Lisp

And now... SBCL 1.4.5 has been released. You can find it here, as usual.

Also as usual, 1.4.5 has lots of new goodies:

  • The core has once again gotten smaller (38,510,336 bytes vs 39,952,344 on x86-64 in SBCL 1.4.4).

  • Foreign calls now have one less register to load.

  • More use of DEFGLOBAL instead of DEFVAR allows for more efficient code in the compiler itself.

  • Lots more compiler bug fixes from stassats that address bugs uncovered by pfdietz's random tester.

  • The heap is now relocatable on platforms that use the Cheney GC.

  • Target compiler error handling code has been moved out of the host compiler.

  • host compiler-specific specialized array code is now restricted to the host compiler.

  • finalzers are now run in a background thread which, presumably, reduces GC latency in some circumstances.

  • lots of x86{-64} and ARM microoptimizations.

  • various set utility enhancements that, presumably, make the compiler faster and use less memory.

  • more judicious use of defconstant-eqx instead of defglobal and defglobal instead of defparameter, as appropriate, allows the compiler to generate more efficient code.

  • The usual batch of GC fixes (race conditions, corner cases, etc...)

  • There's a new editcore feature that allows for the building of "shrinkwrapped" SBCL executables are a single ELF binary file that don't require a separate core file. There were some key bug fixes that make this work for me that were committed post 1.4.5, so expect to hear more about this in the 1.4.6 time frame.

Great work to all of the SBCL developers, particularly Douglas Katzman, Stas Boukarev, and, of course, Christophe Rhodes for keeping the release train rolling, and P. F. Dietz for continuing to find obscure compiler bugs that Stas keeps fixing just as fast as P F. finds them!

SBCL 1.4.4 Released Lisp

SBCL 1.4.4 has been released. You can find it here.

ABCL Error Handling Lisp

There's probably a better way to do this, but I have been having a difficult time trying to, from the lisp side of things, track down the cause of errors signaled from java code.

It turns out that we can use lisp's normal error handling facilities to work with java errors. The following snippet triggers a java NullPointerException and if we just evaluate this in SLIME we don't actually see the java backtrace (or at least I don't see it -- of course it would be nice if there were a way to do so).

(handler-case
    ;; this will throw an NPE
    (java:jstatic-raw "getenv" "java.lang.System" nil)
  (error (e)
    ;; this prints the stack trace to the jvm's standard out, which
    ;; when running under slime, is our *inferior-lisp* buffer.
    (print (#"printStackTrace" (java:java-exception-cause e)))))

But this isn't so great as the stack trace is printed to the inferior-lisp buffer. To see it in SLIME's output buffers, we can use ABCL's getMessage routine as follows:

(handler-case
    ;; this will throw an NPE
    (java:jstatic-raw "getenv" "java.lang.System" nil)
  (error (e)
    ;; this prints the exception type and the stack trace to SLIME's STDOUT
    (print (#"getMessage" e))))

Having this certainly makes it easier to find the source of errors in java code called from ABCL.

Common Lisp and Java Lisp

Tales of Woe

So... in an attempt to use preexisting wheels, rather than reinvent my own at every turn, I've been trying to get a decent Common Lisp environment working with the CDK (Chemistry Development Kit). My abcl-cdk adventures actually went reasonably well and I was able, eventually, to get ABCL talking nicely to CDK. Of course I wanted more than just that, I wanted interoperability between the CDK and my half-round wheel, chemicl, a cheminformatics package I started writing in Common Lisp. This is where the train began to fall of the tracks.

ABCL and cxml-stp

A while back, in an earlier, aborted attempt to get some of my chem/bioinformatics(https://github.com/slyrus/cl-bio) stuff working with ABCL I noticed that plexippus-xpath couldn't be loaded into ABCL. This was fixed, so I was encouraged that things might work with ABCL. (While I'm on a rant, the ABCL trac issue tracker is really slow...). However, cxml-stp seems to break ABCL.

Hopefully this is a fixable bug and some future version of ABCL will work with cxml-stp.

In the meantime...

SBCL and Java

So, I figured I'd try some other approaches to getting Java and a Common Lisp implementation to play nice. I know, you're thinking "why doesn't the dude just use clojure? After all, that's what clojure was designed for!" Well, that's a good question. I did use clojure for some earlier explorations with CDK and, while the java integration generally works well, I have a bunch of existing Common Lisp code I'd like to use and, at the time at least, it seemed like all of the clojure wrappers where thin wrappers around ugly Java libraries. I've grown to know and love many Common Lisp libraries, many of which are nicely available in QuickLisp, and I'd like to be able to use those (things like cxml-stp, plexippus-xpath, opticl, etc...).

Anyway, I tried to get some sort of SBCL Java interoperability working. Three possibilities appeared: 1) jfli, 2) foil and 3) cl+j. Turns out jfli is (was?) Rich Hickey's pre-clojure Common Lisp. I'm guessing that the challenges in getting jfli to work with any of reasonably Common Lisp implementations was part of the motivation behind clojure. In any event, it doesn't seem that jfli works under SBCL.

Next, I looked at foil, which appears to use sockets to communicate to another process running a JVM. This sounded suboptimal but, presumably, workable. Turns out foil looks like some sort of windows-only beast with a bunch of C# files. Not for me.

Finally, I looked at cl+j and it turns out there are some scary warning messages about how cl+j can't possibly work with SBCL's foreign threads handling mechanism. Bummer. This seems somewhat unreasonable on SBCL's part. Surely some amount of engineering should make it possible to have both a JVM and SBCLs runtime running in the same process. Unfortunately, I'm too out of practice with SBCL internals to give this much of a go at this point. Bummer again.

CCL and Java

Ok, next approach. How about cl+j and Clozure Common Lisp (CCL)? Seemed reasonable, but, unfortunately, hung just like SBCL did. Presumably this is more of a MacOS issue than a CCL issue, as cl+j is supposed to work with CCL, but maybe just on other non-mac platforms.

Now what?

So, it seems I'm stuck without a viable approach to using the common lisp libraries I want and the java libraries I want in the same process. Perhaps the ABCL bug will get fixed. Perhaps JVM integration would make a good summer project for the next SBCL Summer of Code.

More fun with CDK and ABCL Lisp

ticagrelor

The drug ticagrelor (marketed as Brilinta by AstraZeneca) is an inhibitor of platelet activation and aggregation that has been shown to reduce the frequency of cardiovascular events in patients with acute coronary syndrome.

The CHEBI page for ticagrelor tells us that the SMILES for ticagrelor is:

CCCSc1nc(N[C@@H]2C[C@H]2c2ccc(F)c(F)c2)c2nnn([C@@H]3C<a href="OCCO">C@H</a><a href="O">C@@H</a>[C@H]3O)c2n1

So we can read that in as follows:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (asdf:load-system 'abcl-cdk))

(cl:defpackage :ticagrelor
  (:use :common-lisp :abcl-cdk))

(cl:in-package :ticagrelor)

(defparameter *ticagrelor*
  (read-smiles-string
   "CCCSC1=NC2=C(C(=N1)N[C@@H]3C[C@H]3C4=CC(=C(C=C4)F)F)N=NN2[C@@H]5C[C@@H](<a href="[C@H]5O">C@H</a>O)OCCO"))

And we can render a 2-d depiction as follows:

(mol-to-svg *ticagrelor* "ticagrelor.svg")
CL-USER> (in-package :ticagrelor)
#<PACKAGE TICAGRELOR>
TICAGRELOR> (mol-to-svg *ticagrelor* "ticagrelor.svg")
"ticagrelor.svg"

ticagrelor SVG

Notice the 6 nice chiral bonds. This is all well and good, but let's jazz things a bit by rendering the molecule on a black bacground with white bonds:

(let ((*background-color* (java:jfield "java.awt.Color" "black"))
      (*default-bond-color* (java:jfield "java.awt.Color" "white")))
  (mol-to-svg *ticagrelor* "ticagrelor-inverted.svg"))

ticagrelor inverted SVG

There, now we have a nice pretty picture of ticagrelor. Thanks CDK!

1 2 3 4 5 Next