Common Lisp, LispWeb and the birth of the WebStations

Most Lisp programmers firmly believe that Lisp is the best existing programming language, and they want to share its power with the users of the applications they develop. Lisp systems are usually designed to be highly flexible, extensible, well documented, and tightly integrated with powerful development tools. Neverthelss, after the golden age of the Lisp Machines, Lisp environments have typically been forced to run on UNIX (or even less hospitable) platforms, and have suffered from a difficult coexistence with the host operating system. In a world that is now mostly C- and XWindows-based, Common Lisp still lacks a consistent interface to library procedures, network functions and graphical capabilities.

We believe that the recent dramatic developments in Web-based network technology can open new perspectives concerning the diffusion and the use of Lisp systems. It is already easy to envision a new kind of workstations (that we may call WebStations) that will emerge in the next few years. Their software architectures will be based on an extension of the classical filesystem model, covering the whole Internet network. Rather than being a fancy add-on, network services will be native, built-in tools, fundamental to the normal operation of the computer. User interaction will take place through some evolution of today's Web browsers, with HTML playing the role of the standard presentation language. Everything, from the interaction with applicative programs to system management to the access to system documentation, will share a consistent hypertextual interface, whereby the boundary between the workstation and the rest of the outside networked environment will become more and more blurred.

The first signs of this upcoming trend are already visible in the most advanced HTTP clients. The latest version of Netscape Navigator, for example, integrates HTTP-based hypertextual presentation with traditional network tools (gopher, ftp), communication tools (email, USENET news), documentation (on-line reference manual), and some simple data-bases (bookmarks, address book). However, adding more and more 'intelligence' to the clients will be useless unless it is accompanied by an equal or greater increase in the power of the servers. In this context, future HTTP servers will be required to act not just as simple 'document providers', but as full-featured application providers able to support complex interactions with large, possibly distributed systems. It is clear that the current CGI technology is not well suited for applications that go beyond simple document editing or retrieval tasks: the script invocation mechanism is rather clumsy, and the scripts themselves have very little control over the server environment and the details of the incoming connections. Moreover, traditional HTTP servers are stateless; while this can be irrelevant or even convenient for document-providing servers (including those that use CGI scripts), it is completely unsuited for application servers. With today's servers, even adding a simple access counter requires a non-trivial apparatus of external scripts.

In our view, Common Lisp is the ideal language for the development of a new generation of HTTP-based Application Servers, able to enrich a distributed computing environment and to eventually become the heart of a hypothetical WebStation. The dynamical and extensible nature of Lisp systems makes it possible to seamlessly integrate user-written applications in the overall environment. Thanks to sophisticated compilers, modern Lisp implementations are able to combine high speed and low memory requirements with the obvious advantages of a 'virtually interpreted' language in terms of debugging ease and incremental development capabilities (by the way, it is little wonder that the best language so far for CGI scripts has been Perl, which shares Lisp's interpreted/compiled language duality).

Let's look more in detail at the implementation and functionalities of a Lisp-based HTTP Application Server. Almost the only requisite for its realization is that the Lisp environment be able to open a TCP socket, to accept incoming connections to it and to represent an active connection as a bidirectional character stream. (Today, most Common Lisp implementations provide such features either natively or through a Foreign Function Interface). Once these capabilities are available, the Lisp system is ready to become an HTTP server. One of the main features that make it different from a standard HTTP server is that, although incoming requests are still expressed in the 'pathname-like' syntax of the standard HTTP protocol, the Lisp server does not require the requests to reference an actual disk file or directory. Rather, the server possesses a dispatch function that, by parsing the request string according to an arbitrary syntax, extracts the name of a Lisp function to be called and the arguments to which it must be applied. The function call is then executed and its return value, that will normally be a string representing an HTML page, will be sent back by the server to the client. This means that user-written applications can be 'linked' to the server by providing appropriate hooks to the dispatch function, and that HTML documents can be dynamically generated according to the program output. A Lisp server should also implement a library of HTML-generating functions, that would take advantage of Lisp's powerful string manipulation primitives (e.g., FORMAT) and macro definition capability. In addition, the dynamical nature of the presented document makes it easy to convert them to different formats. For example, a set of HTML pages containing the reference manual of a software program could be turned into a LaTeX file to generate a high-quality hardcopy just by changing the presentation layer.

The second important characteristic of a Lisp server is that it is not stateless: since the Lisp environment is always running, the result of a specific request can be made to depend on the 'interaction history', that is, on the sequence of previuos accesses to the server. For example, it is possible to have the server store a password string to use for access control without having to write it to a file and re-read it every time a new connection is requested.

Finally, it is important to note that a server like the one just described is a full-rights member of the World-Wide Web: other servers can reference its pages, and its pages can contain references to other servers. Only "internal" references are treated specially by the Lisp server.