Use Homebrew. Don't use MacPorts.

The first thing you find out about writing a server app that uses WebSockets is that your server must support eventing, and this usually involves installing some kind of binary library. Windows apparently will not work with many of these systems, unless you make "modifications" (the blog that said this implied dark overtones). I did my experimenting on the Mac, but Linux should also work for the server -- assuming your browser works OK.

For the Mac, I came across instructions that said -- clearly with a grimace -- that I should use MacPorts to perform these binary library installations. I immediately had what is apparently the common experience: it didn't work. I got error messages that indicated that not only was my version of MacPorts out of date, it was so out of date that MacPorts couldn't update it. I had to do some other kind of installation, and I ended up on the Apple developer site with lots of options, all of which cost 99$. I had vague memories of some way to get around this, but nothing precise. And even if I did pony up 99$ for this one thing, it wasn't clear it would even work or if I'd get any other use out of it. The last time I'd used MacPorts was years ago and I remember it as being messy and uncertain.

Fortunately James piped up and pointed me to Homebrew. What a wonderful improvement! Installation was easy and they seem to have thought of everything. Basically we're moving from last century where the attitude was "programmers are smart, they'll figure it out. And if they can't, they don't belong here." The Homebrew attitude is "package management isn't the problem people are trying to solve. Let's make it trivial so they don't get derailed, and they can use their momentum to solve their real problem!" The only remaining problem is that your old MacPorts, along with anything it installed, remains a thorn in the side of your configuration. The Homebrew command brew doctor will analyze your system and tell you where you might encounter problems along with how to fix them. The best situation is if you've never installed MacPorts and you just start with HomeBrew. If you've ever used MacPorts or Fink then that can cause problems. I seem to have avoided these, but now that I've had the Mac long enough to know these kinds of things, I'm tempted -- whenever I upgrade -- to just start with a clean slate and only install things that I know work.

Use Virtualenv

The second lesson I had is to (finally) start using Virtualenv. My friend Barry Hawkins was raving about this at least two years ago but this is the first time I started using it. Basically it creates an isolated configuration which you can load libraries into which will not collide with any global libraries or other Virtualenvs. If you don't like the result, you can easily blow it away. Here's a nice article describing Virtualenv along with some other valuable tools. This article describes Virtualenv in more depth.

Browser Problems

I began the experimentation phase by focusing on a new (to me) web framework called Flask. This was recommended by one of James' friends at Heroku, who said it was "the one to use." I found the design to be very clean and straightforward; everything you do seems to have direct bearing on the needs of your app and I didn't see anything that looked like handwavy magic. Flask makes very good use of decorators, for example.

Aside: years ago, at one of the Python conferences, a speaker pointed out the huge number of web app frameworks and said that we needed to narrow down the choices because they were too confusing. At the time I agreed because I was confused myself and wanted someone to take me be the hand and point out the "right" web framework. However, the more I read about evolution the more I think that Python's proliferation of web frameworks is a good thing. All the experiments allow different approaches to be tried and -- since the open-source environment is much closer to a true free market than anything we've ever seen -- the best ideas are selected and inherited into the next generation of frameworks. In other language ecosystems, the "right" framework has sometimes quashed this process, very often to the detriment of that ecosystem. It took years for competitors to arise for J2EE because Sun's marketing was so compelling, and billions were wasted implementing bad J2EE systems in the meantime. In the Ruby world, the Rails team seems to continue to make good design decisions and evolve the framework, but this relies on the makeup and characteristics of that team which could change; in the meantime it's very hard to compete in the world of Ruby web frameworks even if you might have a better way of doing things.

After many years of experimenting, I've learned to start with the easiest path possible, and that was an excellent approach in this case. I began looking for example code showing WebSockets examples using Flask, and found this one which looked very straightforward. After jumping through the aforementioned library installation hoops, the server application started up without any Python errors, but the browser was not producing the expected results.

I assumed that it had something to do with Flask, or the library installation, and decided that I should use Tornado instead, because it is designed around eventing in order to handle large numbers of users, and WebSockets are based on eventing. Best of all, Tornado already has a built-in library for handling WebSockets -- how could I lose?

Another Aside: One of the most irritating things about documentation for these frameworks is when the authors use code snippets and say "here's some code that shows how it works." Seriously? Are you saving screen real estate or is it just too much trouble to put a complete working example in your documentation? One compelling reason to put complete examples in is that you can then automatically test and verify that they still work when you create new versions of your framework. Perhaps this would be a nice feature for the Sphinx documentation system that everyone in the Python world seems to be using these days: some way to mark an example as executable code, so that Sphinx could automatically extract it into an "examples" folder.

Eventually I found a Tornado example that was nice and simple but it still didn't work.

Before I had a chance to think about giving up, pair programming came to the rescue. James searched and discovered that there was a new bug report in Tornado -- just one day before -- that said that the latest upgrade of the Google Chrome Browser made some kind of change in WebSockets (apparently a correction) that caused Tornado servers not to work with WebSockets.

James suggested I try a different browser. Firefox, even with the latest update, didn't work and that fit with things we've found on the web about Firefox' current status with WebSockets: not ready yet.

But, surprisingly enough, Safari did work. This makes me a bit nervous because there is a new update of Safari wanting to be installed, so I must be careful not to accidentally install it in case it "fixes" whatever issue that Chrome fixed.

And the aforementioned Flask app still didn't work; it communicated with the server but produced an "Internal Server Error."

All this shows how fragile the current ecosystem is for getting WebSocket applications to work. It's very early days so if you want to experiment with it, keep in mind that the reason your app doesn't work can be anything and everything: your framework, libraries or browser. And the only app you can deploy is one where you control the user's browser (thus, an in-house app). Someday -- we can hope soon -- things will get better.

The Application

So, having jumped through all these hoops, here is what I got working. I just wanted something simple that would show a WebSocket in action, so all my application does is push numbers to the client's browser application.

Let me emphasize here that I have only been able to get this to work on a Mac using version 5.1.1 of the Safari browser as a client. I tried Linux but no go (there's no Safari for Linux, and I wasn't able to run the server in my Linux Virtualbox and the client on Safari in the hosting Windows7 -- there may be a way to configure it to cross over the Virtualbox boundary but I don't know what that is).

By the time you read this, Tornado may have been upgraded so that it works with Chrome, and Firefox might also be working.

Let's start with the client HTML, which is located in the main directory and called CountClient.html:

<!doctype html>
<html>
<head>
  <title>WebSocket Client</title>
  <script type="text/javascript" src="/static/jquery.js"></script>
  <script type="text/javascript" src="/static/countClient.js"></script>
</head>
<body align="center">
  <h1>WebSocket Client</h1>
  <h1 id="data"></h1>
</body>
</html>

This will be sent from the Tornado server and will pass through the templating system, but since we haven't included any templating code it will pass through untouched. See the Tornado templating documentation for details.

The h1 tag with the id of data will be used by jquery to display the data that comes from the server through the WebSocket.

There is one issue that stymied me for awhile, and that is the way that Tornado serves static files, in this case jquery.js and countClient.js. You can't just put them in the home directory and assume your server will find them there; you typically put them in a dedicated directory and configure Tornado so it knows where they are, by creating a settings dictionary (which can also include other configurations):

settings = {
    "static_path": os.path.join(os.path.dirname(__file__), "static"),
}

You'll see this code later, inside countServer.py.

countClient.js, included in the above HTML, contains the code that creates and configures the WebSocket. This is generated from the following Coffeescript, a technology I describe in this article:

ws = new WebSocket("ws://localhost:8888/socket");

ws.onopen = ->
  $('body').append('<div>Opened</div>');

ws.onmessage = (event) ->
  $('#data').html('<div>' + event.data + '</div>');

ws.onerror = (event) ->
  $('body').append('<div>Error:' + event + ' ' + '</div>');

ws.onclose = (event) ->
  $('body').append('<div>Close:' + event.reason + '</div>');

$ ->
  $('body').append('<div>Start</div>');

First, we create a WebSocket object that connects back to our server which is running locally at port 8888. In countServer.py you'll see the handler at /socket.

A WebSocket can capture a set of events, and we override the capture methods onopen, onmessage, onerror, and onclose above. The most common event will be onmessage, and here we extract the data and (using jquery) replace the existing information within the data id in the HTML with the data delivered by the message event.

Once you create the WebSocket, it's connected and available, like magic, for as long as you need. Every time the server has new information it wants to display in the client, it can send it through the WebSocket which will generate an onmessage event.

Finally, let's examine the code for the server that uses the Tornado framework:

import tornado.ioloop, tornado.web, tornado.websocket, os.path

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.render("CountClient.html")

class ClientSocket(tornado.websocket.WebSocketHandler):
    def counter(self):
        self.write_message(str(self.count))
        self.count += 1
    def open(self):
        print "Opened: " + str(self)
        self.count = 0
        self.timer = tornado.ioloop.PeriodicCallback(self.counter, 1000)
        self.timer.start()
    def on_close(self):
        print "Closed: " + str(self)
        self.timer.stop()

settings = {
    "static_path": os.path.join(os.path.dirname(__file__), "static"),
}

application = tornado.web.Application([
    (r"/", MainHandler),
    (r"/socket", ClientSocket),
], **settings)

if __name__ == "__main__":
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

This uses the common pattern of mapping URLs to classes. A typical handler, like MainHandler which maps from /, inherits from the basic RequestHandler. All this does is send CountClient.html back to the client browser.

CountClient.html contains the JavaScript code generated by our CoffeeScript that turns around and makes the WebSocket connection back to /socket, which maps to the ClientSocket class. For each client that opens a WebSocket back to the server, a new ClientSocket object is created. The open method creates a new associated count field and starts a PeriodicCallback to call the counter method every second. If you've seen eventing systems such as Twisted, this sort of programming will be familiar to you: you do everything via the main reactor, which in this case is called tornado.ioloop (if you're not familiar with event-based programming, it takes a bit of getting used to and I don't actually know of any good introductions to the topic -- perhaps readers can suggest some in the comments).

Although this example was a bit frustrating to get working -- mostly because the lack of support in browsers -- the thought that someday we'll be able to expect WebSocket support everywhere is quite exciting, as it will significantly empower us as programmers and make our applications much more responsive.