P16: Practical Progress

Quick demo of Python + WSGI + Nginx

At Daylife, we're running a whole lotta Python code inside Apache using mod_python. This works great so long as each request can be processed quickly, and you don't need your Python processes to keep much of anything in RAM.

However, any real web app inevitably needs to break those rules, at which point you start to venture into a world of increasing pain. Thus arises a need for alternatives to our cozy, familiar, tried-and-true mod_python.

I've set up Lighttpd before, but this time I decided to try Nginx. (It's pronounced "engine x")

Here's a quick demo recipe for running two standalone Python FastCGI+WSGI servers that listen on Unix domain sockets, and configuring Nginx to load-balance between them. This config allows you to run more than one Python VM, but not need N VMs to handle N concurrent requests. It also allows you to restart VMs individually with no service downtime.

"disco", a new map-reduce framework - in erlang+python

Woo! Another alternative to Hadoop crawls out of Nokia Labs:

Disco - discoproject.org

It's yet another Map-Reduce framework. This time the core is written in Erlang, and tasks are implemented in Python. Data transfer between nodes happens simply via HTTP.

There's no DFS implemented at all yet. It seems intended to run on Amazon EC2, where you'd just use S3 as your data store.

This feels like it might be a good fit for our needs, since we're a Python shop, and I'd rather hack on Erlang than Java. :)

Supercomputing for the masses: Nvidia CUDA

This is a post I should have written two weeks ago, but I've been lazy. (Or busy...)

I went to a seminar by Nvidia about CUDA, which is a framework for writing apps to run directly on an Nvidia GPU. Not just graphics apps, either. Nvidia GPUs are massively parallel -- up to 128 hardware threads -- and you can write arbitrary C code to run on it.

The seminar was part of a financial technology conference, so they had a third-party app provider which naturally was building solutions on CUDA. They said they were seeing a 50x speedup (one Nvidia GPU versus one Intel CPU core) when running their compute-heavy applications.

I came home and immediately installed their CUDA SDK for Mac OS X on my MacBook Pro. To my surprise, it ran with minimal hassle (*), and I was promptly playing with particle field and liquid simulations. But my Mac doesn't have enough VRAM to run some of the more intensive demos. (MonteCarlo, BlackScholes)

(*) The only real tweak necessary was: "export DYLD_LIBRARY_PATH=/usr/local/cuda/lib"

Takeaways:

• It's a powerful parallel-programming model, that generates code for the GPU or the host CPU
• You can start developing with commodity Nvidia GPUs, even in laptops.
• You can scale up to higher-end solutions - up to 4 128-thread GPUs per "host" system

One thing CUDA doesn't provide is a way to manage and process massive datasets. The cards have somewhat limited memory, and you have to write an app that runs on the host and feeds the card with data. For search applications like topic clustering -- which I'd like to use this for -- CUDA alone doesn't provide an answer.

Perhaps it would make sense eventually to use Hadoop plus CUDA -- write your map/reduce tasks in CUDA, and rely on Hadoop to distribute data around a cluster of Nvidia-accelerated boxes?

Two spiffy related projects:

• pycuda Python wrapper for Nvidia CUDA
• Mac OS X 10.6 "Snow Leopard" is slated to ship with OpenCL, roughly a version of CUDA that works on multiple GPU architectures

Special thanks to Rich Hecker for posting a heads-up about the CUDA seminar to the NY-newtech list.

AppEngine: Installing PIL on OS X 10.5.3

The just-released 1.1.0 API for Google App Engine contains a new "Imaging" library. Hooray! It works by providing stubs, which use Python Imaging Library (PIL) in development, and some-other-machine-behind-the-curtain when you deploy to Google.

So, if you're developing for App Engine, you now need PIL on your development system too. Google provides a Installing PIL document. Unfortunately, their directions re: OS X are not quite right:

• "Download the Mac installer from the http://pythonmac.org."
• "Install the package"
• "Finish the installation ..."

Alas, that package will refuse to install if you are using the original Python interpreter that ships with Mac OS X.

After hitting this obstacle and wasting a half hour digging through obsolete HOWTOs, I found this nugget in the pythonmac.org Leopard page:

4. In Terminal, create a symbolic link for the system's Python.framework in the location expected by the pythonmac.org tools, like so:

cd /Library/Frameworks
sudo ln -s /System/Library/Frameworks/Python.framework/ Python.framework

Yes, you can skip steps 1-3 and jump directly to 4 if you just want to PIL, not IDLE or anything else. Now run the PIL installer. It should work!

You also need to add the PIL install location to PYTHONPATH. Stick the following in ~/.bashrc.

export PYTHONPATH=/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/

Now open a new terminal window, and type:

python -c "from PIL import Image"

If you don't see an error message, you're done. :)