P16: Practical Progress

"disco", a new map-reduce framework - in erlang+python

Woo! Another alternative to Hadoop crawls out of Nokia Labs:

Disco - discoproject.org

It's yet another Map-Reduce framework. This time the core is written in Erlang, and tasks are implemented in Python. Data transfer between nodes happens simply via HTTP.

There's no DFS implemented at all yet. It seems intended to run on Amazon EC2, where you'd just use S3 as your data store.

This feels like it might be a good fit for our needs, since we're a Python shop, and I'd rather hack on Erlang than Java. :)

Reply to Viktor, re "Tokyo Cabinet + Tyrant"

Viktor Sovietov left an interesting comment on my "More scalable datastores" post the other day:

Why do not try Tokyo Cabinet + Tyrant as per node storage? In combination with Scalaris and tcerl it could work quite well, I think.
Also, using distributed filesystem can be an option as well. I even prefer this approach, because it gives much better performance.

Well, I am currently using a distributed filesystem: MogileFS. The best thing about Mogile is that it's simple and it scales. But it stores metadata in MySQL so this is ultimately a bottleneck. Also, in Mogile, each record is just a separate file on a filesystem; fetching one record requires (1-2) HTTP connections.

Right now I'm using Mogile to store images (photos). This fits nicely with the one-record = one-file paradigm. But I have another dataset which is ~95 GB and growing, append-only; record size is typically 1 kB. If I store these directly as individual files on a large disk, I'm going to get killed by the block size in most filesystems.

(I presently have 6x 700 GB data drives/box times 4 boxes, using ext3, block/fragment size is 4096. So I'd waste 75% of my disk if I'm writing 1kB files.)

Tokyo Cabinet is an interesting suggestion. At first glance, it seems an interesting alternative to BerkeleyDB -- promising performance close to cdb . Tyrant looks like a straightforward network interface to a DB instance, too.

I'm sure it's possible to weave these together with tcerl and make a new persistent backend for Scalaris. But would it be a net win? I am not interested in forking Scalaris; I'd rather see the Scalaris folks express an interest in developing a persistent backend, and giving the nod to this as a potential solution.

To me, an open-source project is only useful if, three years after I've launched a system, there is still a community of people talking about that project and available to help support it. I want to launch a system, then hand it off to someone else for the maintenance phase. If the project doesn't have "legs", then I will the only person who can answer those support calls down the road, and I'd like to avoid that. :)

gcode: Building blocks for Erlang/OTP systems using distributed algorithms

distributerl: Building blocks for Erlang/OTP systems using distributed algorithms.

Hosted on Google Code. Labels: "erlang, distributed, algorithms, vectorclock, consistenthashing, merkletrees, gossip"

Spiffy, will have to keep an eye on this!

More scalable datastores: Scalaris, nmdb

Scalaris is a new distributed key-value datastore, recently announced and code posted to Google Code.

It was announced and demoed at Erlang eXchange 2008. Joe Armstrong (father of Erlang) later wrote on his blog: "my gut feeling is that what Alexander Reinefeld showed us will be the first killer application in Erlang"

Armstrong's summary:

1. They make a peer to peer system based on the chord algorithm
2. They added a replication later using the paxos algorithm
3. They added a transaction layer
4. The injected the wikipedia

5. It went faster that the existing wikipedia

   "Applied to Wikipedia, Scalaris serves 2,500 transactions per second with just 16 CPUs, which is better than the public Wikipedia."

One downside: it's presently a memory-only store, so it's quite useless for permanent data storage. (One full power-outage in a data center will obliterate all of your data. Doh!)

nmdb is yet another distributed key-value store, this one implemented in ~5000 lines of C and using qdbm or berkeley db as the back-end store. It looks simple and stable. Major limitations: it's distributed, not replicated, so is more like a persistent memcache (like Tugela and memcachedb). There is also a hard 64kB size limit on key+value packets.

As you might have guessed from my articles on this topic -- I am looking a "Bigtable-like" datastore that I can recommend to clients. My criteria are:

• It must be reasonably mature: losing data isn't an option!
• It must be open-source, and the project must "have legs" (not abandoned)
• It must be fast enough to serve as the primary datastore behind a web service API
• It must store a few terabytes of data on < 5 machines (not in RAM!), and be able to grow capacity by just adding more machines

I still haven't found anything I'd recommend. Dang it guys, finish one of these projects! :) Maybe I'll have to build something custom on top of MogileFS from scratch after all?

Shockingly cool developments in rich web interfaces

Wow, this gives me warm fuzzies, and some hope that the Internet will do the "right thing" after all -- even if it exhausts all other possibilities first. (Thank you Winston Churchill)

In this case, I'm talking about making closed-technology "rich web interfaces" like Flash and Silverlight a thing of the past. The ability to click "View Source" in a web browser was one of the biggest reasons the web grew quickly, because any curious dabbler could see how a visual trick was achieved -- and then replicate it.

Awesome developments along this front:

1. From the Facebook Engineering Blog - New Facebook Chat Feature Scales to 70 Million Users Using Erlang and Comet. That's a lot of users! Nice validation of Erlang as a scalable back-end for rich web UIs,

2. Processing.js from John Reisig is a port of the Processing visualization language to JavaScript, using the Canvas element. Check out the demos -- I'm stunned to see this kind of smooth animation being created with nothing more than JavaScript and standard HTML components. Wow!

Why do these matter?

1. Adobe has argued that if you want to build rich dynamic web UIs, you need Flash plus BlazeDS or LiveCycle if you want it to scale. Facebook's announcement blows this argument out of the water.

2. Flash doesn't work on the iPhone. But the Processing.js demos will -- and every other device that has a WebKit or Mozilla-based browser, modulo an update or two.

Soon you'll be able to build a rich Flash-like interface that works on... iPhone, Google Android phones, Nokia Nseries, Asus EeePC and Splashtop devices. And all without those companies paying a tax on every device for Flash or Silverlight runtimes!

This means a better platform for developers and cheaper, better gizmos for consumers. Hooray!

PS: Tie these goodies together and my Remote <canvas> Protocol proposal from January is looking pretty feasible. :)

Erlang Studio: Another blogger's impressions

Kevin of hypotheticalabs.com recently attended the Erlang Studio session in Chicago, taught by Joe Armstrong. Lucky for us, he blogged day-by-day a summary of the class!

• Erlang Studio: Day 1
• Erlang Studio: Day 2
• Erlang Studio: Day 3

There are lots of interesting tidbits here... it must have been fun to observe Joe in action. :-) Below is one particularly juicy morsel from Day 2:

Joe made several points during the lecture segments which struck me as totally obvious yet totally unexpected. I can’t wait to trot some of these out the next time I’m discussing the merits of Erlang with my non-enlightened colleagues."

• Shared resource concurrency evolved before networked computers — All this stuff about locks and synchronization comes from the pre-networked phase of computer history. It’s one of the reasons why concurrent and distributed computing in languages with these primitives is so difficult.
• What if the OOP part of other languages (Java, C++, Ruby, etc) had the same behavior as their concurrency support? What if you were limited to only creating 500 objects total for an application because any more than that would make the app unstable and almost certainly crash it in hard to debug ways? What if these objects behaved differently on different platforms? — This is a stunning (at least to me) point. Concurrency is only going to become more important in the future. Why do we put up with such a sorry state of concurrency support in existing languages?

Erlang: Update on "Little Schemer", ways to improve documentation

For those of you tuning in for Erlang news...

I'm still slowly posting updates to my "Little Schemer" in Erlang code. It's roughly complete, save for Chapter 10, which implements an interpreter for a Scheme-like language. I've also added numerous great enhancements suggested by readers like you. :-) If I missed anything, please let me know!

I also got a very nice personal email from Joe Armstrong, who encouraged me to think bigger and turn this into a real book, not just a companion to the original text. He suggested that I contact the original publisher and see if they'd be interested in a rewrite. While this is very gratifying to hear :-) – and I may indeed contact MIT Press – I'm somewhat more interested in self-publishing.

Mark Pilgrim's approach for "Dive Into Python" would probably be my model. He wrote it in DocBook XML, published to the web, solicited lots of feedback... and wound up with an acclaimed introductory text, translated into multiple languages and published by Apress in a dead-trees version. So not only is he now a bona-fide author, it truly helps grow the Python community worldwide, and he never has to worry about his work gathering dust due to an apathetic copyright owner – it's GPL'd!

Joel On Software would be another famous example of self-publishing done right.

(Contrast with Armstrong's earlier book, "Concurrent Programming in Erlang", now out of print. Only the first half of the book is downloadable. Prentice Hall still owns the copyright.)

I've also been causing a small stink on erlang-questions. Stemming from the above thinking, I wanted to find out how the online Erlang manual gets assembled and where the source-code lives. Sadly, the answer is: it's not public yet.

I think the Erlang team keeps everything in a ClearCase repository – which they're surely loath to discard, because they've used it forever. But ClearCase simply can't be exposed as a "public" repository in the way that CVS/SVN/git can. Sadly, until they cross this divide, the process will be more cathedral than bazaar.

User-added comments in the Erlang documentation?

Here's a second idea for improving the Erlang documentation, stemming again from a comparison to PHP.

How about supporting user-added comments in the documentation?

I'm talking about the little "add a note" button at the end of every page of the PHP manual, and the stream of user-contributed "notes" that follows. Or as a friend suggested, look at PostgreSQL: their online manual is available with and without comments.

Allowing comments is just as necessary for online technical manuals as for blogs. It's all about putting Linus's Law into practice: "given enough eyeballs, all bugs are shallow".

Personally, I've spent a lot of time reading the PHP manual in the last 4 years. Even though the quality of the user-added notes varies greatly, I've always found the note-sections helpful, because I never read them unless my question isn't answered by the preceding manual page.

You can think of inline notes as a way to accumulate FAQs (and answers) wherever they're needed within the official manual.

My suggestion for implementing: don't bother to build a comment-handling framework, just borrow some existing framework. Not sure about PostgreSQL, but PHP has a documentation HOWTO. Docbook XML is their core format.

What's Erlang's core doc format? Could we just convert and pour Erlang docs into the PHP doc framework? It looks like DocBuilder is how the HTML docs are generated... will have to dig into this later. :)

Erlang documentation poorly SEO'd?

In the brief time that I've been watching the Erlang world, I've noticed several cases of people saying:

• I have a question about X, which should be covered in the Erlang documentation
• I searched Google, but it didn't return anything
• I found the answer in the manual later, or someone pointed out the location.

The only example I have handy is Russell King's post on Jan 22, where he says, "For whatever reason, google search didn't give me anything."

In my experience, when a search on an obviously-distinctive term fails to yield an obviously-canonical website, the problem is usually that the website is poorly search-engine optimized.

Years ago, I worked at HotJobs.com. When I started, if you went to Google and typed "hotjobs", the results showed our homepage, our discussion boards... and nothing else. Why? Because the most valuable part of our website – the job listings – was not accessible by clicking links. Google couldn't crawl our most-valuable content, and thus it didn't appear in Google.

We eventually, collectively, came to our senses and fixed this bug. Shortly thereafter, we started getting a lot more traffic courtesy of Google, and everyone wondered why we hadn't done this sooner.

I suspect that something similar may be true for the Erlang HTML documentation. It's not hidden behind a search form, but it may be poorly browsable by a web crawler: frames are involved, links seem deeply-nested, possibly lacking helpful keywords. Long documents also tend to be penalized by search engines. Maybe something else is amiss.

Consider for a moment the PHP documentation, which has one page per function call. Choppy, but quickly browsable, with short and descriptive URLs. Search for "print_r" on Google and you get the PHP manual as result #1.

Searching for a similarly-obvious keyword in Erlang yields strange results. "io:format" returns www.erlang.org/doc/man/io.html as the top result, but "erlang io:format" drops it to 2nd place. Wierd!

That's as far as I've investigated. Google's Webmaster Guidelines are a good starting point for further investigation...

Erlang version of "The Little Schemer"

I am learning Erlang, because I want to build web applications that scale easily. Before I can capitalize on Erlang's awesome concurrency support, I need to get comfortable with simpler stuff, like procedural programming.

Erlang at its core is a functional language. As I've noted previously, this means doing things quite differently than in C, Java, or PHP. Recursion is the key building block in functional languages. Thinking in terms of recursive functions is essential for getting anywhere in these languages.

I learned the basics of recursion in college, but it's not a tool I'd instinctively reach for. My attitude was, "Why bother?" Using recursion in most languages just makes the code harder to read, and purely-functional languages like Haskell are for writing PhD dissertations, not getting real work done.

Or so it seemed. Now I'm learning Erlang, and I want to get real work done.

"The Little Schemer", by Friedman and Felleisen has a reputation as a short, sweet book that teaches you to think recursively. That's quite an accomplishment for an approachable book. First published in 1974 as "The Little LISPer", it now uses Scheme, a modern variant of LISP.

I've seen this book translated into other languages before, notably Douglas Crockford's The Little JavaScripter. Since Erlang is a natively functional language, the rewrite should be even easier. Nobody else appears to have published such a thing.

So, here's my take. Call it "The Little Erlanger", if you will. :-)

Google Code: little.erl

I'm including unit tests, to prove that the code works. I also try to use "Good Erlang Style", as best I understand it. Sometimes Erlang offers a shortcut to the recursion-based approach, and I point to these in comments as "Erlang Preferred".

Currently complete through Chapter 7, will finish the rest ASAP.

PS: Erlang fans, I'm sure there is much room for improvement in my style. Any and all comments are welcome!