Chatterbox, Reloaded

This has been the week of major KARL updates.  KARL is the open source collaboration and knowledge system published by the Open Society Institute.  It is an end-user product atop the BFG web framework.

Some recent happenings in KARL-land:

• We are wrapping up a major improvement to the calendar tool. We introduced a concept of sub-calendar “layers” that can aggregate events from other communities.  More visibly, we completely re-did the UI with a new Weekly view and lots of Ajax sprinkling.
• We also did most of the work to re-implement the entire form system using io.formish.  (repoze.bfg.formish to be more precise.)  Big reduction in code and test code.  Still have a lot to think about on form controller patterns.  Those going to PyCon can hear Chris’s recounting of form controller torture in a panel he’s on about form frameworks.
• Along with our friends at Six Feet Up as hosting partners, the KARL team is now operating KARL sites for five organizations.  KARL has a unique approach to customization, the inverse of the traditional Zope approach.  In KARL, a customization package is the starting point, and that pulls in the main software.  Or, doesn’t.  We just rolled out changes to thin out the size of each customization package.
• Chris Rossi has been working on a web interfaces to a number of admin activities, including integration with Six Feet Up’s Zenoss monitoring.
• KARL has a number of periodic admin jobs, for things such as processing incoming/outgoing mail, pulling in feed content, etc.  To date we had been running these as cron jobs.  However, we had cases where hundreds of crons got piled up.  In the latest updates, we converted these to Supervisor-managed jobs.  Risk involved, so we’ll see how it goes.
• And finally, we are going to work with Six Feet Up to install solid-state disks.  Our initial test shows that an SSD alone, with no code refactoring, will completely eliminate our performance concerns on LiveSearch.  As well as benefit other catalog-constrained screens, possibly.  We’ll report back once we live with them for a while.

All in all, KARL (like BFG) is humming along nicely.  Just to emphasize: KARL isn’t a framework.  It is an out-of-the-box product with a strong opinion.  By making such a deliberate choice to not boil multiple oceans, KARL gets to be very compact, very fast, and very stable.

That’s the key takeaway for people working on larger projects that have dynamic performance needs.  Unless your needs fit into Product X’s bulls-eye, you’re probably better off not beating Product X into submission.  Instead, we need an approach where assembling your own custom application, leaving out the parts you don’t need, is more feasible.

Not only is this a win for custom apps, IMO it’s actually a win for Product X.  Instead of having reputation beatdown when it doesn’t excel at Every Possible Thing, it can just say: “We’re good at X. If you want X, you want us.”  Then, focus scarce resources and reputation on being the best possible X.

I’ve enjoyed seeing some writeups on requests/second and memory usage for upcoming versions of Plone.  It’s great to see things trending in that direction.  Hopefully with some tough choices and deprecation, more gains can be made (just my personal opinion.)

I thought I’d give a primitive try at the same numbers for KARL, the collaboration application atop BFG that we’ve been working on and deploying to customers.

Using the ‘ab -n 100 -c 2’ on my first gen MacBook 2 GHz, 2 Gb of RAM, I leveled off at just over 134 requests per second.  Memory usage was 31 Mb.

Obviously it’s not an apples-apples comparison.  The feature set is smaller.  Although we do have cataloging, text search, workflow, security, and the like, there’s a ton of stuff we don’t do.  We’re an end-user application with specific features, versus a framework.

On the other hand, all requests in KARL are authenticated and fully-dynamic.  So the 137 rps above?  That’s our slow number: authenticated, personalized, security-aware, fully dynamic.

For more fun, we recently built an ugly, cheap Core i5 box in the Agendaless office for $600, with 4 Gb of RAM.  In production we deploy under modwsgi, so we fired it up to have 3 processes (for 3 of the four cores).  We also have a script that lets us bulk load 300 sample communities, each containing a bunch of content.

That’s a bit more realistic of a test, since we start paying the price of having content in the catalog.

In that “with content” test, we got 349 requests/second.

Sometime soon we’re going to think a bit harder about a more realistic test.  Pounding the same URL over and over as the same user just doesn’t mean squat.  Well, it’s valuable in so much as it is a veto: if your numbers are pathetically low on the fastest-possible “test”, it’s only going to get worse.  We are slowing building up some Funkload scripts that cover a scenario which includes different users, different activities, and some writes as well as reads.

We need this as we are evaluating various KARL ideas in 2010.  First and foremost, we bought a solid-state disk for the test box.  We had a query (prefix match on text search, where only one letter was entered) which blew up our system previously.  Think, 60+ seconds.  That time fell down to 2 with the SSD.

Next, we’d like to see some before/after on RelStorage using some real-world scenarios.  Finally, I’d like to see some before/after on repoze.pgtextindex, where we swap out just one of our catalog index types (the text one) with transactional text indexing in Postgresql.

Last week at the Plone Symposium East I gave a talk on the KARL project that I’ve been working on.  The basic meme: Plone and its large ecosystem provide a ton of value when your needs match up with its bulls-eye.  What should one do when your needs don’t fit so well into Plone-the-product’s box?

My talking point was, we need to discourage expanding Plone’s bulls-eye to cover generic platform development of any possible application.  Instead, encourage the meme that the technologies (and effort expended learning them) can be used to make a targeted product.

The KARL project adopted that thinking in its switch to BFG (to good effect, as we then focused on building the best KARL we could.)  In describing BFG’s goals, I lifted one directly from Chris:

Documentation: The lack of formal documentation of a feature or API is a bug.

I then went on to explain that Chris released the documentation for BFG before releasing the software, and has made an enormous, constant effort at keeping the wide-ranging docs (API, narrative, example applications) up-to-date as he has refactored.

In making the point, I posited that “Friendly, ample docs make a positive first impression” is part of the reason for swift uptake of Django and other Python web frameworks.  Chris pointed me to a survey that makes that point in spades.

Further confirmation came during the BFG tutorial Tres and I did last week.  Eric Rose clicked the link to the BFG docs in the comprehensive BFG Wiki tutorial and had a very visible positive reaction on his face.

For the last few years I’ve been working with some great folks at the Open Society Institute on a project called KARL.  It’s now open source and has a website with some preliminary information, which means I can chat about it in advance of my presentation next week at the Plone Symposium.

In a nutshell, KARL is a collaboration system for projects and organizations.  We are just wrapping up KARL3 (a rewrite to convert from Zope/Plone to Zope-like BFG application) and we’re doing the migration work.    There’s quite a bit to chat about, so look for some more blog posts as we finish up the process.