The perfect has become the enemy of the sufficient

Posted by Elf Sternberg as Uncategorized

I can’t tell if I’m suffering from the geek equivalent of writer’s block, analysis paralysis, overload, or some combination of all of the above.

I’ve been working on a generalized platform for narrator, the story engine for the Journal Entries and everything else. As I’m working my way through the process, I’m thinking in typical RDBMS fashion, “Okay, the system has Authors and an Author has Serials and a Serial may have more Serials which eventually leads to having Stories.” (To make the whole “Serials” thing clearer: A universe has novels; novels have chapters. “Universe” and “Novel” are synonyms for Serial, and “Chapter” is a synonym, in this system, for “story”– an atomic block of text readable in one sitting. Theoretically.) I have a tree-based heirarchy for Serials working, although I’m damnably unhappy with where in the system the knowledge for “An author may have zero or more serials” is stored.

The actual architecture is sorta backwards: Each story has a foreign key to a serial, and a serial may have a foreign key to a parent, and ultimately all serials have a foreign key to a user, and so forth; the usual one-to-many relationships built within SQL. The nice thing about Django is that it notices these things and builds the backing querysets automagically, so you can traverse either way along this structure without having to write your own code.

And I pretty much have all that put together. I have some performance worries, and have seriously contemplated what this would work like under Mongo, but so far, it seems to be fine.

Time to upload my stories to the system, right? Four “Universes,” one merely labeled “Other Stories.” And I’m thinking, well, I could do it by hand– yeah, all 400 stories or so– or I could fake it just spew the data from the old database, through a transform, to the new one– fast, but misses the point– or, the choice I made, I could write a simple web-based API.


It turns out that there are three mysteries to web-based APIs. First, there is RESTful verb architecture, meaning: do you encode the verb in the URL, or do you go with the HTTP verbs of PUT and DELETE along with the more traditional GET and POST?

RESTful verb architecture rigidifies the interaction with your system in strange ways. Consider my description above as a web page; a web designer would build a limited view of the author’s profile data, plus a tree-based view of the author’s serials and the stories they contain. Now, if your URL for an author is, say, “/author/<author_id>/”, it might make sense in an API to provide a complete view of the author profile and a complete list of his works, or it might make sense to have two two calls– one for profile, one for his works– but there’s no real way to structure the system in such a way that the URL refers some of the author’s information, and a tree of serials and stories. Those are four different relationships, referring to three different objects, one of which recurses.

This interacts with the other complete problem: how do you return data to the client? As… what? JSON? XML? YAML? A blob of text? Remember that one of the nifty miracles of narrator was its ubiquity: you could download the stories highly styled, or simplified for text-to-speech (I seem to have a lot of blind readers), or as text, or as a PDB Docbook format appropriate for ebook readers. Once the system has decided what action is going to be taken on the resource, that resources has to be rendered and returned to the client.

All of this, when dealing with a framework like Django or Rails, also seems to create the headache that they expect the REST commands to correspond to row actions on a table, which is not what’s going on here. An author isn’t just a row entry in the User table; it’s also a resource with other resources. Serializing it isn’t a simple matter of taking the row, the column names, and adding them together into a dictionary.

All of this comes with one surprisingly enormous pressure, from the Django community: DRY. Don’t Repeat Yourself. If you do, yur doin’ it wrong, as you Intarwebfolk like to say. (I am aware of all Internet traditions.) So when I start to write a generic method for handling the XML, YAML, and JSON versions, and have a “special” version for handling HTML, I worry that I’m repeating myself and the anxiety is crippling.

I should just say “Fuck it,” neh? Just write a stupid intake for “add a serial to a serial,” and another for “add a story to a serial,” and live with it. It might even become canon, and if I ever publish this thing, I’ll have people cursing me for using POX. Do everything in JSON (it’s what I know best) and worry about extensions later.

Oh, yeah, there was a third complication: Authentication.  That mystery is less difficult, but still annoying.

Comment Form

Subscribe to Feed



July 2009
« Jun   Aug »