So I've been doing some work on OHM, which is a library for exposing an object as a series of RESTful resources. (It also is a library for consuming such services, but for now don't use that because it's still twitchy -- but the server-side half is good IMHO.)
I realized part way through that it was different from many such APIs that I've seen, in that OHM exposes attributes as resources, not entire documents. This made sense for the thing I was initially creating, as it's basically the configuration API for the entire application, and since the application is farmed (that is, we add new instances programmatically) I wanted to deal with a bunch of these objects in a general way. I could have done it with normal controllers and standard framework stuff, but resources actually map really great to attributes (GET is getattr, PUT is setattr, DELETE is delattr) so it was a very comfortable fit. Also while I could represent the entire configuration as one XML document, I didn't really want to do that as the document would feel very artificial to me.
The downside of exposing a bunch of resources is that, while it's easy to set them each individually, there's no way to indicate that you want to do a bunch of things at once. There's no transactions -- you can update 2 out of 4 things, then fail, and there's no good way to revert. You can reset everything, which should work -- if you are using symmetric methods like GET and PUT, by putting back what you got before you should be safe. There still might be side effects -- things like logs, last changed timestamps, etc -- but if you are being properly RESTful then there shouldn't be any important side effects. That would be possible for me to do here, as it's not likely there will be much concurrency. But that's not a very good transaction.
There's another option I've been thinking about: pack a bunch of requests into one request. MIME already defines a way to do this sort of thing:
POST / HTTP/1.1 Content-Type: multipart/mixed; boundary="random-string" --random-string X-HTTP-Request-Method: PUT X-HTTP-Path: /resource/1 Content-Type: application/xml <xml request body...> --random-string X-HTTP-Request-Method: PUT X-HTTP-Path: /resource/2 Content-Type: application/xml <another xml request body...> --random-string--
WebDAV defines a 207 Multi-Status code, which could be used here:
HTTP/1.1 207 Multi-Status Content-Type: multipart/mixed; boundary="random-string" --random-string X-HTTP-Path: /resource/1 X-HTTP-Status: 201 Created --random-string X-HTTP-Path: /resource/2 X-HTTP-Status: 204 No Content --random-string--
Generally this doesn't make a lot of sense for GET requests, and so the Multi-Status responses are generally a little boring in the case of a success (they should all be some form of 2xx). But for failures it is more interesting:
HTTP/1.1 207 Multi-Status Content-Type: multipart/mixed; boundary="random-string" --random-string X-HTTP-Path: /resource/1 X-HTTP-Status: 403 Forbidden Content-Type: text/html <html>... access forbidden ...</html> --random-string X-HTTP-Path: /resource/2 X-HTTP-Status: 412 Precondition Failed Content-Type text/html <html>... other request /resource/1 failed ...</html> --random-string--
WebDAV uses XML responses for 207, but it's not a very good match for what HTTP already is IMHO -- HTTP looks like MIME, it doesn't look anything like XML.
Of course all this is purely speculative; nothing implements any of this. But I started thinking about this in terms of a WSGI middleware that explodes out the requests, turning that example into two WSGI requests. If all the pieces can agree on a container for the transaction (so the side-effects of the requests can be committed or rolled back as one), then it could almost be automatic -- you just make sure all the requests share the same transaction. This doesn't work over process boundaries really, though with a two-phase commit and some convention for indicating the transaction, it should be doable. Maybe like:
POST /resource HTTP/1.1 X-Transaction-ID: 12345 (globally unique non-guessable ID) ... HTTP/1.1 202 Accepted Content-Type: ... (response body AS IF the commit had happened) POST /resource HTTP/1.1 X-Transaction-ID: 12345 X-Commit: true HTTP/1.1 204 No Content
Two-phase commit is hardly a great solution, or necessarily easy to implement generally, but at least this offers some possible practice for doing this.
Interestingly (at least to me), you can build tools to implement this sort of convention in different environments. To ensure interoperability you'd need some kind of document or something to define what was expected. In the REST vs. WS-* conflict it often seems phrased as though it's purely use-what-we-have vs. build-new-specifications. Or use-what-we-have and build-new-frameworks. There's an underlying (and incorrect) assumption that if you subscribe to REST notions then what we have is all we will have. I think it's more that REST people want to walk lightly when building on web that we have; avoid recreating what already exists, and definitely don't conflict with it. There's a deeper architectural difference as well -- WS-* is focused on RPC, and the over-the-wire protocols defer to the language frameworks, while REST is focused on independent resources and pushes back on language frameworks to fit that model. But that distinction doesn't mean there isn't new code to be written, and new conventions or specifications to be built.
maybe this rest-discuss thread is of interest: http://tech.groups.yahoo.com/group/rest-discuss/message/7780
# Paul Winkler
That thread is very interesting indeed Paul. However it mentions the POE techniques which I really don't see as a good alternative. There are lots of corner cases the POE draft doesn't address at all.
you semi-lost me at "WSGI middleware that explodes out the requests, turning that example into two WSGI requests." does that mean at the HTTP level, theres just one request, then the middleware creates multiple WSGI requests in-process ? since when I read this, the main thing jumping out at me is "you have to make 4 separate requests to set 4 different attributes, hows that going to work for 10 users at once ?". i.e. yes the transaction-ness is one issue, but also just plain memory/time overhead of all those connections.
also, while the whole PUT /my/attribute thing is pretty cool, i must admit my instinct to set 10 attributes at once would be just, POST /my/object .... attr1=val1&attr2=val2&... is that so terrible ?
you semi-lost me at "WSGI middleware that explodes out the requests, turning that example into two WSGI requests." does that mean at the HTTP level, theres just one request, then the middleware creates multiple WSGI requests in-process ? since when I read this, the main thing jumping out at me is "you have to make 4 separate requests to set 4 different attributes, hows that going to work for 10 users at once ?". i.e. yes the transaction-ness is one issue, but also just plain memory/time overhead of all those connections.Generally REST is better optimized for reads than writes anyway. In the particular situation I'm in neither performance is terribly important -- it's just a configuration API, not part of the regular functionality. 4 GETs could be optimized just with Keep-Alive, assuming going up and down the backend stack is fast enough. It depends on the application of course, but multiple requests aren't necessarily a problem. And really while I stuffed a bunch of configuration in one piece of code (WSGI middleware actually), it's really covering about 3 different use cases, and potentially all of those could be handled separately, and then it really should be multiple requests. (Unless I automatically aggregate them somehow, which is what I'm considering here.)
also, while the whole PUT /my/attribute thing is pretty cool, i must admit my instinct to set 10 attributes at once would be just, POST /my/object .... attr1=val1&attr2=val2&... is that so terrible ?Sure, you could do it that way, though each of those attributes is at least moderately complicated (generally each is a JSON document). It's not that different from what I propose, actually, except what I discuss can be understood by an intermediary. Well... actually I can't decide on the difference between multipart/mixed and multipart/form-data, which itself is used in HTML forms already. So it's not necessarily even that different from a POST, just adding in some path and method headers.
# Ian Bicking
You could look into the locking section in the WebDAV specification:
http://www.webdav.org/specs/rfc2518.html#locking
Of course, you'd still need to implement your own transactional mechanisms. Meanwhile, there's also the versioning extensions to WebDAV, which seem to exist to support Subversion, or at least things similar to Subversion:
http://www.webdav.org/specs/rfc3253.html
The problem with the WebDAV specifications is that to be compliant you end up having to support lots of stuff you'll rarely need, I suppose.
# Paul Boddie
I'm tackling a similar issue in http://code.google.com/p/grassyknoll/ Though for me, the issue is less one of transactions than optimization. The backend (PyLucene) performs much better when passed requests in batches. So a request like (pardon the pseudocode):
PUTMANY(doc1, doc2, doc3, doc4)
is much zippier than doing:
- for d in [doc1, doc2, doc3, doc4]:
- PUT(d)
interestingly, this is true for GET as well...
Doing this in an intermediary wouldn't really help then, as it would make the controller unaware of the other requests happening at the same time. Hmm... too bad that. Though in that case you wouldn't have to use a standard intermediary like I propose, you could have a custom one, and presumably client-side support can develop separately without having to pay attention to it one way or the other.
# Ian Bicking
I've been thinking about this issue as well for a project I've been tinkering with, a RESTful database implemented in Python [1].
One idea I had was to add a special transaction resource (at /transactions, or some other agreed-upon location). When you POST to the transaction resource, it sprouts a new child with a unique ID (/transactions/12345). That URL now duplicates the entire structure of the database. So if the database had a resource called /Users/Bob, there is now a /transactions/12345/Users/Bob resource. The client can do all of his database operations within his transaction sandbox, then POST to /transactions/12345 to commit.
I think this is a fairly RESTful way to implement transactions. The best part IMO is that it allows you to add transactions without adding any HTTP headers, additional verbs, etc. The only downside I see is that there are now multiple URLs for a single resource. You might be able to solve that by using HTTP 301 to point clients at the canonical one if they try to touch something that's already been committed.
[1] http://blog.extracheese.org/2007/02/introducing-another-wildly-ambitious.html
Heh. Gary, we've independently invented the same thing :-) Check out the link I posted above.
Yes, I posted my reply here before reading through that thread. :) It seems like the only real difference between our approaches is that when you create a transaction, you tell it which resources you'll want to touch. I prefer to "branch" the whole URL space, rather than some subset. That way, your code doesn't have to think about which resources it'll touch until it does the touching. :)
I haven't got the book yet, but "the transaction is a resource" is a pattern spelled out in this review by Jon Udell: http://blog.jonudell.net/2007/05/24/restful-web-services/ so apparently it's in the book.
Yikes. I propose two reasons you'd rather head down a different path.
- I expect that usually when you want to update several things at once, you're doing something singularly nameable; ie, there's a further abstraction to the group of work that is a resource. In OHM I see this issue, each attribute is individually accessible, but although we've grouped them into an object's attributes conceptually, there's no way to access or offer that conceptual grouping. OHM-level support for record and collection (tuple and relation :) abstractions should be implemented, even if it's just a set of conventions that we don't know yet. If we can name the set of data we're updating (at the simplest level, SO's o.set vs o._set_X), we can REST it in one request. The abstraction part comes from seeing your data as a relational database with idealized views.
- We still have to make use of mechanisms where it is each resource's responsibility to ensure its state transition is atomic when reacting to a side-effect verb, because any abstraction over URL-based referencing means process boundary is no different from server boundary or network boundary. How do you architect this differently when your mindset is thinking about having a change on this site need to set something else on another site? Same as within parts of your site that you may end up partitioning by machine or farm out to a third party. Rhetorically, does the reliability of that inner resource impact the assumptions of the original code when you could count on a local transaction? If updating your marriage status should be reflected in your holy rewards program availability, is two-phase commit with an optimization if we're lucky enough to be on the same process really the mechanism that comes to mind? Mines a little blurry.
Peace, Luke
# Luke Opperman
Ok, found the article I'd printed out (!) on this subject, the original PDF seems to have disappeared but google cache is http://209.85.165.104/search?q=cache:44SdWHeWNXcJ:www-db.cs.wisc.edu/cidr/cidr2007/papers/P15.pdf&hl=en&ct=clnk&cd=1&gl=us
"Life beyond Distributed Transactions" by Pat Helland
# Luke Oppperman
I expect that usually when you want to update several things at once, you're doing something singularly nameable; ie, there's a further abstraction to the group of work that is a resource. In OHM I see this issue, each attribute is individually accessible, but although we've grouped them into an object's attributes conceptually, there's no way to access or offer that conceptual grouping.If you do want to do this in OHM, you can create an attribute that is located at the root of the object (url_path=""). This can be a property that aggregates all the attributes of the object. If they are all JSON, you could do:
class MyObject(object): ... all the other attrs ... all_attrs = ['a', 'b', ...] def complete_state__get(self): return dict((attr, getattr(self, attr)) for attr in self.all_attrs) def complete_state__set(self, data): for name, value in data.items(): assert name in self.all_attrs setattr(self, name, value) complete_state = property(complete_state__get, complete_state__set) class MyObjectWrapper(ohm.server.ApplicationWrapper): complete_state = ohm.server.JSONSetter(uri_path='')With XML you could do something similar, though actually creating the documents involves more documentation and details. But if you have heterogeneous types in your resources this doesn't work so well. For instance, I also attach a file server to the object that accepts PUT, MKCOL, and DELETE. Or some objects may be JSON, and others XML documents, and some operations might really be POST operations, and so forth. Mixing these into one document just doesn't work.
So what I describe here is really the same process, only it's amenable to any kind of document types. And I suppose eclectic HTTP methods.
We still have to make use of mechanisms where it is each resource's responsibility to ensure its state transition is atomic when reacting to a side-effect verb, because any abstraction over URL-based referencing means process boundary is no different from server boundary or network boundary. How do you architect this differently when your mindset is thinking about having a change on this site need to set something else on another site?I certainly would not argue that this lets you ignore what's going on, internally or in the remote server. The failure cases and likelihood of those cases will change. This is really just a tool to make the mechanics easier. That doesn't make the system design easier, except that it might let you focus on the system and not get too bogged down in the mechanics.
# Ian Bicking
I'm new to this - so forgive me if this is a silly suggestion;
Instead of a PUT just to update the existing config attributes, can you also use PUT to create a new config set?
Then use a PUT to make that the new config current ( a 'committ')
Maybe define a GET to retrieve past, current, and new config sets that have been PUT? (So you can rollback to one of them)
If this is silly - please let me know why.
Cheers,
Stephen
# Stephen