Ian Bicking: a blog

I attended PloneConf 2008 recently, to talk about Deliverance. But I’ll talk here more about my observations as a relative outsider to that community. (This post is really written for the Plone community — if you aren’t familiar with Plone this post probably won’t make much sense or be very useful.)

One of the ongoing concerns in the Plone community has been the difficulty of attracting and maintaining developer interest in the community, and generally making Plone easier to work with as a developer. It’s been a very successful community for consulting and integration work, but it has not been as rewarding for developers. There’s the idea of a "Plone Tax", which means different things to different people but just generally speaks to the sense that Plone takes a little off the top — time to restart, time to run the tests, time to dig into code that just goes too deep. There are some distinct problems with Plone, but probably the biggest problem is the quantity of small challenges and the general size and complexity of the system.

At the moment there is no clear path forward to resolve this. A previous effort to fix things is a project called "Five", which referred to Zope 2+3 — backporting libraries and techniques from Zope 3 into Zope 2. Plone is intimately tied to Zope 2, and Five let them use new code without having to do abandon the old environment. But the result wasn’t terribly satisfying: Five didn’t remove any complexity, it only added to it. Even if the new components were superior that doesn’t make them simple. Plone is aching for more simplicity, not more power.

It’s unclear how Plone can actually reform itself as a codebase. Zope 2 is a behemoth, and its metaphors are deeply intertwined with existing code. Acquisition in particular is ubiquitous, essential to lots of the machinery, and deeply confusing. I saw a general interest in two directions: one was to encourage non-content-management tasks to be implemented in a complementary (but separate) technology, another direction is continuing to refactor the existing codebase while somehow trying to maintain backward compatibility.

It’s unclear how to refactor the existing codebase in such a way that it is any simpler, but I suppose these two directions are not exclusive. I want to focus on the idea of a complementary environment. There’s two products in particular that have been attracting interest: Grok and Repoze.

These two products usually go under the umbrella of "exciting new ways to improve the developer experience in Plone" which is a kind of generic positive sentiment. But to my mind they represent two significantly different paths forward, and the differences deserve some more critical thought.

Grok is a layer on top of Zope 3 that attempts to make development in that environment more pleasing. It eliminates most ZCML (ZCML is Zope 3′s XML-based language for declaring the relation of various components in a system). Grok uses conventions and introspection to make Zope 3 look more like a traditional web framework, with simple views and models and templates, and less of the wiring you have to set up in typical Zope 3 architectures. At the same time, you can add all the Zope 3 declarations to break out of the automatic conventions. Grok is a more pleasant layer on top of Zope 3, but it’s entirely focused on Zope 3, and it is led by Martijn Faassen and Philipp von Weitershausen who are both very involved with the Zope 3 community.

Repoze is a more recent project, led by Chris McDonough, Tres Seaver, and Paul Everitt (well, Paul might call himself more of a cheerleader). They all work together at Agendaless Consulting. They’ve been highly involved in Zope 2 for a long time, and are all former employees of Zope Corp and major contributors to Zope 2. I don’t think they’ve ever quite made the jump to Zope 3, and their consulting and experience kept them involved with Zope 2. A while back they got some WSGI religion and started splitting out some pieces of Zope into WSGI middleware and other independent libraries. This included things like pulling out the transaction handler from Zope, reimplementing the Zope 2 publisher so it was more WSGIish, and a variety of other libraries. These libraries are essentially extractions from Zope, or in the case of the Zope publisher and repoze.plone, a way to wrap what would be considered a "legacy" application in the same interface as other newer pieces. Having extracted nearly everything they wanted, they’ve started work on a framework intended to be familiar to the Zope community, repoze.bfg. This framework uses some Zope 3 concepts, but it’s more built from scratch than it is built from Zope 3, and it is attached to Zope 2 only insofar as it uses the pieces they’ve extracted and the ideas they’ve become comfortable with. They’ve described it as the framework they want to use when someone asks them to build something in Zope 2.

Grok and Repoze have a significantly different development methodology. One is a layer on Zope 3, the other is an extraction of ideas from Zope 2 (and a few from Zope 3). In part I think the distinction hasn’t been presented very clearly because the Repoze and Grok communities overlap a great deal, and everyone is quite congenial with each other, and they are reluctant to enter debates about the designs. While I also consider them all colleagues, I also feel pretty strongly about the design differences and I feel a discussion contrasting them is important.

Martijn recently wrote a post on why Plone should consider Grok and a follow-up post. These posts speak to a variety of advantages Grok has over plain Zope 3, which Grok can offer to Plone to manage its existing use of Zope 3 technologies.

I think Plone shouldn’t be so focused on managing the complexity of its stack, but focus on reducing that complexity. And it should reduce that complexity by focusing on content management and moving all the other pieces people have built on Plone out of Plone. It can’t just leave people hanging, which is why developing a clear story for how those other pieces should be developed is essential. Plone the community doesn’t have to map one-to-one to Plone the software. Plone the software should become smaller and more focused. Plone the community doesn’t have to become more focused — the community does what it needs to do, what customers ask for, what is necessary to make a site compelling. To be clear on what I’m proposing here: Plone should have a community-recommended way to build non-Plone applications. This doesn’t mean you have to use those techniques, but it should be much more concrete than just "there’s a bunch of cool things out there, and maybe you should look around and use one of those." By having a community-recommended pattern of development you can maintain and build on the Plone community, which is at least as important an asset as the Plone software.

With this in mind I believe an extraction of Zope 2 and Plone ideas is the right path forward. Extraction is the process of isolating code and ideas, and localizing the effect of that code. These are some of the most important ways to actually increase the simplicity of a codebase. In the Zope 2 (and even 3) codebase the thing that brings the most complexity is the non-locality of code, that parts of the system can effect each other in complex and unexpected ways. Zope 3 formalizes these patterns of non-locality: the Component Architecture is largely about introducing non-localized relationships between code.

Arguably this non-locality of effect is exactly what people want, it’s what enables pervasive customizations. When a client asks you to change some little piece deep in the system, you don’t really want to modify the system’s code — you want to add a little code to the outside of the system that effects the change you desire. The Component Architecture is a formalized way of making these kinds of changes, where Acquisition was a lower-level mechanism to do the same sort of thing. I remain a Component Architecture skeptic, mostly because I think the mechanism is overused. Still I think clearly Plone needs flexibility that a purely bespoke application would not require. But there’s no way out of it: that flexibility has a high cost. In this there is no clear solution. In addition to Acquisition, the complexity of Zope 2 security, the many layers of skinning… is the Component Architecture what Plone needs? From where I stand it seems like a step further in the wrong direction… and yes, it is a better placed step than any of the ones before, but if it’s still a step in the wrong direction does that matter?

I think Plone (at least the community) needs to be conservative in its enabling of these customizations. Sure, the customizability is "powerful," but I don’t hear people clamoring for power. They want simple, predictable, fast, maintainable. That’s not necessarily the same as "easy" — I think sometimes it’s worth making things a little longer and less automatic to make a system more explicit and make code more localized. I think Repoze is a step in this direction, a big (maybe even an intimidatingly giant) step towards simplicity. That’s the step I think Plone should make.

So, I offer this as my suggestion to Plone. I think Plone-the-community has the opportunity to be more than Plone-the-software; I think it must do this to remain viable in the long term. But to get there the community make some choices — you can’t add simplicity.