I attended PloneConf 2008 recently, to talk about Deliverance. But I’ll talk here more about my observations as a relative outsider to that community. (This post is really written for the Plone community — if you aren’t familiar with Plone this post probably won’t make much sense or be very useful.)
One of the ongoing concerns in the Plone community has been the difficulty of attracting and maintaining developer interest in the community, and generally making Plone easier to work with as a developer. It’s been a very successful community for consulting and integration work, but it has not been as rewarding for developers. There’s the idea of a "Plone Tax", which means different things to different people but just generally speaks to the sense that Plone takes a little off the top — time to restart, time to run the tests, time to dig into code that just goes too deep. There are some distinct problems with Plone, but probably the biggest problem is the quantity of small challenges and the general size and complexity of the system.
At the moment there is no clear path forward to resolve this. A previous effort to fix things is a project called "Five", which referred to Zope 2+3 — backporting libraries and techniques from Zope 3 into Zope 2. Plone is intimately tied to Zope 2, and Five let them use new code without having to do abandon the old environment. But the result wasn’t terribly satisfying: Five didn’t remove any complexity, it only added to it. Even if the new components were superior that doesn’t make them simple. Plone is aching for more simplicity, not more power.
It’s unclear how Plone can actually reform itself as a codebase. Zope 2 is a behemoth, and its metaphors are deeply intertwined with existing code. Acquisition in particular is ubiquitous, essential to lots of the machinery, and deeply confusing. I saw a general interest in two directions: one was to encourage non-content-management tasks to be implemented in a complementary (but separate) technology, another direction is continuing to refactor the existing codebase while somehow trying to maintain backward compatibility.
It’s unclear how to refactor the existing codebase in such a way that it is any simpler, but I suppose these two directions are not exclusive. I want to focus on the idea of a complementary environment. There’s two products in particular that have been attracting interest: Grok and Repoze.
These two products usually go under the umbrella of "exciting new ways to improve the developer experience in Plone" which is a kind of generic positive sentiment. But to my mind they represent two significantly different paths forward, and the differences deserve some more critical thought.
Grok is a layer on top of Zope 3 that attempts to make development in that environment more pleasing. It eliminates most ZCML (ZCML is Zope 3′s XML-based language for declaring the relation of various components in a system). Grok uses conventions and introspection to make Zope 3 look more like a traditional web framework, with simple views and models and templates, and less of the wiring you have to set up in typical Zope 3 architectures. At the same time, you can add all the Zope 3 declarations to break out of the automatic conventions. Grok is a more pleasant layer on top of Zope 3, but it’s entirely focused on Zope 3, and it is led by Martijn Faassen and Philipp von Weitershausen who are both very involved with the Zope 3 community.
Repoze is a more recent project, led by Chris McDonough, Tres Seaver, and Paul Everitt (well, Paul might call himself more of a cheerleader). They all work together at Agendaless Consulting. They’ve been highly involved in Zope 2 for a long time, and are all former employees of Zope Corp and major contributors to Zope 2. I don’t think they’ve ever quite made the jump to Zope 3, and their consulting and experience kept them involved with Zope 2. A while back they got some WSGI religion and started splitting out some pieces of Zope into WSGI middleware and other independent libraries. This included things like pulling out the transaction handler from Zope, reimplementing the Zope 2 publisher so it was more WSGIish, and a variety of other libraries. These libraries are essentially extractions from Zope, or in the case of the Zope publisher and repoze.plone, a way to wrap what would be considered a "legacy" application in the same interface as other newer pieces. Having extracted nearly everything they wanted, they’ve started work on a framework intended to be familiar to the Zope community, repoze.bfg. This framework uses some Zope 3 concepts, but it’s more built from scratch than it is built from Zope 3, and it is attached to Zope 2 only insofar as it uses the pieces they’ve extracted and the ideas they’ve become comfortable with. They’ve described it as the framework they want to use when someone asks them to build something in Zope 2.
Grok and Repoze have a significantly different development methodology. One is a layer on Zope 3, the other is an extraction of ideas from Zope 2 (and a few from Zope 3). In part I think the distinction hasn’t been presented very clearly because the Repoze and Grok communities overlap a great deal, and everyone is quite congenial with each other, and they are reluctant to enter debates about the designs. While I also consider them all colleagues, I also feel pretty strongly about the design differences and I feel a discussion contrasting them is important.
Martijn recently wrote a post on why Plone should consider Grok and a follow-up post. These posts speak to a variety of advantages Grok has over plain Zope 3, which Grok can offer to Plone to manage its existing use of Zope 3 technologies.
I think Plone shouldn’t be so focused on managing the complexity of its stack, but focus on reducing that complexity. And it should reduce that complexity by focusing on content management and moving all the other pieces people have built on Plone out of Plone. It can’t just leave people hanging, which is why developing a clear story for how those other pieces should be developed is essential. Plone the community doesn’t have to map one-to-one to Plone the software. Plone the software should become smaller and more focused. Plone the community doesn’t have to become more focused — the community does what it needs to do, what customers ask for, what is necessary to make a site compelling. To be clear on what I’m proposing here: Plone should have a community-recommended way to build non-Plone applications. This doesn’t mean you have to use those techniques, but it should be much more concrete than just "there’s a bunch of cool things out there, and maybe you should look around and use one of those." By having a community-recommended pattern of development you can maintain and build on the Plone community, which is at least as important an asset as the Plone software.
With this in mind I believe an extraction of Zope 2 and Plone ideas is the right path forward. Extraction is the process of isolating code and ideas, and localizing the effect of that code. These are some of the most important ways to actually increase the simplicity of a codebase. In the Zope 2 (and even 3) codebase the thing that brings the most complexity is the non-locality of code, that parts of the system can effect each other in complex and unexpected ways. Zope 3 formalizes these patterns of non-locality: the Component Architecture is largely about introducing non-localized relationships between code.
Arguably this non-locality of effect is exactly what people want, it’s what enables pervasive customizations. When a client asks you to change some little piece deep in the system, you don’t really want to modify the system’s code — you want to add a little code to the outside of the system that effects the change you desire. The Component Architecture is a formalized way of making these kinds of changes, where Acquisition was a lower-level mechanism to do the same sort of thing. I remain a Component Architecture skeptic, mostly because I think the mechanism is overused. Still I think clearly Plone needs flexibility that a purely bespoke application would not require. But there’s no way out of it: that flexibility has a high cost. In this there is no clear solution. In addition to Acquisition, the complexity of Zope 2 security, the many layers of skinning… is the Component Architecture what Plone needs? From where I stand it seems like a step further in the wrong direction… and yes, it is a better placed step than any of the ones before, but if it’s still a step in the wrong direction does that matter?
I think Plone (at least the community) needs to be conservative in its enabling of these customizations. Sure, the customizability is "powerful," but I don’t hear people clamoring for power. They want simple, predictable, fast, maintainable. That’s not necessarily the same as "easy" — I think sometimes it’s worth making things a little longer and less automatic to make a system more explicit and make code more localized. I think Repoze is a step in this direction, a big (maybe even an intimidatingly giant) step towards simplicity. That’s the step I think Plone should make.
So, I offer this as my suggestion to Plone. I think Plone-the-community has the opportunity to be more than Plone-the-software; I think it must do this to remain viable in the long term. But to get there the community make some choices — you can’t add simplicity.
No related posts.
Concerning Repoze “versus” Grok. Yes, there are differences: Repoze does a lot of rewriting in the interests of implementation simplicity. Grok hasn’t rewritten the implementation and focused on usage simplicity. The two systems are learning from each other: Repoze can use bits of Grok’s auto-configuration system. Meanwhile inspired by some of Repoze’s work (as well as frustrations with some of Zope 3′s non-locality), I’ve started looking at simplifying parts of the underlying Zope 3 system (and in general the sometimes complex structure of dependencies in the Zope 3 codebase is a thing Zope 3 developers are concerned about).
Grok’s busy extracting useful bits of code from itself and both Martian as well as grokcore.component are reusable in random Python projects. The rest of the grokcore.* stuff is reusable in Zope 2 and in fact reuse in Zope 2 is driving this process.
You say the use of the component architecture is a step in the wrong direction, but could you be more explicit about why this would be so? What if you use it in moderation? Some form of customization machinery is needed and sticking to the (well-engineered, much used, battle-tested) devil we know seems like the right idea to me.
It’s important for any scenario to move forward is to have a way to evolve from A to B. You can take risks in this evolution, and perhaps larger risks should be taken, but it still will have to be an evolutionary step.
The word “simple” means different things to people. Simple in implementation? Simple in use? Simple to change? Assembly language is simple in implementation and Python adds huge amounts of abstraction and automation over a machine. Yet, Python’s a lot simpler in use than assembly language, even though assembly is so much simpler in implementation. So, of course automation and abstraction can be added to a system to make things simpler! That said, implementation simplicity can increase usage simplicity as a system can become easier to understand and predict, but it’s hardly clear-cut. Simplicity in implementation is valuable when it increases simplicity of development, and simplicity of use.
So, I’ll grant there is value in simplicity in implementation. I also believe Plone’s implementation can be significantly simpler. It’ll still be complex no matter what, though, just less complex than it is now. Any move to simplify Plone will run the risk of trading off one form of complexity for another. There is also value in sticking to the complexity the community knows about.
I believe Plone’s best road to simplification takes two paths:
Towards simplicity of implementation: removing some of the old ways of doing things permanently. Hopefully a point in evolution has been reached that old things can be removed. The removal of acquisition from the system would be a good, though big example. Allow content types that are truly simple and don’t rely on SimpleItem and a whole bunch of other things would be another good step.
Towards simplicity of use: adding easier ways to do things that are too hard right now. Using Grok technology to help with the wiring of the inevitable configuration machinery is a good example.
The alternative would be writing a new CMS taking into account the lessons learned from Plone. That’s an interesting project. Is it the Plone project?
One other road to take if simplification of Plone is not that easy, is to simplify the use of Plone the software for other Python developers not willing to pay the Plone tax. Using ContentMirror, for example, it is possible to serialize Plone content into a relational database and create simple applications on top of that using any other framework.
Granted, this is not an approach that can be used for all kinds of projects, but there are many use cases where separating the content production from its delivery would be a very good idea. Plone is a very good CMS, so I think it would be great if people not willing to use it as a framework can still benefit from it as a piece of software. That would make Plone, the community, even bigger.
In part, this comes down to the old “application vs. platform” argument. I think that technologies like WSGI and Repoze are enabling Plone to be more of an application again, because, as you say, non-CMS functionality can be built elsewhere.
I personally believe that Repoze and Grok are both key to Plone’s future. Roughly, I think the roadmap should look something like this:
Rip out things we no longer need (already happening on trunk).
Where two ways exist to do something, choose one way and remove all other ways (starting to happen on trunk).
Target key use cases for extending Plone-the-application and focus on making these as easy and friendly for integrators as possible (this is where I think Grok conventions and patterns will be beneficial). This includes things like:
Push services that could cross-cut applications out into WSGI middleware so that they become reusable (this is where Repoze comes in)
Promote the use of other, WSGI-enabled frameworks for non-CMS tasks (be that repoze.bfg or Grok or something else)
Promote a way to unify look-and-feel/theming (this is what Deliverance promises).
I whole-heartedly agree with Martijn’s point about evolution. Developers have a strong tendency to favour a rewrite or a clean break, but that’s a sure way to lose those who’ve already invested in the platform. We’re only just now coming to the point where the feature set and the available tools and frameworks cover what I’d consider the basic CMS needs (with a few exceptions), and there’s a real drive in the community now to focus on simplification and the integrator experience.
At the same time, we need to be very careful about throwing the baby out with the bathwater. The Zope 3 CA has an important part to play in making Plone a productive, usable and flexible system. People may not be clamoring for more power, because they have power now. We’d like to unify the way in which that power is provided (e.g. by removing some things that still rely on acquisition when it should rely on components), not take it away.
We also need to be better at indicating what extension points and APIs are aimed at integrators/developers and what things are really just the concern of Plone’s core developers. Right now, “the API” is kind of the whole application, in that you can customise and change anywhere. We need to ring-fence that with something a bit more obvious that gets people started quickly, and only require them to get into that level of detail if they really need to.
Cheers,
Martin
I think Plone should take a carrot approach to simplification, not the stick of removing or deprecating techniques. Right now every Plone site of any substantial size has non-CMS aspects to it. Removing things from Plone is politically difficult as a result. Getting people to implement those non-CMS things outside of Plone seems like the easier approach at the moment, the approach that is on most people’s timelines (even if the dates are different), and it doesn’t have to effect backward compatibility right now. But actually doing this isn’t easy, and probably requires some extra help from Plone as well. So I think it makes sense to figure out that story, to focus on that work for now, because it’s not a small amount of work and it won’t happen with out direct effort. If that’s successful then I think it’ll be much less disruptive if Plone itself is simplified.
As Carlos suggests, I think ContentMirror could be part of that story. The more formally “correct” solution might be web services of some sort, but I think a relational database mirror is probably better understood and less likely to cause performance problems due to its implicitly async design.
+1 on contentmirror, the concept seems great and I’m looking forward to trying it.
In my experience with the use of components and component adaptation in Plone, there are two major problems.
The first is, that adaptation is overused. A portlet is a five-way adapter and there’s no way to change the adaptation order. This rules out a swift solution for most of the more advanced use-cases, simply because you’d have to work very hard with the system to get the adaptation pattern you really want. Adaptation as we know it is simply not a very good multiplexer.
The second issue is that we can’t really ever strike the right balance between explosion and coherence in defining the scope of a component. The best example I know of this is the navigation machinery. This is implemented as a collection of components that each have some scope and function and they’re glued together using the component registry. However, in most cases, I’ve wanted to simply provide a replacement wholesale, and in this case, the exploded code chain is counterproductive and needlessly complex.
To recap, I believe the correct usage of components is to adapt on small specifications, with one-way or two-way being standard, avoid adaptation for multiplexing where possible and finally to avoid exploding our software into too many components. We still have object-oriented programming and Python is still a dynamic language.
It’s pretty hard to do the matrix calculus that makes most of the various Plone constituencies happy while still doing something useful along any meaningful simplification axis. So finding some middle ground where Plone stakes out its ground as-is (“CMS”), and other tasks are delegated to other systems as necessary is probably a good idea.
But at the same time, I’d probably avoid taking this route for new customer projects. Content management features aren’t too hard to create in any of the new crops of Python web frameworks. Writing a form, even auto-generating that form using some framework, accepting input from users, showing errors, managing users, etc: for any given project, satisfying these types of requirements tends to be pretty easy without Plone. It’s often much harder to remove unnecessary features from Plone than it is to create the necessary ones from scratch. It would take a heck of a lot of intersection between a random customer’s requirements and the feature set that Plone provides out of the box for me to use Plone as the “CMS”, and delegate other tasks to a simpler system. I’d tend to just use the simpler system for everything, left to my own devices. That said, for legacy systems that already use Plone, I’d likely go with Content Mirror.
IMO, the killer feature of Plone that no other “framework” has is that it does something out of the box. Folks who install and use it can quickly get a frame of reference quickly, and they can describe (to themselves or others) requirements in terms of modifications to that out of the box experience. And for better or worse, people tend to do those modifications to Plone to satisfy the requirements. So it drives a good deal of consulting business.
So how do we square the circle? I’m not sure. But I don’t think too many tears would be shed if something else were created that was materially simpler and offered some similar set of features out of the box. I suspect that a phoenix-from-the-flames project that aimed to recreate a Plone-ish out-of-the-box experience, but which ditched backwards compatibility would probably be most successful in the long run. There are of course Plone add-ons and such, and these are important, but even without those, an alternative could probably support a healthy community. It shouldn’t be called Plone, obviously. This of course mirrors what we’re doing with bfg vs. Zope, so I suppose my saying any of that is not very surprising.
Obviously one of the biggest issues is a political one. Sure, other people can write a CMS. Someone could even pay a lot of attention to migration, and maybe pick out a handful of the most popular add-ons to also translate, and so make the intention as a successor clear. But that itself doesn’t make a New Plone.
At some future date I think the community could choose to make a radical change like that — and even call the result “Plone” — but it will be easy only if Plone is a CMS and not a platform for building entire web sites. Which is why I suggest this path to making Plone into more of a CMS and less of a platform. And ultimately that question of how to make a complete website must be answered — another part of Plone’s success is that it answers that question. So right now recreating just the CMS of Plone wouldn’t be sufficient to really recreate Plone.