No, I should say forms particularly.
I have lots of things to blog about, but nothing makes me want to blog like code. Ideas are hard, code is easy. So when I saw Jacob’s writeup about dynamic Django form generation I felt a desire to respond. I didn’t see the form panel at PyCon (I intended to but I hardly saw any talks at PyCon, and yet still didn’t even see a good number of the people I wanted to see), but as the author of an ungenerator and as a general form library skeptic I have a somewhat different perspective on the topic.
The example created for the panel might display that perspective. You should go read Jacob’s description; but basically it’s a simple registration form with a dynamic set of questions to ask.
I have created a complete example, because I wanted to be sure I wasn’t skipping anything, but I’ll present a trimmed-down version.
First, the basic control logic:
from webob.dec import wsgify
from webob import exc
from formencode import htmlfill
@wsgify
def questioner(req):
questions = get_questions(req) # This is provided as part of the example
if req.method == 'POST':
errors = validate(req, questions)
if not errors:
... save response ...
return exc.HTTPFound(location='/thanks')
else:
errors = {}
## Here's the "form generation":
page = page_template.substitute(
action=req.url,
questions=questions)
page = htmlfill.render(
page,
defaults=req.POST,
errors=errors)
return Response(page)
def validate(req, questions):
# All manual, but do it however you want:
errors = {}
form = req.POST
if (form.get('password')
and form['password'] != form.get('password_confirm')):
errors['password_confirm'] = 'Passwords do not match'
fields = questions + ['username', 'password']
for field in fields:
if not form.get(field):
errors[field] = 'Please enter a value'
return errors
I’ve just manually handled validation here. I don’t feel like doing it with FormEncode. Manual validation isn’t that big a deal; FormEncode would just produce the same errors dictionary anyway. In this case (as in many form validation cases) you can’t do better than hand-written validation code: it’s shorter, more self-contained, and easier to tweak.
After validation the template is rendered:
page = page_template.substitute(
action=req.url,
questions=questions)
I’m using Tempita, but it really doesn’t matter. The template looks like this:
<form action="{{action}}" method="POST">
New Username: <input type="text" name="username"><br />
Password: <input type="password" name="password"><br />
Repeat Password:
<input type="password" name="password_confirm"><br />
{{for question in questions}}
{{question}}: <input type="text" name="{{question}}"><br />
{{endfor}}
<input type="submit">
</form>
Note that the only "logic" here is to render the form to include fields for all the questions. Obviously this produces an ugly form, but it’s very obvious how you make this form pretty, and how to tweak it in any way you might want. Also if you have deeper dynamicism (e.g., get_questions start returning the type of response required, or weird validation, or whatever) it’s very obvious where that change would go: display logic goes in the form, validation logic goes in that validate function.
This just gives you the raw form. You wouldn’t need a template at all if it wasn’t for the dynamicism. Everything else is added when the form is "filled":
page = htmlfill.render(
page,
defaults=req.POST,
errors=errors)
How exactly you want to calculate defaults is up to the application; you might want query string variables to be able to pre-fill the form (use req.params), you might want the form bare to start (like here with req.POST), you can easily implement wizards by stuffing req.POST into the session to repeat a form, you might read the defaults out of a user object to make this an edit form. And errors are just handled automatically, inserted into the HTML with appropriate CSS classes.
A great aspect of this pattern if you use it (I’m not even sure it deserves the moniker library): when HTML 5 Forms finally come around and we can all stop doing this stupid server-side overthought nonsense, you won’t have overthought your forms. Your mind will be free and ready to accept that the world has actually become simpler, not more complicated, and that there is knowledge worth forgetting (forms are so freakin’ stupid!) If at all possible, dodging complexity is far better than cleverly responding to complexity.
Automatically generated list of related posts:
- WebOb decorator Lately I’ve been writing a few applications (e.g., PickyWiki and...
- Inverted Partials I was talking with a coworker some time ago about...
- What Does A WebOb App Look Like? Lately I’ve been writing code using WebOb and just a...
- Avoiding Silos: “link” as a first-class object One of the constant annoyances to me in web applications...
- WebOb I’ve have it in my head to extract/rewrite parts of...
I like how not just the code, but the response itself is much shorter and simpler.
HTML5 doesn’t eliminate the need for server-side validation. There are four major reasons for this:
Client side validation is a convenience for users. It does not absolve developers of any work. (Except writing JS calendar pickers.)
I assume Ian’s mention of HTML5 Forms refers to the somewhat distant future when you will be able can expect that support from the browser.
And when the browser supports validation, the server-side still has to check the data, but the requirements on how sophisticated the error reporting has to be are much lower: it will only come into play with 1) legacy browsers, 2) malicious user-agents, 3) buggy user-agents, 4) buggy forms. (Even though I don’t think (4) is of validator’s concern at all)
Isn’t the right approach to assume all user-agents are malicious, and that all user input is invalid until proven otherwise? A whitelist-only approach, denying everything that doesn’t match expectations, will still be the most secure approach to data validation. I don’t see how HTML5 will change that.
The checking of submitted input is only a very small part of what form-generators do. On it’s own it’s very simple. What happens when the input is invalid is a different matter. With client-side validation being the norm the other parts of what form-generators/validators do will essentially not matter.
You don’t have to be nearly as graceful about how you report those errors; you still have to check, and I’m pretty sure HTML 5 has ways to do server-side checking as part of its normal workflow, but you can mostly punt on failures.
HTML5 only has one workflow for server-side processing: the client POSTs or GETs the data to the server, and the server does what it wants with the data. No APIs or libraries for server-side code are specified in HTML5.
We actually moved (or at least are in the middle of a move) from an lxml.htm-based htmlfill system to a more “pull-based” one. I was definitely of your opinion when we first started. I thought the designers would just create the templates, and us programmers would use htmlfill to render them.
As the project went on, it turned out that the UI folks created a library of widgets that they used to compose larger pages anyway. They didn’t want to hand-roll every form, they wanted the ability to change a widget in one place to get consistency across forms. In the meantime, the backend programmers were indeed using FormEncode schemas to do validation, and those schemas embedded knowledge about form element names too. So we had to match up the two homegrown systems “by eye” a lot of times. When something went wrong (let’s say the htmlfill thingy couldn’t finish its job due to a missing input field or whatever), the backend programmer had read the indirection code written by the frontend people that pulled together widgets into a form, and had to match that up with the schema data they knew of. This was often quite painful and brainbusting, in a bad way.
The logic inside “questioner” above is pretty shitty to test due to all the conditionals. This was a common theme when we used the old system. There’s an urge once you want to test it to flatten it out a bit. The bit of code above looks lovely in isolation, but once you have hundreds of such functions, and many tests to support them, it’s no longer so lovely. You begin to look for ways to make it less ad-hoc.
We moved over to a system that actually generated forms using widgets, and much of the boilerplate went away. It still sucks, but it sucks slightly less.
There’s definitely merit to the htmlfill based solution, and it will get you a long way. But it doesn’t solve world hunger or something. This problem is just a crappy problem, and sometimes throwing framework at it isn’t the worst thing in the world; often it’s more reasonable than not doing so.
I have one customer project in which a form framework (zope.formlib) is absolutely essential. The customer has a declarative way to declare the forms, which can have sometimes very complicated widgets. The customer doesn’t need to worry about how validation happens, unless they want to create a new widget. This makes the maintenance of the forms and the construction of new applications within that framework tremendously more easy. Since this is built on zope.schema, I can also export and import to XML using the same schema description as the one used by the form, another requirement of the application. This project is so much about forms it has forms in its name.
I have some customer applications in which form frameworks are less of a total killer solution. This is when the forms are mostly small, consisting of a few fields. I wouldn’t say they’re useless even then – I still have to worry less about the validation of form data and the conversion of the form data to useful Python values, and error reporting, and so on.
I’ve been using form frameworks for about 10 years now. I’d say they’re frequently useful, though none of them are optimal handling all use cases. But it’s a fallacy to think that a non-optimal framework is useless.
Oh, and when HTML 5 is there, we’ll replace our complicated server-side widgets with less complicated client-side widgets where appropriate, and we don’t need to rewrite code as we generate our forms.
I’ll note that the state of client side javascript is such that this is already possible today, though of course much of the complexity will move to the client side.
So I was chatting with Ben Bangert last night after posting this, and we were kind of thinking about what it would be like to embrace this no-framework stance. It’s not the same as a no-library stance. Clearly even in that little
validate
function I wrote there’s room for abstraction (e.g., around the required fields). It’s the programmers’ prerogative to make their code better, and that implies introducing the appropriate abstractions. But we don’t write frameworks to do looping or conditionals (we usefor
andif
), and my assertion is that we can approach this problem without frameworks as well. But we must be willing to approach it with a clear mind, we must reject requirements that are antiquated or invalid, and we must pursue requirements that are attached to direct needs, ignoring derivative requirements based on experience with past implementations. But I have a strong intuitive sense that there is a much better solution waiting for someone to find it, and that it won’t be a framework.I find the distinction between a library that helps you construct forms and a framework that helps you construct forms hard to understand. Do you mean by “library” something you call yourself, and that you never register any classes or functions with?
Or do you just call what you like “library” and what you don’t like “framework”, with some intuitions about complexity and composability and who calls who most often and how much work you need to do to integrate it with your way of thinking and your apps and a lot of handwaving?
I am sure that better form frameworks/libraries are possible. I also quite enjoy some of the current frameworks/libraries, antiquated and invalid requirements probably included.
Admittedly the distinction is vague in my own mind, and the line is more intuitive than formal. As a general rule of thumb I’d call something a framework if it calls back into your code, but that’s not necessarily a hard rule; there are examples of such callbacks that I believe are benign (though usually that is self-evident in context because they are also trivial). Though even [os.path.walk](http://docs.python.org/library/os.path.html#os.path.walk) reeks of frameworkness, so it’s not size per se. And probably locality of logic and effect is more important, callbacks are just a code smell related to locality.
there are examples of such callbacks that I believe are benign is kind of illustrative of your attitude concerning such callbacks in general, I take it?
I think it makes sense to keep the discussion more concrete instead of “framework bad, callbacks bad, except when good”. That’s very caveman, not all cavemen necessarily agree with that evaluation. So instead of framing the discussion that way, we can probably more easily talk about concretes.
So let’s talk form systems
One form system (approach A) could work as follows:
we have a description of a form
this description is used to render the form to HTML, including filled in values
this description is used to validate the form input
this description is used to convert the form input into Python objects
this description is used to render feedback messages (validation failure, etc)
the user code is called with the validated, converted form result
There is not much in the way of callbacks into user code going on in this system, unless the rendering/validation/conversion system needs to be adjusted by the developer (create a new widget, say). I know you don’t like many of these systems anyway, I think because there are too many abstractions that you feel you’d need to be able to diverge from in your own code.
The other bit is that the user code is called with these results. That’s callbackery, but it’d be fairly easy to use these systems without such behavior.
Another form system (approach B) works as follows:
we have a description of the form HTML (assisted by a template language).
python objects are pushed into the HTML using the template language.
conversion of form contents from the request can be done by hand, possibly using a few library functions
similar for validation
In this approach, since widgets don’t exist, that area of callbackery into user code is gone. Of course opportunities for reuse are also gone, unless you bring back an abstraction to render a form element, with a well-defined set of inputs, and convert it, and validate it.
I’m interested in considering a form system like variety A which has a very strictly defined data interface (say, JSON) and builds its widgets using a push template language so that they are vary hackable, with as little between the data that goes into the widget and the widget template as we can get away with. It should also has an API that while of course allowing callbackery into user code, exposes its building blocks so that a library-like use is really obvious and short.
Of course all that is a lot more subtle than “throw away your form framework!”. I appreciate frameworks in a way you evidently don’t. I also appreciate good library design so I’m very interested in what you have to say. That’s why I come off so argumentative. :)
Maybe it’d be more useful to talk about where the form ‘frameworks’ fail? I generally get pissed when the following conditions occur while trying to use any of these magic form generators:
Obviously that last one is my biggest gripe. It’d be nice to have a form library that was more what Shannon Behrens has mentioned before, small sets of form ‘macro’s that do little bits of the whole. So you can use less or more as desired, and its more obvious how the whole is composed of just the little parts. Except for formish, none of the framework toolkits seem friendly when you just want to functionally construct a little form. In that respect formish feels more like a library than a form framework.
I totally agree with those gripes. Especially of course the complexity and hackability one – form frameworks tend to be very magic. I think that’s Ian’s point: there are so many edge cases and special features that form frameworks can’t handle them all or is buggy and you end up with more complexity than hand-rolling something instead.
I just wanted to move away from framework hating to actual considerations about what makes a form library/framework/architecture work better.
Honestly, to really figure this out I’d want to sit down with maybe a dozen fully thought out examples from realistic situations, both simple and complex, and start to brainstorm ways to solve that whole set — satisfying the complex ones without sacrificing the simple ones. And without shying away from ad hoc code in the hard situations. Combine that with a few design principles, and I’d be curious what would come out…
With further thought, I think another tension I feel is that I like things to be what they are. For instance, I hate stuff that compiles into Javascript. Similarly HTML is the canonical form of HTML forms.
That sounds like an interesting project. Except I’d have to part ways immediately with design principle #1 (the rest are excellent). If you accept schema-definition as a mental model for what your form toolkit is doing (clearly some don’t prefer it), declarative classes are a natural and intuitive way to declare a schema. And why shouldn’t schema-definition to be a static module-level concern?
Nor am I concerned about intermingling code and data; in fact I think the less distinction there is between the two, the better.
I definitely share your skepticism about form libraries / frameworks. Modern client side frameworks seem like a better way to handle dynamic forms. When I say “client side framework”, I mean things like dojo toolkit and the google web tookit which have their own templating systems, support for data stores, form serialization, etc. I wouldn’t really include jQuery here because it is more of a “library” and less of a “framework”. Following the approach advocated by these frameworks to the fullest, the server simply becomes a dumb json (or other data format) sender / receiver. It makes for a much cleaner separation of presentation / logic imho. Takes a while to wrap your head around the concepts, but once you dive into it you’ll have an “aha!” moment and wonder why you ever did things any other way.
I agree that client-side forms are an interesting area of exploration. It still needs careful design, though the design moves a lot to what the client-server interaction looks like, and of course client-side architecture, away from the server. Especially the consideration of the client-server interaction can hopefully expose the underlying data in a structured manner and make things more hackable.
In my opinion your code’s apparent simplicity comes primarily from the simplicity of the problem that needed to be solved – and not from the methodology itself.
Jacob Kaplan Mosses example has a logical layout with a self documenting nature and an internal structure that another programmer can pick up on and work with.
Your code on the other hand can only be modified once someone has read an understood every line of code.
I, myself burnt my finger with toscawidgets dynamic forms when I am developing a pylons application. I could have used simple html tags. There was no way to bend the form, the way I like.
But I learnt the hard lesson when I want to modify the form contents. I have spent more time to hack and learn the spec.
Simple hand written html form is easy to maintain and easy to read at the end of the day if we have isolated html and css/styles.
In my company, we were used to develop our forms using a home grown form generation framework. But what Ian mentioned many times in his blog happened. We were constantly trying to work around the limitations of our library to correctly handle side cases, constantly fixing bugs, and constantly wondering how to document a such complex library.
Then we have read “On form libraries”, the Ian’s post. It convinces us to ditch our form generation library. We replaced it by a more “low level” solution:
Conclusion: We are very pleased by the solution Genshi + HTMLFormFiller + FormEncode + jQuery + AJAX submission. The code is easy to read and to maintain by a third party. It is highly flexible. And it is well documented and almost bug free.
@Nicolas, I’ll love to see that released as Open Source.
Ian, I’ve been exploring some similar thoughts regarding the need for templates languages in Python web dev. I approached the arguments for using templates with a clear mind and concluded they were mostly antiquated / invalid dogma. Here’s a little rant / example module arguing that Python developers would be much better off writing their presentation layer in plain Python: http://bitbucket.org/tavisrudd/throw-out-your-templates/ I couldn’t figure out what to name it until seeing this blog post today ;)
Related: http://pypi.python.org/pypi/mext.htmlgen
I can’t wait for HTML5 to finally do away with frameworks. I’ll leave all the Tempita and other languages to the programming professionals like yourself. As mainly a front-end designer it’s interesting to see where the future is going with the both ends of the web and frankly it can’t come any sooner.