Ian Bicking: the old part of his blog

On form libraries

Some time ago I wrote a post on form generation. It mostly came from my experience with FunFormKit, early versions of FormEncode, and then what FormEncode is now (which is strictly a validation library).

Now there's some new form generation libraies in the works; ToscaWidgets, Django newforms, and I hear rumors of a few others. I feel bad poo-pooing other people's work, but I don't think this approach is a good approach.

A form generation library is a hard library to write. You have to balance complexity and features. There are a lot of potential features. How do you tweak the forms? How do you control layout? How do you handle repeating form elements? How do you handle elements that require Javascript? Server-side resources like CSS and Javascript? Server-side communication like auto-complete? A simple library cannot handle a lot of these. But very simple forms don't need form generation libraries. So a simple library only handles a narrow band of use cases. A complex library can potentially handle all these cases, but can be difficult to use for the most common cases.

I will assert that as the libraries are usually designed, there is no good balance. Not all problems have feasible solutions. Form generation libraries are one such problem, but they don't seem to be such a problem.

There's a cycle going on in the library development. A developer has certain requirements. Usually they also hate HTML, and may very well be working with an awkward templating language, and maybe they aren't very excited about the task at hand and building a library is a pleasant diversion. But anyway, they Do The Right Thing (or so they think) and try an existing library. It's complex, it doesn't match the way they think about the problem, the bugs in their code are hard to figure out, and adjusting the things they (or their boss or client) cares about is difficult.

And they think "I can do this so much more simply!" The developer is kind of right. They know exactly what they want to do (now), and they can improve their productivity some.

The problem is that no library ends there. It can't end there. Because someone uses the library, and the code must be maintained. Form generation is a hard thing to maintain, because it has to be actively maintained. New fields are added. New cases and exceptions come into being. Anything that touches UI can be redefined in weird and unexpected ways. Good UI is usually eclectic and nonsensical from the perspective of a programmer. UI should be consistent with the user's notion of the domain, not the program's internal modeling of the domain, and it is uncommon that the two really match up.

So you get new requirements over time. A straight list of checkboxes used to be reasonable, but then the number of options expanded too far and now you need two columns of checkboxes. So you add in a little bit of code, both in the library and the application, to handle that case. You want to have a text field that contains its own label when its empty. The validation becomes increasingly complex. Maybe you have a multi-screen form, or maybe a form with optional fields, or who knows. So you add code to the library and the application each time (if you control both of them). It gets more and more complex while at the same time the developer's familiarity with the library code becomes more and more remote. And that's not even to mention when a non-programmer wants to tweak the HTML, or there is developer turnover, or when you don't control the library at all...

It's a mess. And so someone comes along and says "I can do this so much more simply!" and it starts all over again.

The problem domain is wrong. I think there's a better approach, but I'm too lazy to finish it. So I'll lay down the design here, with little expectation that anything will happen because of it, but just maybe this will connect with someone. (Ideally someone who will write code ;)

First, validation: I like how FormEncode works, I think it's the right model. Validation completely separate from the rest of the system, and you can programmatically construct the validation so you can embed it in other systems if you want (and embedding it is probably a good idea).

Second: forms. First, the simplest model: write it in a normal template. But this is annoying people will cry. Why?

  1. Filling in defaults is annoying. You have to do this little dance: is the value in the request (e.g., resubmission)? Then use that. Is it passed in through some defaults? Then use that. Otherwise? Maybe another default, or blank.
  2. Filling in errors is annoying. Similar dance to defaults.
  3. The <label> tag is somewhat annoying to generate, which is perhaps why many very high-traffic websites still don't properly do labels.
  4. I suppose the HTML is annoying in general, though it's only slightly more verbose than the programming code that might replace it.
  5. Javascript can be annoying, and many web developers still aren't comfortable with Javascript.
  6. Getting all the right Javascript and CSS files in place, and linking them up with the appropriate fields is awkward.
  7. Ajax callbacks are really annoying.
  8. Nested and repeating fields compound every other kind of annoying.
  9. It's hard to reuse any of this.

My response is that these issues can be resolved without form generation libraries. formencode.htmlfill addresses 1 and 2. It's not perfect, but it's usable. Things like WebHelpers can help with 4. Neither of these needs to be the implementation, and both could be improved. But while remaining in their scope they can only be improved a little, which is important. They will not become endless time sinks.

Unobtrusive Javascript solves 5 quite well. I think a tool for solving 6 -- the proper inclusion of dependent CSS and Javascript -- could be handy. I imagine <input type="date" js-require="DateInput"> calling some code that figures out where DateInput is (probably building on something like JSAN package metadata) and puts the appropriate code in the head of the page.

7 and 8 are addressed quite nicely by the Web Forms 2.0 WHAT-WG specification, and tools building on those models would be excellent.

And once you have all this, I think 9 is much more approachable. Because you don't need a framework, you just need stuff that generates blank HTML that also uses a few of these simple conventions.

Notably, these techniques embrace the abstractions HTML and Javascript already allow. Also, each tool can be useful on its own, and can be used, not used, or reimplemented on its own. The hard part is that any one tool seems terribly simple and lacking features compared to a form generation framework. But if slow and steady can win the race it might not matter; the other form generation frameworks will suffer the churn of the reimplementation cycle while these simpler and duller tools will better stand the test of time. And even if they don't, unlike a framework the healthy decoupled set of tools can suffer the failure of one piece (in design, maintenance, or implementation) without compromising the entire stack.

Now someone please go build the missing pieces, because I'm trying really hard not to start more projects.

Created 28 Mar
Modified 11 Apr

Comments:

Ian Bicking: The code libertarian.

Besides that (ahem) gem, I pick a nit: nonsensicle => nonsensical.

# Bill Mill

What's your take on WYSIWYG form builders and related libraries in Java and ASP.NET?

# Scott Tsai

I've never tried them or even seen them. It seems reasonable, I suppose, though any framework integration would set off a warning bells. Insofar as you are adding custom attribute to the form elements, it's reasonable. At least what I've seen in ASP.NET seems to do this. What I don't like in the examples I've seen is that it seems to also do control flow through the form itself. That's awful. Templates are good, but they aren't good for control flow. And putting control flow in form libraries is the kiss of death IMHO.

I think form generators aren't all bad, and potentially something less general than a full templating language could be called for when generating large forms (though of course any templating language can be used to programmatically create forms -- there's doesn't need to be any dichotomy between generated and hand-written forms if other tools work on the markup and not any underlying models).

# Ian Bicking

I was just contemplating writing a set of HTMLish widgets for Glade as I read this. They would produce HTML or Mako or something like that. My personal view is that if there were decent UI form designers for HTML, people wouldn't become obsessed with building this particular magic bullet.

# Ali

Agreed on the form generation vs. validation front. I spent the last couple weeks working with ToscaWidgets and while it clearly has an answer for any form I can imagine, it's incredibly frustrating to re-learn how to write a simple HTML form that ends up looking far more elegant in its native markup syntax. Django's newforms was much less frustrating for me, but I dislike the generation aspect in general.

# Brian Beck

I am in total agreement about form generation libraries, and this was something i didnt like about early FormEncode. This is because ive been building forms with all kinds of server side languages for a really long time, and have many times gone down the road of building "generic form generation" systems, as well as using a lot of the early losers like ATG dynamo and stuff like that. its the same 80% solution tune that plays all day long...or at least 80% of the time.

Fortunately for me, I always worked in environments where the non-feasability of form generators was always super-obvious: id get handed enormous HTML redesigns of my forms from designers and had to work around that.

Still, the need to have the ability to just say, "text input element!" somewhere on a page without going through the motions of the repeating the individual HTML and site-specific customizations of that element, is very important. thats why I was so apeshit over HTML::Mason, and then Myghty, and upon Myghty crumbling under its own weight I reimplemented it as Mako: you need to be able to make custom, reusable, nested, inter-aware widgets, in straight templated HTML, immediately.

so ill keep saying it like i always do: the "templating" part is not so hard. With Mako, you create any number of templates containing your widgets as <%defs>, and then you can <%call> them, using whatever nesting schemes you desire. It really works frigging great...its the model ive used for years and component-call-with-content is the "killer feature" of these systems. Its the ability to create custom nestable tags without any python code or external modules. no other template lang ive seen in python can really do this (except myghty of course).

I have been playing with toscawidgets for a gig recently, and its OK but its the same thing. Theyre making me put HTML in my source code, making me dispatch it through confusing classes and datastructures...and thats just not where it belongs.

# mike bayer

Mike, care to add a howto to docs.pythonweb.org about form generation with Mako?

# Max Ischenko

I'm really a fan of treating form libraries as sets of macros. There are higher level macros (perhaps a macro that can output an entire label and field) and lower level macros (just a checkbox, or just the code to output an error). By treating the system this way, the user can get super custom if he needs to (by using the low level macros or even straight HTML), or he can whip stuff out really quickly (by using the high level macros). Need something different? Add more macros based on the lower level macros.

I've used this approach before at multiple companies in developing the common look-and-feel library that is used by several applications. It's a simple idea, but it works out well.

By the way, I too tend to keep form generation separate of form validation. I'm not tied to that idea, but it's worked out well for me in the past.

# Shannon -jj Behrens

I don't really have much of a point with this, just some observations.

One is that a form generation library can have independently evolving components. Let's take the Zope 3 has a form generation library. I'm sure you don't like it. :) It has multiple components that work together:

These components can evolve by themselves. formlib for instance is reasonably new, replacing another system for putting a form together that didn't work as well. It's built on the existing schema system and widget systems, however, so those weren't rewritten. It would also be possible to revise the way the widget system work without an enormous impact on the other pieces, though formlib would likely need some adaptation.

There are two very different extremes of dealing with forms. One is schema-driven, the other HTML driven. Most systems fall in the middle somewhere (which probably creates some of the muddle); Zope 3's system leans fairly heavily towards schemas (which introduces its own limitations). I see benefits in both approaches: HTML driven is very good at being able to adapt to a wide variety of real-world forms. Schema-driven is good at producing lots of forms from small descriptions of data and encourages widget reuse.

It appears that to you the tradeoffs weigh in favor of HTML driven forms. To me, it depends in what you're trying to accomplish.

Note by the way that ToscaWidgets is not necessarily a form generation system. It's a way to package and reuse bundles of python, HTML, CSS and javascript at its core. It has a form library built on top of it.

Finally, I have the suspicion that any set of abstractions that helps you deal with complex real world scenarios in any domain will be vulnerable to your criticism of form generation libraries, including your own proposal on how to handle this particular problem domain. :)

# Martijn Faassen
There are two very different extremes of dealing with forms. One is schema-driven, the other HTML driven. Most systems fall in the middle somewhere (which probably creates some of the muddle); Zope 3's system leans fairly heavily towards schemas (which introduces its own limitations). I see benefits in both approaches: HTML driven is very good at being able to adapt to a wide variety of real-world forms. Schema-driven is good at producing lots of forms from small descriptions of data and encourages widget reuse.

I'm not really opposed to creating forms programmatically. I just think it should be a simpler task than what most form generation libraries have bitten off. It should look a lot more like templating, and a lot less like programming. It shouldn't include things like filling in defaults, filling in errors, collecting Javascript and CSS, and maybe not layout (or at least very simple layout). Once the definition is sufficiently limited you can get away with a very simple and transparent library for form generation. And certainly that is important, since lots of domains require form generation -- stuff like Django's CMS, or user-driven models generally.

As to formlib, I'm afraid I don't really know much. I've seen Zope Schemas, and I think they are fine, though as I remember they did validation without conversion, and I think two-way conversion is integral. Combining validation with widgets and worse data bindings is a really bad path, and you didn't go down that. I'm a little wary of data bindings in general, as it puts the logic in the framework and I hate that; it makes my head hurt. But if it's done really well it's possible -- as long as you aren't squeezing it into a normal web app framework design. I suppose htmlfill is a kind of data binding library; but it's very clearly a library. Anyway, no one will accuse Zope 3 of being a normal web app framework design, so no problem there ;)

# Ian Bicking

I'm sympathetic with your general idea that much that is now done inside widget libraries should be pulled out into independent systems. It will need some thought on my end.

Ian:

As to formlib, I'm afraid I don't really know much. I've seen Zope Schemas, and I think they are fine, though as I remember they did validation without conversion, and I think two-way conversion is integral.

Zope 3's design considers two-way conversion the responsibility of the widget, as it's part of viewing logic. Conversion might be quite different if the data is coming from another source than the web, or is going to another target than the web, after all (an int might already be an int).

Ian:

Combining validation with widgets and worse data bindings is a really bad path, and you didn't go down that. I'm a little wary of data bindings in general, as it puts the logic in the framework and I hate that; it makes my head hurt. But if it's done really well it's possible -- as long as you aren't squeezing it into a normal web app framework design. I suppose htmlfill is a kind of data binding library; but it's very clearly a library. Anyway, no one will accuse Zope 3 of being a normal web app framework design, so no problem there ;)

Zope 3's more normal than it looks like, it's just not presenting itself well. :)

As to data binding, what you mean by this is the code that gets the data to display in the form from somewhere and then gets it back from the web into the application, right? I believe in Zope 3 that's scattered a bit between formlib and the widget system. It'd be nice if it were a more coherent library. It's quite powerful in the sense that it handles nested and/or composite form values well (which can be expressed with Zope 3 schema).

# Martijn Faassen

I think unobtrusive JavaScript should solve numbers 5, 6 and 7. The form should be in no way dependent on JavaScript to function, so that should not be a part of any for generation library (apart from giving the form and its fields sane #ids or .classes to use as hooks from the unobtrusive JavaScript).

Once you've got the form created, you can swoop in and make the form turn cartwheels (including AJAX cartwheels) using JavaScript, which can be included in the <head> of the document from whatever template the form will be sitting in.

# Will

I'm actually pretty lazy when it comes to Javascript fallback. This perhaps is a personal failure. But as an example of how I would deal with date inputs, I'd just throw this in: <input type="date">. Let the Javascript figure that out (which is easy enough). If the Javascript doesn't figure it out, you get a text input and rely on server-side validation. A "fancier" system just knows you don't have Javascript date inputs, and it falls back on select boxes. Which probably don't make anyone any happier than the text box, but somehow it feels like you are trying hard instead of being lazy.

In other cases, like repeating elements, handling that server-side makes my head hurt. It can be done, yes, but it's going to hurt. You'll have to take control from the normal controller and do lots of fancy things behind the scenes. The alternative is not writing the form anything like that, and forcing another half-dozen clicks by the user to do the same thing. That's not a compromise I get excited about either.

I'm not sure exactly why you should make forms that are usable by people without Javascript. I don't think it's accessibility -- I believe "accessible" browsers also interpret Javascript. How well, I'm not sure, but I don't get the impression they really feel like Lynx. If it's to make your site accessible to the Googlebot, well I don't really care about forms. If it's because some people are just stubborn and turn off Javascript, eh. I'm not sure it's worth it.

# Ian Bicking

"I'm not sure exactly why you should make forms that are usable by people without Javascript."

Strong words, and quite shocking in certain ways, too. Many of the JavaScript-only Web applications I've seen aren't likely to be very accessible, and what about people disabling JavaScript for security or usability reasons? To me, JavaScript (and AJAX, which is where a lot of the JavaScript goes these days) is a bonus for those who want a bonus, not a minimum requirement that excludes a number of people for reasons not fully considered by the developer.

# Paul Boddie
Many of the JavaScript-only Web applications I've seen aren't likely to be very accessible

That doesn't mean that you need to bar Javascript in order to be accessible.

and what about people disabling JavaScript for security or usability reasons?

I don't think that makes any sense in the modern web. I don't see any reason to aggressively make things harder for these people, and lots of things should work without problem, but I don't think that's a user choice that I'm going to work to support. There's lots of choices users can make, I can't account for all of them.

# Ian Bicking

I'd like to make a quick edit on the post and some comments to get the package names right ;)

s/ToscaWidgets/twForms/g

I find rather disappointing that one of the points I try to get through in the scarce TW documentation (actually, the description of the package) rarely gets through as intended. ToscaWidgets does not handle form generation.

It only aspires to:

I actually believe it's flexible and powerful enough to serve as a good base to build the dream form library you're sketching out if twForms doesn't satisfy you :)

(I really need to get some docs out soon...)

Alberto

# Alberto Valverde

I've used FormEncode and the associated htmlfill for some time now and have been really really happy with how it works out. When you do complex applications, and you need to be hand editing your html page for specific layout/javascript interaction and it's a godsend when the framework for validation and loading defaults on the page (#1 & #2) just stay out of the way.

htmlfill does a really good job of putting default values on the forms; and formencode is a great way to validate and get those validation errors back on the page (along with setting appropriate display classes on the page). It really understands html as it should, and doesn't mind if it happens to be html that comes through some other framework or page templating language. It's almost like they are just tools and not a framework at all, because they don't ever get in your way when you want to do complex things; they make the complex possible. Personally i use Zope Page Templates to generate my pages (#8) and that helps put some of the dynamic elements on the page; but formencode does quite a bit of heavy lifting; and it is quite up to the task.

...all that said, HTML, Javascript and CSS (#4 #5 and #6) can be annoying until you really have a strong grasp of all three and are comfortable with them all. When you reach that level things start to fall into place, and you realize that HTML, Javascript and CSS are powerful frameworks for doing their respective pieces, and that they can be easier to use directly instead of with some clumsy middle man between the keyboard and the screen

# Jehiah

We have been using Zope at at Avand as the basis of our document management system since 2000, and we have a mature widget system which does client-side and server-side validation (including spell checking as a client-side validation module), handles javascript dependencies (so that scripts aren't imported multiple times), creates complex widgets ... see screenshot

http://www.avand.com.au/user_images/clip_image002_011.jpg

The widgets rendered are the XP style list on the left, the two lists on the main panel, the Display options pseudo-dropdown(which persists user preferences as cookies).

I haven't checked out Tosca, but there might be some potential to cross polinate ideas. The code is fairly tailored towards our needs, and probably not suitable for reuse outside our own project.

# Chui