Bill de hÓra and then Patrick Logan picked up on an old post of mine about monkeypatching.
Patrick’s reply:
I know next to nothing about the specific problems the Ruby and Python folks are encountering with "monkeypatching". However this capability is nothing new for dynamic languages. And it is a frequent desire for me when I program in C-like languages. If you become frustrated using static "utility" methods, for example in Java, that work with "closed" classes (say, String or Object), then you have at least some desire for these "monkeypatches".
See the thing is this capability is a cool feature in many Lisp and most Smalltalk systems. Sorry, dear readers who hate my Smug Lisp Weeniness. But it is true. Not only is it "cool," moreover it is pragmatic.
The truly good implementations of dynamic languages recognize the advantages of these kinds of extensions, and they’ve supported them with good tools for decades. Learn from it, don’t run from it.
Sharp tools are good. I would not want monkeypatching removed from Python. Still, it’s best not to leave sharp tools lying around. It’s best not to mix your butter knives with your steak knives. I don’t resent the safety guards on circular saws.
And sorry Lisp Weenies: your experiences are not so novel anymore. The Python community isn’t new to this dynamic typing thing. We’ve taken some hits and we’ve learned from them. And frankly the problems with runtime patching of methods can’t be specific to Python or Ruby. It only took the Ruby community a couple years to start catching on. Are you telling me Lisp and Smalltalk programmers still haven’t figured this out? Everything you value about modularity is at risk when you monkeypatch code. That risk can be worth it, of course! But do you really need me to explain the benefits of modularity? What’s next, a recap of the problems with GOTO?
One of the things that I think distinguishes Python among the popular dynamically typed languages of the day, is that it’s built — languages and libraries — on a great deal of concrete experience. Experience about developing with Python. There was a time when people tended to define Python as a delta from Java or Perl or C. We don’t need to do that anymore. Sure, closed classes in Java suck. Python isn’t a reaction to Java’s suckiness. That we can do something Java can’t doesn’t get me excited. This feature of monkeypatching has to stand up on its own, and while sometimes its use is justified those cases are few and far between. That’s what we’ve learned: monkeypatching was not dismissed out of hand, it was not dismissed because of anything in Java, it was dismissed because people used it without acknowledging it as a hack, and it sucked.
Of course the use cases are still there. Which is why people are trying new things to address these problems. One benefit of experience is that you know some paths are dead ends. We still haven’t figured out The One Right Path (and we never will), and maybe we’ve only traced out the longest path in a very long dead end in this maze of ideas we are traversing. Since I doubt the maze has any exit (nirvana?) it’s a valid debate about where we are trying to get at all. That said, I suspect we’ve out-explored Lisp. Lisp has been a worthy mentor, an intrepid explorer in his time, but he’s old and doesn’t get out much and only tells stories of where he’s been in the past. There are still things to be learned there, wisdom to be dug out of that environment, but Lisp and Python are not peers.
Automatically generated list of related posts:
- Workingenv is dead, long live Virtualenv! A lot of people have found workingenv useful, but it’s...
- pyinstall is dead, long live pip! I’ve finished renaming pyinstall to its new name: pip. The...
- More Sentinels I’ve been casually perusing Paradigms of Artificial Intelligence Programming: Case...
+1 on GOTO
I do not know python, i probably should. (It does not have lisps powerful macros does it?) But this sentence is a non-argument: “Lisp [...] only tells stories of where he’s been in the past.”
Lisp is still here. Did all those features that were in the story disappear yesterday? Of course pro-lispers repeat the same arguments, its the same language..
Frankly common lisp needs more ease of use in producing (portable) binaries and such. Maybe it need to allow uppercase in (normal) symbols, and a function that includes directly from other languages. It would be awesome if (#include ‘SDL.h) would translate all the functions in the straightforward way. (But not at all straightforward to implement AFAIK.) Also more direct useage of pointers would be nice.(and dangerous) Also something about lisp is that i dont get set, setf, setq exactly, very annoyingly it sometimes does and sometimes doesnt work if i setf an argument of a function. (I do not want to be functional in every case.)
Jasper: that Lisp is still here doesn’t mean it is actively exploring new grounds and tackling the outstanding issues in software development. I haven’t seen anything new coming from Lisp for a long time.
that’s because everything in lisp has been already invented – at least that’s the impression I get when I read lisp evangelists.
Well, I find it amusing when the LISP evangelists come down from the functional-programming heavens to smack us down with smugness.
You can say your programming language is the best in the world 10^89, and still don’t convince me, unless you show me why should I use it instead of Haskell, OCaml, or any of the other functional languages with saner (and more maintainable) syntax and broader library utility. Some let you program your own tools, and some let you have a functional web server in 10 lines or less. The former are OK while in the Academia, but when I work I want to use the latter.
Thing is, as Ian points out, LISP is stagnant. Smalltalk used to be, but with the new Squeak platform, you could argue it’s coming back to life. But LISP? Bless them…
If you think Lisp is stagnant I encourage you to check out [Clojure](http://www.clojure.org/). Functional programming, persistent data structures, concurrency support, full access to the Java frameworks etc.
Ian, you’re missing the point with regard to monkey patching and Lisp.
In CLOS, code does not belong to classes (only the data), so by defining new methods on generic functions, or extending them (:before/:after/:around), you don’t really hit that problem.
See “Warp Speed Introduction to CLOS” at http://eval.apply.googlepages.com/guide.html for a good introduction.
For the C++ people out there who want to become familiar with the concepts from their view, see the playful introduction at http://mikael.jansson.be/journal/2008/03/newbie-guide-to-lisp-oo
And then, of course, there’s always Peter Seibel’s excellent book Practical Common Lisp at http://www.gigamonkeys.com/book
Jasper, Why would you want to setf the argument to a function? Maybe what you /really/ want to do is something like this?
On the other hand, if you really want to use setf then by all means, go ahead! Stupid example follows (key point being the LET form).
LET is useful for introducing local variables. Do you often change the value of arguments to functions in other languages, too?
You need to properly understand how functions are applied to arguments if you want to be able to SETF them. (All arguments are evaluated, unless quoted). So, what happens if you do this?
Exactly, you’ll get the same value as before the call to add. Symbols are just names for the location of an object, not the object itself.
Anyway, [Practical Common Lisp](http://www.gigamonkeys.com/book) should cover it.
David,
Doesn’t functional usually imply “no state” and “no object-oriented programming”? (? is the prompt, = is the output)
I guess not!
To Mikael:
Yes, that’s the definition of functional programming. Pure functional languages like Haskell respect that, and use monads for I/O. LISP, on the other hand, is kind of a mixed beast, more powerful if you will, but with shortcomings nonetheless. Actually, my main complaints about LISP are:
Horrible syntax: maybe not for the initiated, but pretty grim for everyone else. Why, oh why, do I need 5 pairs of brackets just to define a function? Haskell does the same without the bracket-tax.
Dozens of implementations, roughly 50% of them commercial, most of them diverging little or nothing, and none of them a default ‘reference implementation’ (or at least herd-driver). LISP is in the same state COBOL, Pascal and Basic were 20 years ago. It’s a chaotic situation, and every other LIPS hacker seems to think his particular brand of compiler is the best (or, ‘why should I use Arc instead of CLISP? because Paul Graham says so? because it has smaller keywords for the same functionality? or maybe there is some juicier meat underneath I’m missing?’).
Not many mainstream libraries or frameworks, or anything for that matter: where are LISP’s Djangos, NumPys, RoRs, Symfonys, and a long etc.? And then, do they run in my particular brand of compiler? Maybe Clojure or some other ‘LISP on a powerful VM’ will fix this, but I’m still reluctant to pay the syntax-tax.
It’s a niche language. Yes, I know there are X or Y or Z quantity of happy LIPS programmers out there, but it still is a niche. In a way the LISP community is very similar to the Assembly community, gathering around particular implementations (read NASM, FASM, etc.) and trying to tackle the problem of creating full applications with the ‘most powerful language in the world!’ (and how can you argue Assembly isn’t the most powerful language out there, if it lets you do exactly what you want?). Which drives me to the last point:
LISP is powerful, but not empowering: LISP lets you jump through 1000 hoops, but how well does that translate to everyday programming? I’ve yet to see a single big commercial application done in LISP. That shows a tendency: people tend to use whatever makes them productive, not whatever makes their programs theoretically perfect. I won’t use LISP if it takes me 10 times as much time to deploy an application than doing it in Python/Ruby/C#/choose-your-evil.
Don’t take me wrong, I acknowledge my initial post was full of smugness itself. I’m sorry for that. But I hope this will help the hardcore LISP evangelists out there realise why we, the mere code-crunching mortals, stay away from LISP, Smalltalk, and any other such Academic languages.
Cheers, David
Ian, you seem to have misread Patrick’s quote above.
He doesn’t call the monkeypatching support a tool but a feature. The ‘tools’ he is referring to is that your code editors show you where these patches are applied in no uncertain terms and that you can query the system as to which modules have been patched and which modules do the patching etc. etc.
In the case of VisualWorks Smalltalk this means that in the browsers patched methods are shown in red instead of black and that various queries and operations on the patches are provided.
Re-read his last paragraph you quote.
David/#10,
I’m not quite sure I follow you on the “mixed beast” part — you mean a language has to choose whether to support object-oriented, functional, logical, declarative or $paradigm programming?
Adressing your issues:
Syntax.
Cool, I’ll give you that. It’s the price you pay to treat code and data uniformly, and being able to have macros. Still, it’s very much like Python’s significant whitespaces: you’ll love it in the end. Macros is a powerful language feature that enables you to keep the boilerplate code down to a minimum by adapting the language to your problem domain. Moreover, because of the regularity of language-defined functions (e.g. ‘if’), adding your own flavor of ‘if’ that additionally performs extra code will not stand out.
My favourite example is the addition of the keyword WITH in Python-2.5, though… (Hint: how many PEPs has to be written, parsers modified and releases rolled out to support that in Lisp?)
I can not recommend Peter Seibel’s [excellent Lisp tutorial](http://www.gigamonkeys.com/book) for getting past that initial hump. There’s also the friendly freenode|#lisp channel, and comp.lang.lisp (available on Google Groups.)
Implementations.
Yes, you are correct. There are many implementations of Common Lisp, instead of the one true reference design. Instead, the language is a proper ANSI standard, and defined by X3J11, or the HyperSpec. Just ask the PyPy and the Jython people how fun it is to correctly mimic bugs in CPython (i.e., the canonical Python) in order for code running in their implementations to work correctly.
Moreover, having commercial implementations is a good thing, because you get support when you have a problem. Sure, you have the source to Python, but how many are skilled enough to fix that bug with a missing Py_IncRef hidden somewhere in a Python C-ext module?
Libraries.
I recommend Edi Weitz’ code, at http://weitz.de/. There’s also, of course semi-official lists over at http://www.cliki.net/Library and http://www.cl-user.net/asp/tags/dev-tools
Getting productive.
Common Lisp is a large language. You can, of course, do much in it without learning it all. But, you still need to swallow a piece that probably feels larger than the C-like languages. Much of that feeling comes from–at least, it did for me–the fact that you’re not quite sure that if you start using these constructs, are you perhaps missing out on something that’s even more efficient? In the end, all that matters is to actually start hacking. You’ll uncover more of the language by hand.
(In fact, [some people (recent discussion on the topic in c.l.l)](http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/3cd44f31eb2b69bc/5381a0339e1f6847) say you have to be smarter/more passionate than your average Java code monkey to be able to grasp Lisp — after all, Java uses single dispatch, no closures, no macros, no lambda, no MOP, etc. But, saying that wouldn’t be politically correct and a wee bit to smug, so I won’t.)
@David: Read some more Paul Graham and you’ll hear various stories about successful, commercial Lisp applications.
For example, the biggest that comes to mind is Orbitz’s search engine.
Reinout: I noticed the tool aspect, though admittedly I didn’t really reply to it. Perhaps this is because I don’t rely on many tools for my development.
I think there is a lot in Smalltalk particularly, but also in Common Lisp, that is not transferable to an open source environment. When you buy a system and libraries and then develop on top of it and keep what you develop internal, you can do development where you are hacking around the system. Then you can build tools to make it manageable. But when you are creating reusable libraries that are being made public, released, and combined by different people in different ways, I don’t think the process works.
I also kind of talked about Lisp, because it’s clearer to me that there’s not a whole lot of ideas to mine from there. Some, just not a whole lot. Smalltalk probably has more stuff left in it. Though I do think Smalltalk has a core structure that is just incompatible with open source development. Making an IDE a core part of the environment is part of this — and part of what makes the tooling for this feature possible. But that’s another substantial topic of its own.
Ian,
As the author of the (excellent!) ORM SqlObject, you should know what it means for productivity to work in a higher abstraction level: Python instead of raw SQL. Now, imagine being able to abstract the actual syntax of the language away, and move even closer to your problem domain.
A good example of this is the (define-binary-class …) macro Peter Seibel (in the book Practical Common Lisp) creates for his ID3-parser. You essentially define a bunch of classes and functions, automatically, by just defining the layout of the binary file. Enter a bunch of clever macros which then fill the rest out for you.
Even though you might not actually start using Lisp, I strongly suggest you read Practical Common Lisp, as I believe it will make you see a lot of things in a new light. (It did to me, and I’ve written my fair share of Python, including a commercial TurboGears-based deployment, attended EuroPython and been quite the Python advocate.)
You seem to be saying ‘monkeypatching’ is a problem in CL, and that we haven’t figured it out. You think we haven’t figured it out ? Look at emacs! (elisp is a lisp, just as CL is a lisp, scheme is a lisp, and python, believe it or not, is widely considered a lisp by the lisp community). It’s the oldest example, and it’s used as the example of the right way to do it in the ‘monkeypatching is ruining ruby’ article you mention. (Lets forget, for a moment, that elisp is dynamically scoped :) ). No sane lisp programmer would ‘monkeypatch’ something and expect other things not to break. A sane api for extending functionality of pre-existing libraries (like the hook approach in emacs) is the way to go.
Ian:
In your older article you linked, you wrote that you don’t consider Perl a high-level language, merely a dynamic one, and that all it’s really about is regular expressions. Do you still hold that view? If so, can you give me any example of an abstraction that Python can facilitate [but Perl cannot](http://hop.perl.plover.com/ “Higher-Order Perl, by Mark Jason Dominus”)?
Well, maybe I should soften some of my statements. I see 3 general reasons for monkeypatching:
Something is simply missing from an API. An example in Javascript is that
Array.pushisn’t defined in some versions of IE, and some libraries monkeypatch it in.You want to add new functionality, completely separate from other functionality. Monkeypatching might be most aesthetically pleasing (i.e., a method instead of a function).
You want to modify existing functionality.
In the case of 1, it’s usually fine: you are ideally simply improving the situation. Of course it depends on the correctness of your confidence: if it is not completely self-evident how the missing feature should be added, other people are likely to add it in slightly different ways and conflicts will happen. I think a naivety in among Rails developers has led them to write a lot of code believing they are in case 1, when really they are in case 2.
In the case of 2 — I suspect the most common case in Smalltalk — it is fine at first. But you have to worry about naming conflicts, and some behavior — like fallback behavior for classes you didn’t know about when you constructed the patch — is only implicit, based on the inheritance structure. This works fine when you are working with code that is centrally developed (except for your own code). When there is a community of developers sharing code I don’t think it works.
However, Generic functions in Common Lisp solve both problems with case 2 well: fallback behavior is very clear, and each function is independent and in its own namespace, and located in a way that makes it clear who is responsible for the function.
In the case of 3 it’s quite dangerous. I don’t see how generic functions make it any less dangerous.
So yes, I agree that CLOS is an important and interesting system worth looking into, and generic functions are a notable alternative to methods for object-oriented programming. OTOH, CLOS isn’t getting any more interesting over time.
Now, as to something like Clojure: perhaps another parallel I could have made in the [article I linked to](https://ianbicking.org/because-unanswered-problems-are-always-hard.html) is that where some languages chose to accept side effects, and some didn’t, there is another divide just as substantial as intrinsic typing and source typing. Here I’m not sure what the procedural languages can do: either we’ve chosen the right path, or we haven’t. I don’t know if knowledge can cross over that easily, just as there’s whole practices of code for dynamic languages that don’t really cross over well to statically typed languages.
Aristotle: regular expressions were only an example, of course, not a summary of Perl. I think Perl sags badly under the weight of Very Bad Decisions, like references. A language without well behaved nested data structures is just silly. Of course you can write all kinds of high-level stuff with Perl. You can do so with C too. I don’t say that just sarcastically, as people have made complex dynamic systems with C. (Maybe it’s unfair to lump Perl with the much-less-expressive PHP.)
The “high level” in Python that I’m trying to refer to isn’t really that high. Not crazy programming techniques, just high enough that you can safely trust that all the basics of programs — functions, conditionals, loops — work safely and well, and you can focus on the stuff that is more challenging. So I would argue that low level abstraction problems in Perl are more concerning than the ability to do highly abstract code. Once you start using closures and metaclasses and other things you have to watch what you are doing, in either language. But I feel much safer doing these high level things when I know the low level stuff is robust. Perl hasn’t made it robust.
There might be a point when even these high-level abstractions are robust. For instance, I’d feel more comfortable with closures if they could be introspected better. That the state in a closer is just inferred (at least in Python) makes it an awkward abstraction. We move up by small steps, and “high-level” and “low-level” is really just a way of placing language on a spectrum specific to this current time in development. I would certainly hope that the high level of today becomes the low level of tomorrow.
I know the Python party line about anonymous functions is “Why would you ever want to do that?”, and I know the workaround for read-only closed-over variables is to use an aggregate, but I find the irony particularly delicious when a Python programmer complains that Perl doesn’t have robust “low level stuff” with regard to closures and metaclasses.
(The explicit referencing semantics in Perl 5 seem to me to stem from autoflattening in list context, which is a deliberate design decision. They’re an improvement over nested structures in Perl 4, but they’re definitely one of the least beautiful parts of that version of the language.)
chromatic: by low level stuff, I meant stuff like decent function signatures and nested data structures. Perl’s data structures are indefensibly awful. I spend most of my programming time fiddling with data structures, not doing metaprogramming. Even if Perl’s metaprogramming facilities were better than Python’s (maybe they are, I don’t know), so long as its basic programming facilities are so flawed it hardly matters.
Ian (#18),
“”"In the case of 3 it’s quite dangerous. I don’t see how generic functions make it any less dangerous.”"”
Quoting [Peter Seibel](http://www.gigamonkeys.com/book/object-reorientation-generic-functions.html) again:
“”"The methods I’ve been discussing so far are called primary methods. Primary methods, as their name suggests, are responsible for providing the primary implementation of a generic function. The standard method combination also supports three kinds of auxiliary methods: :before, :after, and :around methods. An auxiliary method definition is written with DEFMETHOD like a primary method but with a method qualifier, which names the type of method, between the name of the method and the parameter list.
[...]
Auxiliary methods are just a convenient way to express certain common patterns more concisely and concretely. They don’t actually allow you to do anything you couldn’t do by combining primary methods with diligent adherence to a few coding conventions and some extra typing. Perhaps their biggest benefit is that they provide a uniform framework for extending generic functions. Often a library will define a generic function and provide a default primary method, allowing users of the library to customize its behavior by defining appropriate auxiliary methods.”"”
Something like that, maybe?
Mikael: I see the extension of generic methods, at least in the safe non-monkeypatchy way, and using before/after/around, as more like subclassing and overriding methods. Monkeypatched code has the same patterns, though they aren’t explicitly named.
Ian,
You are correct.
In fact, monkey patching isn’t necessary in Common Lisp, because of/thanks to multiple dispatch (which follows from methods not belonging to classes, i.e. the reason for having to monkey patch).
It seems the problem doesn’t happen at all. I wonder what similar issues happen in Lisp?
rj, I’ve programmed Lisp in an academic context (toy programs) and I’ve written serious programs in Python (gps interface). I would never, EVER consider Python to be ANYTHING like Lisp. Ever.
Mikael Jansson: “As the author of the (excellent!) ORM SqlObject, you should know what it means for productivity to work in a higher abstraction level: Python instead of raw SQL. Now, imagine being able to abstract the actual syntax of the language away, and move even closer to your problem domain.”
Who says SQL isn’t the right abstraction level for many problems? Maybe the average “CRUD” application benefits from things like SQLObject, but the most significant complaint about most ORM solutions is how they reinvent SQL badly in the solution developer’s language of choice, and how you have to jump through hoops to write something that may be difficult in SQL, but which is natural in that language. I’ve done a fair amount of work with relatively large databases of late and I can’t see how hiding the database system (and its very useful query analysis tools) would have been beneficial.
“A good example of this is the (define-binary-class …) macro Peter Seibel (in the book Practical Common Lisp) creates for his ID3-parser. You essentially define a bunch of classes and functions, automatically, by just defining the layout of the binary file. Enter a bunch of clever macros which then fill the rest out for you.
I think declarative structure parsing is very nice, and there are solutions like this in Python.
“Even though you might not actually start using Lisp, I strongly suggest you read Practical Common Lisp, as I believe it will make you see a lot of things in a new light. (It did to me, and I’ve written my fair share of Python, including a commercial TurboGears-based deployment, attended EuroPython and been quite the Python advocate.)”
I hope you’ll make it back to EuroPython this year, too. ;-)
Expanding on the SQLObject reference: certainly SQLObject uses a number of metaprogramming techniques, and especially when it was written quite a few of those techniques were very novel. I am glad I had those new features (in Python 2.2) to aid in that development. But I also know that, in retrospect, many of the techniques I used, I wouldn’t use if rewriting it. Of course I still would use many of those techniques, and it’s hard to know ahead of time which will work well, and when they will work well.
This is really what I’m referring to: the wise use of advanced features is essential. Python, Common Lisp, Ruby, etc, all give programmers the ability to create very clever and unmaintainable code. I wouldn’t blame any one feature for this either — for instance,
evalis very dangerous, but I certainly wouldn’t remove it. Runtime patching of code is similar. Or, to give an example from SQLObject, I do operator overloading there in a particularly peculiar way, using operators to build expressions instead of calculate expressions. Generally I would say that’s a dangerously clever technique. Specifically for that problem (constructing SQL expressions) I think it works well. Experience and experimentation is generally the best guide for figuring this out. But that experience doesn’t have to be entirely personal; the community has experience and can offer important guidance.As an example in Ruby and Python, community experience will come down on anyone using
evalfor anything that doesn’t absolutely requireeval. In Ruby it still is seen as a valid metaprogramming technique. I don’t know the Common Lisp community well enough to know what community convention accepts and avoids, but I’m sure there as in any community there are particular standards.My gut feeling is that eval is used because there is no proper support in the language to achieve whatever you’re trying to do with the code passed on to eval.
Generally, you don’t want to use eval in Common Lisp, but among newbies it’s common to think it should be used in many places where you should do it in another, easier & cleaner way. Have a look at http://www.faqs.org/faqs/lisp-faq/part3/section-13.html (“When is it right to use eval”) for more info.
Same thing goes for macros: newbies tend to think of them as working like their crippled relatives in the C world, and as such use them as a way to inline code. With strange results. :)