Ian Bicking: a blog :: Opening Python Classes

{ 2007 08 08 }

Opening Python Classes

So, I was reading through comments to despam my old posts before archiving them, and came upon this old reply to this old post of mine which was a reply to this much older post.

I won’t reply to that post much, because it’s mostly… well, not useful to respond to. But people often talk about the wonders of Open Classes in Ruby. For Python people who aren’t familiar with what that means, you can do:

# Somehow acquire SomeClassThatAlreadyExists
class SomeClassThatAlreadyExists
def some_method(blahblahblah)
stuff
end
end

And SomeClassThatAlreadyExists has a some_method added to it (or if that method already exists, then the method is replaced with the new implementation).

In Python when you do this, you’ve defined an entirely new class that just happens to have the name SomeClassThatAlreadyExists. It doesn’t actually effect the original class, and probably will leave you confused because of the two very different classes with the same name. In Ruby when you define a class that already exists, you are extending the class in-place.

You can change Python classes in-place, but there’s no special syntax for it, so people either think you can’t do it, or don’t realize that you are doing the same thing as in Ruby but without the syntactic help. I guess this will be easier with class decorators, but some time ago I also wrote a recipe using normal decorators that looks like this:

@magic_set(SomeClassThatAlreadyExists)
def some_method(self, blahblahblah):
stuff

The only thing that is even slightly magic about the setting is that I look at the first argument of the function to determine if you are adding an instance, class, or static method to an object, and let you add it to classes or instances. It’s really not that magic, even if it is called magicset.

I think with class decorators you could do this:

@extend(SomeClassThatAlreadyExists)
class SomeClassThatAlreadyExists:
def some_method(self, blahblahblah):
stuff

Implemented like this:

def extend(class_to_extend):
def decorator(extending_class):
class_to_extend.__dict__.update(extending_class.__dict__)
return class_to_extend
return decorator

Automatically generated list of related posts:

2 Python Environment Experiments two experiments in the Python environment. The first is virtualenv,...
The Shrinking Python Web Framework World When I was writing the summary of differences between WebOb...
Python HTML Parser Performance In preparation for my PyCon talk on HTML I thought...
A Python Web Application Package and Format (we should make one) At PyCon there was an open space about deployment, and...
Woonerf and Python At TOPP there’s a lot of traffic discussion, since a...

21 Comments

Michał Kwiatkowski says:

August 8, 2007 at 2:37 pm

The main difference between Ruby and Python here is that in Ruby you can modify built-in classes, while in Python (CPython at least) you can’t. That’s where all those nice railish 7.days.ago method calls come from – by extending base classes (Numeric in this case).

Of course this power comes with a cost. Because any piece of code can modify built-in classes, a single require can completely change the meaning of the code below it. In Python import statements are much more safe in a sense that you can expect what changes they will bring to the following code. In Python you decide what names will be imported into your current scope. In Ruby you have to guess (or better reference the documentation). Most of the time it’s not a big problem, but can be a source of confusion when you monkey-patch to much.

Eric Florenzano says:

August 8, 2007 at 3:43 pm

I don’t see why it’s so hard to dynamically add methods.

class abc(object):
    def jkl(self):
        print "hello, world"

def ghi(self, string):
    print string

abc.ghi = ghi

a = abc()
a.jkl()
a.ghi("this is a test")

would print

hello, world
this is a test

Ian Bicking says:

August 8, 2007 at 3:48 pm

I agree it’s not very hard, but somehow it is perceived to be very hard.

Re: global implicit monkeypatching, the [mathn](http://www.ruby-doc.org/stdlib/libdoc/mathn/rdoc/index.html) module is the one that really gets me. And it’s in the stdlib. It changes Fixnum/, i.e., number division. Globally.
Simon Willison says:

August 8, 2007 at 4:25 pm

open-uri is equally scary – it patches the built-in open() function (used to read files from the filesystem) and replaces it with one that also knows how to pull in data over HTTP if the “filename” begins with http://.
Martin Aspeli says:

August 8, 2007 at 4:50 pm

Language flame wars are always really lame because people argue about what you can do in language X or Y. Comparing languages like Ruby and Python, there are very few things you can do in one that you can’t really do in the other.

What’s much more important is what the language encourages you to do. I think a language that encourages monkey patching of built-in stuff (kind of like some people do in JavaScript with prototypes) has a cultural problem, for example. It may be neat to feel like you have the power to change anything, but when building complex systems, using third party libraries (which may have been written by morons, that just happened to be morons with the time to spend writing the thing you can’t find the time to perfect), and sharing a project with dozens of developers across the globe… well, I want predictability and consistency.

My biggest reasons for liking Python are that the core concepts are easy to grasp, they are applied consistently, and the syntax is (for the most part) consistent and predictable. It is very, very rare that I come across Python code that makes no sense to me. It’s hard to write obfuscated Python. That means a lot to me.

I’m not familiar enough with Ruby to make some kind of comparison, and in any way, it’d be stupid to do so. But I think there are lessons to be shared about how a language (natural or programming) influences the way people think about problems and their solutions.
Ian Bicking says:

August 8, 2007 at 4:51 pm

Technically you could do that in Python:
```
real_open = open
def url_open(fn, mode='r'):
    if fn.startswith('http:') and mode.startswith('r'):
        return urllib.urlopen(fn)
    else:
        return real_open(fn, mode)
__builtins__['open'] = url_open
```
Of course, if you try that sort of thing when you are working with other programmers they’ll rightfully kick your ass.
Jess Austin says:

August 8, 2007 at 6:53 pm

I don’t really know Ruby, but does this Python code do what you’re talking about?
```
a = SomeClassThatAlreadyExists()

class SomeClassThatAlreadyExists(SomeClassThatAlreadyExists):
    def some_method(blahblahblah):
        stuff

b = SomeClassThatAlreadyExists()
```
I suspect the difference is that in Ruby both object a and object b will have some_method, while in Python only b will have it. That strikes me as a point of sanity in Python, although for all I know Ruby may have other mechanisms that alleviate problems with global class modification.
huangyi says:

August 8, 2007 at 7:09 pm

Both a and b will have the method, because what’s changed is class, not instance.

Carl says:

August 9, 2007 at 2:51 am

@huangyi:

That may be the case in Ruby, but not so in Python. Here’s the cut and paste from my Python shell:

))) class SomeClassThatAlreadyExists(object): pass
...
))) a = SomeClassThatAlreadyExists()
)))
))) class SomeClassThatAlreadyExists(SomeClassThatAlreadyExists):
...  def methody(self, arg): pass
...
))) dir(a)
['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__',
'__hash__', '__init__', '__module__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__']
))) b = SomeClassThatAlreadyExists()
))) dir(b)
['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__',
'__hash__', '__init__', '__module__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'methody']
)))

If you want a to take the new method, you have to modify the class in place, not just have a new class take the old class’s name.

))) class SomeClassThatAlreadyExists(object): pass
...
))) a = SomeClassThatAlreadyExists()
))) dir(a)
['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__',
'__hash__', '__init__', '__module__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__']
))) def newmethod(self, arg): pass
...
))) SomeClassThatAlreadyExists.method = newmethod
))) dir(a)
['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__',
'__hash__', '__init__', '__module__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'method']
)))

Fredrik says:

August 9, 2007 at 4:16 am

> __builtins__['open'] = url_open

If you want to monkeypatch, please do it the right way. Builtins live in the __builtin__ module (singular); the __builtins__ object is a CPython implementation detail, and is not always a dictionary.
Paul Boddie says:

August 9, 2007 at 8:50 am

“That’s where all those nice railish 7.days.ago method calls come from”

Ah yes, domain-specific languages on the cheap: the source of many requests for changes to the Python language, just so people who can’t write parsers can pretend to almost write English or some other natural language directly in their programs. Reminds me of articles from The Register about Yahoo! such as “Yahoo! denies! China! claims!” which would presumably be permissible under some syntax-busting DSL-fuelled language amendment suggestion.
Ian Bicking says:

August 9, 2007 at 9:06 am

At one point I got into a small argument with some guy about how I thought "markdown string".to_html was a really stupid idea, and he just thought it was obvious and wonderful, and that I clearly didn’t understand object oriented programming.

I think it’s when people try to tell me about how I don’t understand object oriented programming (because how could someone understand such things if they use Python?) that gets me a bit testy.

ActiveResource’s Hash.to_xml method is similarly nutty, as though there was just some canonical way to turn any hash into an XML structure.
Michał Kwiatkowski says:

August 9, 2007 at 9:24 am

Ah yes, domain-specific languages on the cheap: the source of many requests for changes to the Python language, just so people who can’t write parsers can pretend to almost write English or some other natural language directly in their programs.

Why such strong words? Even people who can write parsers would probably not do so in Python for implementation of those kind of language enhancements, because it would be time expensive, inconvenient, non-standard, etc. In Ruby it’s much easier to extend the language itself, which in part gives you more freedom in designing API for specific needs (like date comprehensions in the 7.days.ago example). This power comes with a cost. Sometimes you want to pay it and sometimes not, so there’s a place for both Python and Ruby in this world. No reason to bash one or the other.
Ian Bicking says:

August 9, 2007 at 9:35 am

There is, however, lots of room to bash 7.days, since I believe it is equal to 7x24x60x60==604800. That is, it doesn’t even return any kind of time delta object, it just converts a number of days to a number of seconds with no units. Adding .ago it becomes equivalent to time.time() - 7*24*60*60.

All of which is fine for some code you hack out. Maybe there wasn’t a good time/time-delta object in Ruby when this code was written. It would certainly have been a poor choice to have fixed that problem when DHH was initially writing Rails. But because of the way in which this was implemented it can’t be changed, you can’t really tell where it comes from (ActiveSupport::CoreExtensions::Numeric::Time, it turns out), and now you can neither really be sure the methods will be there (they’ll only be there if you are using Rails), nor can you add your own improved methods without causing all kinds of frustrating incompatibilities for yourself and everyone else.
Michał Kwiatkowski says:

August 9, 2007 at 10:05 am

Actually 7.days.ago will evaluate to a Time object, so it’s not equivalent to time.time() - 7*24*60*60 (which is an integer in Python).

Methods will be there if you require 'active_support' at the top of your program or you use Rails (which imports active support for you), it’s that easy. You don’t expect to get datetime class in Python without importing datetime module, do you?

You can change them, because (as you found out) you know where they are. Look at ActiveSupport::CoreExtensions::Numeric::Time and override methods that interests you.

About the incompatibilities argument – you’re right it would be problematic to patch around this, but I don’t believe it’s a problem of the language, but rather of packaging. If those time comprehensions were packaged separately (or there would be a method to cherry-pick in the require call) you could easily exclude them from the import and define your own versions. In the way or another – it’s not the problem with open classes, it’s just an inconvenient way DHH bundled different things together into active support. And remember you’d have to face those problems in any other language as well – messing with other modules internals is risky, no less in Python than in Ruby.
fijal says:

August 9, 2007 at 12:00 pm

Hum. we do have such a thing for in-house use in pypy.

You say:
```
class SomeClass(extendable):
  def one_method(self):
    pass

class __extend__(SomeClass):
  def second_method(self):
    pass
```
and just works (by metaclass trick), doesn’t work for builtin types.
Paul Boddie says:

August 9, 2007 at 1:24 pm

Michal, Ian: They may seem like strong words, but the Ruby flavour of DSLs (which some people argue are not really proper DSLs) tends to lead people to suggest things like making parentheses optional for function calls in Python, just so they can lay out their code in a way which looks like some prose in a natural language or another language, but is really a combination of coincidences and side-effects. Meanwhile, with pyparsing and similar tools, how hard would it be for such people to write a proper parser for a proper language?

Listen to this podcast for some deeper thought into DSLs:

http://www.se-radio.net/index.php?post_id=211178

And take a look at EasyExtend for a more convincing Python-related approach:

http://www.fiber-space.de/EasyExtend/doc/EE.html

P.S. Apologies for the ASCIIfication of your name, erm, Michal – Ian’s software doesn’t like the last letter of your name, preferring to substitute it with another character which it won’t let me reproduce either.
Michał Kwiatkowski says:

August 9, 2007 at 2:57 pm

Paul,

Optional parentheses are actually very useful, I use them all the time while hacking in [ipython](http://ipython.scipy.org/) ;-).

just so they can lay out their code in a way which looks like some prose in a natural language or another language, but is really a combination of coincidences and side-effects

Ruby works. Really. So does Lisp. It’s up to a programmer to properly use language capabilities. You can write beautiful code in C/Python/Ruby/Lisp/whatever, but it’s completely up to your skills if you shoot yourself in the foot or not. Some APIs (or DSLs if you like) people write in Ruby are in fact a combination of coincidences and side-effects, but most of them are not. It’s really not that different from Python or any other language.

You agree on the usefulness of DSLs – why not have the ability to program them in the same language as the rest of your application? Why force the programmer to design/learn new language while he is already comfortable with the one he uses? Messing with built-in classes is risky, sure, but a programmer should know that already. Things shouldn’t be hard to implement just to scare the clueless newbies (they will misuse the language in a way or another).

Now just don’t get me wrong. I believe Python has much of its value in its purity. I personally like predictability Python brings, because it means I can easier understand and hack on other people’s code. It’s really a great language to maintain code in. That means I also would strongly reject ruby-ish changes to Python. It’s a different language than Ruby and there is value in this. At the same time I really like the way you can quickly try out DSL/API ideas in Ruby (or Smalltalk/Lisp for that matter). There’s a place for both Python and Ruby in this world and there’s no point in saying that people using the other one are crazy/uneducated/stupid. There are different needs and so there are different tools to satisfy them.
Paul Boddie says:

August 9, 2007 at 4:57 pm

I’m not arguing about whether Lisp or Ruby work, but whether more flexible syntax plus guesswork on the part of the language plus environment make for a sane DSL or a sane environment.

I guess it’s just a case of “not in my Python” on my part, but I’m tired of seeing excuses for DSLs really amounting to the fact that people need new toys to be bothered to write new libraries. It’s not as if we’re using Java where patterns and boilerplate are needed all over the place: you can get quite far with Python, even if you need to actually write Python code – shock horror!

Seeing inline dialects in code reminds me of things like Pro*C Embedded SQL, although DSL proponents would have you write the SQL in some bizarre variant of Python – something which you can see in various Python projects, in fact. A diversity of languages is a good thing, I agree, but I dispute the DSL movement’s emphasis on where the diversity should be.
Jim says:

August 10, 2007 at 2:06 am

> Optional parentheses are actually very useful

Do you really mean that though? Useful? You may very well prefer them, but unless they actually provide functionality, they aren’t useful.
Martin Aspeli says:

August 13, 2007 at 5:38 am

“About the incompatibilities argument – you’re right it would be problematic to patch around this, but I don’t believe it’s a problem of the language, but rather of packaging. If those time comprehensions were packaged separately (or there would be a method to cherry-pick in the require call) you could easily exclude them from the import and define your own versions. In the way or another – it’s not the problem with open classes, it’s just an inconvenient way DHH bundled different things together into active support. And remember you’d have to face those problems in any other language as well – messing with other modules internals is risky, no less in Python than in Ruby.”

No, it’s a problem of the language. Or rather, of the culture of the language and the way people use it. It’s infeasible in any language to have import/require statements that are so detailed you need to know lots and lots of detail about each module. If you had to “require” the “days” syntax and the “ago” syntax and also the “forwards” but not the “backwards” because you needed the one from module “foo” but not from “bar”… well, you’d spend more time working out what to import than you’d spend writing useful code. In general, importing a module from which you only actually need a subset should not have bad side-effects. That places an unnecessary burden on both the author and the user of the module.

Like people have pointed out, you can do this in all kinds of languages, including Python, but in Python this kind of thing is frowned upon: monkey patching is a necessary evil that you do in moderation and aim to refactor away (for the most part).

Predictability has a lot of benefits. I hate languages (like VB) that make parentheses optional in most cases, because it’s hard to see what’s a function and you end up having to learn various special cases about when something is required and when it’s not. That doesn’t make things easier, it makes them confusing and inconsistent. People who write code learn (either explicitly or implicitly) the underlying structure of the lanaguage (much like for a natural language) and then intuitively understand how to apply that structure to new problems that are like ones they’ve seen before. The fewer rules and more internally consistent they are, the more predictable the language and ultimately the less time the developer wastes on debugging things that don’t work the way he/she thinks it should work.

Ian Bicking: a blog

Opening Python Classes

21 Comments

Home

About

Archives

Categories

Recent Posts

Recent Comments