I should preface this entire post by noting that I haven’t used Erlang, just read about it, and I handle most concurrency using Python threads, a model for which I have no great affection (or hate). But I was reading two posts by Patrick Logan on Erlang and it got me thinking again.
From my reading and what I’ve heard from other people, Erlang the language-and-syntax seems quite crufty. Erlang is not the Complete Package, the language that will make you leave your wife, the language that will have you walking into telephone polls because it’s just so hot that you can’t take your eyes off it. Erlang is not that language. People seem to get excited about two things in Erlang: the concurrency model and pattern matching. I’m not quite sure why people are excited about pattern matching. Pattern matching isn’t particularly hard to implement, though maybe it interacts with other aspects of the system in a way I don’t understand. I remain skeptical that it’s anything special.
Then there is the concurrency model (and its associated message passing), and this does strike me as special. You can kind of implement that model in Python, but with none of the concrete benefits.
These are the useful features I see in the concurrency model:
- Erlang processes are share-nothing.
- The processes are light weight. You can start lots of processes, and they start quickly and can die off quickly without any great overhead. They aren’t OS-level processes. These are sometimes called green processes or microprocesses. There are many implementations of microthreads in Python, but you lose the benefits of share-nothing.
- The runtime makes light weight processes feasible as well. The OS overhead of a Python process is actually just a small part of the total overhead when spawning a process. Loading and initializing the runtime libraries of a Python program of even modest complexity is problematic. I’m not sure exactly how Erlang does this, but I suspect it’s because libraries can be safely shared because Erlang is a functional language. (PHP is also like this, with most of the library written in a sharable C library.)
1 and 2 can be handled with the right VM. This isn’t a particularly common VM design, but it’s certainly doable. 3 is tricky, and effects the language design.
I don’t think it effects the language design as greatly as to require a functional language. It requires that sharable code itself be a functional (i.e., immutable) subset of the language. What the code does doesn’t have to be functional, only what the code is.
To explain this, functions in most languages are immutable. Python functions aren’t actually immutable, but they are close enough that hardly anyone would notice if you made them immutable (right now you can overwrite their name, compiled code, and some other stuff — but almost no one actually does this). The function may have side effects, but as long as the function object is immutable then it can be shared safely among processes.
You can extend this to the module as a whole. This means you couldn’t monkeypatch objects in the module, and module-level assignments would effectively be constants. Any module-level objects would have to be immutable. Things like classes, which are also a kind of module-level object, would also have to be immutable — meaning things like class variables would be immutable. It could still be possible to do module-level code if there was a way of freezing the module after its instantiation (this would be important for Python, as even decorators are a kind of code run at import time). Adding a general concept of "freezing" to the language might be the most general and expedient way to make modules sharable.
The other half of the concurrency model in Erlang is message passing. Erlang processes send messages around like other systems send methods. It would be possible to use exactly Erlang’s system, as it’s not Erlang-specific and has been ported to many languages. Though I’d be somewhat more inclined to use bencode as it has some cleverness in its design. Or perhaps JSON, just because. Regardless of the format you are always passing around data, which I think is important. Systems that pass around "objects" become complex, mostly because you just cannot pass objects around and so those systems are just complex facades built around data exchange.
Lately I’ve become fond of thinking of objects as views over a more fundamental data structure (WebOb is strictly based on this pattern). Methods are details of the implementation, but only the data can mean something to someone else. At least this is the worldview you start to internalize when you think about REST a lot. (My post Documents vs. Objects is also about this.)
Arguable a plain-data-with-views world is just the dynamic-weak-typed anti-pattern. That anti-pattern is one where you get all the disadvantages of dynamic typing, and all the disadvantages of static typing, all in one ugly bundle. Erlang’s records remind me of this anti-pattern. Instead of dictionaries everything is a tuple, and getting a record is just syntactic sugar mapping record names to indexes — all in the context of a dynamically typed language where you could mistype a value and get strange output as a result.
The strong-static typing solution to the type problem involves complicated contracts, aka "service descriptions", the stuff of WS-*, CORBA IDL, etc. The strong-dynamic typing solution is that every object have an explicit type. This is fine for things we all agree on: lists, dictionaries, bytes, numbers. Sadly even a basic thing like unicode strings cause problems, as do dates, but at least those have straight-forward solutions (you add more basic types to the message format). More complex types, like a domain object (e.g., a user record) are difficult. Are they all just dictionaries? Dictionaries plus special metadata? XML namespaces address this problem in some ways, but are also the point at which XML starts to just piss people off and make them want to use JSON. And for good reason, because XML namespaces force you to define entities and responsible parties very early on, possibly long before you know what you are actually trying to do. Besides XML, are there other systems that aren’t just naive/unextendable (bencode, JSON) and are still basically dynamically typed?
This is the kind of thing I would really love to see PyPy experiment with.
Automatically generated list of related posts:
- Runtime vs. Test time Defenders of static typing have long claimed that it helps...
The other big feature you’re missing is hot code reloading. Hot code reloading is hard in Python for a lot of complicated reasons, but it’s easy in Erlang for three key reasons:
the VM can keep two versions of any module in memory (so that you can finish out processes running code written for an old version)
you pass around data structures that aren’t entangled with particular versions of code (e.g. Not Objects), and pattern matching makes it easy to upgrade old data structures
since it’s a functional language everything is on the stack AND the way to write a loop is to do a tail call so you have lots of opportunities to upgrade running code (the next time you call the function you simply use the new version of the function — of course this is opt-in depending on whether you’re using a local or absolute call).
Records aren’t really used for interop, they’re typically used within a single module as an internal data structure (opaque to anything else) or at worst within an application. Sometimes this bleeds through and becomes an API (e.g. Yaws or the file module) but that’s the exception and not the rule. In any case, they suck to use because they’re not extensible and the syntax is annoying, but they do have some nice performance properties.
It’s worth noting that since Erlang started supporting SMP, its processes are M:N microprocesses per core. Any and all Python microprocess implementations are going to be M:1 microprocesses per core due to the GIL if nothing else.
The other big, very difficult to copy thing is the error handling. Erlang handles remote and local failures in an impressively smooth and integrated way. The model of “failure is caught by some other monitoring process” is not at all easy to replicate without fairly deep support from the VM.
very nice exposition in Bob Ippolito’s comment.
I would add, Erland is also designed for application uptimes measured in years. Erlang assumes you want the application to run on two or more machines, you can add more machines, and bring down machines at will
(obviously you have to program in a particular style to get the benefit, where anything and everything is assumed to fail at one time or another, and some other process is called to pick up the dropped ball. You can write in Erlang, in poor style or with poor engineering, applications that cannot do this.)
It seems like hot reloading would be feasible simply by way of having cheap processes…? With cheap processes you can kill processes with old code and start new ones. Of course messages-as-data is also essential. Is code reloading on a smaller granularity also important?
Maybe another feature is support for asynchronous calls. Or at least support for that pattern. At least, that seems helpful for message passing, and seems similar for error handling, though I don’t really know any of the details of Erlang on that level. Is Erlang good at async? I’m thinking about things like all the awkwardness of Twisted, which is a parallel awkwardness of both function calls and exceptions, so it feels like both problems could be solved at once, and work over the process barrier as a result.
Hot code reloading is not feasible simply by having cheap processes. You don’t want to kill old processes that run old code and start new ones. That’s the same as shutting down and starting back up, you want a seamless upgrade.
Erlang ONLY does async. Synchronous message passing in Erlang is a pair of async message send and blocking selective receive.
You really just need to learn Erlang, much more useful than talking about how you think Erlang might work. It’s not a big language.
I guess I need something that is worth using Erlang for. Learning a language without a purpose (even a kind of silly purpose) feels fake.
Is everything async? Like, ever function call, every anything? I guess on some level you couldn’t tell if you are in a system with no side effects… it’s just an indefinite pause either way.
Ian, with regard to async everything, there are two parts to erlang. The pure functional sub section is known as ‘sequential’ erlang and has pretty much the same semantics you would expect from any language where you can do FP (python, lisp, JS etc). The send operator (
!) andreceivesyntax let you write processes that communicate and it is anything built on these that are async.funny that we were talking about this on the #pypy channel today (and I just read your post now). Stackless do have corotines that are interesting as you don’t have any problem with the gil (as it is running just one thread) and if you have the scalability of erlang (various process on the same or other machines passing messages to each other) you end up with a similar system of concurrency. The remaining problems are the shared memory for corotines.
Pypy people are usually interested in this kind of stuff, but they all seem to be doing other projects right now (like working on the JIT and on making ctypes fast). I think they would welcome you there if you want to stop by on freenode.net for a chat (I certanly would).
About the twisted thing, they have now a corotwine package that mixes twisted and corotines, which might be interesting to you.
Just to be clear, when it comes to stuff on this scale I’m a talker, not a doer ;)
“Adding a general concept of “freezing” to the language might be the most general and expedient way to make modules sharable.”
Interestingly, this general concept of “freezing” is also what came up recently as the royal road to optimization of dynamic languages as I was idly chatting with Peter Norvig a short while ago — that’s what they did at Harlequin for Dylan, and it worked wonders (apparently gets great synergy with type inferencing and jit/specializing a la psyco). Which is in weird contrast with what I heard from Ruby experts years ago as I lauded their language’s having that “freezing” concept — apparently it isn’t (wasn’t?) really used much and/or didn’t work well (I wonder if it’s just the fact that, particularly a few years ago, optimization and good threading were pretty bad in Ruby’s implementation, and at the same time monekeypatching appears to be a strong cultural meme in Ruby, much more than in Python…).
I think you are missing the interaction of many features:
Pattern matching. Yes it is possible to add pattern matching to many languages, but in Erlang (as in many functional/logic languages) it is an integral part of the language and used everywhere.
Error handling. This is possible only because of share-nothing processes. The error handling is an integral part of the language as well and definitely affects how you build applications.
Immutable data. This actually makes the world much easier to comprehend once you get around the mental block. It also means that there are no back doors to process communication, there is only message passing. Which means that you can make distribution transparent, …
Etc.
The problem is not that you can’t implement these features on other languages, in many cases you can, or that the concepts are new, they’re not (actors are older than erlang). No, the problem is to get all the benefits of these features as you do in Erlang you need them built in and integrated with each other from the beginning. Just looking at each one separately does not show you the power of them working together. This also means that adding this feature or that to another language will not get you all the either. It will just not cut it.
Its true that pattern matching can be added to any language but it is ingrained in erlang and is used consistently throughout.
For me, the syntax of Erlang is beautiful, in the same way that LISP is, it is simple and elegant and does not require an encyclopedic knowledge of sytax.
Add that to the powerful libraries, particularly OTP and mnesia and you have the makings of a distributed toolset rather than language.
If you can’t tell, I am a pretty big fan.
Re: crufty syntax. Think of how wrong all the people who complain about Python’s significant whitespace as being crufty syntax are. Maybe you are like me and was one of those people until you started using it and now you really hate languages that have needless punctuation like {} END.
After learning / lightly using Erlang for a month I hardly notice the syntax anymore. And I’m starting to get annoyed that in Python I have to surround the atom equivalent with quotes, but only in some places!
Also, It seems I write less Erlang than Python expressing the same code (Pattern matching is big reason for that). So, even if it is crufty there’s less of it to deal with.
Finally, there are macro packages that let you write Erlang in Lisp like syntax and one that is Python indentation like.
Several good comments on the many benefits of Erlang and especially the combination of those into the sum greater than the parts kind of thing. One more to point out…
Messaging and pattern matching in Erlang are deeply intertwined, i.e. “selective pattern-based receive”. If you just accept messages over various pipes and you just pattern match on various language data structures, you still don’t obtain the benefits of an intertwined mechanism for pattern matching within the “per thread” (shared nothing thread in Erlang) mailbox.
Erlang is subtly, specially simple in its own way, just as Python’s various object, meta, syntactical foo makes it expressive in its own way. What we really need are simple, efficient ways to interact between Erlang, Python, and other systems, not necessarily, if ever, running in the same OS process. We need to easily spin each of these things up to do their own work and use each other.
Such a mechanism exists for Python and Java and C to interact with Erlang, but we need more general foo.
The problem with “all these pieces work together perfectly” is that — according to hearsay — they work together perfectly to write protocol stacks and distributed databases and stuff like that. You know what? I don’t care about that kind of stuff. I would like to write stupid crap like web applications. And — according to hearsay — Erlang isn’t good at that. Relevant aspects of the language (and/or libraries) sound quite painful. So something is off. Is it just too immature an environment (at least for that domain)? Or is there something wrong with it? I can write some functional code when I’m dealing with a Very Hard Problem. But you know, most problems aren’t Very Hard, and I’d rather not twist my brain to solve those problems. Web programming is just Not Very Hard, even given the concurrency issues.
I know enough [Twisted](http://twistedmatrix.com/trac/) to know that’s the wrong way to write web applications. Maybe it’s the right way to write servers that interact directly with sockets. Maybe. I don’t know much Twisted at all, yet I feel pretty confident about my opinion. Erlang takes a somewhat different approach, and obviously has the language support to back that approach where Twisted does not. But while I know less about Erlang than I do about Twisted — and I should rectify that — I can’t help but notice a lack of people claiming Erlang is the Complete Package, or even anywhere close to it.
So… what should I expect to find out, ultimately? That Erlang is super-awesome and I’m going to stop using anything else? That share-nothing is nice, but in order to get it I’ll need to use something more like CGI (ala App Engine)? That some hybrid approach is workable, using Erlang for things like a database (e.g., [Couch](http://incubator.apache.org/couchdb/)) but writing all my code using traditional concurrency techniques? If the last one is the case (I guess what Patrick is proposing) then… do I need to care about Erlang? If I deploy CouchDB and use it from Python, what should I care what CouchDB implemented in? I don’t worry a great deal about what MySQL is written in.
“That some hybrid approach is workable, using Erlang for things like a database (e.g., Couch) but writing all my code using traditional concurrency techniques? If the last one is the case (I guess what Patrick is proposing) then… do I need to care about Erlang? If I deploy CouchDB and use it from Python, what should I care what CouchDB implemented in? I don’t worry a great deal about what MySQL is written in.”
Yeah. I don’t know that I was actually thinking about anything specific, but this is essentially it.
The “application specific” parts of a web application seem easy in Python, Ruby, etc. The “system specific” parts seem fairly easy in Erlang. What would be great would be the ability to integrate these two easily.
So rather than a Django application running in all Python (I assume — don’t know much about Django) instead the distributed, supervisory, scale-outing stuff in Erlang and the app per se in Python. But the Python developer does not have to know that the system stuff is in Erlang.
And the system stuff in Erlang could just as well be running apps in Python as well as Ruby or who-knows-what.
To work with MySQL you have an integration protocol. Same with Erlang and other languages. Although it could be easier.
Another example I’ve dealt with in the past: the CLIPS rules engine, and Python has a nice integration: PyCLIPS. I’d like to have a farm of rules engines, talk to them through some scalable, distributed, supervisable mechanism just like what Erlang/OTP provides. But talk to them as Python/PyCLIPS “services”.
It’s very “do-able” but fairly ad hoc. Especially if part of the system is in Ruby, part is in Erlang, and part is in Python (and part in CLIPS (lisp, if no using PyCLIPS)).
Dealing with a very heterogeneous system can be quite challenging, but I think it’s doable. Using a protocol like HTTP (what CouchDB uses, for instance) is probably quite helpful, as it’s a protocol that all languages and environments are familiar with. JSON or XML or something along those lines has a similar advantage. Sure, you can make other systems understand Erlang’s messages, but you don’t get the tool support, existing libraries, developer knowledge, etc.
Maintaining a heterogeneous system is also difficult, but a separate problem. Maintaining anything is a problem, it’s just one that happens to scale O(n) for n kinds of systems.
“Dealing with a very heterogeneous system can be quite challenging, but I think it’s doable.”
Yes, I think that’s the current big challenge that is being ignored by the continuing excitement over “many languages in a single VM”. We need better ways to do “many languages in many VMs on many nodes”.
Primarily that seems like a job for HTTP and XMPP plus some JSON/XHTML conventions plus some management/operations capabilities.
“Maintaining anything is a problem…”
I assume I can quote you on that… ;^/
I think a lot of the cases where people want many languages in a single VM are heavily motivated by systems maintenance. Especially with the JVM, after people make claims about All The Great Libraries You’ll Be Able To Use, the advantage noted is the ability to run everything in one container, one build process, etc.
Oddly, when people think about this second problem they seem to ignore the possibility that instead of changing runtimes dramatically they could just fix their build process. (The challenging part is that fixing the build process in this way usually involves pushing complexity out of the build and up into the language, framework, and application — but it seems to be worth it.)
the VM can keep two versions of any module in memory (so that you can finish out processes running code written for an old version)
(Pardon the long, late post…)
I’m primarily a Python programmer getting my feet wet with projects combining Python and Erlang. Probably the best way to get the gist of Erlang is to read/skim Joe Armstrong’s thesis: http://www.erlang.org/download/armstrongthesis2003.pdf.
I’ve dabbled in Scheme, Common Lisp, Ada, and other languages, but Erlang is the only language I’ve decided to actually use for real work in the past 10 years other than Python. In my experience, languages like Python and Erlang complement each other. Neither language does everything. It’s important to remember that all languages have design trade-offs due to their emphasis.
Erlang is most definitely worth learning, but not because it is “better” than favorite language X, or because it is designed to completely replace your favorite language, but because it made trade-offs that other languages didn’t, that solve the big difficult problems of software that are very difficult or impossible to fully solve in other languages without making the same extreme trade-offs. Let Erlang make the trade-offs so your other language doesn’t have to. :-) A realistic and common approach is to combine Erlang with other languages, like Python.
Here is a comparison of some of the trade-offs and reasoning behind the different languages. (For more details on the Erlang side, please read the thesis PDF.)
Python and concurrency: Python has the GIL to make it easier for C to interact with Python (from what I’ve heard). The GIL could be removed, but supposedly writing C towards Python would have been made more complicated. The advantage of the GIL is that Python is a great glue language as a result. The disadvantage is reduced ability to take advantage of threads. But I think Python got it right when deciding to go multi-process as a solution, and Erlang also agrees on this design philosophy of using processes. :-)
Erlang and concurrency: People should keep in mind the context in which Erlang was created. It was for telecoms, where the system must be reliable and never go down, while also supporting a high amount of reliable concurrency. Deferring concurrency to OS-level threads or OS-level processes wouldn’t work, either because they’re too slow, or because shared-state is unreliable. Deferring to the OS also can make concurrency inconvenient or sacrifice portability, or at least consistency in performance across platforms. The conclusion was to create lightweight, runtime-level processes. These are not the same as many tasklet, coroutine, pseudo-thread, etc. implementations. The processes must be truly concurrent, not cooperative, and they cannot share state. This separation adds to the additional goal of fault isolation. Processes aren’t just a feature, they are the unit-of-work much like how objects are for pure object-oriented languages. Processes communicate through messages. Supposedly, message-passing was a key concept of object-orientation in Smalltalk that was lost in C++, Java, etc. So, processes are analogous to objects (or you can consider them objects from a certain point of view), except they are naturally concurrent and provide more physical separation than multiple objects within the same process in other languages.
Python and libraries: Python readily binds to C/C++ libraries, and this is convenient. But what happens if that library itself has a serious bug? I’ve personally experienced times when (especially a C++ library, such as earlier versions of wxPython or beta-quality 3D engines) would crash Python, even though Python itself is stable. The trade-off is that Python readily talks to all sorts of libraries easily, but risks being at the mercy of failures from those libraries. As another example, try using ctypes where the C library is coded badly. For many people, the convenience is worth the risk. For others where the system must not go down (e.g. telecoms, mission-critical servers, control apps), the convenience isn’t worth the risk. It’s a different emphasis/trade-off.
Erlang and libraries: To talk to other languages, you use a “port” in Erlang, where the external language is a separate process that acts as an Erlang process from Erlang’s point of view. The trade-off is that you get fault isolation (which is an explicit goal of Erlang, but not an explicit goal of Python, although desirable) at the expense of some extra work. External code won’t crash Erlang if they’re utilized as ports.
Python error handling: Python uses error-trapping with try/except, like other languages that have try/catch, etc. Because a process is not the de facto unit-of-work for Python, it’s natural that error handling is performed within the same process (by default). Although, you could probably act on the exception with a distributed call, the error-trapping is always essentially performed in the same process.
Erlang error handling: Fault-tolerance is an explicit Erlang goal. That means guarding against hardware failure, and not just software exceptions. The only way to guard against hardware failure is through distributed programming, i.e. redundancy and fail-over. This is more than just watching a heart-beat, it can involve any sort of error or exception. The requirement that the error-trapping occurs in the same process as the error makes distributed programming more difficult or less robust. Often the process can’t continue and shouldn’t even try, and so the sensible thing to do is to detect that is the case, crash the process, and restart the process or groups of related processes, where this might even be done from a non-local supervisory process. The exact semantics of how this is done is subtly different than other languages. Maybe you could do it, but not without a lot of work. Technically you can do anything in C that you could do in Python, but the point is to use the best tool for the job. For fault-tolerant sorts of things, Erlang is most likely the best tool for the job.
There is much more than this (behaviors, supervision hierarchies, etc.) but by now it should be apparent that Python and Erlang do their jobs well, but they were meant for different jobs. But there is no reason that such jobs aren’t complementary. For example, I’d looked at Twisted too, and it is so foreign from most normal Python code that you might as well learn another language. Especially because it doesn’t have the built-in facilities for distributed message passing at the language level, distributed error handling, share-nothing semantics, etc.
If you want one reason to learn Erlang, it is this: no other language has seemed to solve the problem of reliable concurrency anywhere near as well. And that is kind of important. Erlang seems to be the only language that comes close to readily scaling linearly in performance with multi-core and distributed. Little things like pattern matching and single-assignment are part of the goal for reliability, but in themselves aren’t the whole point.
> Besides XML, are there other systems that aren’t just naive/unextendable (bencode, JSON) and are still basically dynamically typed?
Thrift, while not dynamic in the classic sense, seems to do a good job at creating extensible interfaces with a defined but flexible contract (IDL) that are usable in both dynamic and static languages. All in all, from what I’ve seen so far, it’s a pragmatic compromise weighting the different forces in the right way (IMHO, anyway).
Unlike Hessian, it also doesn’t favour one language. Hessian does so for Java, all the clients for other languages are fig leafs.
Lisp