I haven’t received a great deal of specific feedback on toppcloud (update: renamed Silver Lining), only a few people (Ben Bangert, Jorge Vargas) seem to have really dived deeply into it. But — and this is not unexpected — I have already gotten several requests about making it more agnostic with respect to… stuff. Maybe that it not (at least forever) require Ubuntu. Or maybe that it should support different process models (e.g., threaded and multiple processes). Or other versions of Python.
The more I think about it, and the more I work with the tool, the more confident I am that toppcloud should not be agnostic on these issues. This is not so much about an "opinionated" tool; toppcloud is not actually very opinionated. It’s about a well-understood system.
For instance, Ben noticed a problem recently with weird import errors. I don’t know quite why mod_wsgi has this particular problem (when other WSGI servers that I’ve used haven’t), but the fix isn’t that hard. So Ben committed a fix and the problem went away.
Personally I think this is a bug with mod_wsgi. Maybe it’s also a Python bug. But it doesn’t really matter. When a bug exists it "belongs" to everyone who encounters it.
toppcloud is not intended to be a transparent system. When it’s working correctly, you should be able to ignore most of the system and concentrate on the relatively simple abstractions given to your application. So if the configuration reveals this particular bug in Python/mod_wsgi, then the bug is essentially a toppcloud bug, and toppcloud should (and can) fix it.
A more flexible system can ignore such problems as being "somewhere else" in the system. Or, if you don’t define these problems as someone else’s problem, then a more flexible system is essentially always broken somewhere; there is always some untested combination, some new component, or some old component that might get pushed into the mix. Fixes for one person’s problem may introduce a new problem in someone else’s stack. Some fixes aren’t even clear. toppcloud has Varnish in place, so it’s quite clear where a fix related to Django and Varnish configuration goes. If these were each components developed by different people at different times (like with buildout recipes) then fixing something like this could get complicated.
So I feel very resolved: toppcloud will hardcode everything it possibly can. Python 2.6 and only 2.6! (Until 2.7, but then only 2.7!). Only Varnish/Apache/mod_wsgi. I haven’t figured out threads/processes exactly, but once I do, there will be only one way! And if I get it wrong, then everyone (everyone) will have to switch when it is corrected! Because I’d much rather have a system that is inflexible than one that doesn’t work. With a clear and solid design I think it is feasible to get this to work, and that is no small feat.
Relatedly, I think I’m going to change the name of toppcloud, so ideas are welcome!
Automatically generated list of related posts:
- toppcloud (Silver Lining) and Django I wrote up instructions on using toppcloud (update: renamed Silver...
- toppcloud renamed to Silver Lining After some pondering at PyCon, I decided on a new...
- Silver Lining: More People! OK… so I said before Silver Lining is for collaborators...
- Core Competencies, Silver Lining, Packaging I’ve been leaning heavily on Ubuntu and Debian packages for...
- A new way to deploy web applications Deployment is one of the things I like least about...
A few brainstormed (no pun intended) name ideas:
Cloudcover, Stratosphere, Jetplane
I couldn’t agree more.
$ python -c “import this” The Zen of Python, by Tim Peters
Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren’t special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one– and preferably only one –obvious way to do it. Although that way may not be obvious at first unless you’re Dutch. Now is better than never. Although never is often better than right now. If the implementation is hard to explain, it’s a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea — let’s do more of those!
You are locking everything down so that it will always work? Then maybe this is like the iPad but for Python in the cloud. Thus: cloudPad? PadCloud? PyClad? Cloupad? Something like that?
I’m pretty sure that was a sarcastic comment. But I’ll answer anyway. The “lockdown” is in the form of a well-known-system so it’s more like how the Apple should be. That is a BIG read button saying this is unstable are you sure you want to install it? If you want to set a comparison I think it’s more like firefox plugins, you can go ahead and fork it to add your stuff (read install plugins) but the core system will be stable and sound without it (vanilla firefox).
No, it was supposed to be a fun comment! Apple is doing something to help out people who want the Internet to “Just Work” without breaking all of the time, and Ian is doing something for us web developers so that we can churn out deployed web apps in what is surely the Python equivalent of the pretty white packaging that Apple is famous for! I should obviously have used some smileys with my comment, and also remembered that some people are angered by the iPad and would think I was saying bad things about toppcloud. So, here are some belated smileys: :-) :-)
I still think Ian should name it pyclad. It sounds like “AllClad” and has a nice, hefty feeling. Like something made of metal.
A few (cloud based) ideas:
Nobody is locked and it cannot be compared to the iPad because this is open source software (a clear license is missing though). If anybody disagree with Ian technical choices, they are free to fork and follow their vision. If a vision is proved to be valuable by an implementation, I have no doubt inclusion in the initial project may be an option. Thanks Ian for designing Toppcloud the way YOU think it serve its purpose the best.
Ultimately I really want it to be a consensus opinion; I am not the best sysadmin, and I certainly don’t have the entire breadth of experience about how all these pieces might work. So I hope that it can retain a single codebase (at least if you squint enough to ignore forking chatter ;) and that codebase can represent something that is really workable for most people. There are different options for how to factor things as well; for instance, long-poll/Comet-style interaction is not feasible with the setup, nor do I think it’s worth pursuing, but a message hub service that does long polling would be a nice addition.
While I do think a consensus is possible, and that there really are best practices people can agree on (not always exactly-optimal practices, but practices that are widely acceptable), there are also bike shed disagreements that will have to be shelved. For instance, I like Ubuntu, I don’t think there’s anything wrong with Red Hat, but supporting both is not feasible.
Ian, have a look at the standalone Python code example I include against that mod_wsgi issue. Tell me why the behaviour it shows (outside of mod_wsgi) is in some way sane.
It doesn’t really matter if it is sane or not; the problem exists and is particularly noticeable in mod_wsgi (probably because mod_wsgi is doing more in C, and so it encounters some weird ordering issues that mostly-Python servers do not). Even if it was fixed in Python it wouldn’t actually fix the problem, as existing Python deployments would still cause this to happen. Maybe it’s a workaround for someone else’s bug, but it doesn’t really matter. And I’m not really complaining about putting a workaround in toppcloud — the workaround has to go somewhere, and as far as I’m concerned it is now “fixed” with respect to toppcloud. If it goes in mod_wsgi then it will be fixed both places. If it goes in Python it will be fixed for future Python releases. It’s not so much a judgment as a matter of practicality.
To say it is because of more being done in C code isn’t strictly characterising it right. The issue is that mod_wsgi itself doesn’t implement a web server and leaves that up to Apache. With other WSGI hosting mechanisms such as Paste or CherryPy WSGI server they implement a web server the operation of which more than likely causes the affected modules to be imported at some point prior to any request actually being handed off to a WSGI application.
As far as implementing a workaround, as pointed out the in the mod_wsgi issue, the strptime module may not be the only module for which this could theoretically occur. Do you address just the one observed instance seen so far and not address the others? If not, eventually someone is likely to be tripped up by the others. What do you do then, progressively add more and more workarounds? From a maintenance perspective, that is going to be come a pain after a while.
Yes, that is what you do, because there’s no other way to fix the problem. I’m not saying it’s pretty ;)
Changing the name sounds like a good idea, if only because I am having a really hard time googling for your project. (Tried “topcloud python bicking” and “site:ianbicking.org top-cloud”)
toppcloud double P
“toppcloud will hardcode everything it possibly can”
I totally agree and am happy having someone impose sensible defaults. Deployments quickly bloom from ‘it works for me’ to complete unmanageability and wrenchedness.
How bout cumulus, acumulus, cloudless, cloudly … cloudian?
Thanks for this post and the insight to your design decisions. I think it’s the right way to hardcode as much as possible of the toolchain. But I still have some reservations about Ubuntu as the choice of operating system. And every other fixed operating system would arouse the same reservations.
I mean its a question of where you want to go with toppcloud. Should it be a proof of concept or will it eventually advance to a widely used tool for webapp deployment to cloud environments? I am looking forward to the latter and with this in mind I think the hardcode-everything-stand needs to be softened at least sometime in the future. And in my opinion a fixed operating system, especially if it’s Ubuntu, is a no-go for several reasons.
Personally, I think sticking with Ubuntu is silly. The last stats I have show RHEL 5 and it’s clones in the lead for live server deployments.
I’m guessing that RHEL/Centos’s default Python 2.4 is a show stopper? RHEL 6 should be out in Q2 or Q3 with 2.6 support. Hopefully Ian or someone else will create a RHEL6 fork.
Admittedly there’s a bit of a “let’s storm operations!” to this; this tool expresses developer sensibilities. Among developers RHEL gets no love. The Python 2.4 thing kind of shows the symmetry: RHEL/CentOS doesn’t seem to love developers either. Ubuntu works well; I’ve heard no claims to the contrary, and have encountered no evidence to the contrary.
As to toppcloud’s widespread use… I’m not sure that matters to me. The only time that will really matter is if some contributor I really value is between a rock and a hard place — in this case, between toppcloud and operations/IT. It’s not impossible that contributor may be me in some future circumstance. But it will be disappointing if the tool is compromised because of organizational/political challenges (not surprising, but disappointing).
I totally agree with that RedHat/CENTOS is horrible for deployment, it’s outdated and “stable” I for once will not have looked at this if it depended on that.
On the other hand I have been using ubuntu and love it because “it just works” So for me toppcloud is the way to change that and make the top deployment tool.
I also love the way ubuntu works, but I try to avoid tools that tie me to a single vendor/distro. And there is more to linux/unix than RHEL and Ubuntu of course.
Fedora for one is another very popular distro. Especially amongst devs as its the base for future RHEL releases/technologies. But than there is also gentoo, debian, suse, mandriva, slackware, arch and also *nixes like FreeBSD and OpenSolaris. Not that I am crying for support for all those distros but I am looking for an abstraction layer to write own backends for them.
I think you are missing the point. The reason of having ubuntu as the distro is not “we love ubuntu” is it works and works great. It’s stable, has enough packages and is well maintained. Is that better than X? maybe, maybe not. The point is that it’s good enough for this system and it’s one less thing to worry about. I will love to plugin nginx as the frontend for static files but that will make the system more complex for everyone at the expense of something I will like to have. That said as Ian pointed out the idea is to have things by consensus if in the future “the best” deployment method for webapps is to run nginx with reverse proxy to tornadoweb, then there is a chance the system will switch to that if everyone involved agrees it’s the best way to go.
Then you probably want Puppet instead of this tool.
How about Argent Nimbus (silver cloud in Latin, though the conjugation is almost certainly wrong, maybe in Latin-ish). I like the idea of combining metal + cloud. Maybe even heavy metal clouds? Aurum Nimbus. Other people’s clouds have silver linings, our linings are gold.
Best wishes.