Projects

This is a list of projects I am working on, or have worked on. All are open source.

Recent(ish) work
Projects I’ve Retired From:
Small projects
Toys

Recent Work

Firefox Voice: Firefox Voice was our project out of the now-defunct Consumer Voice Products team, in the now-defunct voice program in Mozilla, part of the now-defunct Emerging Technologies division.

Firefox Voice is a voice assistant for the browser, allowing the simple commands like “new tab” as well as routing and interpreting messages to Google (“what’s the weather”), integrating with music services (“play David Bowie”) and many other commands.

At least for a while (I’m writing this in August 2020) it’s available on addons.mozilla.org.

I wrote some thoughts coming out of this project in Thoughts on Voice Interfaces.
Test Pilot Firefox experiment backlog: Although I was able to ship some things in Test Pilot, there was a long backlog of ideas, and experiments I started but didn’t work out or weren’t able to be shipped. I made a list of them!
Firefox Screenshots: I started Page Shot (which became Firefox Screenshots) in October 2014. During that time I went from doing it as a small side project, to managing the project during its nascent stages, then when I left management I rejoined development to get it launched as a Test Pilot experiment. During that period it went from being a page freezing tool (similar to the Way Back Machine) to a screenshot tool.

It was popular inside Test Pilot experiment channel, and it wasn’t much later when it re-launched as a core Firefox feature in September 2017. When it launched it was the first Mozilla product built that took unencrypted user content, that used Google Analytics, the first Firefox feature to ship as a WebExtension, and the first use of Sentry to monitor errors in the browser.

For myself, Screenshots has been a lesson in managed expectations, but also a lesson in providing simple tools without worrying about how hard they are to create.

Despite being popular, in 2019 the server component of Screenshots is being shut down, though the tool will otherwise remain in Firefox. During its life 36 million people used the Screenshots server with over 100 million screenshots uploaded. About 20% of screenshots created are uploaded (instead of being downloaded or copied to the clipboard), so it would be reasonable to estimate 500 million screenshots taken in the course of two years.
Side View: Another Test Pilot experiment, this makes it easy to open a mobile view of a website in the Firefox sidebar. I’m mostly proud that we kept it simple and usable.
Email Tabs: Another Test Pilot experiment. This was strongly influenced by Mozilla’s user research. The results of our research was clear: everyone loves to save and share things by email. And yet it took me a few years to sit down and quickly hack out a tool that makes emailing articles work well. Working with email, and poking around in other company’s email products, is a bit like science: you have to see what works, nothing works by default or because you want it to work a certain way.

I’m excited by Email Tabs, in no small part because of the simplicity and power of prepopulating compositions. Composing (but not sending) an email is transparent, manipulable, and open-ended. I hope I have the chance to use this lesson elsewhere. The browser is a great location for this, because the browser can integrate with systems in the same way a user integrates with systems, which isn’t just handy but also is a kind of working in the open: you can see and understand what the automation does, because it does what you might have done. I wrote some thoughts in Thoughts on the Firefox Email Tabs experiment
Personal History Archive: I’ve been poking around at this project for a long time, never really finishing it or deciding exactly what to do with it. The underlying goal is to support local, personal experimentation with the data that can come out of a browser. Most collection like this emphasizes data collected from many aggregate sources, either from select sources or with anonymization, and with only a rough outline of user behavior. A user agent (like a browser) sees everything! It doesn’t just see that you visited a site, it sees exactly what that site displayed. It doesn’t just see that you have an email, every email you read is displayed by the browser.

So this is a tool to try to collect all that intimate data in a way that supports personal experimentation. Especially experimentation that can produce tools uniquely suited for a user agent that belongs to the user, touching data you’d never want to give to an external service.

Also I have just been poking around with creating highly usable data dumps, where the data is clean, always well-formed, and where behavioral and more static data is mixed together in a usable way.
Other GitHub miscellany: Working in the open at Mozilla I have had the opportunity to try things out. Sometimes they work out, and sometimes they don’t. At that stage you’ll find my work in GitHub, which is updated more frequently than this page.

Projects I’ve retired from

Download stats aren’t necessarily accurate, but as of June 2019 pip and virtualenv (which I originally wrote) were downloaded 980 million and 150 million times respectively. Computers are very good at downloading things! The balance of packages I have written/started have been downloaded roughly 70 million times, most notably WebOb, Paste, PasteScript, PasteDeploy, WebTest, Tempita, SQLObject, and INITools.

Collaboration / general

Hotdish: This took some of the ideas of TogetherJS, some of the tech from Browser Mirror, and thinking about them all in the context of a browser session. It’s implemented as a Firefox addon. The model we came to is one where a group of people share everything they do in a specific browser, with mutual awareness of what each person is doing (both in general, and with in-page feedback using TogetherJS), and people can join and present to each other in this context.

Our longer-term goal was for people to be able to understand what each other are doing — what pages they are interacting with, but also how they are interacting with those pages, similar to how you can understand what a person is doing when they are doing physical work.

Another goal is to look at how, using the same data we capture in order to share a person’s actions with everyone else, we can also use that data to create a richer record of what people are doing. We wanted knowledge capture and transfer between group members that can encompass not just one site or document, but a session, discussion, and research.

I narrated a screencast that we made to explain some of the concepts and things we learned from the experiment.

Some blog posts on the topic: Defaulting To Together, and Hubot, Chat, The Web, and Working in the Open.
TogetherJS: TogetherJS was my last (shipped) project at the first iteration of the now twice-defunct Mozilla Labs. TogetherJS is a library and service to enable real-time collaboration on any site. It can still be seen on JSFiddle where it is used for live collaboration.

The larger vision for TogetherJS falls under the category of Cobrowsing: basically experiencing the web or a website collaboratively. We really wanted to explore what it meant to interact with things side-by-side, moving in and out of synchronous collaboration.

Several blog posts on the topic: TogetherJS as a Postmodern Programming Tool (talking about heuristic-based integration), Nouning the Verb of Browsing (talking about using the on-the-wire messages as a kind of record), Hubot, Chat, The Web, and Working in the Open (more about creating a record of activity), Collaboration as a Skeuomorphism for Agents (about using the metaphors and mechanisms of collaboration to introduce automated agents), and my thinking concludes in Towards a Next Level of Collaboration (which provides a taxonomy and discussion of several specific aspects).
doctest.js: A testing framework for Javascript, inspired by Python’s doctest. Something of an alternative to BDD, based on examples and expected output. Blog post: Why doctest.js is better than Python’s doctest.
walkabout.js: This is something like a UI/DOM fuzzing library. It inspects the DOM (or uses rewritten Javascript code) to determine what parts of the page are “live” (and in what way). Then it can choose a random action to perform, like clicking on an element or entering text in a field.
SeeItSaveIt: An experiment in extracting structured data from web sites. Works as a Firefox addon. This is basically furloughed (as of April 2013) pending progress on Web Activities. The addon allows custom scrapers to be run on sites, with several levels of separation to both allow scraping scripts to run safely and isolated from the site receiving the data. There is some more information on this site.
The Cut-Out: A synchronization library. This is intended for HTML/Javascript applications that keep all their data locally, but want to be able to synchronize that data between devices. But it’s perhaps overpowered for that minor task. This includes several pieces, a client library, a protocol, a server, an on-disk database format, and some rough replication and balancing support. It was an experiment in minimalism.
Browser Mirror: This project has largely been superceded by TogetherJS.. This is like a screensharing system, except it works with the DOM instead of pixels - the page you are viewing is transmitted to the other party, but not a “live” page, literally just the things you see. Things like clicks are transmitted back to the original browser. Just like screensharing…? Started out as a “this can’t possibly work” project, but then it kind of worked.
webannotate: An offshoot of BrowserMirror and related ideas, taking a snapshot of the DOM and saving it, then allowing annotation of that static page. Like inline commenting without any of the tricky web overlay ideas.

Python Packaging

I don’t currently work on these Python projects, but I authored some important (and now core) parts of the Python packaging ecosystem.

pip: The Python Installer Program. A very popular installation tool for Python. I wrote this as a kind of response to easy_install, not because I hated easy_install, but because I knew it should be better and that it had relatively small problems that made people hate it. Pip solved problems people wanted solved, and it’s been very popular since then.
virtualenv: This is an environment isolation tool for Python. The functionality is now going to be built more directly into Python itself, but virtualenv remains very popular for managing projects. workingenv was an early attempt at the same concept, but virtualenv stuck because it works really well. It’s just the right level of hack to get everything to work consistently and well.

Python Web

There’s a bunch of projects that I’ve worked on in the web space, either authoring or making major contributions. I’m not actively working on any of these myself any more.

Some of the projects in this list are popular, some I think were influential. Several of them I wish were influential.

WebOb

WebOb is a Python request and response library (along with some other miscellaneous pieces).

WebOb was an extraction of the ideas in Paste (as seen below), but with a bit more opinion. It’s also the result of letting a bunch of ideas gel for a while. WebOb has a very specific scope, and implements everything within that scope in an incredibly thorough way. Its completeness exceeds any other Python project I’m aware of (and other languages to the degree I’m aware fo them). WebOb understands HTTP really well - and it knows how to both read and write HTTP, client and server. It’s primarily focused on being a library for servers, but it can generate requests and parse responses as well, making it uniquely appropriate for middle layers.

WebOb is the basis of many Python web frameworks, Pyramid/Pylons probably foremost among them. (Django and Werkzeug/Flask being the notable exceptions.)

Paste Core

Paste, as well as Paste Deploy and Paste Script, were part of a general effort to write a web framework toolkit. Each contained low-level routines for different tasks.

Paste core is primarily concerned with a set of independent tools for use with WSGI. This includes things like simple routing, static file serving (later turned into webob.static), exception reporting and debugging (later turned into WebError), validation (would become wsgiref.validate), logging, testing (would turn into WebTest), CGI routing, and other stuff.

Where possible much of this has been moved into WebOb or other packages built on WebOb.

The context in which Paste was created was one with a lot of competing Python web frameworks, where there was little overlap in the technology those frameworks used. Paste tried to create a basic foundation for sharing technology, while still allowing diversity in the actual web frameworks themselves. It succeeded to a degree.

Paste contained the first web-based interactive debugger for Python (if or when similar tools were built for other languages, I don’t know). Other tools are better now, but Paste pioneered the idea.

Page Deploy

Part of the Paste project, this is a configuration system for web apps. It has been used for a few things; for configuration during actual deployment and for application composition primarily.

Paste Script

Also part of the Paste project, this is a kind of framework for building command-line tools to build applications, and some useful commands to go with. It includes things like a web server container (does that make sense?), and by far the most popular piece has been paster create which helps setup boilerplate files for new projects.

Silver Lining

This was an attempt to create a general web application hosting framework. Vaguely like Heroku; you’d lay your application out in a certain way, and using Silver Lining you could create a new server instance and upload and update your application. It supported both Python and PHP, with the potential for other languages in the future. I thought it was really cool, but no one else really got on board. Maybe once you decide to give up this level of control you would rather just pay for the service instead of using an open source product. I’m not sure. This was kind of my last hurrah for server-side web projects.

WebTest

This is a WebOb rewrite of paste.fixture. WebTest is a tool to make functional tests of your WSGI-using web application (not specifically WebOb applications). It makes it easy to create artificial but full HTTP requests, send them to your application, and provides a bunch of helpers to inspect the results. One feature I think is notable (not challenging, just notable) is that it always contains an implicit “this request should work” assertion. Other similar frameworks force you to write assertEqual(resp.code, 200) all the time; what nonsense! In WebTest this is implied unless you say otherwise. It does form parsing and cookie storage and other stuff too.

FormEncode

A form validation and conversion toolkit for web applications. It’s built on the idea of a two-way validation/conversion pipeline, structuring an destructuring the content of HTML forms. It also includes formencode.htmlfill, a library to rewrite forms to insert form values; a novel approach, one that I think deserved to be used more, but like much of FormEncode probably obsolete now as client-side validation is more appropriate. Like many Hard Problems of web development in the previous decade, this hard problem is best avoided instead of solved.

lxml.html (and cssselect)

I didn’t write lxml (the general XML and HTML Python library, based on libxml2). But I did write the HTML-specific wrapper in lxml, which exposes HTML-specific semantics. This adds knowledge of links to lxml, and forms, and some other stuff - all of which has made lxml an excellent screen scraping library

I also wrote lxml.cssselect. libxml2 contains an XPath parser, and cssselect translates CSS3 selectors to XPath expressions. If you are curious what this looks like, css2xpath is a little webapp to do those same translations. I hope you will agree that XPath is ugly.

Commentary

An experiment in commenting on web pages. Worked as an intermediary to the web page, which turned out to be extremely fragile. Also this was before we had good HTML parsing available, so it had to use the Old Terrible Tools (like an HTML SAX parser). I think Rietveld’s UI was inspired by this.

htconsole

This was a more ambitious take on the interactive web-based debugger. Instead it was a generalized interactive console. It had some clever ideas, like inline live function editing. It exceeded by Javascript skills in some ways at the time, such that it was just a bit frustrating to work on, and I wasn’t sure to what end I’d be using it.

SQLObject

It’s been a long time since I worked on this. This is one of my first major open source projects, that actually had pickup. SQLObject is an ORM (Object Relational Mapper). It pioneered many metaprogramming techniques in Python, allowing a declarative class statement to be mapped to SQL, including SQL expressions. (NumPy championed some of these features, but I believe SQLObject was the first to use these features for SQL expressions.) The project is still active, but not very active. But SQLObject was an inspiration for features in frameworks like SQLAlchemy.

Tempita

A very small templating language. I sometimes wanted to generate strings and simply string substitution could be too hard, and I wanted a no-frills no-complications template language. I think Tempita is really pretty slick. It also avoids the unnecessary punctuation other template languages have. It also supports structured content, I think sqltemplate is really cool, though I stopped using SQL before I wrote it.

referrertrack

An App Engine tool that uses the Google Analytics API to get a comprehensive list of referrers. You could use this to watch for every referrer that came to your site, and check them off as you checked each one out. No one ever cared, but I found it useful (until it bit rotted).

Deliverance

This project is now pretty much gone. It was an HTTP proxy that would rewrite all the outgoing requests to apply styles to those requests. The idea was to allow a diverse set of applications to be styled in a consistent way, to make up a “site”.

Python Testing

Like packaging, I love and hate testing. As a result I’ve tried a bunch of tools.

ScriptTest: A script testing framework. Lets you run a script in a subprocess, and inspect the results of that script: what its output was, stderr, error return codes, and any file changes.
MiniMock: A mock library specifically for use with doctest. Using doctest, I found it was possible to make a pretty good mock library in almost no code. (It’s gotten a little bigger since then.)
WebTest: Mentioned above

Small Python Projects

In addition to those bigger projects, there’s a bunch of small stuff I did that is no longer active.

PyLogo: A Logo interpreter written in Python. A pretty complete implementation of the language, with a lot of support for calling into and out of Logo to Python.
apppkg: A very incomplete design of an “Application Package” for Python. Some of the motivation described here. The spec is probably the most interesting part.
ObConfLoader: An attempt to extract a best-practice format for locating Python objects with a string, something that Paste Deploy did in a kind of half-assed way.
CheckURL: Check for bad URLs, why not?
DebugHeaders: A little WSGI debugging tool to show HTTP headers.
CmdUtils: This was intended to be a little framework to make it easier to build high-quality command line utilities.
fassembler: This was a general build tool I built at The Open Planning Project. A dead end I suppose, but I liked the architecture. Now mostly I try to avoid building anything.
INITools: A parser toolkit for .ini files. Much nicer parser than ConfigParser.
WebError: This was an extraction of paste.evalexception, the interactive debugger in Paste.
DevAuth: A small WSGI tool to try to protect “developer” backdoor access to web applications, in a generalized way.
EmailIt: A small application to email a page to a person, kind of “share by email”. Uses lxml to capture a page in a way that can be embedded in an email.
FlatAtomPub: An Atompub server library. Atompub was supposed to be the culmination of REST thinking. This was my attempt to assimilate those ideas. No one ever actually cared about Atompub though, go figure? This was an interesting experiment, but never useful.
OHM: The Object HTTP Mapper. An attempt to create an ORM-like layer for exposing objects over HTTP. ORMs are a little suspicious, OHM was even more suspicious. These days I’m pretty over REST, mostly because attempts like this to embrace REST were so unsatisfying.
TaggerClient and TaggerStore: An attempt to create a generalized tagging library. Part of a concept of postmodern, ad hoc, and eclectic components making up a web site.
VaingloriousEye: My best project name ever. This was a little piece of WSGI middleware (a wrapper you could apply around an application) to give live referrer statistics, so you could obsess over who came to your site.
WaitForIt: A silly hack of a WSGI middleware. This would spawn threads so that a request that took a really long time could progress, while still serving up some status information to the user. People use Celery queues and stuff like that these days, but for a little while some admin tasks could take a couple minutes, causing timeouts that would then abort the task.
WSGIFilter
WSGIOverlay: An experiment with HTML Overlays, kind of a microformat-style way of applying a template to a site, using simple directives to lay the content of a page over a template. Deliverance was a far more complete implementation of the idea, and one that didn’t require application cooperation, but this is the more refined and simple implementation of the idea.
WSGIProxy: This would become webob.client. This is a way of taking a WSGI request and sending it to another server.
wphp: Embeds PHP in Python, as though a PHP app is another Python WSGI resource. Basically a hack to send a FastCGI request to an embedded PHP process.
HTTPEncode: An overly ambitious attempt to embed structured data in web requests, in a way that they could be extracted “efficiently” in certain server layouts. Kind of optimizing an HTTP request into a function call when both endpoints were on the same server.
RhubarbTart: Another framework! A clone of the CherryPy interfaces. It was kind of an attempt to bully CherryPy into using WSGI better.
CherryPaste: An attempt to wrap CherryPy in Paste-like semantics.
DjangoPaste: An attempt to apply Paste containerization to Django.
TracPaste: Trac with Paste Deploy configuration.
PoorMan: A framework I wrote! Just for the heck of it.
Wareweb: Another silly little framework.
PickyWiki: A little App Engine wiki. wikistorage is the underlying storage library for App Engine. I wanted to play around with the sandboxing App Engine provides, to allow more promiscuous code execution on a wiki.
hReviewCollector
hReviewParser
PageCollector
PageSplitter: This would take a really large page and try to split it up into smaller pages.
scrapy: This never really got anywhere, but it was my aborted attempt to create a scraping framework for Python. SeeItSaveIt would be my current thinking on the matter.
ScriptTranscluder: This is a web application that lets you transclude content, by basically providing an endpoint that will fetch a given page and parse and rewrite it so it can be included inline in a page.
dtopt: My most terrible monkeypatching hack ever! It allows you do to from dtopt import ELLIPSIS inside a running doctest, and add the global ELLIPSIS option based on that. Actually looks up the stack frame and injects bytecode into a function to accomplish this.
FitLoader: Remember FIT? Nah, no one does. This was my attempt to create something like that for Python. It’s a silly idea all around.
appengine-monkey: When App Engine first started it had some things that just felt needlessly different from “normal” Python. Probably foremost among these was that it had its own HTTP client library. This library expressed the normal HTTP libraries in terms of this App Engine specific library. That code was eventually integrated into App Engine itself. (App Engine’s own framework, webapp and now webapp2, also happen to use WebOb.)
GeoWebDNS: This was an Open Plans project to give a small REST service to map locations to areas. It used PostgreSQL, had some import tools, did it’s thing. My first and last real geospatial project.

Miscellaneous / Small Retired Projects

Twill/Javascript: Twill is an old timey functional testing DSL for web applications. This was an attempt to use the same DSL in the browser for functional testing.
WebClippy: An attempt to provide instructional overlays, so multi-step instructions on how to use a web page could be created.
htranscluder: This did client-side transclusion using the microformat <a href="resource" rel="include">content</a>

Small (maybe not current) Stuff

Misc recipes: Before GitHub when I want to try out an idea, I throw it in an SVN directory. There’s a grab bag of experiments here, all from quite a while ago.
whrandom: A small Javascript library for a pseudo-random generator.
Javascript style guide: Not a “project” per se, but my own customization of the Airbnb style guide, represents my current and ongoing opinion on Javascript style.
Browser Commander: An incomplete prototype of a kind of file manager built in the browser. Includes a small server and client library to provide Node.js APIs in the browser (which are proxied over WebSockets to the server).
git-sync: A small script to use git more like rsync for the purpose of deployment. Motivation explained some in this post.
receiptverifier: Not very active right now. This is a library to do client verification of Open Web App receipts. It’s not very exciting, but it does do a heck of a lot of error handling. It was an exercise in thoroughness. Maintained by other people now.

Toys

Wordpool: A little HTML/Javascript word-search-like game. The product of a two-hour hackathon.
xmassapp: I wanted to compile Christmas carols that I liked, without any fluff. Also it’s an example of an Open Web App.
deducer: A small Javascript/HTML game of logic. Based on the Sherlock game. Has no instructions, is not really usable ;)
fatfingers: An attempt to turn ambiguous touch events into modal selections, as a client library.