Project ideas for (what’s left of) 2020

While I’m conflicted with the need to find a new job, versus using this break from employment to do things I don’t otherwise do, one way or another I feel compelled to make something. I started writing down all the things I’d like to do and now I have to choose one or two.

I’ve embedded Hypothes.is on this page, which allows inline comments. The controls are floating in the top right of the page. If you have thoughts or resources related to any of these ideas that would be much appreciated.

I also hope some of these ideas will also spark your own imaginations, or, as so often happens to me, brings back to mind some forgotten thought you had meant to pursue.

Job
1. Find a job
2. Figure out what job I want
Web projects
For the kids
Natural language
Programming languages
AI
1. Binocular vision and image processing
2. Neural networks from scratch
Not (just) computing
Conclusion

Job

Find a job

I’m unemployed. It’s probably a good idea to find a job.

Pros: I do need to find a job

Cons: I usually have a job, and this is a unique moment when I don’t have a job

Figure out what job I want

I’m unemployed, I should probably figure out what I want to do.

I have a sense of what I want to do, an intersection of a Kind Of Work, and Topics Of Interest.

The Kind Of Work I want to do probably goes under the title of Product Engineer or Research Engineer, but those titles aren’t used universally or particularly well understood. I want to make new things, engaged individually and as part of a team in the ongoing discovery of the product scope. I’d like to be part of a small, diverse, multidisciplinary team.

The Topics Of Interest would include:

New user interfaces (including voice)
Edtech with a focus on pedagogy (I’m not as interested in tools supporting bureaucracy of education)
Efforts directed at executive function
Productivity tools, using the lens of executive function (how can our tools support the executive function of users)
Psychological well-being
I have a technical interest in user-empowering automation tools
I have some interest in collaboration tools
I do enjoy making development tools, but I’m not sure it’s where I want to go with my career

My tasks here are (a) working better at describing my goals, and (b) verifying the goals. What do I really want to do, and what do I want to want to do?

Pros: Sets me up to find the right job for me

Cons: Encourages me to focus on something that might not exist

Web projects

Web-based document store

This is a way for applications and personal documents to interact.

This thinking came out of the blog post We Need Open Hosting Platforms.

The idea for the system is really just documents. These could be notes, essays, images, slideshows, or anything that the user authors. It’s not like a filesystem that also contains all kinds of system files and programs that don’t belong and aren’t managed by the user. It is like Google Drive or Dropbox or whatever, but… open?

It’s also not just named blobs. There’s lots of options for data storage, but it’s just a bunch of bytes and a content-type. Here we want to include other information:

How can the document be edited?
How can the document be validated?
Core metadata (e.g., title)
Extensible metadata (e.g., images have sizes)
How can the document be viewed in public?
How can it be embedded in a web page?
What permissions do other people have? What permissions are possible?
Versioning
Difference display
Copying and linking
Commenting
Collaborative editing

The idea here is that there is a framework for creating new kinds of documents, with both general UI (applicable to any document) and specific UI for different document types.

Things like “editors” are static bundles of JavaScript and HTML that obey certain protocols. Once you’ve got a document you might upgrade the editor, but you’ll also have a static copy of the editor in perpetuity.

A goal here is to allow open source developers to create new and interesting interfaces and interactions, without any of the worries of hosting. And users can use them without any worries about being bound to a service, and without placing any burden on the open source developer to maintain any kind of service.

Pros: I think the F/OSS community lacks innovation in the area of user experience, and the lack of a place to do that work is part of it. Thinking critically about documents in the abstract is interesting to me

Cons: It’s pretty ambitious and I probably don’t have the endurance to make this work really happen. It’s only as good as other people might make it

Directed and automated exploratory testing of web applications

I think this would be a great business, and the while the automated testing environment grows year-by-year and new ideas emerge, I still think there’s huge unmet potential.

The web platform is pretty well-defined and sandboxed. At any moment a web application page supports dozens, but not thousands, of reasonable “next operations”: a click, keyboard entry, pasting, dragging, so forth. The protocols that support web debugging also expose much of the metadata you need (the DOM itself does not). So I believe automated exploration of a web is feasible and useful.

I hate scripted testing, such as Selenium. Most breakage of Selenium tests are because of tooling reasons, and of the remainder most are because of the test detecting intentional and correct changes. This testing punishes positive change instead of enabling change.

The most conservative attempt to fix this test breakage problem is “resilient selectors”, that deal with fragile selectors. This is about the most modest fix I can imagine!

A testing system built close to the web rendering system itself could fix many of the tooling issues, and many of the timing issues. Puppeteer is an improvement… but it’s a change to the paradigm that I think would really help: instead of testing against a script, the automation framework should just try things with an emphasis on reporting the result instead of comparing the function against a separately-maintained ideal (in the form of a scripted test).

I’ve put a lot of thought into this at Mozilla, though my pitches fell on deaf ears. I’m as optimistic about the idea now as I was many years ago when I started experimenting. But this is a Hard Work idea, not a Have Fun idea. I’m not opposed to hard work, but I’m not going to do it just for the fun, because it isn’t that fun!

Pros: I’m personally convinced of the potential, and there are big interesting unanswered questions

Cons: It’s too big. It’s a developer tool, which is fine, but that’s probably not what I want to do long-term

Simple, reflective CMS

Can a CMS be self-hosting? If you just do it as a file editor with the right sort of authentication, I suppose so. But can a quality CMS be self-hosting? One that doesn’t invite you to corrupt the app itself, that can be updated, that is extensible without going deep into the code, that handles both the individual bits of content and the aggregate content and the change of content over time.

The CMS wouldn’t actually be fully self-hosting, I imagine a an underlying storage and rendering layer that is fixed, but not UI-specific. And it requires some things like a rescue UI that can’t itself be destroyed.

I’d want to prioritize a small codebase. Just making a featureful CMS isn’t a useful exercise for me.

Pros: I’ve been thinking about this sort of thing a long time

Cons: I won’t do anything with the result, and the world is full enough of CMSs that no one else will either

Personal web automation

Most of the web ideas I’ve had at Mozilla now leave me feeling bored. I’m not trying to be a professional right now.

But web and browser automation is still stuck in my head. Right now can I make something useful for just me? What processes should I be optimizing? What interactions need different controls? How would I want to access those controls?

I might start by simply coding up the extension, and extending it in-place instead of creating any user-programmable tool.

Pros: Could be personally useful. Thinking this stuff through is interesting to me, and now is a good time for self-experimentation

Cons: It’s just diddling around with the same stuff I’ve been doing for a long time

Contribute to a web clipper

I came upon Rumin Web Clipper recently (though I haven’t reviewed the current state of the art). It’s the sort of thing I’ve worked on before, and I have a lot of ideas in this area, but I don’t want to be in charge of anything.

I could find a site or two to try to scrape from and submit some PRs. Then the next step is to take some of that data and put it into a site: the other half of structured data extraction.

Data extraction and injection is an interesting lens on automation.

Pros: My contributions could be useful even if I don’t stick with it. I can focus more on what I care about and less on infrastructure

Cons: I might be trying to colonize someone else’s project

Remake Hotdish. Also make bots

One idea I didn’t get to with TogetherJS was Collaboration as a Skeuomorphism for Agents: using collaboration the mechanism for smart bots to act on your behalf.

Also all of Hotdish felt like something not-quite-complete. Unfinished work hangs around in my head.

Could I remake Hotdish in a simple way, with an eye to external bots?

Using a Firebase backend would make deployment fairly light.

Pros: Gets some old ideas out of my head and into code. I feel better at a lot of the technologies than I was before. I’m really interested in the bot angle

Cons: Maintaining and using an extension is kind of a pain. I’d need people to actually use it with me. I can always recruit my wife, but will that be sufficient? And Chrome doesn’t have sidebars, which means the primary UI element I imagine wouldn’t be portable. I’m pretty sure the work would disappear into obscurity

Hypercard-like thing

This idea is perennial for many people. I don’t think we all agree what it means.

To me this is a composition tool that can create simple, content-oriented interactive sites, with some simple core metaphors like buttons/links and cards.

The use of slideshow tools in schools make me more confident that this remains an important use case. I really hate that slide decks have become a significant medium in schools, but the positive side is that they are a good medium for short-form multimedia documents. The slide is a valid unit of structure, akin to the paragraph. The composition tools are flexible and don’t privilege text over everything else. It’s almost an accident that the purported purpose is giving a presentation.

I see aspects of this embedded in newer online educational tools like Flipgrid and Seesaw. They raise the bar, but maybe also prove the point.

Pros: Feels of-the-moment. Some of the technology feels familiar. Other parts feel familiar but tedious (I’m not going to write a good drawing tool). Jumping onto someone else’s project might be possible (but whose?)

Cons: Large scope. Everyone projects different ideas onto Hypercard. Cannot be finished

For the kids

Child-focused Wikipedia browser

Recently my father asked me what ever happened to OLPC. I looked it up to be sure – the domain is there! – but mostly the idea has passed. I found this retrospective about OLPC from 2016 and it got me thinking.

Was OLPC too focused on constructivism? That is, the idea of students actively constructing their education. I like constructivism, but now I’m skeptical if it’s actually correct. Also: is it the right role for a computer in these environments?

A computer can also be a font of wisdom, a deliverer of information. Maybe that’s enough. And of course Wikipedia is a nexus for information.

If we just plopped Wikipedia on a kids computer, how well would that work? Certainly it doesn’t work if you are pre-literate, but that’s only one of many problems.

What would it look like to make something that a kid could sit in front of, as their first ever computing experience, and experience Wikipedia? Probably there would be a place for speech output, maybe speech input. Nonlinear presentation of articles might be good. Using simple.wikipedia.org might be useful (though I assume it’s only in English). Returning to and repeating content will be important for kids.

It feels like it would be easy to get started on this, while the final product would be a long journey.

This could turn into a different browsing experience than the traditional web browser. But it is truly browsing, Wikipedia exemplifies the Old Web: linked documents containing information. But the content is more normalized and consistent (including editorially consistent) than the web, making it easier to reimagine.

WikiReader is an example, though not an example of much.

Pros: Interesting to think about. Getting started is easy. I live with some experiment subjects. I think Wikipedia is one of the best things we got going

Cons: I feel a little under equipped. It doesn’t satisfy my desire to do some technical deep dives

A foundation for a series of microworld environments

Mindstorms introduced the idea of Microworlds, but it never went anywhere. Where could it go? Is there an environment that could support many microwords? Would it look like Scratch, Logo, or something different? Or a framework for building environments?

It might be unreasonable to even consider a foundation without first having several actual microworlds. But supporting small domains and ideas is part of the goal, so this isn’t a blocker. For instance:

Buzzfeed-style quiz generator
Eliza-style chatbot
Word explorer (e.g., what words are palindromes? What words have 6 letters where the third is T?)
Simple music generation
Cypher building
Math worksheet creator

It’s possible that it would be better to design these and not worry so much about the implementation. As long as I’m not implementing maybe there can be more ambitious examples:

Projectile physics
Ecology simulation
Map generation, erosion
Equation exploration (explore things like how numbers increase by square/cube/exponent)
Writing cues
Choose your own adventure
Foreign language exploration (what happens when you translate words vs sentences?)

Pros: I think we’ve leaned in too far to “learn programming” but not nearly enough into “learn with programming”

Cons: I haven’t taught these things myself, I just noodle around with the kids. I don’t feel pedagogically competent

Jslogo environment / IDE

This Logo interpreter is a decent generic logo interpreter. (I learned Promises while contributing async support.) But when I use it with my kids it’s kind of awkward, and we always lose our place.

It would be nice to make use of the interpreter in a different environment.

First, I’m thinking less interactive/REPL and more about creating a full program. Right now the typical program we write is like:

cs
repeat 4 [forward 100 rt 90]

That is, always start with cs (clear screen) and then a bunch of commands so it can be re-run from the beginning after adjustments.

I imagine making these repeatable code blocks the standard interaction pattern, treating them something like unnamed functions. Then if you want you can name them and compose them as well (without the clear screen of course).

It might be worth revisiting the history to look for other ideas (or mine some comments).

Pros: I know the language and am familiar with the implementation. Block programming has its advantages, but Logo also has affordances for children. The environment is a really big deal and I have an intuition on some improvements, but there’s also lots of active questions. Deploying is fairly easy, so it could exist in the wild even if I don’t pursue it indefinitely

Cons: Logo is a fairly obscure language. I’ve been impressed with how well the kids do with block programming. Turtles are OK, but also a little eh

Lo-fi text web page creator

This is something my younger daughter vaguely requested, and makes sense to me.

Imagine an 80s computer or roguelike, where the screen is your canvas and characters are your paint. So only absolute positioning on a strict grid. Emoji are the only graphics. Letters can be sized 1x, 2x, 3x, etc.

I’m not sure if this requires a monospace font, and I’m not sure what the exact spacing of the font would be in that case; does a clean letter aspect ration matter (e.g., 2x1)?

I think there would still be click events and that sort of thing, so you could add dynamicism to the page (that’s the fun part). I tried creating a literal grid and was unhappy with it, so it would still be absolute positioned divs, just with fixed alignment.

This would require some emoji picker, like emoji-pane or emoji-picker.

Pros: Fun to try out with the kids as they build web pages for themselves (which they are interested in). I think I’d like the aesthetic of it. I see a lot of advantage in simplifying compositional tools for kids

Cons: Deployment can be annoying. Glitch makes it easy to get a site up, but not easy to build editors, so any CMS has to be a whole production. Kids might only play with it a little then we all forget it

Can computing feel like it has a sense of space, a home?

When I’ve tried to interrogate my own sense of nostalgia in computing, something I’ve wanted to recapture is the sense of space and a home in your own computer. A hard disk or even a set of floppy disks give this sense of a place, where you acquire and place things, and where you move around. BBSs had similar feelings, even if it was a clubhouse instead of a home. This was part of the early web as well, but now the web is like visiting a public space. I can visit Facebook, Google Drive, or any other social forum, but I’m just a visitor.

I’m especially noticing this while watching kids use iOS and Chrome OS, where the experience feels particularly generic and impersonal.

I don’t know how to bring back this sense of personal place. I know it’s not flat, reorderable lists. It’s not algorithmically ranked items. Maybe it’s important to be spacial. Or maybe it’s just important that it’s sticky and personally constructed. A place with personal paths, rediscovery.

Pros: This is an attribute I’d like to have in things-I-make, and I think there might be general lessons to be learned. So even if I come up with something that doesn’t work well or I abandon, I think I will have learned something

Cons: I don’t know what I’m looking for or how to know it when I find it

Program your parent: a board game

This is similar to Robot Turtles, but relies on the parent to be the computer.

Like Robot Turtles the programs are built with cards. Though maybe it would involve more printable and consumable cards (e.g., things you write numbers on, or “functions” that are a page with slots for parameters).

I imagine two possible goals: one is simply to make your parent do silly things. The second is to solve puzzles, both specifically and generally. (Maybe the puzzle prize is to get your parent to do something silly, like you win silly card/routines)

This is going to rely on the parent both faithfully executing programs, and doing it with enough transparency to help their child debug the program.

Pros: Fun to do with the kids. Easy to share

Cons: I don’t want to create yet another puzzle-based beginner programming thing

Create a Scratch extension

You can build extensions to Scratch. Some speech-related extensions would be easy to imagine. There’s also Snap! to consider.

I’ve also wanted an equation block, that doesn’t require dragging a ton of math-related blocks together to make a simple equation like 100 - (x / 4). I bet it wouldn’t be too hard. The fact variable names are free text and editable does make it harder though, not to mention the fundamentally unnamed aspect of many getters. Maybe this is a bad idea.

Pros: Without too much work I could make an interesting change to a programming environment my kids enjoy

Cons: It’s hard to distribute extensions. Probably what I learn won’t be generalizable to anything else

Natural language

Chat bot builder

I think it would be cool to let the kids write a little chat bot. Wiring it up on the web is really easy (well, easy in Chrome).

I’ve already extracted a library from Firefox Voice and played around with a site on Glitch.

I want to allow some of the programming to happen as part of the dialog. For instance, if the bot doesn’t respond well, you should be able to say: “you should have said [X]”, and add to the program on the fly.

I’ve had a hard time focusing the kids on question of what kind of entity they’d like to create. Does it have a personality? Does it have a purpose?

Pros: I have all the pieces, and it’s stuff I’ve been doing lately. Simple and hopefully fun

Cons: The kids aren’t as excited about this as I think they should be. Not sure about the phrase matching, but I don’t want to turn this into Dialogflow

Use an intent parser to build a text adventure

I checked out and did some tests with intent parsers while making Firefox Voice, but I never actually seriously used one. If I did use one it would be Rasa.

Making a text adventure (ahem, writing Interactive Fiction) using an intent parser seems interesting. Parsing always frustrated me as a user, and maybe these new techniques can help.

Pros: Fills out my domain knowledge a bit, while also using things I’ve somewhat familiar with. The result should be fun and playful

Cons: I don’t have a story idea! Publishing something based on Rasa may be hard (since I don’t want to maintain a server)

Voice-activated news reader

One of the concepts from my work in Consumer Voice Products was a voice-activated news reader. It would be a way to browse newsy sites and get articles read to you, with a voice interface to control the read back (“skip”, “summarize”, “expand” etc). The ideal place to do this is on the phone, as a kind of podcast alternative, but deploying on the phone requires lots of boring engineering I’m uninterested in.

So instead I could try making it in the browser. Maybe I could use Howl or something based on Speech Commands for voice recognition. Or just leave the mic open for transcription as a prototype phase, though feedback might make that impossible.

“Give me my daily update” is a common home voice assistant feature, but it does nothing for me. I imagine this as something you’d interact with as an alternative to radio.

Pros: I don’t think this would require a lot of work to prototype. I think the idea is a good one. It could be an interesting demo

Cons: I think it will rely on keyword spotting and it’s not feasible for me to collect new words to train models, so my commands will have to rely on what’s already out there. Also maybe this isn’t fun

Mock out a broad natural language interface

What if we did most of our computing using natural language? This is mostly a brainstorming exercise. Take a some typical computer interactions and reimagine them entirely with natural language.

The result probably imagines a refactored experience. Things that are distinct steps now would be combined, while other situations would require conversational or other multi-step interactions. It might imply new things, like named staging locations, or other shortcuts.

Pros: I think there’s a certain naiveté to our voice interfaces, but I don’t know what we’d even be trying to get to. Maybe I’d have some more ideas?

Cons: The result is just… a blog post?

Brain-in-a-jar dialog game

The premise: a person has had their consciousness extracted, or at least simulated in a computer. You, a temp/Mechanical Turk, have been hired to make this artificial consciousness feel better. If it realizes it is a robot it will be rebooted.

Technically I’m not sure how to do much of this. But it could be fun to try. Different utterances would increase or decrease the existential skepticism points.

Pros: Learn some more NLP. Play around with fictional emotional ideas. Maybe I can share a rough playable game

Cons: I’m not sure how to drive a useful plot forward. There’s a game design aspect that I know nothing about

Interview chatbot

Not a job interview! This chatbot doesn’t pretend to be human, doesn’t even pretend to really talk to you. Instead it interviews you. It asks questions, maybe asks followup questions, and saves the responses. Might operate better over email than chat. Make your own oral history!

Pros: I think I would like the result of this. Self-documentation is important. It would be easy to manually (Wizard Of Oz) prototype

Cons: Most people (including myself) will just try it for a week or two then forget about it, or worse feel guilty for not continuing

Emotional Tamagotchi

Your little entity/bot gets lonely and sad. Instead of feeding it fake food and whatnot, you must feed it emotional support and positive feelings.

Secretly it’s all sent to actual sad people to make them happy. A language transformer covers up the humanity.

Nah, I probably wouldn’t do the second part.

Pros: Kind of easy. Maybe involves plugging existing models together. I’m curious what it would feel like to use

Cons: Easy to hack. Lacks actual emotional core

Programming languages

Try to come up with a new category of programming syntax

There’s only a few syntaxes in programming languages:

Algol
Lisp
Forth
Smalltalk
APL
SQL?

People keep revisiting these basic syntaxes, and even then it’s like a power law of popularity. Just out of curiosity, are there other things we could try?

Smalltalk-72 had some interesting ideas. It was very expansive and inclusive. Smalltalk-76 (which is pretty much what we have today) pulled a small core out of that, but maybe there’s other foundations of syntax to be mined from those ideas?

I would probably just use this as a brainstorming exercise, and then mock up some results and write a blog post.

Pros: Pretty simple and finite. Hopefully fun. If I can’t come up with anything, so it goes

Cons: Just a silly little exercise

Functional core notebook

Order of execution is a common problem in notebooks, as well as experimentation. Notebooks are often exploratory, and you want to try different versions of functions and processes.

This would be a lot easier with functional and deterministic code. You could memoize results and understand dependencies.

I also feel like lots of the imperative and stateful parts of notebooks are the setup, such as acquiring data. So I imagine a kind of “shell” in the notebook where you do imperative setup and the cells where you reshape data and do calculations.

The name has some similarity to Functional Core, Imperative Shell and probably some similar motivations, but encodes the idea directly into the tool and languages.

I’m wondering if I could do this with a deterministic subset of Python. Things like “import” are environment setup, and maybe even function definitions, while there’s innards that are deterministic.

It may be OK for the code to stateful, if the statefulness is part of the analysis. That is, you could have a cell like:

most_populous_cities = []
for city in all_cities:
    most_populous_cities.push((city[“population”], city[“name”]))
most_populous_cities.sort()

It both creates a new global variable, and modifies it in place. But if all_cities is the same then the result of this calculation will be the same. You could consider its input to be [all_cities] and its output is an addition to the environment [most_populous_cities]. (Also city is set and almost certainly is not important, but that’s the kind of detail we’d need to figure out. Also it’s not OK for another cell to modify most_populous_cities – modifications need to create new global variables.)

I think Observable has bits of this in its concept?

Pros: I’ve played with ASTs and found it interesting and productive, and this general concept is one I’ve been thinking about for a long time

Cons: Do I make a new notebook interface? Maybe… and maybe it’s okay if it’s scrappy, but it’s kind of a distraction

Something with symbolic execution

I’ve never seriously done symbolic execution or even used it. But I’ve found the Wolfram Language interesting, and find it really interesting as something quite distinct from more formal functional languages.

I guess I should check out this list, SymPy, pySym (?!?), Sage, ExpoSE (JavaScript), Prepack (JS partial evaluation), peval (Python partial evaluation).

Pros: There’s lots of material to start with, and it’s easy to come up with early experiments. It’s the kind of problem I can sit down and tackle for a day at a time

Cons: Not very human-centric. Lots of prior art is actually about a kind of speculative execution of binaries for security checking, which I find boring

Attempt to reverse execution

Given a return value from a function, what inputs or actions might have led to that?

I presume this is pretty hard, and possibilities explode quickly. But I’d like to understand why it is hard, and the easiest way might be to try until I fail.

This will require new “possibility” objects, and probably kinds of partial evaluation (partial unevaluation?).

Maybe I’d pick a subset of JavaScript (maybe Python) and use the AST.

Pros: I can just jump in and start

Cons: I know the result is failure! I just don’t know where and when

Partial evaluation as the execution engine

I think this might be what I actually am thinking of when I think of symbolic execution.

Partial evaluation is taking an expression and evaluating the parts you already know. So if you see this:

# My apologies for this lazily constructed example...
def calculate(x, y):
  if x:
    return x + 10
  return y

print("Answer:", calculate(10, some_func()))

Then you can look at how calculate() is called with one known value (10) and then see that if x: will be true, and so on, reducing it all to print("Answer:", 20).

There are limitations to how well you can do this in an imperative language. For instance in this case we’ve optimized away the call to some_func(), and that might not be correct: the function might actually do something important, even if its return value is thrown away.

But this is also a feature: by eliminating some_func() you can simplify the program, or at least find the simpler specific program that lives in the complete program.

A long, long, long time ago I made something called tcl->scheme that was a TCL interpreter written in functional Scheme, with the idea that if you apply partial evaluation you can compile TCL into Scheme. This only works to the degree that TCL doesn’t use overly dynamic features (like eval), but even if it does you still have a working intepreter to fall back on.

Maybe combined with the functional core notebook concept this could create something interesting.

To take this a bit further:

The language has to have the ability to be expanded out into expressions. S-Expressions are very appealing, though more eclectic syntaxes aren’t that much worse. But Python’s indentation-based syntax is problematic.
All objects need source code representations. It’s important to me that we can turn all the intermediate forms into actual source code, to see how the source actually evolves. <SomeObject at 0x0292a43e> isn’t going to work.
But maybe representations can also be richer than text.
It would be nice to track where literals came from, and use that to detect if they are editable.
It would be interesting to include tracking information so you could see why an object became what it is.
There doesn’t have to be any undefined errors: if you refer to something that doesn’t exist it could be an as-yet-unspecified input. Or code that hasn’t been written yet. The system can “think about” (partially evaluate!) incomplete code.
Because execution happens opportunistically the language can’t be imperative, it can’t be a sequence of things being executed. Technically it could, and partial evaluation systems can take that into account, but the rewritten code is too complex for our purposes.
You can imagine a multiple worlds execution, and doing diffs of the result. If I have three different inputs, and diff the resulting partially evaluated programs, what do I get? If I have two implementations of a function, how do they change the overall execution? This could be an interesting way to think about code changes.
There’s going to be lots of “smart” primitives: routines that are built into the execution engine and know how to execute the source itself. There’s definitely a powerful and general execution engine (the thing I write), and a much less powerful user-programmable expression engine.

I think Smalltalk-72 played around with these ideas, and looking into it would be helpful.

The JavaScript parsing and AST-rewriting ecosystem is healthy, and I think I could use it as a starting point.

Pros: I’ve thought about this a long time, and writing it down makes me excited. This is fun programming

Cons: I’m not sure I can get it to be a useful environment (simply because it’s a lot of work), and if it’s not useful then I can’t confirm the utility of the concept itself

Update:

I’ve started on this some, using Recast to operate on JavaScript ASTs. Working in github/ianb/partial-language.

Right now it’s stuck to JavaScript syntax, though not necessarily JavaScript semantics. I expect this may be a problem. A series of transformations looks like:

// Original input, note order of declaration is not important:
x == 1 ? func(10) : !y;
const x = 1;
function func(p) {
    return 1 + p ** 2 / 4;
}

// Rewrite 1 (function and variable definitions consumed and remembered):
x == 1 ? func(10) : !y;

// Rewrite 2 (x and func substituted):
1 == 1
    ? ((p) => {
          return 1 + p ** 2 / 4;
      })(10)
    : !y;

// Rewrite 3 (1 == 1 substituted, parameter moved into function):
true
    ? (() => {
          const p = 10;
          return 1 + p ** 2 / 4;
      })()
    : !y;

// Rewrite 4 (?: evaluated, p substituted):
(() => {
    return 1 + 10 ** 2 / 4;
})();

// Rewrite 5 (arrow function with one return removed):
1 + 10 ** 2 / 4;

// Rewrite 6 (math):
1 + 100 / 4;

// Rewrite 7 (more math):
1 + 25;

// Rewrite 8 (last math):
26;

It’s cute, doesn’t take too much code. I’m not sure how far it can go. Having more control of the syntax would be nice, for instance substituting func(10) for an inline arrow function is not the best, but there aren’t block expressions in JavaScript.

AI

Binocular vision and image processing

For some reason binocular vision isn’t a common tool in robots and vision. Different distance-sensing tools like Lidar seem more common. Lidar is more expensive, but it creates a concrete depth field. But is a depth field even enough? It seems interesting to actually include visual information and depth information together.

I’m specifically interested in how binocular vision works with Convolutional Neural Networks (CNNs). CNNs are the standard tool in image processing, where you have a neural network where you not only train some kind of detection, but you also have a bunch of “convolutional” layers that are essentially image processors. These typically learn to find edges or other features in the image, by using a “windows” (fixed size squares, 3x3, 5x5 etc, taken from the image) and training matrices that process those windows. (I guess for this case wide windows would make sense, there’s no need for the windows to be square.)

In a typical setup you pass in a red, blue, and green layer, and you let the training process discover things like the meaning of color. And if color isn’t important then it can learn to just add them all together. Instead of these three layers, you could just pass in six: red-left, blue-left, green-left, red-right, blue-right, green-right. And then the CNN can learn how to use the differences between those layers to understand its environment.

The first thing I’d have to do is create a binocular camera. There’s some expensive options intended for robots. Unfortunately I can’t fit two Raspberry Pi cameras on a single unit. Maybe I just need two cheap USB cameras. Or maybe there’s a clever way using mirrors to turn a single camera into a binocular camera?

Next I have to figure out a training process. What am I even training for? One possibility might be to use an ultrasonic distance sensor to get a real distance, and then just train a model to see if it can predict that number from the image. The ultrasonic sensors aren’t very accurate and only see a small cone, so I’m not sure if it’s great training data. But if I train it to make a depth field, and only train to the part of the image that I have a measurement for, will it magically give accurate numbers for the other positions? It would be better if I had a full task in mind to train against.

Pros: It’s kind of interesting, it’s nice to think about real physical things instead of just information on the web

Cons: Figuring out how to get images and ground truth data for training is hard, I don’t have many ideas

Neural networks from scratch

I took a course that covered neural networks, and it was OK – it got me to go through a bunch of exercises and covered the material in a somewhat structured way. There’s a huge amount more I could try to do as a practitioner: learn PyTorch, try different architectures, do more data shaping and feature selection, etc. But there’s also the basic theory, some of which I felt pretty comfortable with, but not all. Backpropagation especially: I get that a differentiable algorithm is useful, and in theory how that would lead to backpropagation, but there are so many details that are vague in my mind.

Building a neural network system from scratch seems like a great way to solidify my theoretical understanding, even if I’ll never again work at that level and just use frameworks in any practical setting.

I’ve seen a couple resources on this: Neural Networks From Scratch, a more casual Machine Learning for Beginners: An Introduction to Neural Networks (with another for RNNs and CNNs), a similar How to build your own Neural Network from scratch in Python, and another.

Pros: I can just follow along with someone else’s instruction. I’m pretty sure what I learn will be useful to me

Cons: Not directly creative

Not (just) computing

Physical workspace optimizations

In most ways my computing behavior has become less optimized and more aligned with broad consumer trends over time. This is a result of (a) not being willing to spend time customizing things, and (b) wanting to experience normal things so I can relate to normal computer users.

But there’s a part of me that thinks I’ve overdone it. Maybe micro-optimization are silly, but they can also be a tool to focus. The awkward in-between experiences in a computer are genuinely dangerous for focus. I feel my mental load is finite, and using it up on stupid window fiddling and such spends down my mental capacity.

I’m interested in what it might look like to customize my physical environment. Things like the Bloomberg Terminal keyboard are interesting, or an editing keyboard or what have you.

Minimal keyboards are kind of the style right now (I’m using one myself). But maybe that’s a silly optimization. Lots of stuff on a 101 key keyboard isn’t for me: function keys I don’t use, a number pad I don’t use. Does it actually follow that lots of keys aren’t useful? Or is it that I haven’t put in the time to make sure the keys are useful for me?

It’s pretty easy to just add more USB devices. I can have multiple keyboards attached, multiple mice and trackpads. I don’t have to assemble a single custom device (which I’ve decided is infeasible), I can just throw things together, apply some labels. Maybe it’ll just be silly but I don’t know.

One goal would be to use keys for “absolute” controls. As an example instance ⌘-[ moves one tab to the left (very relative), while ⌘-1 moves to the first tab of the active window (somewhat relative). Instead I want a “focus Gmail” key, and my current focus shouldn’t matter.

Obviously there’s a lot of general physical arrangement to be figured out as well.

These 16 key kits seem like an interesting macro option. Interesting things here too.

Progress: I’m trying switching back to my old Kinesis Advantage keyboard. The actual keyboard is a bit old and smushy and not 100% reliable, so I might have to re-up if I decide I like it. I’m not sure I do? The movements come back quickly though. Interestingly you can fit the Apple Magic Trackpad in the space between the two hand-wells. I’m not sure if that’s actually an improvement.

I’m trying to make it easier to move between sitting, standing, and using a treadmill. I’ve had the treadmill a long time, but haven’t used it that much because it means moving from one workspace to another. This is OK, but puts my productivity in conflict with healthy body positions. That’s not a good way to set oneself up to make healthy choices. So instead I’ve made a platform that can be slid over the treadmill, with a chair on top, and I move that back and forth while everything else stays in place.

Pros: I’ll become an expert hacker! Maybe I can type with a different keyboard for each hand. Cryptic neon-green characters will flow from my screen

Cons: It makes my environment less portable. I might not like the result

Something involving executive function

This probably starts with study, and leads… somewhere I can’t determine right now. I’m just starting to think about these things.

I’m very interested in the mental models that underly our action.

Pros: I’m pretty sure I could use what I learn in almost anything I do. I’m sure there’s important theory for me to learn

Cons: This doesn’t satisfy my desire to make. That desire is considerable

SMS-based dispatch system for the local neighborhood watch

There’s a watch/patrol/safety group in my neighborhood. It’s a little unclear how residents can get in contact. I played around with some text automation using Integromat and Twilio and while some parts were quirky, it wasn’t too hard to do, nor very expensive to maintain. Maybe I could set something up for the group using this?

Pros: Helps connect me more with the neighborhood. Right now safety is an important political issue, and this particular effort seems concrete and positive. The hard parts are all in working with and designing with these other people that I don’t know well

Cons: Nothing particularly technically involved. Doesn’t let me do the kid of somewhat reclusive and focused work I’m looking for

Freshen up Southside Greenway proposal

Some people made a pretty involved proposal to turn my street into a greenway. That would be totally incredible, even though it feels outside of the bounds of what we can imagine in this city right now given everything else that is going on.

The proposal has a lot of good work, embedded in a pretty hard to find location. I could make a site and take that content and maybe help keep it alive in the process.

Pros: The material is all there, so this feels like there’s a lot of leverage to the work

Cons: The material is all there already

Long-form slow-paced email correspondence platform

Reading about famous correspondences of the past, I am somewhat jealous. We have social media but it’s a little light on interpersonal communication. Obviously we can just communicate, just do it, but I do think there’s something useful about an understood and agreed-upon structure with some supports and reminders.

I’m thinking about correspondence that is rate-limited to once per week. So if I receive your letter on September 1st, I can send a reply any time but it can’t possibly arrive before September 8th. This is the opposite of texting. Because of these shared expectations the conversation never has to “finish” but it also isn’t left dangling.

This is probably best prototyped with a couple people and no actual automation.

Pros: Prototyping would also presumably lead to some interesting correspondences. The whole thing is very simple

Cons: Well, I do have to find people to prototype it with. What if it turns out to be boring? Or one of us ghosts the other one? Is this a guilt factory?

Game world

Ultima VI really caught my imagination as a child. Mostly because it felt both complex and comprehensible: you could see the implementation, but it still felt alive.

Nostalgia and escapism makes me want to recreate that kind of simple emergence. A little world that can exist without me and yet is by me. Tiny beings with their own tiny internal world.

I’m sure there’s game frameworks that would be a great basis for this, but there’s so many options that I’m a little lost. What abstract level lets me do the parts I am excited about and skip the parts I’m not interested in?

Pros: This has been in my head much longer than any of these other ideas! It feels fun

Cons: Escapist and it’s unlikely to lead anywhere

Possible resources: fgmk - game and editor, fgmk engine, pixel editor, Reldens top-down MMORPG, JS RPG engine, GitHub category, tutorial on MDN, another tutorial, tile editor, artwork, especially LPC

Remembrance Agent

I came upon this concept in an Invisibilia podcast episode, where they talked to Thad Starner about his 90s personal wearable note-taking tool, the Remembrance Agent.

The idea of note-taking and retrieval is simple enough, but as presented the tone feels different from other note taking, more stream of consciousness, more about connecting the present self to the past self. Maybe it’s just the word “remembrance” that sticks with me.

I honestly don’t know what I would do in this area. Maybe nothing, but I like thinking about doing something.

Pros: Seems cool

Cons: What am I doing again?

Arcology requirements and constraints

You might know arcologies from SimCity. I encountered Arcology: The City In The Image of Man in my college library’s oversized book section. Or one of Buckminster Fuller’s proposals like the Old Man River’s City project.

It’s an old fantasy and it seems entirely unreasonable – even undesirable – now. But I still wonder… there’s all these visual concepts, but what would it really require in practice? What amount of space or energy is required for the mechanics? What kind of movement and transportation is required? How much intake of material would a community need? What kinds of services?

I guess the result would be a spreadsheet?

Pros: Not computers, something different. Fairly accessible research process

Cons: Not sure what I’d do with the result

Play with fractals

What’s the state of the art in fractals these days? And chaos theory? These were fun exciting new things in the 90s.

While it’s kind of an aside, I came upon this tutorial for compiling to WASM that looked interesting. It steps through creating a simple language (tokenizer, parser) and compiling it directly to WASM. The example language it uses it suitable for rendering the Mandelbrot set (and not a lot else), but extending it only for other fractals would probably be feasible and interesting.

Pros: It would be nice to get clearer on WASM, and fun to do a project around it

Cons: I could probably just look up some fractal explorers, play with them, and call it a day

Write fiction

I’ve twice tried and failed Nanowrimo. Maybe I should try it again? Fantastical alternate reality or absurdist political thriller?

Pros: I do think about this often. Unlike many things here it’s not something I’d satisfy in my professional life. It’s escapist, disconnected from the current moment

Cons: It’s really big task and has no outward result unless I finished, which is unlikely. Maybe I’m between jobs, but even doing silly projects does move me forward in my career, even if it only clarifies my own feelings towards my work. It’s escapist, disconnected from the current moment

Document my beliefs

I think I believe things, but what are those things anyway? If I write them down maybe I’ll find out.

Pros: I think about these issues often, and am constantly trying (and am overwhelmed) to determine what my place is in the world. I’m going to put that effort in regardless, but doing so in a conscious and structured way might at least move me forward

Cons: Maybe I’m not ready to decide what I believe, maybe it simply has not yet arrived? Is this indulgent? It would probably not be fun, it would be hard work

Conclusion

Now that I’ve written all these down I’m worried I won’t get through any of them (except hopefully a job). Am I looking to expand my own thinking, my own skills, make something that is fun, or something that has future potential?

Job