2022

Published 2023-01-05

(See other years).

Office hours

From January to March I ran office hours every friday morning. I wrote at the time:

I'm trying to virtually recreate the cafe/salon culture I loved before moving to a tech backwater during an epidemic.

I ended up talking to 35 people. Most of the time it was awkward and exhausting, much more so than in-person conversation. I think this is just a corollary of what many people found in the switch to remote work - video calls are strangely tiring.

Many people wanted to ask for advice on a specific topic and usually I didn't have anything useful to offer. I think there's an illusion of expertise where the amount of time and research I put into writing an article is invisible, so it's easy to assume that I know much more about a subject than I actually do.

I talked to a couple of people who were having a hard time and just needed someone to talk to. I can't regret offering that, but I don't feel that I did a good job of it either.

HYTRADBOI

I ran a virtual conference called Have you tried rubbing a database on it.

I made a mess of the infrastructure on the day (postmortem), but the talks were excellent and it seems a lot of people really liked the format (dense pre-recorded 10 minute talks + dicussion via text chat).

One of my main goals for 2022 was 'try more things that I expect to fail'. I don't know if I want to run a conference again, but I'm glad that I did it once and I'll definitely try more experiments in this vein.

imp

In 2021 I focused mostly on the editing experience with the live repl. This year I switched focus to figuring out how to interact with mutable state.

I wanted something like a database schema, but I also wanted to be able to pass around complex structured values and work with heterogenous, loosely-typed data. And I wanted to avoid having nominal types because they're difficult to store and transmit between programs.

I ended up with a gradual typing system based around heterogenous sets of tuples. Structure is supplied by symbols - scalar values whose type is unique (eg the type of :foo is :foo). This is sufficient to model lots of useful concepts like structs, tagged unions, first-class modules etc while still preserving the property that arbitary data can be sent to a different program without losing type information. (Details here)

Next I wanted to figure out how to store code and data together (like a spreadsheet). Here I ran into a wall. I had assumed until now that that I would somehow insert code as a scalar value in the database but this leads to a lot of questions around how to refer to other chunks of code, when does code get evaluated, what is the evaluation environment etc. It felt really clunky.

preimp

After much doodling (the shape of data, what is a database) I had an epiphany. The language already has data literals and variables. It can express data and computation. Why do we need a separate database? Just store all the data in the source code!

This requires that it be possible to efficiently and correctly locate the value in the source code.

I explore the idea first in a simple datalog where the interaction mechanism was to 'checkout' a source file from the database, modify it and 'commit' the diff. This interaction has some legs, but it became clear that to make things really smooth I was going to need direct interaction via a structured editor.

Making a language, database and structured editor is starting to feel like a lot of work so I explored this first using clojurescript as the language. I drafted an interactive demo in the program is the database is the interface. It looks at first like a notebook interface, but:

I liked this model a lot but the clojurescript + html implementation had serious performance issues and using an existing language implementation limited how pervasive I could make interaction.

So back to boiling the ocean! I still haven't figured out how all these ideas mesh with imp's flexible syntax and hetoregenous sets so I went instead for a simple lisp. Here's a little demo:

On the left is a structured editor for the source code and data. On the right is the evaluated result. Values on the right remember where they came from in the source, so they can be directly edited and the source will update.

Evaluation is always live as-you-type - not just when editing code on the left but also when editing arugments to functions/forms on the right. That requires a way to handle errors without stopping evaluation, so there is an error type that is propagated through all function calls and builtin forms. Many errors report the values involved, so it's often possible to fix an error by directly editing the values in the error.

And that's about as far as I got. Playing around with the demo again I still really feel that this is a fruitful direction. But it's just going to take a lot more ocean-boiling before reaching something usable enough to convey to someone else.

Miscellaneous projects

Focus continues to accrue small improvements:

For my HYTRADBOI jam project I tried to write a sql frontend in one week that could pass the ~6 million tests in the sqllogictest suite. This was totally pointless in terms of output, but it was really fun to push myself hard for a week and see what I'm capable of.

I built a little accounting webapp in zig using h2o and sqlite. I'm keeping the code private for now because it's full of api keys and passwords, but eventually I'd like to separate out the h2o bindings.

Sponsorship

I was too lazy to make a graph this time :)

My github sponsors hit ~2600 USD/month at the end of 2021. It tailed off a bit when I made all my writing public and hovered around 2300 for most of the year.

When I started working at tigerbeetle I left my sponsors account open but suggested people might want to end their sponsorship. That has been happening steadily, but the total is still at ~1700 at the moment.

Advising

Most of my income in 2022 came from consulting as an advisor. I wanted to focus on imp, which didn't leave enough hours to write code for anyone else. But I found a couple of contracts where I'd bill for 8 hours a month in exchange for being available to answer questions on slack/discord and joining the occasional design meeting.

This seemed most valuable in the early stages of a project where I could save clients a lot of time by pointing them at the right papers, sharing undocumented gotchas, steering them around mistakes I've made in the past etc. Later on there was less I could do without having the time to read their code and dig into concrete changes.

Overall it seems like a fairly low-cost, sustainable way to fund research.

Grant

In June I applied for an Emergent Ventures grant to work on imp. I didn't expect much, but the application only took a few hours. So I was surprised a few weeks later to be awarded 50k usd.

And then just 5 months later I dedided to give up on imp and work at tigerbeetle. I returned the grant, feeling a little sheepish about my flip-flopping. But I think working at tigerbeetle was a good decision and it would have been silly to keep working on imp instead just out of fear of embarassment.

TigerBeetle

I've been working alone for nearly three years. The majority of that time was spent on imp, and while it continues to produce interesting ideas I don't feel any closer to having a useful tool. In the second half of this year I really struggled to maintain the motivation and confidence to keep going. I wasn't making progress on imp anymore but I couldn't bring myself to give up and work on something else. Instead I just stagnated.

Meanwhile, Joran' and Isaac's HYTRADBOI talk about TigerBeetle led to a conversation with some investors who were attending the conference, and eventually to forming a company and hiring a small team. Including me.

It was a hard decision, because I'd always wanted to do something independent and I was finally reaching the point where that was finanically sustainable. But at the same time I was so tired of working alone and I was full of doubt about whether I could ever. And I was really impressed by the design of TigerBeetle, I've always wanted to build a storage engine, and I liked their approach to building it (marathon pace, small teams, low dependencies, reduce the state space, lean heavily on deterministic simulation testing). Plus it's a tool with clear value - lot's of companies have ledgers but most of them are too slow and lack fault-tolerance (I recently heard that during peak hours one company will accept fake credit cards because their payment processor can't handle the load synchronously).

It was too good to pass up.

So far I've been focusing entirely on the storage engine - adding tests and fuzzers, fixing bugs and improving performance. I'm learning a lot, and I'm also enjoying the simplicity of having a clear goal and knowing (roughly) how to get there.

Climbing

Despite wrestling with a bewildering range of chronic joint problems, it was a very succesful climbing year. I set what I thought was a very ambitious goal - to send a 12a by the end of the year. I knew I was strong enough so for just a few rainy winter months I cut down on fun social bouldering and focused almost entirely on lead mileage on moderate routes to build up better fitness, efficiency and headgame.

It was a staunch reminder of the unreasonable effectiveness of actually trying to improve. In the second week of the climbing season I sent my first 12a with only two attempts. By the end of the season I sent five 12as, two 12bs and was starting to feel pretty close on a few 12cs. I also had a chance to work through my friends 13a project on toprope and was surprised to find that while it was obviously difficult, it didn't feel outrageous. I didn't even aspire to ever climb that hard before, but now it feels not just attainable but likely.

And that's just from a few months of semi-structured training. It makes me wonder in what other fields I'm leaving that kind of improvement on the floor.

2023

Most of my computering time will be spent on tigerbeetle. The storage engine still needs plenty of performance work. There's a whole query engine yet to be designed. Probably many bugs yet to be discovered before we're release-ready.

I have a week per month for other projects. I don't think I'll put any time into imp this year, or any other mega-projects. I need to spend some time rehabilitating my motivation, which calls for short projects with clear goals and a high chance of success (or at least of an interesting outcome). I have a list of ideas but I'm not going to commit to anything. I'm also hoping to do more collaborations and contribute more to existing projects.

I had been resigned to the idea that joint issues are just an inevitable consequence of aging, but the more I read the more it seems that it's not age itself but decades of mostly sedentary living that causes issues, and that it's possible to age much more gracefully. So for 2023 I'm going to try to reinforce my knees and shoulders while they still (mostly) work.

I'm going to try a year without reading fiction. It's not bad in itself, but it's an infinite source of effortless sedentary entertainment. I think it would be good for me to be bored more often, lest I become boring.