Welcome to everyone who signed up since the last update. Thank you for giving me money. I will spend it on sensible things like beans and running water.
New posts:
- The intro section for the post on correlated subqueries. I'm still working on the remaining sections. I've tested detailed explanations on several guinea pigs and, uh, let's just say I'm now planning to stick to conveying the important intuitions rather than diving into the code.
- Data-structures for the focus text editor.
- Notes on 'Working in Public' by Nadia Eghbal.
Why am I doing this? If I want to do research there are plenty of academic and industrial positions that will happily fund me. But many of the things I want to do fall between the cracks of existing funding sources. Being independent allows me to:
- Explore ideas that are unpublishable or unfundable. Ideas like bloom, which has become my standard mental model for distributed systems, lie fallow because the hard implementation work that would be required to bring them to production would be a publication-free death sentence for any academic career, and because the value produced would be hard for a for-profit company to capture.
- Do good evaluations of existing ideas. And also discover and share how to do good evaluations. My experience working as an industrial researcher has been that the vast majority of academic database papers have evaluations that range from misguided to actively misleading. Industrial evaluations of eg streaming systems are often similarly poor. Much of this comes down to poorly aligned incentives - if your paper doesn't show you trashing the strawman competition then it won't get published and you won't get that post-doc. Poor evaluations make it harder to learn which ideas are worth building on.
- Distill research debt. Even if the mountain of available papers were all written honestly and had good evaluations, it's still an enormous amount of work to read it all to find the few pearls of wisdom. Very few people take the time to create a map of this terrain for the people behind them (usually because they have no time - publish or perish). It's an enormous waste of time for everyone involved.
There is a clear tension here - the whole point of this work is to share information, but people are much more willing to pay for things that are exclusive. My current thinking on how to balance that is:
- Keep the newsletter and all work-in-progress projects as sponsors-only - this is where the sausage gets made, a space to make mistakes and figure things out.
- Release stable, polished projects publicly once I'm sure everything is correct.