I got back to work Aug 8. The rest of the month has been blessedly uneventful.
preimp
I got as far as I could with the clj version.
The main obstacle is provenance. The main feedback from the essay was that it would feel much more natural with direct manipulation. Why write code to make a toggle button for the status of a todo? Why can't you just click on the status and change it?
Doing that requires tracking, for each value, where in the source code it came from so that we can map edits back to that location. Doing this in clojurescript seems very difficult. Clojurescript's strings and numbers are just javascript strings and numbers. There is no way to attach any additional information to them. I could put wrapper objects around all strings and numbers in the source code, but then existing functions won't work on them.
Performance is also an issue. I used preimp to build an accounting tool that we actually use, but we have to use it on my fancy laptop because on a $200 chromebook it takes multiple seconds to render a few months of transaction history.
Here is a static recreation of a page that even on a brand-new 12th gen i7 takes 250ms to render. Probably something about the nested tables makes layout super-linear, but the browser profiler just gives me a single box labelled "layout" with no further information.
The slow layout is compounded by the fact that having a codemirror editor in the layout often causes forced reflow of surrounding elements. I think this is something to do with the hacks that codemirror has to use to measure the size of text elements, since it can't just ask the browser to layout text in isolation. But I have no idea how to verify that theory or prevent the extra layouts.
Clojurescript is also slow in general. The self-hosted compiler takes a long time to compile small amounts of code eg 5ms for a single line of code on my new i7. The self-hosted compiler also doesn't do any optimizations, so the generated code is slower than regular clojurescript - running the below code over a vec of 790 transactions takes 4.6ms. Naturally all these times are much worse on a $200 chromebook.
(def untagged
(into []
(for [transaction transactions
:when (nil? (:tag transaction))]
transaction)))
The performance issues are probably workable, but if I have to write something from scratch to get provenance anyway I might as well aim for less opaque performance while I'm at it.
So this month I wrote a very crude tree-walking interpreter for a simple lispy language, and built a minimum-viable structured editor in dear imgui. It's a long way from usable but I have the direct manipulation working:
In the video above you can see me defining some data, making a simple view over that data and then directly editing the output of the view. On hovering over a value in the output you can also see the highlight on the location in the source from which it originated.
(The video also demonstrates a perfectly useful interaction that breaks all of the lens axioms. I delete the first todo and when the view updates I see the next todo. In a well-formed lens deleting the first todo should leave me with a blank space where the first todo used to be. That makes almost all of the existing lens research useless for this kind of interaction! Similar disallowed interactions include opening a list of unread email and marking one of them as read, or changing the search in a search box and seeing the search results change. get(putback(l, v)) == v seems like a totally reasonable rule that noone could disagree with, and yet when you apply it to everyday GUI patterns it breaks everything.)
There isn't enough of a language yet to recreate the accounting app, but I very roughly compared the frontend performance by just pasting this output into both.
In the cljs + html version:
- The frame where I hit eval takes 427ms total, which breaks down into:
- 160ms in javascript
- 25ms to parse and eval the input
- 145ms in reagent and react
- 64 ms in recalculate style
- 150 ms in layout
- 42 ms in pre-paint
- 160ms in javascript
- The frame immediately beforehand takes 267ms.
- The overall breakdown is similar, except that all the javascript time is spent in codemirror.
- The layout in this frame is marked as forced reflow, I believe caused by codemirror.
- With no inputs, remaining frames take fractions of a millisecond
- If I scroll down, I get frequent dropped frames during 300ms+ intervals where nothing appears to be happening:
In the zig + dear imgui version it takes:
- 7ms to parse the input
- 15ms to evaluate the input
- 8-12ms to draw and render each frame, regardless of scrolling
Of the two, the html version is the one where I've spent a bunch of time trying to understand and improve the performance. In the dear imgui version I've done the easiest possible thing at every step and the performance is still fine.
And I know that dear imgui can't do all the things that html can, but it can do all the things I need to do to test out research prototypes and it does them without requiring any mental contortions or trial-and-error debugging from me.
When I was confused by how the layout was behaving, I stepped through the layout code in a debugger. If I have performance problems in the future, I'll be able to profile the layout code in detail rather than being presented with a single box saying "layout". And while the documentation is poor-to-none the code itself is pretty readable, so I just jump-to-definitioned whenever I was confused. Being able to print out the internal state of the layout engine was also pretty handy.
framework
My old dell precision 5510 is still mostly working after a little over six years. The speakers died in the first year, but I don't really need them. The left shift key cracked in half, but it still works. Two years ago a 1 foot drop onto a soft cushion caused the hinges to snap away from the tiny piece of internal plastic that held them in place, breaking off part of the case and bending all the connectors on the right side of the motherboard. But replacing both the case and motherboard would be pricey so I just applied duct tape and lived without the right-hand ports. Finally, this year the 3rd battery is totally exhausted and I have a battery life of about 11 seconds - short enough that sometimes the power cable falls out and my laptop dies before I can get it back in. I priced out a full repair at ~900 CAD of parts, including shipping and tax. Which is pretty reasonable, especially since the vast majority of the cost is the motherboard.
Instead I was seduced by one of the 12th gen framework laptops. Definitely an unneccesary upgrade, but one that I've enjoyed.
The aluminium outer case covers the side of laptop, where the dell had a plastic edge that broke with the hinges. The hinges themselves are attached to a metal part on the lower frame, rather than the thin plastic screwon in the dell, and the lever is much longer which should reduce the force on that internal connection. The ports at the side are sacrificial - the actual connection to the motherboard is deeper in the frame where it seems well protected from bending. An accident of the kind that crippled my old laptop looks like it would be a minor repair here.
I'm not confident that framework will still be around selling replacement parts six years from now, whereas dell will probably still have a few dusty 5510 batteries left in stock long after I die. But I want framework to succeed, and I feel happy taking a risk on them.
Also I won't drop this one.
dotfiles and backups
Setting up a new laptop prompted me to find a nicer way to manage all the random crap in /home/jamie that I can't make public.
The solution I settled on is to make /home/jamie a git repo with:
> cat .gitignore
/*
!.bash_profile
!.bashrc
!bin
!.config/
.config/*
!.config/alacritty
!.config/fish
!.config/nixpkgs
!secret
!.gitconfig
!.gitignore
!.ssh
!.sway
!tax
!.gdbinit
!.mbsyncrc
!.newsboat
!texts
!tower
!.mozilla/firefox/o6svvwi3.dev-edition-default/chrome/userChrome.css
!.mozilla/firefox/o6svvwi3.dev-edition-default/logins.json
zig-cache
zig-out⏎
That ignores everything except folders that I explictly track, but doesn't require me to individually add every new file in those folders.
I commit with:
> cat bin/mu
#!/usr/bin/env bash
(git add ./ && git commit -am 'Mu') || true
I don't push the repo anywhere, because it seems weird to send all my ssh keys to 3rd party. But I do encrypted backups to tarsnap with:
> cat bin/backup-remote
#!/usr/bin/env bash
set -ex
cd ~
mu
git ls-files -z | xargs -0 \
tarsnap -c \
--cachedir /home/jamie/tarsnap \
--keyfile /home/jamie/secret/tarsnap.key \
-f $HOSTNAME-`date +%Y-%m-%d_%H-%M-%S`
And very occasionally backup to two usb sticks:
> cat bin/backup-local
#!/usr/bin/env bash
set -ex
cd ~
mu
cd /run/media/jamie/$(ls /run/media/jamie/)/$HOSTNAME
git pull ~
One usb stick lives on my keychain and the other with all my important paperwork. I should probably encrypt them.
I run backup-remote at the end of .bash_profile so that it starts when my laptop turns on and is finished by the time I start doing any serious work.
I have daily tarsnap backups going back 3 years. It typically costs me $2/month so I've never bothered cleaning up old archives.
links
Weather in the mountains in BC is wildly unpredictable. Often the official forecast will tell me that it's sunny all day while I'm being rained on. Once I got snowed on for an hour, from a totally blue sky on a sunny day.
I found a weather app called flowx that is far better than any other I've tried. It can show forecasts from all available models side-by-side (useful as a kind of error bar - on the days where all the forecasts agree they're usually accurate). It shows hi-res models (useful for when it's only raining on some of the crags). It can show actual weather for the past 3 days and the angle of the sun at various times (useful for judging which crags will have dried out). It syncs when it can but also works with stale data offline. The free version is already very useful but I was plenty happy to give them $2/month to unlock all the extra models.
Counterexamples in type systems is a collection of interesting soundness bugs in various type-systems.
Most educational material focuses on positive examples. But often the easiest way to understand the design of a system is to understand how it is trying to dodge all the flaws and cracks in it's predecessors.
Luau is a fork of lua developed by Roblox that features gradual typing, sandboxed execution and many interpreter optimizations.
Towards a General Database Management System of Conflict-Free Replicated Relations builds a CRDT layer underneath sqlite, so that application code can treat it like a regular sql database that just happens to do something sensible when syncing with other copies.
It's not clear how much of it is actually working at this point, but the idea is intriguing.
Many examples of situated software in this thread: