The program is the database is the interface

!!! DRAFT !!!

I do my accounts each year with a simple script. Something like this:

(ns accounts
  (:require 
    [clojure.string :as str]
    [clojure.pprint :as pp]))

;; converted from statement.csv
(def txs
  [{:date #inst "2022-05-13T11:01:56.532-00:00"
    :amount -3.30
    :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}
   {:date #inst "2022-05-12T10:41:56.843-00:00"
    :amount -3.30
    :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}
   {:date #inst "2022-05-12T00:01:03.264-00:00"
    :amount -72.79
    :text "Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA"}
   {:date #inst "2022-05-10T10:33:04.011-00:00"
    :amount -20.00
    :text "e-Transfer to: John Smith"}
   {:date #inst "2022-05-11T17:12:43.098-00:00"
    :amount -90.00
    :text "Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER"}])

(def date->tag
  {#inst "2022-05-12T00:01:03.264-00:00" :things})

(def text->tag 
  {"Coffee" :eating-out
   "Range Physio" :medical})

(defn tx->tag [tx]
  (or
    (date->tag (:date tx))
    (first 
      (for [[text tag] text->tag
            :when (str/includes? (:text tx) text)]
        tag))))

(def txs-with-tags
  (vec 
    (for [tx txs]
      (assoc tx :tag (tx->tag tx)))))

(def total-per-tag
  (reduce 
    (fn [totals tx]
      (update-in totals [(:tag tx)] #(+ (:amount tx) (or % 0))))
    {}
    txs-with-tags))

(def untagged 
  (vec
    (for [tx txs-with-tags
         :when (nil? (:tag tx))]
      tx)))

(pp/pprint 
  [[:untagged untagged]
   [:total-per-tag total-per-tag]])

There are many things about this which are nice.

But the workflow isn't always great. When I run the code above, I see:

> clj accounts.clj
[[:untagged
  [{:date #inst "2022-05-10T10:33:04.011-00:00",
    :amount -20.0,
    :text "e-Transfer to: John Smith",
    :tag nil}]]
 [:total-per-tag
  {:eating-out -3.3, :things -72.79, :medical -90.0, nil -20.0}]]

That transfer to John Smith isn't covered by any of the tagging rules. So I select the date, switch to the editor window and paste the date into the date->tag definition:

(def date->tag
  {#inst "2022-05-12T00:01:03.264-00:00" :things
   #inst "2022-05-10T10:33:04.011-00:00" :eating-out})

Now I see:

> clj accounts.clj
[[:untagged []]
 [:total-per-tag
  {:eating-out -23.3, :things -72.79, :medical -90.0}]]

Multiply this by a thousand transactions and it becomes tedious.

Pretty-printing also makes it difficult to decide the amount of detail I should print. Sometimes I want to see which transactions contribute to each tag total. But if I always print them all then it's hard to see the totals themselves without a lot of scrolling.

It's also hard to share this workflow with someone non-technical. I have to setup and maintain the correct environment on their machine, teach them how to use a text editor to change the tagging rules, how to interpret syntax errors, how to use git to share changes etc.


I could solve these kinds of problems by writing a web app and storing txs, date->tag and text->tag in a database.

Then I could put controls on the transaction itself that allow changing the tag in place. And I could add an expandable section next to each total, so that it's easy to see the transactions for that tag when I want to but they don't take up space by default.

Plus sharing becomes trivial - everyone has a web browser.

But this is a big leap in effort:

None of this is insurmountable, but it's definitely much more work than the original script.

So a simple script is low-effort but produces a low-quality experience. And a custom app can produce a high-quality experience but is high-effort.


So I made a thing.

It is very hacky and easy to break. But it will hang together just long enough to convey the idea.

It's kind of like a notebook. There are cells where you can write code and the resulting values will be nicely rendered:

;; converted from statement.csv
(def txs
  [{:date #inst "2022-05-13T11:01:56.532-00:00"
    :amount -3.30
    :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}
   {:date #inst "2022-05-12T10:41:56.843-00:00"
    :amount -3.30
    :text "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER"}
   {:date #inst "2022-05-12T00:01:03.264-00:00"
    :amount -72.79
    :text "Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA"}
   {:date #inst "2022-05-10T10:33:04.011-00:00"
    :amount -20.00
    :text "e-Transfer to: John Smith"}
   {:date #inst "2022-05-11T17:12:43.098-00:00"
    :amount -90.00
    :text "Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER"}])
(defs date->tag
  {#inst "2022-05-12T00:01:03.264-00:00" :things})
(defs keyword->tag
  {"Coffee" :eating-out
   "Range Physio" :medical})

Funtions render a little differently.

(defn text->tag [text]
  (first
    (for [[keyword tag] keyword->tag
          :when (clojure.string/includes? text keyword)]
      tag)))

If you type "Joe's Coffee Hut" (including the "!) into the textbox above and hit the text->tag button, you'll see the result of running the text->tag function on that input.

Functions aren't limited to just returning values though. They can modify the values stored in other cells:

(def txs-with-tags
  (vec 
    (for [tx txs]
      (assoc tx 
        :tag (or
               (date->tag (:date tx))
               (text->tag (:text tx)))
        :actions [(fn ignore []
                    (edit! 'date->tag assoc (:date tx) :ignore))
                  (fn tag [tag] 
                    (edit! 'date->tag assoc (:date tx) tag))]))))

If you type :eating-out into the one of the textboxes above and then hit the tag button, it will change the date->tag cell to contain an entry for the date of that transaction with the tag :eating-out.

And then any downstream cells will update, so you'll see the tag for that transaction change to :eating-out.

These actions are just values so they can be passed around like any other value. For example, if I make a list of untagged transactions then I'll still have access to the same actions:

(def untagged
  (vec
    (for [tx txs-with-tags
         :when (nil? (:tag tx))]
      tx)))

We can also attach metadata to values to control how they render:

(def hidden-vec
  (with-meta
    [1 2 3]
    {:preimp/hidden true}))

If you click on the + above it will reveal the contents of the vec. This is useful for controlling the default level of detail.

(defn hidden [v]
  (with-meta v {:preimp/hidden true}))
(def total-per-tag
  (reduce 
    (fn [totals tx]
      (if (= :ignore (:tag tx))
        totals
        (update-in totals [(:tag tx)] 
          (fn [total+txs]
            (let [[total txs] (or total+txs [0 (hidden [])])]
              [(+ total (:amount tx))
               (conj txs (update-in tx [:actions] hidden))])))))
    {}
    txs-with-tags))

You can click on a + above to reveal the transactions for that tag.


The demo on this page is ephemeral - you could do something similar in many notebook environments using mutable data-structures.

But the preimp repo contains a server which persists the entire history of the notebook to disk, and also syncs changes between different clients to allow (coarse-grained) collaborative editing.

The server also allows reading and writing cell values over http. Here's the script that I use to upload my bank statements:

(ns wise
  (:require [clj-http.client :as client]
            [clojure.data.json :as json]))

(def endpoints {
  ;; FILL ME IN
})

(defn api-get [user path]
  (let [domain (get-in endpoints [user :wise-domain])
        url (str domain "/" path)
        response (client/get url {:headers {"Authorization" (str "Bearer " (get-in endpoints [user :wise-token]))}})]
    (assert (= 200 (:status response)))
    (json/read-str (:body response))))

(def now (java.time.Instant/now))

(defn get-transactions [user]
  (into []
        (for [profile (api-get user "v2/profiles")
              :let [profile-id (get profile "id")]
              balance (api-get user (str "v4/profiles/" profile-id "/balances?types=STANDARD"))
              :let [balance-id (get balance "id")
                    statement (api-get user (str "/v1/profiles/" profile-id "/balance-statements/" balance-id "/statement.json?intervalStart=2022-01-01T00:00:00.000Z&intervalEnd=" now "&type=COMPACT"))]
              transaction (get statement "transactions")]
          transaction)))
          
(defn update-preimp [user]
  (let [transactions (get-transactions user)
        cell-name (symbol (str (name user) "-wise-transactions"))
        cell-value (pr-str `(~'defs ~cell-name ~transactions))
        body (json/write-str
              {:cell-id (get-in endpoints [user :cell-id])
               :value cell-value})]
    (client/put
     (get-in endpoints [user :preimp-domain])
     (merge (get-in endpoints [user :preimp-headers]) {:body body}))))

(defn update-preimp-dev [_]
  (update-preimp :sandbox))

(defn update-preimp-prod [_]
  (update-preimp :jamie)
  (update-preimp :cynthia))

Finally, you can export a preimp notebook to produce a perfectly valid clojurescript program.

You can run this program in the repl:

> clj -M -m cljs.main -i ./exported.cljs --repl --repl-env node 

preimp.exported=> (require 'clojure.pprint)
nil

preimp.exported=> (clojure.pprint/pprint txs-with-tags)
[{:date #inst "2022-05-13T11:01:56.532-00:00",
  :amount -3.3,
  :text
  "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER",
  :tag :coffee,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}
 {:date #inst "2022-05-12T10:41:56.843-00:00",
  :amount -3.3,
  :text
  "Card transaction of 3.30 CAD issued by Milano Coffee Roasters VANCOUVER",
  :tag :coffee,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}
 {:date #inst "2022-05-12T00:01:03.264-00:00",
  :amount -72.79,
  :text "Card transaction of 72.79 CAD issued by Amazon.ca AMAZON.CA",
  :tag :things,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}
 {:date #inst "2022-05-10T10:33:04.011-00:00",
  :amount -20,
  :text "e-Transfer to: John Smith",
  :tag :eating-out,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}
 {:date #inst "2022-05-11T17:12:43.098-00:00",
  :amount -90,
  :text
  "Card transaction of 90.00 CAD issued by Range Physiotherapy VANCOUVER",
  :tag :medical,
  :actions
  [#object[preimp$exported$iter__897_$_ignore]
   #object[preimp$exported$iter__897_$_tag]]}]

nil

This tiny extension to the notebook model allows writing simple ugly crud apps with very little effort. You simply provide the logic and preimp gives you storage, sharing and UI for free. But the result is not an opaque image, nor is it tied to the preimp environment - you can export everything into a regular script and run it in the repl.

TODO rest of this section

required: rich data model, which can be roundtripped through text some way of attaching metadata to values, to control their rendering without interfering with execution in the repl

needed: no declaration order no mutable environment no side-effects (other than edit!) integrity constraints perf (compiler latency) (thoughput only needs to be better than spreadsheet)

extensions: much more ui options selection (+ actions on the side) dropdowns understand types / destructuring render values as copyable text (cf ?) undo/vc (have whole history in db) live repl, sandboxing distribution - single binary, single-file db, optional server (like fossil)

research: finer-grained collaboration (what data model?) bidirectional editing / provenance


We have to do the other demo too. You know the one.

(defs next-id 2)
(defs todos 
  {0 {:text "make a cool demo" :status :done}
   1 {:text "step 2: ???" :status :todo}})
(defn new-todo [text] 
  (edit! 'todos assoc next-id {:text text :status :todo})
  (edit! 'next-id inc))
(defn named [name fn]
  (with-meta fn {:preimp/named name}))
(defn status-toggle [id]
  (let [status (get-in todos [id :status])
        other-status (case status
                       :done :todo
                       :todo :done)]
    (named (name status)
      (fn [] (edit! 'todos assoc-in [id :status] other-status)))))
(defs filter :all)
(def toggle-filter
  (named (str "viewing " (name filter))
    (fn [] (edit! 'filter #(case filter 
                             :all :todo
                             :todo :done
                             :done :all)))))
(def filtered-todos
  (vec
    (for [[id todo] todos
          :when (#{:all (:status todo)} filter)]
      [(:text todo)
       (hidden [(fn set-text [text]
                  (edit! 'todos assoc-in [:id :text] text))
                (status-toggle id)
                (fn delete []
                  (edit! 'todos dissoc id))])])))
(def app
  (vec
    (apply list
      new-todo
      toggle-filter
      filtered-todos)))

TODO spreadsheets don't have state problem, but not interactive (at least in same way)