After much delay erl-telehash is getting some love with a 0.1.1 release. At this point the core features are implemented and seem to be working. I'm now working on some simple demo apps that I can use for larger scale testing.

Lately I've been porting some code from c++. The code in question is a compressed bitmap used in swift to track which parts of a download have already been retrieved. To reduce the memory usage the original uses lots of pointer tricks. Replicating these in ocaml is interesting.

This spawnfest entry is inspired by traveling. I love the idea behind sites like chatroulette and omegle but if I had an internet connection I wouldn't be bored enough to use them. I want a version I can use entirely over the phone network to while away the hours spent stuck in airports and train stations.

I have a new secret project. It's called Mist. The goal is to make it easy to build, use, distribute and maintain small-scale p2p apps. By small-scale I mean apps where each invocation has at most a few dozen users. You would use Mist to build the next etherpad, not the next bittorrent.

Now that we have all the necessary datastructures we can build the router itself. Most of the routing table logic is handled by the bit_tree and bucket modules. The router just ties these together and handles I/O.

I ran into some tricky bugs caused by a misconception I had about gen_event. Since this is not explicitly stated in the gen_event documentation I will say it here: gen_event does NOT spawn individual processes for each handler. Each handler is run sequentially in the event manager process.

The other half of the routing table is the buckets which store node addresses.

It has been suggested that the bit_trees presented in the last post are overly complicated. Indeed, in the cold light of the morning there is absolutely no need for that zipper. Without further ado, here is the much simpler version.

The next step in building a switch is managing a routing table. Actually, the next step is handling sessions via _ring/_line but I'm still mulling over the protocol so we'll skip to the routing table.

The next step in building a telehash switch is being able to dial.

TeleHash is a p2p network based on the Kademlia DHT that provides addressing and NAT traversal. These are problems that every p2p app has to deal with, including my poppi. Unfortunately there is no erlang implementation yet so I have to roll my own. The code so far lives here In this first post I'll just cover the absolute basics - sending, receiving, encoding and decoding messages.

This is a hugely overdue post about an interesting system I worked on almost a year ago whilst at Smarkets and never got around to writing about. Unfortunately I don't have the code in front of me but the overall idea is simple enough to explain without examples.

In Shanghai I found that ssh was blocked at the protocol level so even running sshd on port 80 doesn't work. I don't whether this is widespread or whether it was our hotel in particular that was blocking it. Regardless, I found a workaround using httptunnel.

In my last post I discussed the new search algorithm behind texsearch. There is a significant speed improvement over previous versions but it now consumes a ridiculous amount of memory. The instance running latexsearch.com wavers around 4.7 gb during normal operation and reaches 7-8 gb when updating the index. This pushes other services out of main memory and everything is horribly slow until they swap back in.

Texsearch is a search engine for LaTeX formulae. It forms part of the backend for latexsearch.com which indexes the entire Springer corpus. It is also crazy slow, until today.

I want to write a little about the documentation system I wrote for the smarkets API. The main concern I had with the documentation was that it would be incorrect or become out of sync with the code, especially since I didn't really understand the system when I started documenting it. To prevent this I built a couple of documentation tools that have paid for themselves many times over.

Since the beginning of June I have been working at smarkets. Smarkets is a betting exchange, something like a cross between a traditional bookies and a stock exchange. You can buy and sell contracts like 'England will lose the world cup' whose value is determined by events in the real world. Trading continues right up until the contracts are settled so you can watch the market prices change in real time during a live event.

At the moment all the posts on the front page are displayed in full. It's already getting a bit crowded so I want to just show the a small part of each post. Not particularly exciting but a little fiddly.

Scampy was originally intended to be a smart bot. I had visions of data mining conversations and inventing dsl's for chat scripts. This all takes time however, so in order to get up and running quickly the prototype just selects responses at random from a prewritten list. This turns out to be depressingly effective.

The bodies of these posts are written using Mauricio Fernandez' simple_markup. This is pretty easy to get running as an erlang port. Most of markup.erl is just gen_server fluff. The init function opens a port which uses packets preceded by their length in 4 bytes.

Normally, code is the default and strings have to be escaped with ". In erltl files strings are the default and code has to be escaped. The result compiles into a normal erlang module with a 'render' function which accepts a single argument and returns an iolist.

Webmachine is what initially attracted me to erlang. The basic principal is that the best way to make restful apps is to just mimic the http spec. It's a deceptively simple idea.

Mnesia is bliss. It may not be as powerful as a full relational database but in this context the ease of use more than makes up for it.

It seems that html is the easiest and best way to create cross-platform interfaces. It's high time I got the hang of doing it.