It’s been a rough month. Winters are always rough with the lack of sunlight, but it was just getting progressively worse. I was suffering from brain-rot. I needed a brain twister.

A year ago, Shinyuu asked me what it would take to make a simple bank app for her roleplay project. With the requirements being extremely trivial, I figured I’d challenge myself to write a bank using a blockchain. That’s how dracones blockchain became a thing. It was a very non-trivial rust project with a huge codebase (mostly coming from the underlying substrate blockchain), all the complexity of Ethereum, with WASM sprinkled on top. It was a fun project to do, and I somewhat enjoyed tearing out the “expensive” consensus and figuring out how to replace it with a trivial Raft-based state machine. In the long run, it showed that blockchains are mostly useless as databases.

When Shinyuu approached me with a question of what’d it take to add a simple frontend to that bank, I knew I didn’t want to continue the blockchain going. There was too much effort maintaining the underlying stack, and even more effort to keep the solidity bits running on EVM in check. I decided that it’d be a good time to push out of my comfort zone and learn something new once again.

Requirements

While I wanted to focus on studying the new technologies, I still tried to adhere to the common NALSD principles (after all, you can get Google out of your search bar, but not out of your engineering thinking). I had a pretty good understanding of scale for an initial prototype and, given the daily active users numbers from the previous bank implementation, I had a very solid understanding of the expected traffic. I could even scale it up easily to meet 100x the demand with no hardware investments whatsoever.

So here’s the deal. We’re building a full stack banking application: a web frontend, and a processing center. It also has to talk to the mucklet, which means using the slightly annoying to deal with Resgate protocol. And here are the things I decided early in my design:

  • I’m doing the frontend in ClojureScript to see how it behaves on frontends

  • I’m doing the backend in Clojure, because I really ought to try using the Java stack at least once

  • I’m not using any SQL storage, because it’s boring

  • I’m not doing complexity for complexity’s sake, but I still plan for scale.

To elaborate a bit on the SQL part, I have a bunch of personal and professional experience dealing with various RDBMs (SQlite, MySQL variants and Postgres). I wanted to try something different to learn about alternatives, and I absolutely did not want to consider mongodb.

I have hardware limitations on how much I can dedicate to this project, so I had to make sure it fits into a “reasonably small” container. I was willing to sacrifice more resources to demonstrate the scalability potential, though.

What I learned so far

ClojureScript is reasonably fun for the frontend. I tried using reagent (a ClojureScript wrapper around React) and Tailwind CSS and the common LLMs (Cody and GPT-4o) were pretty good at quickly drafting the layouts. While I see Tailwind as a bit of a mess, using LLMs to generate it worked out great: I skipped the boring part, delegating the code monkeying parts to the machine. GPT-4o could even produce hiccup from screenshots! I suppose, it’d be not as bad writing raw CSS either, but that’d require more context. With Tailwind styles, LLMs were almost always spot on.

To manage the state, I went with re-frame for the generic storage, and reitit as a router. Re-frame is very much like JS’s Redux, and, while I always preferred MobX to Redux, in this case the extra code doesn’t feel too annoying. It’s natural to compose things from small functions in Clojure and reasoning with the re-frame’s data flow was easy (also, the docs are ridiculously good). Reitit integrated the history API into the same re-frame managed store with no issues, too. If I hit any issues storing my whole state in a single atom, I can consider re-posh too, which is a DataScript backend for re-frame, allowing you to write your query logic in Datalog. You can see the outline of the frontend here.

ClojureScript’s frontend game isn’t much different from the JS world. You get all the debug tooling and similar concepts. The main thing it shines in is evaluating code in your REPL directly, which is akin to writing JS in globals only while coding from the Chrome developer tools and having the console always open.

Communicating with mucklet over Resgate was a backend-only task, but there’s no Resgate client library for Java, and, given I wrote one for Dart before, I was definitely not looking forward to tackling the protocol again. That meant I needed a nodejs-based microservice to use the official Resgate client. I already had the shadow-cljs infrastructure for ClojureScript, so it wasn’t a huge issue. An issue was the communication protocol.

Early on, I decided to base all the communication on gRPC. There’s a clojure-native library called Protojure to do protobufs and gRPC, but I found its design lacking. Protojure is based on a generic HTTP2.0 client and server, and thus misses a whole lot of bells and whistles you’d expect from a gRPC library (e.g. the client stubs do not reconnect automatically, the server side requires you to write trailers by hand to return errors, etc.). On a positive side, it does gRPC-Web out of the box.

After experimenting with it, I opted out to use the Java/JS bindings instead. They weren’t too bad (after I figured the whole concept of thread pools in Java), and I could write some pretty clojure-looking code with the help of several macros to convert the protos to and from maps. This shows an incredible flexibility of Clojure hosted on a JVM.

Writing easy-to-use libraries is hard, especially for something as massive as gRPC. In here, Clojure shines by allowing to effortlessly bridge into the Java APIs and by having an extensive macro language that allows to wrap those Java APIs in something that looks more Clojure-like.

With my trivial mucklet proxy done, I turned my eye to organizing the project code, which was quickly becoming a mess of clj/cljs, Java, and compiled classes. To solve the growing concerns, I opted in for a Polylith approach. Polylith is an architecture to think functionally at scale, and it reminded me of some of the nicer microservice development stories in Google. With Polylith, you develop every part of your stack in isolation (but in a monorepo) as small contained bricks, and you combine the final app from those bricks like building a LEGO model, all while having the full scope of Clojure REPL at your hands.

While Polylith solved the “how do I structure the code”, mount answered the “how do I assemble it on boot” problem. Mount is an extremely easy to use library that defines how you bring a component up and down, and it helps you to manage state without ever restarting the REPL.

It feels superb to compose the apps from functional bricks again. I’ve missed this approach and I haven’t seen anything like this implemented in the rest of the industry. Polylith is just a method that immediately makes sense and it works.

The pieces of the app started to come together. I use the devenv to keep my development environment versioned, and it helped a lot not only with pulling the local tooling into my shell (protoc packages are still a mess), but also managing the stack of the sidecars I now needed to make things work: a redis for session storage, mucklet-proxy as a standalone running process (because the cljs REPL was for the frontend), envoy to convert gRPC-Web to gRPC, and postcss to generate the CSS from Tailwind classes.

Amazingly, I could develop both the backend and frontend from the same editor, in the same language. VSCode’s Calva automatically figures if you’re in a .clj or a .cljs file and switches the REPL scope accordingly. Things just update on their own. Where they don’t, mount (bound to an editor’s hotkey because in Calva you can run snippets in the REPL) would restart them. You really stop leaving the REPL because everything is at your fingertips, you can eval any bit of code both on the backend and the frontend side. Imagine running an RPC call from the frontend from your editor, without having to get the app in the browser in the right state as easy as you’d call a fetch() from the developer console but with all of your functions and data at your fingertips, and then observing how it updates the Redis by querying it from the same REPL. Or, you could craft the JWT using the code you already wrote for the backend and verify it worked when sending it in client requests without even having a functional mechanism to pass it to the client yet.

The missing puzzle pieces

The bit that I struggled the most with was the persistent storage. On one hand, Clojure has a good variety of native databases. On the other hand, I ended up having a decision paralysis.

Given my previous version of the bank app used a blockchain, I wanted something immutable with a detailed history. That crossed many otherwise interesting options like datalevin off. What I was left with were Datomic and Xtdb and it was a really tough call to pick one.

Datomic is the more mature solution, and pretty much the original Datalog database for Clojure. I really like Datalog. It’s an elegant logic-based language that I was first introduced to in LogSeq and enjoyed using ever since. With it, you outline the requirements for the data, and then the form of what to pull from the database and it does the rest. It’s so much more a pleasure to play with than SQL. Datomic is a proprietary solution with a free-to-use license, and it comes in either the Local (in-process) or Pro (networked) options. The annoying bit here is that these two use different APIs. Datomic Pro’s Peer API is generally more feature-full, but it comes with a requirement to run a dedicated transactor and, potentially, a separate storage layer (e.g. Postgres). The transactor is somewhat memory-hungry, and the Peer API is, too, because Datomic performs the querying client-side. The Client API, on the other hand, allows the use of in-process Datomic, which is so much more easier to set up and manage, but it (oddly) also enforces limitations on query patterns. The history travel API is also different, meaning no easy migration from Local to Pro.

On the other hand, there’s Xtdb, which seems to be designed straight after a blockchain DB sans the blockchain bullshit. Almost a perfect choice for my use case! Where Datomic uses a strong schema, Xtdb opts into a schema-less document model—you don’t need to define your schema, but you also have to use transaction functions if you want to do trivial document updates. Datomic’s transactions are unordered, so you absolutely cannot perform e.g. a balance transfer between two accounts twice in the same transaction. Xtdb’s are not. The log also differs. Where Datomic stores the final shape of the transaction (facts added and retracted) after applying the transaction functions, Xtdb stores the intent (i.e. the calls to the transaction functions and their output). This is mostly due to transaction functions being part of the database store in Xtdb (and thus versioned in the DB) as opposed to regular namespaced functions for Datalog local.

I’ve been going on this back and forth for longer than I should have, but, in the end, I seem to have settled on Xtdb for now. My concern is that Xtdb v2 (in beta at the time of writing) takes a different approach and is way more SQL-like than I’d want. They will support v1 for the foreseeable future, of course.

Where am I now?

Clojure has proven to be a great platform for developing a full-stack application with ease and for managing a growing development environment with multiple concerns. Testing your code live as soon as you write it is something I never had experience before, especially at the frontend side (backend-wise jupyter notebooks might come somewhat close). Most importantly, the language allows me to have fun and there are plenty great libraries to sort out all the concerns.