Albert Zak

Run, Build, and Grow
Small Systems
Without Leaving Your Text Editor

Presenting at LIVE @ SPLASH 2024

Watch on YouTube (15 min) >

Clojure, but more alive

See the value of any expression on any node – in your code.

A REPL, but it's connected to all your nodes at the same time (1:n)

• Query macro expression
  • rewrites itself to show the data inline
• Declarative deployment
  • Say "this function should run on that node"
• Supervision & reconciliation
  • Stop a process? Comment it out.
• Effects via object capabilities
  • All host platform caps passed in as argument
• The editor is also a node
  • You can run code in it
• Define and use custom inline editor visualizations
• Values have history

Taglines#

• "Grug-brained Erlang"
• "Debug mode is the only mode"
• "A personal Bank Python" [todo / future work]
• "(i) see inside (ii) see across time (iii) [see across possibilities]"
• Baby's First Situated System
• Worse is Better [?]
• If Raspberry Pi were as simple as Arduino / P.C.C

Status#

Hacked together with SCI, rewrite-clj, zprint, Lezer and CodeMirror. Everything barely holds together.

I am ashamed of the code. But I might clean it up and release it if enough people are interested.

There's a babashka script ./new-commit-every-minute.bb that does what you'd think.

I'm building this in public, scrappily.

Follow along, or get notified when this is ready:

Related work#

• Smalltalk
• Lisp
  • Emacs
    • org-mode
• TaskTXT - via Potluck - roundtrip state through plain text
• APX - trace points + context
• Cap'n Proto - IDs in code to stabilize expressions
• music live coding environments
  • ixi lang - self-rewriting code
  • Motifn - bidirectional viz, keyboard actions
  • Gibber - inline viz
  • Strudel - viz+sliders
• Eve - viz, feature flags, programming for humans
• Dark classic - redeploy-on-every-keystroke, production traces in the editor
• preimp, aka: The Program is the Database is the Interface - self-rewriting Clojure notebook

More related work that's missing from the video#

• Glisp - bidirectional self-rewriting Clojure-ish code, custom editor UIs for types
• Quokka.js, Console Ninja - inline runtime values, inline logging
• DreamBerd - question marks! 🐸

Future work (todo)#

• error handling - at least like Sentry, maybe even a poor Smalltalk debugger or Common Lisp condition system
• persistence story - accretive bitemporal facts like XTDB or something local-first?
• communication - infer netcode like Electric
• codebase management - hashed like Unison or Scrapscript
• collaboration / ocap code sharing / delegate authority to change part of the system
• control over when and where changes deploy: staged rollout, feature flags, "under construction" development preview areas
• actually secure security
• release as extensions for common editors (maybe as language server?)

FAQ#

• What is this? A Language? An IDE? A new REPL?
  • It's just ClojureScript, with a handful of additions:
    • ! to redeploy the next top level form on every change
    • ? the query macro, rewrites itself to show values inline
    • node to assign functions to physical places
    • context scopes queried values to only show those matching some node context
    • with-context attaches custom metadata to node context
  • You can call it a different kind of REPL
    • or a programming environment with a runtime
    • for building distributed information processing systems

• Is there any chance something like this can become an Emacs mode, VS Code extension, or NeoVim plugin?
  • Yes. Editor plugins are necessary. It should be fairly straightforward to port, apart from the inline html views.
    • The editor API surface for plaintext interactions is tiny right now: callback on every change, get buffer contents as string, (async) replace position span with string.
    • Inlining graphics in text always breaks the editing flow anyways. I don't like the heavyweight feel of blocks, so I never use this feature.
    • Just checking the rough shape of data in plain text is good enough usually.
    • It's also easy to hook the query macro into Clojure's tap system, so non-text output renders in a sidecar viewer like Portal (or Clerk, but haven't tried that)

• It's always connected to all the running processes?
  • Yes, conceptually
    • nodes can be anywhere, behind firewalls, in production, etc.
      • communication goes through a central broker
  • It feels like a REPL that's connected to all nodes at the same time.

• Can I just eval forms like normal?
  • Not trivially. I'm still thinking about the UX of targeting specific nodes.
  • Evaluation in this system should feel more like nodes pulling your specification (code).
    • Instead of you imperatively pushing code to nodes that they should execute,
    • they all pull the latest code from you
      • (and decide themselves how to best fulfill it)

• Can it do transient sessions, like browser clients?
  • Not yet. The set of all nodes has to be hardcoded.
    • But the solution is simple, and I'll implement it soon
      • Introduce random sessions IDs
      • and keep the (then non-unique) node IDs to tell them to run the same code

• Does it run on Clojure on the JVM / CLR / babashka?
  • Almost. There's no reason it couldn't run anywhere SCI does.

• Why Clojure / Lisp?
  • (Watch any of Rich Hickey's Talks)
    • (start with Are We There Yet)
  • s-expressions
    • code being data
    • trivial for the query macro to rewrite
  • Clojure emphasizes simple plain data throughout your system
    • plain data is trivial to inspect
      • and to hold on to
      • copy/pasteable
      • serializable
      • safely experiment with it
    • functions take plain data and return new plain data
    • see the data in every step in between
  • semi-structural navigation via keyboard shortcuts
    • just works across code and data

• What does it take to set up and connect a node to this system (eg how are the node's capabilities specified)?
  • it's a library you embed into your application. call it with:
    • a random node ID
    • the central message broker's URL
    • and an optional map (dict/object) of this node's capabilities for effects you want to expose
      • caps are function closures or references to host platform objects
  • or just download ready-to-use runtimes (just give them node IDs)
    • docker containers
    • standalone binary (Windows, Linux, Mac)
    • single board computer SD card image (e.g. Raspberry Pi)
    • babashka library/pod?