Davis Haupt's Blog

Ideas for an Agent-Oriented Programming Language

Sun, 15 Feb 2026 17:00:00 GMT

Software development is changing. Tool calling, inference scaling and RL with Verifiable Rewards have combined over the past year to enable agent harnesses like Claude Code which can reliably navigate, modify and contribute to large codebases.

LLMs scale amazingly well with the amount of training data you throw at them. But I’ve been thinking about how to build tools that work alongside the characteristics of LLMs rather than language models needing to learn how to work with existing human-centric tools during training.

I have a hunch that a programming environment built around the strengths and limitations of autoregressive LLMs can lead to cheaper and higher-quality agent-powered development. How could we prove out that hypothesis? One would first need to design a language that aligns with how LLMs “think”. What would such a language look like? In this post I put forward some ideas for a language called Markov that I think would fit the bill.

Humans should be able to read and edit Markov code

First and foremost, Markov will still be human-readable and editable. Agents excel at generating boilerplate, but the core business logic of any system is where the most attention to detail will always need to be. Most programmers can get by today without ever digging into TCP packet captures or assembly code. That doesn’t mean there’s never a good reason to do so. I wrote a few months ago in Thinking about Thinking with LLMs:

[T]he best programmers aren’t the ones that make the widest use of the highest abstractions. That’ll continue to be those who dig down and understand what’s happening at a deeper level — that understanding will always lead to more deft use of any tools that programmers have at their disposal.

Transparency into the source code is as much ideological as it is practical. Over the next few years agents will likely start independently building systems end-to-end. In that world, interpretability of AI-generated systems becomes as important as interpretability into the models themselves. We may not have transparency today into the reasons an LLM might make the decisions it does,¹ but we should discourage models from developing incomprehensible “neuralese” and keep developing tools which allow for human insight to continue to play a role.

Sum types for fast compiler feedback

Agents thrive on feedback. It helps them stay on the assigned task without falling down unproductive rabbit holes. Unit tests are important here, but static analysis will always faster and less error-prone than running code. Markov will have strong, static types and exhaustive pattern matching to encourage making illegal states unrepresentable.

Rust’s reliance on sum types and compiler-enforced exhaustiveness checks in pattern matching enables fearless refactoring and extension of existing code. A human can spend a few minutes adding a variant to an existing type after thinking hard (or planning with an LLM) about the best way to represent some new functionality. The agent can then propagate that change outward through the codebase, completing now-inexhaustive pattern matches and following boilerplate patterns it encounters.

Markov should lean into this pattern: a human (or very smart LLM) makes the decisions about core data structures and interfaces, and an agent will propagate that change throughout the rest of the codebase. Today this second step needs to be carried out by a frontier model but in the future a smaller/cheaper/faster model can probably do this job sufficiently well.

This is a pattern that has been extremely effective in my own work with coding agents since Claude 4.5 was released: focus on core types and abstractions and let the agent plumb it through. With a coding agent I don’t have to be as careful about future-proofing my types or using fancy language features to avoid boilerplate when adding new functionality. Strong interface boundaries and full sum types help keep an LLM on guard rails.

Edoardo Vacchi wrote a post late last year called The Return of Language Oriented Programming about LLM-enabled DSL toolchains. He describes how an LLM can generate a spec and an interpreter for a DSL and then generate business logic in that DSL. This is reminiscent of narrow waist software engineering that I talked about in my newsletter in reference to Mitchell Hashimoto’s article about non-trivial vibe coding. A well-defined protocol between different parts of the system — in Mitchell’s post, a view model; in Edoardo’s, a DSL — reduces the “cognitive” load and can make either a human programmer or an LLM much more effective.

I prefer embedded syntax within a strongly typed language² to a DSL backed by a full compiler toolchain, but I agree with the main thrust of the article. This pattern is probably most commonly associated with more academic languages like OCaml and Haskell. However, embedded DSLs form the foundation of modern machine learning with Python-based DSLs like PyTorch and Triton. Edoardo addresses this towards the end but asserts that:

The cost of defining your own syntax instead of leaning onto the host language’s is now much lower; in fact, I would even dare to say that you might want to prefer flexible syntax, so that you will be able to optimize for token cost.

I think Edoardo presents a bit of a straw man argument. Writing a lexer, parser and interpreter might be cheaper with LLMs, but maintaining that stack is still a nonzero amount of effort that relying on a host language with good DSL support can obviate.

Compiler errors as prompts

Compiler errors are human-readable. There’s short explanations with error codes and links to external documentation to stay within a small terminal window, and often some ASCII art drawing an arrow to the specific problematic row and column within a file. LLMs can only interpret these errors since Stack Overflow and other sites are present in the training set and the aggressive post-training with verifiable rewards that conditions them to understand these errors.

But what would compiler errors for LLMs look like? They would be phrased as prompts with suggestions on how to fix an error formatted as diffs rather than ASCII diagrams. When possible the compiler should explain the cases that a specific error may happen and give the agent possible directions of investigation.

Token optimized syntax

Markov’s syntax should be optimized to make the most effective use of the context window. There’s already some promising results for TOON, an alternative rendering of JSON whose grammar uses elements that LLMs need to represent across multiple tokens. Token-optimized code is cheaper for agents to process³ and also potentially easier for them to understand: TOON shows slightly better retrieval accuracy compared to JSON even though the models have been trained on heaps and heaps of JSON.

“The Return of Language Oriented Programming” touches on token-optimization, too. LLMs trained on a general corpus of English use fewer tokens to represent common words than they do to represent shorter abbreviations that human authors often use for the sake of write-ability, like Rust’s fn that marks functions definitions. I firmly believe that code should be optimized for readability over write-ability, and it’s pretty cool that if Markov was designed to be simple for LLMs to write efficiently would also probably be more readable for humans than one designed for humans to write efficiently. This nicely feeds back into our first goal.

LLMs are horrendously bad at counting,⁴ though, and so Ruby’s begin/end lexical scopes will probably be preferable to Python’s indentation-based scoping.

Martin Alderson surveyed existing languages and compared their token efficiency. Python was close to the top, but Clojure came in second place and Haskell and OCaml both beat out TypeScript. So there’s some evidence that functional programming idioms can be beneficial on this axis, too.

Interoperability is a non-goal

Most popular new languages of the past decade have all piggy-backed on existing software ecosystems: Swift on Objective-C; Kotlin on Java; TypeScript on JavaScript.⁵ Allowing programmers to gradually migrate over their existing codebases and rely on existing trusted dependencies has been the characteristic behind each of those languages’ rises.

This benefit degrades with coding agents. Today’s agents have been shown to reliably port existing libraries to new languages with a fraction of the effort. Drew Breunig even experimented with a software library that has no code checked in to source control at all. When you can generate new functionality in less time than it takes to research the right library, the ability for Markov to call into an existing ecosystem without an FFI is less critical.

Conclusion

Well, you may be wondering now, where are the code snippets? Unfortunately not in this post. I’ve had these ideas floating around in my head for months now, and wanted to get it down on paper and out into the world.

Armin Ronacher recently wrote up A Language For Agents which touches on similar themes. Armin’s post is worth a read, but Armin focuses on different aspects of a language and its ecosystem than I’ve done in this post.

We’ll see where all these ideas go! Luckily, coding agents make starting a new language easier than ever.

This fact keeps me up at night. My writing in this article might seem more accelerationist than my usual tone. I’m not sure that this future is coming, but I think it’s important to be prepared for and for these tools to be crafted intentionally rather than stumbled in to. ↩
Enforced with sum types or their more expressive cousins, GADTs ↩
The underlying LLM calls are often billed per-token. ↩
How many “R”s are in strawberry, again? ↩
Rust is the exception here (and is slightly older than the others), but it really filled an empty niche rather than improving on the ergonomics of a predecessor. ↩

Reviewing Project Hail Mary by Andy Weir

Tue, 10 Feb 2026 17:00:00 GMT

I read Project Hail Mary for the first time soon after it came out and devoured it in a single sitting — ruining my night’s sleep in the process. Even though I already knew the twists and turns of the plot, I still couldn’t put it down on this second read-through.

Just like The Martian, the scientific method infuses everything in this book. Moreso than Weir’s first novel, Project Hail Mary has a much more compelling “human element” about the ways we can work together across difference.

The narration and dialog both have a great and witty rhythm to them that keep the story fun to read throughout. I’m excited to see how it translates to the screen next month!

Reviewing Culpability by Bruce Holsinger

Tue, 03 Feb 2026 17:00:00 GMT

Who’s at fault in a fatal accident involving a self-driving car? Culpability investigates the abstract morality of the trolley problem while exploring the emotional fallout among the family who were the lucky survivors of the crash.

The book is firmly rooted in the perspective of the narrator Noah, a successful but run-of-the-mill lawyer at a Maryland law firm, his brilliant wife Lorelei, a digital ethicist at the top of her field, and their three teenage children. It’s an engaging book about parenthood and coming of age that has almost as much to say about 21st-century privilege and the American Dream as it does about ethical algorithms and artificial intelligence.

I don’t think I’ve read anything that spends as much time on the everyday considerations and dynamics of marriage or fatherhood. The book was 350 Kindle pages and I was able to finish it in a few sessions over the course of two days.

In my own media bubble I hear a lot about three-letter acronyms like Artificial Superintelligence (ASI) and Universal Basic Income (UBI); forces that some argue will move nations in the coming decades. Holsinger reminds us that we ought to give the same attention — if not more — to how the systems we’re integrating into our lives today will affect us and our relationships with the people we care about most.

Digital systems by definition are predicated on ones and zeros: absolute truth and absolute falsehood. Even still, the emergent behavior of modern distributed systems on top of these simple rules can be so uninterpretable to most human understanding that the best our existing moral frameworks and legal systems can do is just throw up their metaphorical hands and shrug their anthropormorphized shoulders.

During the COVID vaccine rollout, a hospital system put blame on an algorithm for a blatantly unfair apportionment of vaccine resources that benefited tenured medical faculty over resident physicians on the front lines. LLMs and diffusion models are trained to mimic authors and artists. This can be argued to fit in the bucket of copyright fair use in our legal system while still feeling wrong to many of us. Our system is built upon the assumption that mortal, moral humans are the progenitors and beneficiaries of intellectual property. That hasn’t been true for a while (the Walt Disney Company famously put their finger on the scale), however Generative AI models that take millions of dollars to train but then can be scaled out and deployed widely make the limitations of that assumption even more apparent.

Culpability also touches on the limits of our legal system, especially how it intersects with power and privelege. Noah is a lawyer and the story’s backdrop is the the police investigation of the car crash. Law itself is a field that has attempted to coerce human experience into discrete judgment for virtually all of recorded history. Machine code has voltage differentials giving way to ones and zeroes; neural networks have probabilistic predictions flattened into deterministic output. The legal system has judges and juries which dispense innocent and guilty verdicts.

I’m reminded of Broken Volume, sculpture by Cheyney Thompson I saw at The Whitney in 2019. 3D random walks were generated by CAD software and cast into concrete. One of the pieces broke in half upon installation, where it stood for the length of the exhibition. Random walks have the simplest of rules and can create beautiful shapes, but they don’t know about gravity. Algorithms follow defined rules that are encoded within them. All digital systems are based on our own incomplete and imperfect assumptions about how the things we create will behave when they encounter the world that we inhabit. Even when deep learning systems start to pick out their own patterns in their training data, they’re learning from the biases and imperfections of what we choose to include and what we choose to omit from their training.

People die in a car crash where a self-driving car is involved. Who’s at fault? We get closer and closer to an answer throughout the story, but never quite reach it. Can there be a singular answer? Do we need there to be one?

Reviewing Mistborn: The Final Empire by Brandon Sanderson

Sat, 31 Jan 2026 17:00:00 GMT

After last year I wanted to give Sanderson another shot. This time I chose one of his most popular series rather than a newer, spin-off work.

Mistborn stuck pretty closely to some common fantasy tropes. I thought Sanderson’s main characters had more depth to them in the book, which contributed to me enjoying the book slightly more than Tress. Allomancy is an interesting and certainly well-thought out magic system, but ultimately the softer, more mystical magic of Lord of the Rings or A Song of Ice and Fire is more to my taste these days.

I struggled the most with the dialog in this book. Sanderson’s word choices for his characters often felt grating to me and disarmingly modern for an early industrial fantasy setting. The way that members of the city’s criminal underground talked about their profession felt awkward in the way they leaned in to modern tropes the reader would understand rather than in-universe exposition. I don’t think any character exactly said that they were “doing the heist” or something that cheesy, but many exchanges weren’t far off.

I didn’t really put my finger on this until starting to read The Well of Ascension, the sequel which I picked up but have stopped reading for now. But here are a few quotes that hopefully capture what I mean:

So I began spreading rumors in the Western Dominance, then made myself one of Lord Cett’s advisors

The way this glosses over an indeterminate amount of time, and how a character casually talks about penetrating the inner circle of a rival, did not feel like how someone would talk about an operation of this importance or risk.

Maybe this won’t work — maybe after a couple months of being besieged, we’ll just end up giving away the city anyway.

“Being besieged” just really grated on me here. I could think of twenty ways to phrase this that sound more natural, starting with “maybe after a couple months under siege”.

These examples might seem like nitpicking, but I picked up at least one of these every chapter or two during Mistborn, and it really did affect my immersion in the world.

Reviewing A Tree Grows in Brooklyn by Betty Smith

Sat, 20 Dec 2025 17:00:00 GMT

A Tree Grows in Brooklyn is a quintessential coming of age story. Not just for Francie — the protagonist who we follow from the age of 10 through her going off to college — but also for New York itself, which we see change through industrialization along with the characters.

The first part (of five) is on the slower side, with barely-connected vignettes that paint a picture of what it was like to grow up in an immigrant community in Williamsburg around the turn of the century. These ensure that you really feel connected to Francie’s family and neighborhood by the time the proper story gets going in the second part.

I wonder why this wasn’t a part of my high school English curriculum: the book has a lot to say about everything from childhood and motherhood to the American Dream. I’m happy I picked it up, I enjoyed this more than I expected to!

Reviewing The Strength of the Few by James Islington

Wed, 19 Nov 2025 17:00:00 GMT

This is a tough sequel to review without any spoilers for The Will of the Many, so I’m not going to attempt it. Spoiler warning for Book I below!

The Strength of the Few picks up right where The Will of the Many left off: Vis, the protagonist, has been simultaneously copied into two other worlds called Luceum and Obiteum while his story also continues back in the world of Book I, called Res. Somewhat necessarily, each storyline is not as deep as the story in the first book.

The politics and motivations that made me excited about The Will of the Many fade into the background of this installment. The Catenan Senate in Res is relegated to a third of the chapters, and Vis’s unfamiliarity with the other two worlds make him much less of an astute observer of the political machinations around him there.

I mentioned in my last review that the world of Res felt shallow for how physically large it was. I can better forgive some of that that shallowness here considering how different Obiteum and Luceum feel. The Strength of the Few shows that Islington can build deep history and culture into his worlds when he wants to, but he sometimes thinks its better to keep readers at an information disadvantage for the sake of the plot.

Luceum

I absolutely loved the Luceum storyline. Readers are introduced to an entirely new world with just enough detail to keep us engaged and excited to learn more without overwhelming us with exposition. The way Islington uses Gaelic to represent Luceum’s primary language is great. From most readers’ perspective it’s the same as if he’s made up his own language without needing to be a philologist on top of being a fantasy author.

The worldbuilding in Luceum felt much more like what Ursula K. Le Guin utilizes in her Earthsea books rather than the exposition-heavy information dumps that Islington uses elsewhere that reminds me more of The Broken Earth Trilogy’s latter entries.

This version of Vis also has some real character growth here in a way that was completely missing in Obiteum and unsatisfying in Res.

Obiteum

Obiteum reminded me of Cittagazze from The Subtle Knife: the world is clearly broken, and something related to the existence of multiple worlds was what broke it. In general I saw more influences from His Dark Materials than Red Rising throughout The Strength of the Few. The initial imagery of Gleaners parallel Spectres in a very fun way.

Once we move from the rebel hideout of Qabr into the city of Duat, though, things start to fall through the cracks. My book club all agreed that we needed a map of Duat to go along with Vis’s descriptions of how he is moving through the city. I would love to have learned more about the society in Duat, its religion, and the totality of how Will works in Obiteum. So much time is spent on the undead Iunctii, but Vis’s entrance to Obiteum introduces us to a variety of Will-based devices that aren’t mentioned at all in parts II and III.

Res

I enjoyed learning more about the mechanics of Will and how Catenan society operates outside the confines of the Academy. Vis is fully on the inside of “the system” now, and seeing how Will affects everything from politics to athletics to combat was great.

The Catenan civil war completely changes the setting and unfortunately as much as we spent learning about Catenan politics in Book I and the first half of Book II, there’s so many more forgettable Catenan senators introduced whose motivations are explained mostly through exposition dumps.

All in, Part III’s Res storyline felt very rushed. Vis makes some consequential decisions which feel like they come out of nowhere. A new plan from Vis and Eidhin in the final 100 pages which I assumed would form the Res plotline in Book III is somehow wrapped up in less than a chapter. While there are strong emotional points, I ultimately don’t truly understand why Vis makes the decisions and alliances he does.

Splitting the Book

There were two very disorienting time jumps in the book: 6 weeks in Obiteum between parts I and II, and a month in Res between parts II and III. The narration refers to events and people that happened in the meantime without any indication of missing chapters in between — both times I had to double-check that my Kindle didn’t glitch out and miss a few dozen pages. Ultimately I think the location of these time jumps mark out a good place where the book itself could have been split in two:

Book II would follow the entire Luceum storyline, Vis’s time in Qabr in Obiteum, and the Res storyline up until Excesius is killed. This can dive more deeply into how exactly Will is used in Caten and flesh out the politics of the Senate even more.

Book III would cover the Catenan civil war and Vis’s infiltration of the city of Duat in Obiteum.

The downside of this approach would be that the three storylines would not proceed in lockstep in time. Maybe Islington is planning some bigger payoff in the next book, but there was one moment of “synchronism” that did not feel narratively or emotionally powerful, so I don’t think the parallel narrative gimmick is particularly worth it.

Conclusion

Overall I enjoyed learning more about the world(s) that Islington is building. My excitement about Luceum mostly counterbalances my confusion in Obiteum and my frustration with how Res feels even smaller and more condensed than in the first book.

Backstory isn’t bad in and of itself, but learning more history shouldn’t be the reason we keep reading. I don’t like when characters (and readers) learning new information becomes the main plot tension in a story. When revealed, the “right” answers are often not as satisfying as the possibilities and theories that readers come up with along the way. Right now I don’t know what’s a plot hole and what’s a conscious omission that will be filled in later to support a twist or some other dramatic effect. The only way to find out is to wait for the rest of the series to be published.

The Expanse balanced this well: its final installment gave some satisfying answers about the Protomolecule while leaving enough to interpretation and keeping the conclusion of the book mostly about the characters we’d gotten to know.

Considering how the focus of The Strength of the Few shifted from the motivations of individual people and groups in Res to the cosmic forces connecting and separating the three worlds we’re now following, I’m not as optimistic that Islington will stick the landing.

Thinking About Thinking With LLMs

Sat, 25 Oct 2025 17:00:00 GMT

After reading Developing our position on AI from the Recurse Center, the corresponding entry in my newsletter grew long enough I’ve decided to break it out into a full blog post. Above and beyond its specific findings, I think the post charts a path for a more civil and considered mode of discussion that we should all strive for on the Internet and in our own lives.

I’ve long been fascinated by the Recurse Center. Throughout my career I’ve been consistently impressed with alums I’ve crossed paths with, so I’m not particularly surprised that probably the most balanced and thoughtful take on AI and software engineering that I’ve read so far has come out of Recurse.

Discourse on the Internet

I think the article presents a great way to deal with contentious issues within organizations and how to write about those issues publicly. First, listen to everyone. Then, acknowledge disagreements and emphasize nuance to make an effort to find common ground.

The goal doesn’t need to be having The Correct Take at every moment. People have different life experiences which often leads them to different top-line beliefs. But beyond those headlines, where reasonable people state anything from “we’ll all be living in ASI-powered utopia by 2028” to “LLMs are slurping up all our fresh water”, there is a surprising amount of common ground.

In the case of this article, most people agreed that using LLMs to learn new things needs to be done with some caution.

Step-Functions in Abstractions

This paragraph from the post really resonated with me:

[T]utorials and teachers can only get you so far. Ultimately, you must build your own mental structures. Stack Overflow can be a helpful resource, but blindly following or copy-pasting from it does little to help you learn. While you can get a lot farther mindlessly using LLMs than Stack Overflow, the same principle holds true.

The comparison to StackOverflow is one that I’ve seen pop up again and again in discussions around LLMs. While LLMs can feel revolutionary, it’s not the first time we’ve seen a step change in software development resources.

Google made the skill of searching and skimming manuals for reference materials almost obsolete. O’Reilly still exists, but reference books are a much smaller part of an engineer’s education than shorter-form blog posts focused on specific tasks¹.

StackOverflow made it possible to accomplish many common coding tasks without much upfront thinking. Most consider copy/pasting StackOverflow answers to be an unfortunate shortcut. StackOverflow filled a niche, but it didn’t obviate the need for a deep understanding.

Software engineering has always had a component focused on automating “white collar” human work like airline logistics, processing payroll, or organizing and maintaining libraries². Programming as a profession has never been immune to this — assemblers automated away calculating memory offsets for jumping to procedures; garbage collectors automated memory management; Google Search changed what it meant to RTFM.

New tools always make it easier to code with a shallower understanding of the system. Claude Code, Cursor and their ilk are bringing programming closer to natural language just like C and Java and Python once did.

This is a good thing! It’ll lead to the continued democratization of programming and ever more people in conversation with computers. But I don’t think this changes the fundamental reality that the best programmers aren’t the ones that make the widest use of the highest abstractions. That’ll continue to be those who dig down and understand what’s happening at a deeper level — that understanding will always lead to more deft use of any tools that programmers have at their disposal.

All four of types of documentation certainly existed before the advent of search engines, but shorter-form content thrives in ranked searches and recommendation algorithms. ↩
The ones filled with stacks and stacks of books, not the ones with pre-written functions you can call from your programs ↩

Reviewing The Will of the Many by James Islington

Thu, 18 Sep 2025 17:00:00 GMT

The Will of the Many started off feeling like a cross between the political setting of Red Rising and the magic system of The Fifth Season. The world felt big, but not particularly filled in — the author throws out place names and countries, and we meet characters from different corners of the now world-spanning Catenan Republic, but the places don’t feel particularly differentiated or deep. A map would have helped here, but maybe the author has a reason to hold one back.

While the worldbuilding wasn’t as deep as what I’m normally interested in, I ultimately found myself drawn in by the web of intrigue and competing political motications that the narrator/protagonist finds himself surrounded by. He has no “correct” choices. He’s unsure of his own goals beyond survival. And while he’s uncovering the secrets and motives of those around him as he goes, it doesn’t feel like the suspense is just the drip-drip-drip of information — no other character has all the pieces to the puzzle, either.

The plot is messy in such a satisfying way. I’m not sure if the focus of the second installment will continue to be on the same story or if it will shift its attention elsewhere. But I’ll be excited to dive back in when it comes out.

Reviewing The Telling by Ursula K. Le Guin

Fri, 15 Aug 2025 17:00:00 GMT

Back to science fiction with another installment in the Hainish Cycle! The Telling is about orthodoxy, totalitarianism, and cultural revolution. It felt different than other sci-fi I’ve read from Le Guin. Instead of being a man from an alient planet like Hain-Davenant or Annares, the narrator is an Indian-Canadian woman from Earth. While the story takes place on the recently-uplifted planet Aka, Earth and the narrator’s rememberings and attitudes towards our home feature strongly in the book.

The Hainish Cycle is all about celebrating how our differences make us stronger while emphasizing our common humanity. Bringing Earth to the forefront of this installment helped drive the lesson home in a new way.

Reviewing The Farthest Shore by Ursula K. Le Guin

Sun, 27 Jul 2025 17:00:00 GMT

Compared to the first two books in Earthsea I was not as excited about The Farthest Shore. Some of the scenes felt — for lack of a better word — a bit trippy, which made the narrative a bit harder to follow.

Although I do understand that Earthsea is marketed for a late-teen audience, I may be getting a bit tired of the coming-of-age tropes. “The Farthest Shore” still continues to fill out the world of Earthsea in an interesting and satisfying way.

Reviewing The Tombs of Atuan by Ursula K. Le Guin

Fri, 25 Jul 2025 17:00:00 GMT

The second installment of the Earthsea series, I enjoyed this book about as much as A Wizard of Earthsea. Le Guin puts a lot of effort into fleshing out the world of Earthsea, but unlike other fantasy worlds, she’s doing it through the eyes of her characters rather than through exposition or lore dumps.

Ged, the protagonist of book one, and Arha, the protagonist of this book, grew up in completely different circumstances. They see the same world and same events through their own differing and often conflicting perspectives. While it may sound disorienting, the overall world feels even more coherent and real for it.

Reviewing A Wizard of Earthsea by Ursula K. Le Guin

Sun, 20 Jul 2025 17:00:00 GMT

Until now I’ve only read Le Guin’s science fiction work, and have wanted to give her fantasy series Earthsea a try for a while.

I really enjoyed this first installment. The societies that the protagonist ventures through felt lived-in in a way that feels missing from some other high fantasy like the Lord of the Rings.

The way that magic and the sorcerers and witches who perform it are weaved into the world was really well done. “Sorcerer” is a vocation like any other, although one that’s highly valued in many places. Sorcerers and witches often fill in the role of high technology, whether that’s powered vehicles or medicine.

The magic system itself felt like it had its own internal logic, But enough mystery was left for there to be places for Le Guin to fill in the blanks in future novels without backing herself in a corner.

Beyond the top-level genre, A Wizard of Earthsea is a coming of age story, which are always enjoyable in their own way. Even if the “twist” was a bit predictable, the worldbuilding was top-notch, and I’m excited to pick up the second book in the series!

Reviewing You Dreamed of Empires by Álvaro Enrigue

Mon, 14 Jul 2025 17:00:00 GMT

A retelling of the initial encounter in Tenoxtitlan between the Aztecs and the Spanish. I found it a bit hard to get into this book. Translations are always tough since translators are rarely as skilled with words as the original authors and there’s two levels of intention and meaning to work through.

One thing I did enjoy was how Moctezuma and Cortés, both giant historical figures, were brought down to human scale through their point-of-view chapters.

Overall, I found the pacing quite slow and the climax somewhat rushed. I kind of think that a sequel would be more interesting from a plot perspective — although I expect that plot was not what Enrigue was optimizing for.

Reviewing A Fisherman of the Inland Sea by Ursula K. Le Guin

Sun, 25 May 2025 17:00:00 GMT

I read many of Le Guin’s more classic entries in her Hainish Cycle back in college and since then have been fascinated with the world she built for novels like The Disposessed and The Left Hand of Darkness. A Fisherman of the Inland Sea is a series of short stories of which only the last three have anything to do with the universe of the Hainish Cycle. These three all revolve around the development of instantaneous, faster than light travel that Le Guin calls “churten theory” — notably not capitalized most of the time. Le Guin doesn’t focus on the effects of FTL travel on the society of the Ekumen. Instead, the imprecision of the new technology opens the door to stories about love, community, and how the narratives we tell ourselves affect the way we move through the world.

These short stories weren’t as fully baked as Le Guin’s full-length novels, but in some ways that helped the core insights behind them shine through even clearer.

Reviewing Tress of the Emerald Sea by Brandon Sanderson

Mon, 12 May 2025 17:00:00 GMT

Part fantasy, part pirate novel, part fairy tale, this was the first Brandon Sanderson book I’ve read, and I’m not sure it was the best one to start on. The main things I enjoyed in this book were the worldbuilding and the writing style. Unfortunately, I had a hard time disentangling the character of the in-universe narrator with the author, who took almost every opportunity to drop “wisdom” and moralize the current situation. The story itself felt like it predictably flowed along the lines of fairy tale tropes even when it was subverting them.

Reading the acknowledgements section after finishing the book, it’s clear that Sanderson was trying out a new first-person narrator and also targeting a “The Princess Bride”-style fairy tale pretty explicitly here. I’ll probably try out another one of his novels sometime soon to see if the things I did like about this book shine through some more.

Reviewing Zen and the Art of Motorcycle Maintenance by Robert M. Pirsig

Wed, 07 May 2025 17:00:00 GMT

I really enjoyed this book — its final lines even surpassed The Great Gatsby as my favorite closing lines of a novel. But I couldn’t find myself justifying a rating higher than 4/5 when I broke it down and tried to explain what exactly I enjoyed so much in it. I caught myself after a few minutes, smiled, and realized in some ways that’s what Zen and the Art of Motorcycle Maintenance was all about.

I’m a very analytical person. I think that isn’t particularly surprising considering I started programming computers way before my brain finished developing. While analytic, dualistic, “dialectical” thinking has benefited me in a lot of ways, I’ve come to realize over the past few years that not everything in life can or should be intellectualized or broken down into repeatable understanding. Zen and the Art of Motorcycle Maintenance gave some form and vocabulary to a lot of things I’ve been thinking and sometimes writing about over the past few years. I found a lot of resonance in how the book describes how those of us with more analytical tendencies can widen our perspective.

A description of a cross-country motorcycle trip unfolds alongside the narrator’s philosophical musings to the reader, and the two threads intersect and complement each other at various times throughout the book. The “present-day” story starts off slow but picks up as the philosophising half (the “Chautauqua”) starts giving more depth back to the characters in the story-half. At the end of some chapters, I found myself doing something I’ve rarely, if ever, done before: I put the book down just to think through and digest what I’d just read rather than powering full speed ahead to see what happens next.

I can’t remember the last time I wanted to start a book over again as soon as I finished, but I feel like this is a book that would really read differently the second time around with the full context that’s revealed mid-way. I think I’ll just sit for a little longer with that first impression, though.

Reviewing Abundance by Ezra Klein and Derek Thompson

Sun, 13 Apr 2025 17:00:00 GMT

I listen to both Ezra’s and Derek’s podcasts regularly and was pretty familiar with the outline of “Abundance” the concept from Klein mulling over his thoughts over the past four years on the Ezra Klein Show. I’m broadly sympathetic to the aims of the book but I felt like Abundance fell short of crafting the most convincing argument for “a liberalism that builds.”

I don’t know if the felt lack of insight from the book is because I listen to the authors on a weekly basis or if the book was aimed at a wider audience who have been less exposed to these ideas. Regardless, I wish they had focused a bit more attention on explaining why an Abundance agenda will be difficult politics.

The “Build” chapter glosses over safety regulation pretty quickly without engaging at all with how workplace accidents can become media sensations or the real the tradeoffs that would need to be made to make safety regulations less onerous.

In “Invent”, much of the focus is on how the NIH and the American academic establishment let Katalin Karikó and her research on mRNA vaccines slip through the cracks. While the $450 million Stimulus/DoE loan that Tesla recieved is mentioned, the similar grant paid to Solyndra before the company went bankrupt never is. Wanting government to be less risk averse is a good aim that I support, but taking on risk and uncertainty does lead to a higher likelihood of undesirable outcomes. There’s a political communication problem in helping the public understand that a single failed company in a startup program doesn’t indicate waste or foul play.

The specific DoE program that funded Tesla and Solyndra made it into the black by 2015, but beyond the economics of this one program, the ways in which it supports American businesses should be accounted for separately. Tesla very likely wouldn’t be where it is today without that Stimulus loan, and I think that’s worth at least half a dozen Solyndras.

A specter hanging over the book is that the authors seem to be advocating for something that on the surface has the same goals as Elon Musk and Vivek Ramaswamy’s Department of Government Effeciency had when it was announced after the election. This book is both timely and ill-timed — the authors have had to spend a lot of time in interviews distancing themselves from DOGE as Musk has “deleted” federal programs and agencies left and right. I worry that any effective implementation of Abundance on the national level will be held back by negative associations with the ramifications of this push for “effeciency” that the country will be feeling for years go come.

I think Abundance is still worth reading, if only to understand its place in the recent discourse, and especially if you’re not particularly familiar with the authors or topic areas. The book is more of the beginning of the discussion rather than the final word.

Reviewing Golden Son by Pierce Brown

Sun, 16 Mar 2025 17:00:00 GMT

I found Golden Son to be even more of a page-turner than Red Rising. The setting of the book changes pretty significantly and the cast of characters around the protagonist mostly shifts as well, but I was kept interested by how the author continues to build out the world of the series.

What brings my review down a star is how Brown moves from relying on themes and ideas from other sci-fi series to borrowing some pretty major plot points. The way that this second book leans more in its influences than the first has made me less excited to continue the series from here — I started book three, but I put it down a quarter of the way through, and I’m not feeling much motivation to pick it back up.

Reviewing Red Rising by Pierce Brown

Sun, 02 Mar 2025 17:00:00 GMT

A fun sci-fi novel that brought together a lot of parts of other series and franchises into an interesting world: a solar-system spanning oligarchy with a caste system backed by geenric engineering whose ruling class reveres the ancient Romans more than any enlightenment-era political theorist.

Some parts will stick out as a bit derivative: the main setting of the book is a Hunger Games-esque battle royale, and the strategies and political theory the main characters use to win felt very reminiscent of Ender’s Game. Even so, the book was able to weave in its own themes. I found the narrator’s internal struggles interesting to follow along with and generally the book was a page turner.

2024 in review

Sat, 04 Jan 2025 17:00:00 GMT

I wrote most of this post during the week between Christmas Eve and New Year’s Day when enough people in New York have left the city that it makes it quite difficult not to turn introspective. It’s been a pretty big year for me and for this blog, so I thought it would be fun to take a look back on the year as we all move into the second half of the decade.

Blog Posts

I wrote four long-form blog posts this year: a fun post on Dune for April Cools and three in my Nix on macOS series. I’ve gotten a ton of really encouraging feedback from friends, acquaintances and internet strangers about my Nix articles.

Somehow my TIL post from November 2023 started ranking well on Google and my site is one of the first that shows up when most folks search for “nix darwin”. I’m proud my writing has been useful to folks. It’s a good reminder that I can find an audience in a niche just writing about what I want to write about before worrying about metrics at all. I still have at least three more posts to write from my original series outline, but we’ll see if I switch tach and write about other things in 2025.

One of my goals for 2024 was to write more. I’m not always motivated to write deeper long-form technical content that I’m happy to put up on the main blog, though, so I took a page out of Arne Bahlo’s playbook and started a link curation newsletter. It’s a genre that provides structure while still gives me space to express myself, share something interesting, and flex my writing muscles some more.

My blurbs about each link follow a bit more from Simon Willison or John Gruber’s link blogs than Arne’s summaries — I try to add some analysis and sprinkle in nuance where I can. The original goal was to expand some of these blurbs into full blog posts. I haven’t felt the burning desire to do that yet, but we’ll see how next year goes.

The newsletter is called Weekly Links, and after a 52 weeks there’s… well.. 17 entries. The first 10 came quickly and easily, but I started falling off with the habit in early March and ultimately had a six month gap between #12 in early May and lucky #13 in November. I don’t want to beat myself up about not following through with the weekly cadence — I want to make sure I’m writing these for myself first and foremost, and I’ve really been enjoying getting back into the habit of writing.

What I Learned

I barely had enough articles saved up to fill up my first newletter back from hiatus in November. When I sat down to write #14, though, I had more links saved up from the week than I could fit in one post. It’s been a great example of how writing for me is a positive sum activity — the more I write, the more my mind is attuned to finding interesting things to write about.

Discovering positive-sum feedback loops has been the real theme of the year for me. Whether it’s writing each week, running a mile in the morning, or working on interesting projects at work, I’ve discovered first-hand how activity begets energy when you’re working on the right things. I’ve struggled with motivation and energy levels in the past, but this year I’ve learned that sometimes you just need to kickstart the flywheel.

In that vein, the article that’s had the greatest impact on me this year is probably Work hard and take everything really seriously by Tom MacWright (featured in Weekly #6). It helped me crystallize how I’d been feeling around work and leisure time — how “balance” is more complicated than we often give it credit for.

All of these habits may be self-reinforcing in some ways, but they also require active maintenance. My main goal for next year is to do the things that excite me more often.

As always, thanks for reading! I hope everyone had a great new year, and I’m excited to see where 2025 leads.

Reviewing Tomorrow, and Tomorrow, and Tomorrow by Gabrielle Zevin

Fri, 20 Dec 2024 17:00:00 GMT

A book about life, love, and video games. I honestly didn’t know what I was getting into when I started this book, but by the time they described the twist behind Sadie’s first video game in the second chapter I was hooked. Some people may be put off by how the characters and the third-person narrator talk about video games as a high art form but I love hearing and seeing how digital media can expand our artistic horizons rather than shrink them.

A friend who read the book around the same time described it as “a romance, I guess?” and that ambiguity is at the heart of what makes this book feel real. As we follow the main characters from their college days into their thirties, the ways their views about the world and each other change does a lot towards making the characters feel like real people.

Ultimately I got a lot out of what this book had to say about friendship. It’s natural for miscommunications to accumulate into misunderstandings over the years despite our best intentions. What’s important in the end is to remember that it’s never too late to return to the foundation and try to grow something new.

[TIL] nix flake update: Updating dependencies of a Nix flake

Sat, 30 Nov 2024 17:00:00 GMT

I just realized today that I haven’t updated VSCode on my Mac in almost a year. It turns out that if you install any program through Nix, its version is determined by the version of nixpkgs in your flake.lock file.

To update nixpkgs and get newer versions of all your programs, cd into the directory with your flake and run nix flake update. It’s the pacman -Syu of nix!

Configuring VSCode with Nix on macOS

Sun, 24 Nov 2024 17:00:00 GMT

Welcome back to the Nix on Mac series! By the end of this post, you’ll be able to fully configure VSCode through your Nix flake via home-manager, which we set up in part 2. This includes:

Custom keybindings and settings
Installing themes and extensions from a nixpkgs overlay
Properly aliasing VSCode and other macOS applications to /Applications for Spotlight

Without further ado, let’s dive in!

Installing VSCode

Nix won’t install non-free software like VSCode without you opting in. Add this line to your nix-darwin configuration module:

configuration = { pkgs, ... }: {
    # ...
    
    nixpkgs.config.allowUnfree = true;
    
    # ...
};

After that, installing VSCode is just one line of code in our home-manager config:

homeconfig = { pkgs, ... }: {
    programs.vscode = {
        enable = true;
    };
};

I’m sure many of you who are following along are using VSCode to edit your flake.nix file. I should note that installing VSCode through home-manager will likely blow away your existing settings when you call switch. Now would be a good time to back up your existing settings and extension list so you can replicate your setup from within Nix.

VSCode will be installed in /Users/$USER/Applications/Home Manager Apps/¹ after running switch.

If you’ve previously installed VSCode, go ahead and dump the copy from /Applications in the trash. Where we’re going, we don’t need ~~roads~~ globally installed programs!

Editor Settings and Keybindings

You can add user settings and keybindings pretty easily:

programs.vscode = {
    enable = true;

    userSettings = {
        # This property will be used to generate settings.json:
        # https://code.visualstudio.com/docs/getstarted/settings#_settingsjson
        "editor.formatOnSave" = true;
    };
    keybindings = [
        # See https://code.visualstudio.com/docs/getstarted/keybindings#_advanced-customization
        {
            key = "shift+cmd+j";
            command = "workbench.action.focusActiveEditorGroup";
            when = "terminalFocus";
        }
    ];
};

It almost looks like JSON! And in fact, converting Nix types to and from JSON is part of the Nix standard library.

Extensions

A huge part of VSCode is the bountiful extension ecosystem. home-manager lets you install extensions similarly to how it installs packages. The full VSCode marketplace isn’t present in nixpkgs by default, so we’ll need to install an overlay.

Nixpkgs overlays let you override and add new entries to nixpkgs. We can add the nix-vscode-extensions overlay by adding a line to our nix-darwin configuration:

{
  # ...
  inputs = {
    # ...
    nix-vscode-extensions.url = "github:nix-community/nix-vscode-extensions";
  };

  outputs =
    inputs@{ self
    , nixpkgs
    , nix-darwin
    , home-manager
    , mac-app-util
    , nix-vscode-extensions
    }:
    let
      configuration = { pkgs, ... }: {
        # ...
        nixpkgs.overlays = [
          nix-vscode-extensions.overlays.default
        ];
        # ...
      };

      homeconfig = { pkgs, ...}: {
        # ...
        programs.vscode {
          # ...

          userSettings = {
            # ...
            "workbench.colorTheme" = "Dracula Theme";
          };

          # ...

          extensions = [
            pkgs.vscode-marketplace.jnoortheen.nix-ide
            pkgs.vscode-marketplace.dracula-theme.theme-dracula
          ];
        }
      }
  # ...

vscode-marketplace is one of the properties that the nix-vscode-extensions overlay added to nixpkgs. Any VSCode extension in the marketplace should be accessible from pkgs.vscode-marketplace.$AUTHOR.$EXTENSION, where $AUTHOR.$EXTENSION is the same as the itemName property in the extension’s URL on the extension marketplace website.

Cleaning up lists of attributes with the `with` clause

Writing pkgs.vscode-marketplace in front of every extension will get tedious as as your list of extenions gets longer and make the list harder to read. Luckily Nix has a language construct to help with this: in front of any expression, you can type with <attribute set> to bring all attributes within the attribute set into scope. Our extensions list can look like:

extensions = with pkgs.vscode-marketplace; [
    jnoortheen.nix-ide
    dracula-theme.theme-dracula
];

Much cleaner!

Playing nice with Spotlight

You might have noticed that the version of VSCode that Nix installs doesn’t show up in Spotlight. Why is that? It’s unfortunately pretty simple: All artifacts that Nix and home-manager add to your system are symbolic links, and Spotlight won’t index symlinks. There’s a very long thread about this on GitHub if you’re interested in reading. From the various options discussed there, mac-app-util is the easiest way to get Spotlight working as expected.

We can add it as another input at the top of our flake and then modify our flake output to load mac-app-util in both nix-darwin and home-manager:

{
    description = "My system configuration";
    inputs = {
        # ...
        mac-app-util.url = "github:hraban/mac-app-util";
    };

    # ...

    in
    {
      darwinConfigurations."$HOSTNAME" = nix-darwin.lib.darwinSystem {
        modules = [
          # ...
          mac-app-util.darwinModules.default
          home-manager.darwinModules.home-manager
          {
            # ...
            home-manager.sharedModules = [
                mac-app-util.homeManagerModules.default
            ];
            # ...
          }
        ];
      };
    }
}

After running switch, VSCode should be properly copied to /Users/$USER/Applications/Home Manager Trampolines and should be available the next time Spotlight refreshes.

Stepping Back

We just included someone’s third-party Nix utility in our flake to do some custom behavior when Nix installs macOS GUI apps. You probably didn’t notice that mac-app-util is written in Common Lisp. I’ve never written Common Lisp, and my guess is you haven’t either. Even so, with fewer than five lines of code we were able make use of some code that someone wrote in their favorite niche programming language without needing to figure out how to build or run that code. Nix handled it all for us! I, for one, think that’s pretty amazing.

Conclusion

Our Nix config is now able to fully configure and manage VSCode, including installing extensions and themes. home-manager has just as deep an integration with many other utilities. If you haven’t already definitely go through and check out home-manager’s config options to find out what you can do with your favorite program.

You can find the full flake.nix for this installment on GitHub here. See you in the next one!

Notably, this isn’t inside the normal /Applications folder since home-manager can only install programs under your home directory. Even stranger, Spotlight can’t pick up our alias! We’ll come back and fix this at the end of the post. ↩

The Spice Didn't Always Flow

Mon, 01 Apr 2024 17:00:00 GMT

Dune: Part Two hit theaters recently, which got me thinking about my own experience reading Dune in the past, what I enjoyed about it, and my own personal headcannon that made this novel more enjoyable for me. And hey, it’s April Cools, so why not put pen to paper here?

Credit to this reddit thread for the initial discussion, but I hope I’ve expanded on the concept below.

Questions

I’ve read Dune a few times over the past ten years, and as immersive as the world and politics are, a few discontinuities kept bothering me.

The universe’s best-kept secret

Spice is commonly referred to as a geriatric drug that slows down the effects of aging before Paul escapes to the desert, and this is the explanation given to the reader and to Paul as to why it’s so valuable a commodity. In the final chapter, it’s considered a big reveal that the Guild Navigators are addicted to the spice, and have the Eyes of Ibad just like the Fremen, carefully concealed by colored contact lenses even in front of the Emperor:

The taller of the two, though, held a hand to his left eye. As the Emperor watched, someone jostled the Guildsman’s arm, the hand moved, and the eye was revealed. The man had lost one of his masking contact lenses, and the eye stared out a total blue so dark as to be almost black. (Book 3, Chapter 48)

If the Spice has been around for thousands of years, how is it still not common knowledge among at least the Great Houses that spice induces hallucinations and limited prescience?

When was spice discovered?

Melange is described as essential to space travel, and yet the narrator and many characters talk about the time of its discovery as if it was close to within living memory.

He thought of the filmbook Yueh had shown him — “Arrakis: His Imperial Majesty’s Desert Botanical Testing Station.” It was an old filmbook from before the discovery of the spice. (Book One, Chapter 9)

Beyond the logistics of a filmbook being preserved for over 10 millennia, it’s described as “old” rather than “ancient”, the preferred word used throughout Dune to describe the incomprehensible age of structures like Castle Caladan and languages like Chakobsa.

Evidence of a human presence on Arrakis before the discovery of the spice, but after the establishment of the Empire in the wake Butlerian Jihad, brings up another question: how did humans even make it to Arrakis without spice in the first place?

The Theory

My own personal headcannon does in fact contradict the later books, but makes Dune as a standalone story more consistent to me. Rather than being an ancient part of the traditions of the Fremen, the Spacing Guild, and the Bene Gesserit, Spice is a relatively recent discovery by the Empire, probably in the last hundred years or so.

Arrakis itself was discovered at some point in the more distant past. It remained mostly an ecological and scientific oddity rather than a center of commerce and resource extraction. The Fremen’s ancestors were expelled from their previous home at some point in the past and discovered the spice soon after arriving on Arrakis integrating it into their spiritual rituals and replacing other awareness-spectrum narcotics. As much is described by Jessica after she becomes a Reverend Mother of the Fremen:

… [T]he Fremen culture was far older than she had suspected.

There had been Fremen on Poritrin, she saw, a people grown soft with an easy planet, fair game for Imperial raiders to harvest and plant human colonies on Bela Tegeuse and Salusa Secundus…

And she saw the thread of the past carried by Sayyadina after Sayyadina — first by word of mouth, hidden in the sand chanteys, then refined through their own Reverend Mothers with the discovery of the poison drug on Rossak… and now developed to subtle strength on Arrakis in the discovery of the Water of Life. (Book Two, Chapter 37)

Spice is just one awareness-spectrum narcotic in Dune. What makes it special and valuable is its potency and lack of side effects. Especially notable is its addictive properties. Once you use the spice, any other drug fail to have the same awareness-expanding effect. Paul’s lecture to Gaius Helen Mohaim and the Spacing Guild representatives at the end of the book makes way more sense in this context:

[The Spacing Guild] might have taken Arrakis when they realized their error of specializing on the melange awareness-spectrum narcotic for their navigators. (Book 3, Chapter 48)

And pages later:

“Even your Bene Gesserit Truthsayer is trembling,” Paul said. “There are other poisons the Reverend Mothers can use for their tricks, but once they’ve used the spice liquor, the others no longer work.” (Book 3, Chapter 48)

Considering the life-extending properties of spice and large-scale extraction only beginning 80 years before, it’s likely that Mohaim and the Guild navigators or others in their generation were likely the ones who made the decision to go all-in on Spice and Arrakis, rather than Paul admonishing their predecessors thousands of years in the past.

While it’s not really consistent with later books in the series, I feel like this retcon/headcannon gives the characters of Dune a lot more agency in the history of their world, and makes the Imperium feel like a more dynamic human society rather than ossified in a single pattern for millennia.

Managing dotfiles on macOS with Nix

Sat, 17 Feb 2024 17:00:00 GMT

In part one of this series we installed Nix, set up our system configuration with nix-darwin, and installed some packages at the system level. In this post, we’ll set up home-manager.

Unlike nix-darwin, home-manager is cross-platform: it works across NixOS, macOS, and anywhere else Nix can be installed. It was difficult at first for me to understand how home-manager and nix-darwin should interact. While there is definitely overlap with what these two Nix libraries can do, nix-darwin is used for managing system-wide settings and applications: it brings the power of NixOS to the Mac. home-manager on the other hand is most useful for managing user-level configuration and dotfiles.

By the end of this post we’ll have installed home-manager and used it to set up configuration for vim, zsh, and git.

Install `home-manager`

home-manager will be another input to our flake. In our inputs attribute, add home-manager under nix-darwin in flake.nix:

{
    description = "My system configuration";
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
        nix-darwin = {
            url = "github:LnL7/nix-darwin";
            inputs.nixpkgs.follows = "nixpkgs";
        };
        home-manager = {
            url = "github:nix-community/home-manager";
            inputs.nixpkgs.follows = "nixpkgs";
        };
    };
    # ...

We will then add the minimal home-manager config as another module under configuration:

# ...
    outputs = inputs@{ self, nixpkgs, nix-darwin, home-manager }:
    let
        configuration = {pkgs, ... }: {
            # ... nix-darwin configuration from part 1 
            # remains unchanged here
        };
        homeconfig = {pkgs, ...}: {
            # this is internal compatibility configuration 
            # for home-manager, don't change this!
            home.stateVersion = "23.05";
            # Let home-manager install and manage itself.
            programs.home-manager.enable = true;

            home.packages = with pkgs; [];

            home.sessionVariables = {
                EDITOR = "vim";
            };
        };
    in
    {
    darwinConfigurations."$HOSTNAME" = nix-darwin.lib.darwinSystem {
        modules = [
            configuration
            home-manager.darwinModules.home-manager  {
                home-manager.useGlobalPkgs = true;
                home-manager.useUserPackages = true;
                home-manager.verbose = true;
                home-manager.users.$USER = homeconfig;
            }
        ];
    };
  };
}

Our darwinSystem setup now includes another module that bridges together home-manager and nix-darwin. Remember to replace $USER with your system user!

Run darwin-rebuild switch --flake ~/.config/nix to ensure everything is set up correctly.

Managing `.vimrc`

Let’s manage our first dotfile! First, we’ll create a new file in our repository and add it to git so that Nix can recognize it:

$ cd ~/.config/nix
$ echo "set number" > vim_configuration
$ git add vim_configuration

Since this isn’t a vim configuration series we’ll just have a small config that will demonstrate how home-manager manages dotfiles. You should be able to see line numbers when running vim after this configuration is applied.

Inside our homeconfig block before the closing };, add this line:

home.file.".vimrc".source = ./vim_configuration;

This one line highlights a few subtleties of the Nix language. Let’s dig in a bit before moving on.

Paths

In Nix, unlike most languages, Paths are first class primitive values that are not inside of quotes and start with ./ for paths relative to the current file or / for absolute paths. ./vim_configuration points to the file that we created in the same directory as flake.nix.

You’ll run into type errors if you try passing strings to options where paths are expected.

Setting attributes in Nix

Attribute sets are probably the most ubiquitious datatype in Nix. They’re the equivalent of dictionaries in other languages. Something that’s pretty unique to Nix is that setting nested attributes are supported with some clever syntactic sugar. Desugaring our vim dotfile line above, we would get:

home = {
    file = {
        ".vimrc" = {
            source = ./vim_configuration;
        };
    };
};

This shorthand is useful when our attribute sets are sparse, and they can be combined with the “full” attribute set literal along the path. If we had two dotfiles, this:

home.file.".vimrc".source = ./vim_configuration;
home.file.".bash_profile".source = ./bash_configuration;

Is equivalent to this:

home.file = {
    ".vimrc".source = ./vim_configuration;
    ".bash_profile".source = ./bash_configuration;
};

There are other options to pass to dotfiles besides source like onChange and recursive. See the home-manager documentation for more details.

Once you run darwin-rebuild switch --flake ~/.config/nix, run vim ~/.config/nix/flake.nix. You should see line numbers in the left gutter:

  1 {
  2   description = "My system configuration";
  3 
  4   inputs = {
  5     nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
  6     nix-darwin = {
  7         url = "github:LnL7/nix-darwin";
  8         inputs.nixpkgs.follows = "nixpkgs";
  9     };
 10     home-manager = {
 11         url = "github:nix-community/home-manager";
 12         inputs.nixpkgs.follows = "nixpkgs";
 13     };
 14   };

Not the most exciting change, but we’ve configured an application using home-manager! You can exit vim by hitting ESC followed by :q!<enter>.

Configuring zsh with a `home-manager` DSL

You’re probably getting tired of typing out or copy-pasting darwin-rebuild switch --flake ~/.config/nix all the time. Now, let’s add a shell alias! Our fingers will thank us.

We could create a zsh_configuration file and use home.file.".zshrc" to symlink it to the right spot, but I want to demonstrate how to use some of the built-in configuration that home-manager provides. Add the following inside our homeconfig block:

programs.zsh = {
    enable = true;
    shellAliases = {
        switch = "darwin-rebuild switch --flake ~/.config/nix";
    };
};

After running darwin-rebuild switch --flake ~/.config/nix for the last time, you’ll be able to just run switch from now on.

When programs.zsh.enable is true, home-manager will install zsh to your PATH and use the different options under the attribute set to generate a .zshrc for you. If you already have a .zshrc, Nix will ask you to move the file so it’s not overwritten.

Configuring git

Let’s add another stanza for configuring git with some common options while we’re here. Make sure to put in your name and email address:

programs.git = {
    enable = true;
    userName = "$FIRSTNAME $LASTNAME";
    userEmail = "me@example.com";
    ignores = [ ".DS_Store" ];
    extraConfig = {
        init.defaultBranch = "main";
        push.autoSetupRemote = true;
    };
};

Installing packages with `home-manager` vs `nix-darwin`

When should you prefer to use home-manager versus nix-darwin? This is really a matter of opinion, but it’s important to keep in mind that former is cross-platform while latter is locked to macOS.

I generally default to using home-manager for configuring anything that is not macOS-specific. All your home-manager config will continue to work if you ever decide to bring your config to a NixOS system.

Conclusion

You can find the full configuration after this blog post here.

It’s a matter of preference when you use home-manager’s provided DSLs for configuring your apps and when you’d rather just write dotfiles and sync them to the right spot. Personally, I like the DSLs, even if they lock me more into using Nix. Feel free to explore the home-manager configuration options to see what’s possible!

In Part 3, we’ll use home-manager to fully configure VSCode — declaratively managing extensions, themes and user settings, all through Nix. See you then!

Package management on macOS with nix-darwin

Mon, 29 Jan 2024 17:00:00 GMT

I think Nix is really cool. Nix the package manager and functional configuration language is most often associated with NixOS the Linux distro, but nix-darwin makes it almost as easy to declaratively configure macOS as it is to configure NixOS installations. Even if you’ll still relying on Homebrew for package management and never touch nixpkgs, I’d say that Nix with nix-darwin provides the best way to manage packages and system configuration on macOS.

Unfortunately, the resources for getting started and integrating different parts of the Nix ecosystem are not particularly approachable for beginners. When I started out I would often use GitHub’s code search to trawl through other people’s configs and try different snippets until I found what actually worked. Inspired by Arne Bahlo’s Emacs from Scratch series, I wanted to create a guide to help folks get started with Nix on macOS from scratch, step by step.

Throughout this series we’ll create a declarative system configuration with Nix where you can manage anything from your shell aliases to what VSCode extensions you have installed to running daemons with launchd. We’ll build up to this incrementally: by the end of this post, you’ll have Nix installed on your system and be able to declaratively install system-level packages from either Nixpkgs or Homebrew.

Installing Nix

I recommend using the Determinate Systems Nix installer. They have a command-line utility and also recently came out with a graphical installer if you’d prefer that.

Setting up `nix-darwin`

nix-darwin is a Nix library that makes it easy to configure macOS through Nix.

Nix is a programming language, and Nix configurations are programs. All programs need an entrypoint, we’ll be using a flake¹ to provide the entrypoint to our configuration.

Below is our minimal flake that calls nix-darwin. Make sure to replace $USER with your username and $HOSTNAME with your system’s hostname.

You can place this flake in any directory you’d like. For the purposes of this series, we’ll assume that the flake lives at ~/.config/nix/flake.nix.

# ~/.config/nix/flake.nix

{
  description = "My system configuration";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    nix-darwin = {
        url = "github:LnL7/nix-darwin";
        inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  outputs = inputs@{ self, nix-darwin, nixpkgs }:
  let
    configuration = {pkgs, ... }: {

        services.nix-daemon.enable = true;
        # Necessary for using flakes on this system.
        nix.settings.experimental-features = "nix-command flakes";

        system.configurationRevision = self.rev or self.dirtyRev or null;

        # Used for backwards compatibility. please read the changelog
        # before changing: `darwin-rebuild changelog`.
        system.stateVersion = 4;

        # The platform the configuration will be used on.
        # If you're on an Intel system, replace with "x86_64-darwin"
        nixpkgs.hostPlatform = "aarch64-darwin";

        # Declare the user that will be running `nix-darwin`.
        users.users.$USER = {
            name = "$USER";
            home = "/Users/$USER";
        };

        # Create /etc/zshrc that loads the nix-darwin environment.
        programs.zsh.enable = true;

        environment.systemPackages = [ ];
    };
  in
  {
    darwinConfigurations."$HOSTNAME" = nix-darwin.lib.darwinSystem {
      modules = [
         configuration
      ];
    };
  };
}

Activating our `nix-darwin` config

One of the stranger footguns when using Nix flakes is that all files referenced by a flake must be checked into source control. This means that you’ll need to have git installed before we set up our Nix environment². Files just need to be staged to be accessible, not committed, so git add is sufficient until you want to back up your config to a remote repository.

$ cd ~/.config/nix
$ git init
$ git add flake.nix

Once all this is set up, we can run nix-darwin to activate our configuration:

$ nix run nix-darwin --extra-experimental-features nix-command --extra-experimental-features flakes -- switch --flake ~/.config/nix

nix-darwin requires sudo, so you’ll be prompted for your password. Nix may error out if there are files that already exist at paths that it’s trying to replace. Feel free to either rm these or mv them to a backup location, and then re-run the line above.

Once the command succeeds, open up a new terminal window to pick up the new zsh environment and confirm that darwin-rebuild is installed on your path:

$ darwin-rebuild --help
darwin-rebuild [--help] {edit | switch | activate | build | check | changelog}
               [--list-generations] [{--profile-name | -p} name] [--rollback]
               [{--switch-generation | -G} generation] [--verbose...] [-v...]
               [-Q] [{--max-jobs | -j} number] [--cores number] [--dry-run]
               [--keep-going] [-k] [--keep-failed] [-K] [--fallback] [--show-trace]
               [-I path] [--option name value] [--arg name value] [--argstr name value]
               [--flake flake] [--update-input input flake] [--impure] [--recreate-lock-file]
               [--no-update-lock-file] [--refresh] ...

Congrats on setting up nix-darwin! Our configuration is active, but it doesn’t do anything useful yet. Let’s change that.

Installing your first Nix package

It’s time to install our first package from the nixpkgs repository. Update the list of systemPackages declared in flake.nix:

environment.systemPackages = [ pkgs.neofetch pkgs.vim ];

Here, we’re setting the attribute environment.systemPackages to a list. It’s important to point out that lists in Nix are space-separated rather than comma-separated like most other languages.

pkgs refers to nixpkgs, the standard repository for finding packages to be installed with Nix³. Both neofetch and vim are derivations within nixpkgs.

To rebuild our Nix config, we don’t have to use the super long nix run command from above anymore since nix-darwin added the darwin-rebuild command to our PATH. From now on, we just need to run:

$ darwin-rebuild switch --flake ~/.config/nix

Once this runs successfully, we now have a new command in our PATH:

$ neofetch -L
                    c.'
                 ,xNMM.
               .OMMMMo
               lMM"
     .;loddo:.  .olloddol;.
   cKMMMMMMMMMMNWMMMMMMMMMM0:
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.
 XMMMMMMMMMMMMMMMMMMMMMMMX.
;MMMMMMMMMMMMMMMMMMMMMMMM:
:MMMMMMMMMMMMMMMMMMMMMMMM:
.MMMMMMMMMMMMMMMMMMMMMMMMX.
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.
 'XMMMMMMMMMMMMMMMMMMMMMMMMMMk
  'XMMMMMMMMMMMMMMMMMMMMMMMMK.
    kMMMMMMMMMMMMMMMMMMMMMMd
     ;KMMMMMMMWXXWMMMMMMMk.
       "cooc*"    "*coo'"

Now we’re really cooking!

Searching Nixpkgs

Check out nixpkgs search to find other packages you might want to install.

Installing packages from Homebrew

Nixpkgs is expansive, but some programs are still only available from Homebrew. nix-darwin provides what I think is the best interface for Homebrew formulae, casks, and even Mac App Store apps⁴. Let’s add this right under environment.systemPackages:

homebrew = {
    enable = true;
    # onActivation.cleanup = "uninstall";

    taps = [];
    brews = [ "cowsay" ];
    casks = [];
};

Running darwin-rebuild switch --flake ~/.config/nix again will install the Homebrew formula specified in the brews list. Try it out:

$ cowsay "homebrew and nix can be best friends"
 ______________________________________
< homebrew and nix can be best friends >
 --------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

If you’re on a Mac where you’ve been using Homebrew for a while, you can run brew list and brew list --cask to list your installed formulae and casks. Once you’ve added every package you want to carry over to the corresponding lists in your Nix config, uncomment onActivation.cleanup = "uninstall". Your homebrew config in nix-darwin is now declarative: only the packages specified in your flake.nix will be installed, and if you ever remove a package from the lists here it will be uninstalled the next time you reload with darwin-rebuild switch.

Nixpkgs vs Homebrew

While nix-darwin makes it easy to install packages with Homebrew, I’d recommend trying to find the corresponding derivations within Nixpkgs when possible rather than relying solely on Homebrew formulae. As the series goes on we’ll see how native Nix derivations are easier work with in Nix.

Additional configuration

You’ve probably gotten tired of entering your password everytime you reload your config. Luckily, there’s a one-liner to enable Touch ID for sudo, which you can put at the end of your configuration:

# ...
let
    configuration = {pkgs, ... }: {
        # ...
        security.pam.enableSudoTouchIdAuth = true;
    };
in
# ...

All of nix-darwin’s configuration options are worth exploring — we’ll go more in-depth into some of them in future installments of this series, but in case you’re curious, the you can explore all the configuration options and start making your config your own!

If you’d like to see the full file that we’ve built up over the course of this post, you can find it here.

Now that we have a handle on our system configuration, in the next post we’ll set up home-manager and use it manage dotfiles and other program configuration.

Until next time!

Since this series will focus on system configuration rather than development environments, we’ll only be creating this one flake and won’t cover flakes in-depth. If want to read more about flakes, feel free to check out Julia Evans’s blog post on flakes and the Zero to Nix wiki page. ↩
If you’re on a brand new machine, the first time you run git in the terminal you should be prompted to install Xcode Command Line Tools, which includes git. ↩
While it’s not necessary to fully understand this right now, the configuration value that we’re defining is a Nix module. The nix-darwin.lib.darwinSystem function that’s called at the bottom of the file is responsible for passing nixpkgs through to configuration with the name pkgs. We’ll dive deeper into Nix modules in a later post. ↩
While we won’t be installing any App Store apps in this post, you can check out the description in the nix-darwin documentation for more information. ↩

What I Like About Nix

Tue, 12 Dec 2023 12:00:00 GMT

I got a new computer recently and decided to take the plunge setting it up with Nix¹. I wrote up a snippet on how I set everything up and you can find my whole configuration on GitHub. I’ve only scratched the surface of what Nix can do – In this post I wanted to focus on my subjective impression of Nix so far and why I feel I’ve been enjoying it.

Long-term, shared memory

No need to worry about remembering what switch you flipped in settings six months ago after reading an OSXDaily.com post, or what command you ran in a shell at 3am the morning you unboxed your new computer. Any change you’ve made to, for example, hide the dock by default or allow for repeating keys in VSCode and other editors is there for later review.

On macOS these system preferences are normally either configured in the point-and-click System Preferences interface or specified in loosely documented defaults write invocations that you can find all over the internet. nix-darwin is a Nix tool I’m using that lets you configure virtually all of these settings from a Nix file.

Making your own configuration declarative is good enough, but on top of that people have built up modules and abstractions around more complex configuration like enabling sudo to use TouchID or running daemons through launchd that you can enable with a single line of code.

A complete picture of your environment

All of my dotfiles, editor themes and extensions, and command-line programs are defined in a single repository. It makes it a lot easier to see how it all interacts with each other and hopefully avoid situations where different dependencies conflict. No one wants to end up here, but it happens all too often, at least to me:

Synchronization between machines

I have a desktop and a laptop. They’re now running virtually the same Nix configuration, out of the same repository, with a few tweaks in host-specific overrides. I can work out the kinks in getting Emacs to play nicely with the Yabai window manager on my desktop and have it all work on my laptop in a matter of minutes once I pull down the updated configuration repository and reload my Nix environment.

I should mention these two machines are running different versions of macOS, and virtually everything has worked flawlessly across both.

Reducing manual toil

I’ve used Karabiner Elements for years to remap some keys on my keyboard. More recently I’ve moved my configuration over to Goku, a DSL for Karabiner on top of the JSON configuration format. Every time I wanted to change my configuration I would need to remember to run the goku command. Now that command is run automatically by home-manager whenever my karabiner.edn file changes. When I change my skhd or yabai configuration, Nix will automatically restart those services so they pick up their new settings.

Standardizing the refreshing of all configuration to a single command (darwin-rebuild switch) and running everything only when it’s needed has been a small change that’s significantly reduced the cognitive load of having a lot of subsystems that may require new configuration, occasional restarts, or other babysitting from time-to-time.

Conclusion

If the entirety of Nix was a DSL for a package manager and system configuration system I’d be satisfied. But there’s so much more to Nix, from development environments to production artifacts. I’m excited to dive more into those corners of the ecosystem going forward, and hopefully write about it too both here on the blog and over on the TIL page!

Nix isn’t the easiest thing to introduce since it can refer to a few different distinct components or the whole ecosystem. In case you haven’t heard of Nix before or just want to learn more, I find the conceptual documentation on Zero to Nix to be more approachable than the official docs. ↩

[TIL] nix develop -c $SHELL: Run zsh with 'nix develop'

Sat, 18 Nov 2023 17:00:00 GMT

I built my development flake for this website with the FlakeHub CLI. That flake uses pkgs.mkShell to load in binary dependencies to the development environment. I run zsh defined and configured by nix-darwin, so it was pretty jarring getting thrown into a plain bash shell when running nix develop. Luckily the fix is simple. Rather than just running nix develop, run:

nix develop -c $SHELL

and your normal shell will be run instead of the default bash shell.

(via this discussion)

[TIL] nix-darwin and home-manager: Setting up Nix on macOS

Fri, 17 Nov 2023 17:00:00 GMT

I recently got a new computer and am spending some time actually digging into Nix for managing dependencies. My main issue the last time I tried using Nix was how tough it was for me to wade through the documentation, so I was determined to try a different path this time. I started off with the very opinionated Zero to Nix tutorial series, which I thought was a great introduction.

After learning about Nix and Flakes, and getting my website building locally with a nix-powered development environment courtesy of the guidance in Part 6, the next step was to see what else I could do. I’d heard of two different tools, home-manager and nix-darwin. I started off setting up home-manager, but I was confused on how I should be installing graphical applications like VS Code, and system utilities like Karabiner Elements. It turned out nix-darwin was the answer there.

To paraphrase a Reddit thread I found, nix-darwin manages system-level programs and configuration, while home-manager manages user configuration. You can actually get nix-darwin to call into home-manager as part of its own configuration to use both at the same time. This is described in the home-manager reference docs, but like many Nix resources, it’s not very simple for a beginner to figure out how to actually put this snippet to use in their own configuration.

Before following these instructions, you should follow the Zero to Nix guide or install Nix on your system in some other way.

I’m going to lay out the basic steps that I followed below while sharing each configuration file.

Create a nix-darwin flake file. From the README:

mkdir -p ~/.config/nix
cd ~/.config/nix
nix flake init -t nix-darwin
sed -i '' "s/simple/$(scutil --get LocalHostName)/" flake.nix

The default flake puts the configuration inline to the flake.nix. I decided to copy/paste that configuration out into a separate file called darwin.nix:

# darwin.nix

{ pkgs, ... }:

{
    # List packages installed in system profile. To search by name, run:
    # $ nix-env -qaP | grep wget
    environment.systemPackages =
    [ pkgs.vim ];

    # Auto upgrade nix package and the daemon service.
    services.nix-daemon.enable = true;
    services.karabiner-elements.enable = true;
    # nix.package = pkgs.nix;

    # Necessary for using flakes on this system.
    nix.settings.experimental-features = "nix-command flakes";

    # Create /etc/zshrc that loads the nix-darwin environment.
    programs.zsh.enable = true;  # default shell on catalina
    # programs.fish.enable = true;

    # Used for backwards compatibility, please read the changelog before changing.
    # $ darwin-rebuild changelog
    system.stateVersion = 4;

    # The platform the configuration will be used on.
    nixpkgs.hostPlatform = "aarch64-darwin";

    users.users.davish = {
        name = "davish";
        home = "/Users/davish";
    };
}

You should remember to change your username and home directory to match.

Modify flake.nix to work with home-manager. The example code can be found in the home-manager docs. In case that link breaks, here it is:

# flake.nix

{
  description = "Darwin configuration";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    darwin.url = "github:lnl7/nix-darwin";
    darwin.inputs.nixpkgs.follows = "nixpkgs";
    home-manager.url = "github:nix-community/home-manager";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = inputs@{ nixpkgs, home-manager, darwin, ... }: {
    darwinConfigurations = {
      hostname = darwin.lib.darwinSystem {
        system = "aarch64-darwin";
        modules = [
          ./configuration.nix
          home-manager.darwinModules.home-manager
          {
            home-manager.useGlobalPkgs = true;
            home-manager.useUserPackages = true;
            home-manager.users.jdoe = import ./home.nix;

            # Optionally, use home-manager.extraSpecialArgs to pass
            # arguments to home.nix
          }
        ];
      };
    };
  };
}

Create a home.nix file. Here is the one that was generated when I tried to set up home-manager on its own. I’ll replicate it here because the comments were useful as a tutorial on its own:

# home.nix

{ config, pkgs, ... }:

{
  # Home Manager needs a bit of information about you and the paths it should
  # manage.

  # This value determines the Home Manager release that your configuration is
  # compatible with. This helps avoid breakage when a new Home Manager release
  # introduces backwards incompatible changes.
  #
  # You should not change this value, even if you update Home Manager. If you do
  # want to update the value, then make sure to first check the Home Manager
  # release notes.
  home.stateVersion = "23.05"; # Please read the comment before changing.

  # The home.packages option allows you to install Nix packages into your
  # environment.
  home.packages = [
    # # Adds the 'hello' command to your environment. It prints a friendly
    # # "Hello, world!" when run.
    # pkgs.hello

    # # It is sometimes useful to fine-tune packages, for example, by applying
    # # overrides. You can do that directly here, just don't forget the
    # # parentheses. Maybe you want to install Nerd Fonts with a limited number of
    # # fonts?
    # (pkgs.nerdfonts.override { fonts = [ "FantasqueSansMono" ]; })

    # # You can also create simple shell scripts directly inside your
    # # configuration. For example, this adds a command 'my-hello' to your
    # # environment:
    # (pkgs.writeShellScriptBin "my-hello" ''
    #   echo "Hello, ${config.home.username}!"
    # '')
  ];

  # Home Manager is pretty good at managing dotfiles. The primary way to manage
  # plain files is through 'home.file'.
  home.file = {
    # # Building this configuration will create a copy of 'dotfiles/screenrc' in
    # # the Nix store. Activating the configuration will then make '~/.screenrc' a
    # # symlink to the Nix store copy.
    # ".screenrc".source = dotfiles/screenrc;

    # # You can also set the file content immediately.
    # ".gradle/gradle.properties".text = ''
    #   org.gradle.console=verbose
    #   org.gradle.daemon.idletimeout=3600000
    # '';
  };

  # You can also manage environment variables but you will have to manually
  # source
  #
  #  ~/.nix-profile/etc/profile.d/hm-session-vars.sh
  #
  # or
  #
  #  /etc/profiles/per-user/davish/etc/profile.d/hm-session-vars.sh
  #
  # if you don't want to manage your shell through Home Manager.
  home.sessionVariables = {
    # EDITOR = "emacs";
  };

  # Let Home Manager install and manage itself.
  programs.home-manager.enable = true;
}

And there you have it! You can install nix-darwin with:

nix run nix-darwin --experimental-feature nix-command --experimental-feature flakes -- switch --flake ~/.config/nix

You can now run darwin-rebuild switch --flake ~/.config/nix to rebuild all your configuration whenever you make a change. Another useful link is the list of nix-darwin configuration options. I believe this is the best place to go for information on what can be configured with nix-darwin.

Honing Your Craft

Sun, 21 May 2023 09:53:48 GMT

Pablo Picasso was one of the most famous modern artists of the 20th century. While many of his paintings border on the abstract, he sharpened his artisitc skill on more realistic and traditional styles. Novelty and creativity are important parts of great art, but just as necessary to making anything great is the technical ability to take an idea and represent it in the real world — even abstract expressionists could paint unbelieveably realistic landscapes if they felt the need.

I’m certainly no artist, but the lesson here applies more broadly: practicing technical skills on their own is an important part of the creative process.

Science and Charity by Pablo Picasso, 1897 (age 16)

Guernica by Pablo Picasso, 1937 (age 56).

I’ve got more than a few ideas for open-source projects and apps that I’d like to implement. I also have more than a few languages, frameworks and technologies that I’m itching to learn how to use. For a long time I’ve taken to heart the advice that the best way to learn a new programming language or framework is to build a larger-scale project with it that you’re excited about. What I’m realizing, though, is that innovation tokens apply just as much to side projects where you’re looking to grow and learn as they do for production applications. The more mental energy spent designing a UI and triaging bug reports, the less energy there is for diving deep into a new language, investigating performance bottlenecks and learning other tools.

My thoughts on this really solidified as I started a read-through of Crafting Interpreters with Kotlin. Even though the book fully spells out the code and structure of the interpreter, just translating the code to a different language has helped me learn a ton about language implementation in general and Kotlin in particular along the way.

This isn’t to say that you shouldn’t push forward on your own ideas. But in order to put the best work out there, it’s important to also dedicate energy to honing technical skill on its own with some more well-defined projects that still require a lot of depth of implementation. Challenging Projects Every Programmer Should Try and its sequel are good lists to start off with.

[TIL] freeport: Kill a process that's hogging a port on MacOS

Sat, 22 Apr 2023 09:54:46 GMT

Every once in a while, a development server on my laptop gets stuck open when the terminal that spawned it has already closed. I normally have to take 5 minutes or so to search through StackOverflow to find out how to kill the process that’s hogging up that port, so I finally made a script for it:

#!/bin/sh

if [ $# -eq 0 ]; then
    echo "usage: $0 <port number>"
    exit 1
fi

lsof -t -i tcp:"$1" | xargs kill -9

You can also just copy/paste the last line and replace $1 with the port number in question, if bash scripts aren’t your thing.

[TIL] Json: Typescript type for the JSON spec

Sun, 09 Apr 2023 09:45:16 GMT

Some JavaScript types like dates are not representable directly with JSON and require extra care to parse back out to a JavaScript object after they’re serialized. If you want to make sure that you are only stringify-ing JSON-safe types, you can use this type to represent objects compatible with the JSON spec:

type Json =
    | { [key: string]: Json }
    | Json[]
    | string
    | number
    | true
    | false
    | null;

function safeStringify(obj: Json) {
    return JSON.stringify(obj);
}

If you have some type that is not JSON-compatible that you want to serialize, this type will help you write a serialize() function where the Typescript compiler confirms that the function is producing a safe-to-stringify value:

type MyType = {
    date: Date,
    title: string,	
}

function serialize(obj: MyType): Json {
    return {title: obj.title, date: obj.date.toISOString()};
}

[TIL] typesEqual: Ask the Typescript compiler how two types are different

Thu, 06 Apr 2023 17:00:00 GMT

type FirstType = {};
type SecondType = {};
const typesEqual = <A, B extends A, C extends B>() => {};

// This will fail to compile if FirstType and SecondType are not equivalent.
typesEqual<FirstType, SecondType, FirstType>();

Found in this StackOverflow answer by Almaju.

I was replacing a hand-written validator with Zod and wanted to make sure my new type inferred with Zod was equivalent to my old hand-rolled type. The assertion on the last line of this snippet will fail to compile if the two types passed in are not equal. The error will explain exactly which properties are missing and/or incompatible, so you can adjust your new type as needed to match.

Asking the Right Questions to ChatGPT

Thu, 30 Mar 2023 14:26:43 GMT

This is my generative AI take. I’m sure there are many like it, but this one is mine.

When it comes to social trends I’m often on the far side of Moore’s Chasm. I signed up for Snapchat a year after all of my friends, and joined Instagram two years after everyone else. So it was only appropriate that it took weeks after the internet was set on fire for me to see the value that applications of Large Language Models like ChatGPT and GitHub Copilot bring to programming. But now that I’ve given it a real shot, it’s clear to me that if you have enough knowledge to ask specific tactical questions about a well-defined technology, framework or library, ChatGPT can be a huge force multiplier.

When ChatGPT dropped a few months ago, I asked it a few questions right away, like “can you compare OCaml and Haskell?”, and “What is COVID-19?”. I wasn’t super impressed with the depth of the answers, and hallucination was pretty obvious, even if the grammar and vocabulary was suprisingly coherent and humanlike.

This week I decided to give ChatGPT another shot. I’ve started working on an API built with Django REST Framework this week for the first time in about two years. I still had a notion of what I wanted to do even though my working knowledge of how to build stuff with the framework was rusty. While the Django and DRF documentation are both thorough, it’s often frustrating to search through them for specific examples when you know what you’re looking for.

ChatGPT was able to provide me with surprisingly good answers to my intermediate-to-advanced Django questions and prompts, like:

Write a model definition with a created_at field that is set to the current time when the model is created.
I have two django models, X and Y. Y has a ForeignKey relationship to X. When creating an X in the django admin I want to be able to create Ys in-line. Is this possible? (Yes, with an example ModelAdmin implementation)
When should I use a through model in Django?
I have a django model Y with a decimal field A and foreign keys to a User model and a model X. Is it possible to get the sum of A for all Ys in an X and the sum for all Ys linked to a User in an X in one query?
I want to create a page where users can create a new X with a any number of Ys inline. How can I do this with django generic class-based views?

When searching for an answer to a technical question on Google it’s often best to try and abstract your goal into a general question to see if there’s any answers out there on the internet even tangentially related. With ChatGPT, I got the best answers when I gave it specifics about my use case and the models I was using. It was able to plug in my model names and properties that I gave it without any problem. It was also a lot easier to phrase my questions naturally without having to generalize them out like I would with Google.

I see two main components that led ChatGPT to being a great copilot on this Django project. Django and DRF are old, stable, and popular. I’m sure that tons of blog posts, StackOverflow answers, and books on the framework made it in to GPT-3’s training corpus. Just as importantly, my own knowledge enabled me to ask precise questions that were within the bounds of possibility for the framework. I had worked with Django for years and knew the general vocabulary of the framework and what it was capable of. I stayed away from asking for ChatGPT’s opinion or asking high-level design or architecture questions, and I suspect those kinds of questions are where the model is more likely to hallucinate.

For now, at least, LLMs are most useful when the prompter has sufficient domain knowledge to form precise questions. I don’t think the robots are here for all our jobs just yet. But this is a fast-moving field: we’ll see what happens in the next 5 years – or even 5 months.

[TIL] ListSerializerMixin: Specify a different viewset serializer for lists

Tue, 28 Mar 2023 17:00:00 GMT

I often want to include related nested objects in my Django REST Framework ModelViewSets for CRUD operations, but don’t want those related objects cluttering up the list view. To accomplish this, you can return a serializer from get_serializer_class() that doesn’t include the nested relation when the ViewSet action is list. I ended up doing this on a bunch of viewsets, so I factored the logic out into a separate mixin:

from typing import Union
from rest_framework import viewsets

class ListSerializerMixin(object):  
    list_serializer_class = None  
  
    def get_serializer_class(  
        self: Union[viewsets.GenericViewSet, "ListSerializerMixin"]  
    ):  
        if self.list_serializer_class and self.action == "list":  
            return self.list_serializer_class  
        return super(viewsets.GenericViewSet, self).get_serializer_class()

On your viewset, you can inherit from ListSerializerMixin¹ then add the list_serializer_class attribute next to serializer_class instead of overriding the method directly:

class ObjectViewSet(ListSerializerMixin, viewsets.ModelViewSet):  
    serializer_class = ObjectDetailSerializer  
    list_serializer_class = ObjectListSerializer  
    permission_classes = [IsAuthenticated]  
  
    def get_queryset(self):  
        return Object.objects.all().prefetch_related("related_items")

This leads to serializers that are easier to read at-a-glance, and slightly easier to write.

Make sure to add the mixin before the ViewSet in your superclass list. ↩

[TIL] getUrl(): URL routing for content collections

Sun, 26 Mar 2023 17:00:00 GMT

Astro added content collections in v2.0, and while it’s overall a big improvement for pages like blog posts where there’s a uniform schema, it makes generating links to your content just a bit harder than file-based routing. I added a helper function and some Typescript types to content.ts to make it easy to map content collections to URLs in a typesafe way that your editor can help auto-complete.

export const collections = {
  // Your collections here
  // see https://docs.astro.build/en/guides/content-collections/#defining-multiple-collections
  coll1: defineCollection(),
  coll2: defineCollection(),
  coll3: defineCollection(),
};

type UrlMap = Partial<{
  [key in keyof typeof collections]: string;
}>;

const urls: UrlMap = {
  coll1: "/path/to-coll1/",
  coll2: "/path-to/coll2/",
  coll3: "/path/to/coll3/",
};

export const getUrl = (collName: keyof typeof urls, slug: string) =>
  urls[collName] + slug;

In your Astro components, you can then call getUrl("collName", slug) to get the URL.

While it’s just a bit of string concatenation, the added type safety can make a big difference in a larger project. Even on this website, I’ve found the autocomplete for the collection name to be very helpful.

[TIL] form_valid() overload: Handle uploads with class-based generic views

Sat, 25 Mar 2023 17:00:00 GMT

The Django documentation describes how to bind uploaded files when using function-based views. There’s no easy equivalent I could find on how to do the same for generic, class based views like CreateView. The right way to do it is to override the form_valid() method on the view:

def form_valid(self, form):
    file = self.request.FILES["file"]
    self.object = form.save(commit=False)
    self.object.file = file
    self.object.save()
    return HttpResponseRedirect(self.get_success_url())

Configuring VSCode as a Keyboard-Centric IDE

Tue, 24 Jan 2023 17:00:00 GMT

One of my first posts on this blog in 2020 was about my experience switching from VSCode to Emacs for all of my text/code editing work. I still agree with all of the benefits to Emacs I wrote about back then, and my posts on Obsidian show my respect for org-mode as a format and the ecosystem around it – especially org-agenda. However, as you might have noticed when the file extension on my posts switched from .org back to .md, I’ve moved on from Emacs for now. But I’m happy with how I’ve moved my good habits back to VSCode. Spending time on config gave me more confidence in my ability to configure software I use to my liking, and make software work for me rather than make me work for the software.

At work, I’m always editing code while SSH’d into a remote machine. TRAMP with lsp-mode on a very large C++ codebase just had too much lag for me. Even accessing Emacs compiled with native-comp running on my remote machine had way too much input delay when waiting on responses from clangd or ccls, two C++ language servers. Now, it’s more than possible I was holding it wrong – I do have colleagues that work all day in Emacs. But it seems like my bad UX experience was due to the limitations of Emacs’s concurrency support – calls to language server were blocking other interactions with the UI. On the other side of the hedge, VSCode really just works. I can still type and otherwise interact with the editor while waiting for clangd to process a request. The remote SSH extension is the state of the art. The clangd integration was plug-and-play. And I’ve never experienced the lag that I was feeling every day on Emacs.

While I love the smoothness of the VSCode UI, I’m not a fan of how easy it is fall back on the mouse while navigating around the editor. My favorite part of the Doom Emacs distribution was the keybindings: it made me feel like I was playing chords on the piano when splitting panes and navigating around. While I’d configured Emacs to look and feel like VSCode in my original Emacs post, I ultimately got rid of the project explorer and visual tabs to embrace more Emacs-y buffer and project management. I found myself reaching for the mouse way more often when I moved back to VSCode, putting myself at risk for RSI and generally interrupting my flow. So I set out to make my VSCode configuration as close to my Doom Emacs setup as possible. There were only a handful of changes I needed to make:

Installed VSCode’s Vim plugin. I’ve used vim keybindings all over the place for almost four years at this point.
Installed VSpaceCode + WhichKey. Together, these plugins replicate the chords/leader bindings that Doom borrowed from Spacemacs really well. I’ve remapped a few bindings that were different in Doom and added a few more as well.
Disabled Tabs. Tabs are a visual abstraction that are hard to use effectively without reaching for the mouse. I got used to Emacs’s buffer management, where I can fuzzy-search for recently opened files in a reverse-chronological list. VSCode’s analog to buffers are called Editors and are normally mapped on to Tabs; after disabling Tabs I can still navigate open Editors with the command “View: Show Editors in Active Group By Most Recently Used”, which I’ve mapped to <leader> , with VSpaceCode.
- I removed tabs by adding "workbench.editor.showTabs": false to my settings.json file.
- The command ID for “Show Editors in Active Group” is workbench.action.showEditorsInActiveGroup.
Disabled the Activity Bar. I can pull up sidebar Views like File Explorer and Find in Files with a shortcut, and dismiss them with cmd-B.
- Add "workbench.activityBar.visible": false to your settings.json.

The fact that VSCode can do all this is a testament to its extensability and the power of a more limited but still deep plugin API. A settings.json file is no match for init.el when it comes to expressiveness, but I’ve still been able to mold VSCode into an editor that I feel at home using.

I’m sure I’ll keep playing around with Emacs going forward, but the all-consuming nature of the Emacs philosophy – the idea that everything from email to web browsing to journaling should happen inside Emacs – doesn’t resonate with me in practice. I can use VSCode for most of my development, Obsidian for editing Markdown, and even PyCharm for Python/Django development (check out intellimacs for JetBrains IDEs if you’re interested in my setup there). With just a few hours of set-up, I can use the right tool for the job and still feel at home regardless.

Improving My Writing Style One Python Script at a Time

Tue, 20 Dec 2022 17:00:00 GMT

One of my pet peeves about my natural writing style is how I lean into complex sentences divided by commas. Left unchecked, my prose starts looking like it might be ChatGPT’s attempt at writing a blog post in the style of s-expressions. I thought it would be neat to try and write some code to help me proofread for this specific issue and improve my posts.

So much of my Python experience is from writing apps with Django that I forgot how quick and easy it is to whip up a small script that does some text processing with nothing but the standard library. As much as I appreciate static types and exhaustiveness checking in larger programs, being able to ignore edge cases that I know don’t appear in the specific input I’m concerned with is a relief for scripts like this.

The first script I wrote looks for sequences of multiple sentences that have too many commas in them. If there are more than three commas in two adjacent sentences, I need to reword something. Here’s its output on one of my first blogposts, Switching to Emacs:

***
One thing I started to worry about recently has been my reliance on a closed-source, and, once I'm out of college, pretty expensive editor.
Sure, many employers pay for licenses, but it would be awesome if I could configure an editor that worked well for me and was also completely open-source.
- 5 commas
***
Sure, many employers pay for licenses, but it would be awesome if I could configure an editor that worked well for me and was also completely open-source.
A few months ago, I switched my Python development to VSCode, where I had already been doing frontend JS/TypeScript dev for a few years.
- 4 commas
***
After learning vim keybindings this summer, I would've definitely liked to also be using the keyboard more often.
Primarily, though, I'd heard really cool things about [org-mode](https://orgmode.org/), and wanted to try it out.
- 4 commas
***
Primarily, though, I'd heard really cool things about [org-mode](https://orgmode.org/), and wanted to try it out.
Full support of org-mode is only present in emacs, which in the midst of quarantine would be a bit of excitement to check out.
- 4 commas
***
The Python language server was a bit finnicky, but after setting it up, along with JavaScript, I'm already at a point where I can be productive.
I enabled a few more Doom modules, like a terminal emulator (`term`) and a file explorer (`treemacs`).
- 4 commas

Since these windows are all processed in order, I can see without any extra processing that there’s actually two sequences of three sentences with 7 and 6 commas respectively.

As you might expect, I ran this post through the script before publishing. It had no notes! I’m improving already.

Mocking Obsidian's Markdown Parser

Tue, 20 Dec 2022 17:00:00 GMT

I’ve been working on a plugin for Obsidian called Obsidian Full Calendar on-and-off for the past 10 months or so. For most of that time the plugin has had no unit tests, and I finally got around to adding some test coverage during a big refactor.

Tests are easiest when code doesn’t have side effects since filesystems and network calls often aren’t available in the environment the tests are running in. Obsidian’s core code is closed-source and can only be run from inside the Electron app, so plugin developers who want test coverage aren’t left with many options but to test their plugins completely outside of Obsidian. Unfortunately for me, my plugin is mostly a pile of glue sitting between FullCalendar as the view layer and the Obsidian filesystem APIs for persistence. I would need to mock out the relevant APIs from Obsidian if I wanted to have any meaningful test coverage of my own code.

There isn’t yet any comprehensive mock Obsidian API for use in a testing environment that I could reach for, so I went ahead writing my own!

Taking Stock

There are two Obsidian APIs that I make heavy use of:

Vault: The filesystem API. This API allows you to create, read, update and delete files. The core type that is passed around these APIs is TAbstractFile, which can either be a TFolder or a TFile.
MetadataCache: To support everything from file search to graph view to the backlink count, Obsidian parses every Markdown file in a Vault and saves relevant information about its structure in the MetadataCache. I’m most interested in:
- YAML frontmatter, parsed into a dictionary
- Position and task status of lists and list items
- Position and text of Markdown headings

First Thoughts

Mocking out relevant parts of the Vault API isn’t that bad, but keeping a mock MetadataCache that was in sync with file contents felt like it was going to be a big pain. Every item in a cache entry records its location in its file, consisting of line/column information as well as the character offset. Writing cache entries by hand would require recording the position of every single heading and list item within the file – a single character change would have a cascading effect on the whole cache entry.

My first thought to avoid this was just to write a simplified metadata parser for testing purposes. My mock API would just take in file contents and would parse out the metadata itself. I wasn’t particularly in the mood to write a Markdown parser just for unit tests. A library like mdast might have made this easier, but that can be a rabbit hole for another day.

The `FileBuilder`

The solution I landed on is to build up a file’s contents line-by-line while also constructing the metadata for each line at the same time. It’s sort of the inverse of parsing – the file is constructed in a structured format that’s easy to write, and then serialized to a string and to the metadata format at the same time! The resulting builder DSL is super readable for the verbosity of the output it produces.

The following expression:

const [contents, metadata] = new FileBuilder()
  .frontmatter({ hello: "world" })
  .heading(2, "Journal")
  .text("this is a journal entry!")
  .text("and a second line!")
  .heading(2, "My list")
  .list(
    new ListBuilder()
      .item("first list item")
      .item("second list item")
      .item("to-do", false)
      .item("done", true)
      .item("nested list")
      .list(
        new ListBuilder().item("nested list item").item("another nested item")
      )
  )
  .done();

Renders this Markdown:

---
hello: world
---
## Journal
this is a journal entry!
and a second line!
## My list
- first list item
- second list item
- [ ] to-do
- [x] done
- nested list
	- nested list item
	- another nested item

And constructs the following metadata, which I'll hide in a dropdown.

{
  "frontmatter": {
    "hello": "world",
    "position": {
      "start": { "line": 0, "col": 0, "offset": 0 },
      "end": { "line": 2, "col": 3, "offset": 20 }
    }
  },
  "headings": [
    {
      "position": {
        "start": { "line": 3, "col": 0, "offset": 21 },
        "end": { "line": 3, "col": 10, "offset": 31 }
      },
      "heading": "Journal",
      "level": 2
    },
    {
      "position": {
        "start": { "line": 6, "col": 0, "offset": 76 },
        "end": { "line": 6, "col": 10, "offset": 86 }
      },
      "heading": "My list",
      "level": 2
    }
  ],
  "listItems": [
    {
      "position": {
        "start": { "line": 7, "col": 0, "offset": 87 },
        "end": { "line": 7, "col": 17, "offset": 104 }
      },
      "parent": -7
    },
    {
      "position": {
        "start": { "line": 8, "col": 0, "offset": 105 },
        "end": { "line": 8, "col": 18, "offset": 123 }
      },
      "parent": -7
    },
    {
      "position": {
        "start": { "line": 9, "col": 0, "offset": 124 },
        "end": { "line": 9, "col": 11, "offset": 135 }
      },
      "parent": -7,
      "task": " "
    },
    {
      "position": {
        "start": { "line": 10, "col": 0, "offset": 136 },
        "end": { "line": 10, "col": 10, "offset": 146 }
      },
      "parent": -7,
      "task": "x"
    },
    {
      "position": {
        "start": { "line": 11, "col": 0, "offset": 147 },
        "end": { "line": 11, "col": 13, "offset": 160 }
      },
      "parent": -7
    },
    {
      "position": {
        "start": { "line": 12, "col": 2, "offset": 163 },
        "end": { "line": 12, "col": 22, "offset": 183 }
      },
      "parent": 11
    },
    {
      "position": {
        "start": { "line": 13, "col": 2, "offset": 186 },
        "end": { "line": 13, "col": 25, "offset": 209 }
      },
      "parent": 11
    }
  ]
}

Testing the test code

I’m never sure of what the best practice is for testing code that you’ve written for the purpose of testing, but this case I’d constructed a complex enough API and written enough code that I decided to see how deep this rabbit hole goes.

The main thing I wanted to avoid was manually writing out the output that I was expecting. Constructing metadata by hand was what I wanted to avoid in the first place, and I didn’t want to track down small typos in my transcription of the Markdown documents. I came up with a testing strategy that minimized the amount of typing I had to do for each of the output formats, allowing me to add a lot of test cases with minimal effort.

Text output

Jest comes with built-in snapshot testing functionality that allows test authors to simply visually validate the test output after the test is run the first time. On subsequent runs, the result is compared to the accepted output to make sure there isn’t any regression.

We get the option of using separate snapshot files or keeping the snapshots inline. I opted for inline snapshots since the output isn’t particularly long for any given test. Here’s an example of what the inline snapshot assertion looks like after it’s accepted:

const [contents, metadata] = new FileBuilder()
  .heading(2, "First heading")
  .heading(2, "Second heading")
  .heading(2, "Third heading")
  .done();
expect(contents).toMatchInlineSnapshot(`
  "## First heading
  ## Second heading
  ## Third heading
  "
`);

Parsed metadata

The goal of this whole builder is to make sure that the metadata produced is identical to the metadata that Obsidian constructs from parsing a Markdown file. To verify that, I wanted to use Obsidian’s actual parser output as the source-of-truth for these tests.

I copy/pasted the inline snapshot result for every test into a blank Obsidian note and added a small Obsidian command to my local copy of my plugin to grab the current note’s metadata and copy the full assertion to the clipboard:

this.addCommand({
  id: "full-calendar-FCTEST",
  name: "FCTEST",
  callback: async () => {
    const leaf = this.app.workspace.getMostRecentLeaf();
    if (leaf.view instanceof MarkdownView) {
      const file = leaf.view.file;
      const cache = {
        ...this.app.metadataCache.getFileCache(file),
      };
      delete cache["sections"];
      if (cache) {
        console.log(cache);
        await navigator.clipboard.writeText(
          `assert.deepStrictEqual(metadata, ${JSON.stringify(cache)})`
        );
      }
    }
  },
});

Adding an assertion for metadata to each test only took a copy-paste and five or so keypresses!

The code for this is still on an in-progress branch in Obsidian Full Calendar, but I think it would be generally usable for other plugins, so I might split it out into a separate package in the future.

Deploy a Django app with fly.io

Tue, 04 Oct 2022 17:00:00 GMT

I started a new project a few days ago built with Django. Heroku is killing their free tier, and I’ve read that Fly.io is the cool new thing, so I decided to try it out.

Overall, Fly has great documentation. Their Language and Framework Guides are pretty comprehensive, but in a list that includes popular frameworks like Rails and Laravel alongside less boring options like RedwoodJS and Remix, I couldn’t help but notice Django’s conspicuous absence.

I was able to get everything working great with Django and Fly.io after some trial and error, so I wanted to write up my process to make it easier for people going forward.

If you want to get everything set up when starting a new project, use the startproject template I made which you can find at github.com/davish/django-flyio-template. Read on if you’re interested in a more in-depth look!

As opionated as Django is, there’s still some choices you need to make when starting a project. Here’s my stack that I’ll use throughout this post:

Poetry for package management.
Daphne for the production webserver.
Docker to describe the deployment.
- Fly also has support for heroku-style buildpacks and Procfiles.
Postgres for the production database, since Fly has built-in support for Postgres.
- Local development still uses SQLite.

0. Initialize the Poetry project and Django App

If you’re starting an app from scratch, then run these commands from inside an empty directory where you want to store your project:

$ poetry init
$ poetry add django dj-database-url daphne psycopg2-binary
$ poetry run django-admin startproject <project> .

1. Add the Dockerfile

Create a file called Dockerfile in the project’s base directory:

FROM python:3

ENV PYTHONUNBUFFERED 1

WORKDIR /app

COPY poetry.lock pyproject.toml /app/

RUN pip3 install poetry
RUN poetry install --no-root

COPY . .

ENV DJANGO_SETTINGS_MODULE "<project>.settings"
ENV DJANGO_SECRET_KEY "this is a secret key for building purposes"

RUN poetry run python manage.py collectstatic --noinput

CMD poetry run daphne -b 0.0.0.0 -p 8080 <project>.asgi:application

It’s also important to also add a .dockerignore file, especially if you’re storing your site in a Git repository.

.git/
__pycache__/

fly.toml
*.sqlite3

# Files that store secrets
.env

# macOS file
.DS_Store

# Editor-specific configuration
.idea/
.vscode

2. Configure the Fly.io app

Run flyctl launch --no-deploy to create a project and get a name and URL. --no-deploy prevents flyctl from deploying our app right out of the gate. We have to make some more changes before we can do that.
The last command will have generated a fly.toml file. Open it up and add these lines, starting with the existing empty [env]block:

[env]
  DJANGO_SETTINGS_MODULE = "<project>.settings"

[deploy]
  release_command = "poetry run python manage.py migrate"

[[statics]]
  guest_path = "/app/static"
  url_prefix = "/static"

3. Link a Postgres Database

$ flyctl postgres create
# Make a note of the db app name you choose.
$ flyctl postgres attach <db-name>
# db-name should be what you selected in the previous step.

The postgres attach subcommand creates a properly permissioned database user and generates a connection string that’s stored as a secret named DATABASE_URL.

4. Configure Django settings

Back in our Django project, there’s a bit of configuration that we’ll need to do in the project’s settings.py file.

The `DATABASES` dictionary

import os
import dj_database_url
# ...
DATABASES = {
    "default": dj_database_url.config(
        default="sqlite:///" + os.path.join(BASE_DIR, "db.sqlite3")
    )
}

dj-database-url will pick up the connection string that flyctl postgres attach set in our environment and properly configure our database connection for us. To avoid dealing with Postgres locally, however, we’ll default to a SQLite database when we don’t have a database URL defined.

Static files

Django collects static files through the collectstatic command that we run in our Docker build process. We both need to tell Django where to store static files on the filesystem (STATIC_ROOT) and at what relative path these static files will be served from (STATIC_URL). These need to match The guest_path and url_prefix from our fly.toml file, respectively. guest_path is /app/static because we placed our app to the /app directory in our Dockerfile.

STATIC_URL = "static/"
STATIC_ROOT = "static"

With Heroku and some other hosting providers, you’d have to set up and configure WhiteNoise and serve static files through Django. I think it’s great that Fly makes it so easy to bypass the running web server and serve static files directly with so little configuration. It’s a nice bonus that it maps so cleanly on to Django’s own configuration as well.

Allowed hosts

Run flyctl status if you’ve forgotten what domain Fly.io has set aside for your app. Once you’ve got that hostname, set these two settings:

ALLOWED_HOSTS = ['hostname.fly.dev']
CSRF_TRUSTED_ORIGINS = ['https://hostname.fly.dev']

Generate a new secret key

Django generates a secret key by default and stores it in plaintext in your settings file. This is dangerous in production, since secrets should never be stored in source files where they can get caught up in verison control. As its name implies, secret keys should be kept secret, and are important for validating session cookies and other cryptographic uses throughout the framework.

SECRET_KEY = os.getenv("DJANGO_SECRET_KEY")

And then run:

flyctl secrets set DJANGO_SECRET_KEY=<generated secret key>

You can store any other sensitive values in Fly’s secrets — check out the docs for more info.

5. Prepare for deploy

Fly.io will migrate our database for us on every deploy, so now is a good time to make any changes to your app locally that you’d like to have before that first migration. Most notably, this will likely include a custom User model.

If you’re adding configuration to an existing Django app, there’s probably nothing to be done here.

6. Deploy to fly.io

We’re finally ready to deploy our app! Run flyctl deploy --local-only, and the tool will build your Dockerfile and push it up to Fly.io. I needed to use --local-only because the remote builder in my account had shut down and I had no way of turning it back on, but your mileage may vary. Local builds require Docker to be installed and running locally.

Try visiting your site, and see if you can see the default Django welcome page!

7. Create admin user

The last thing that you’ll need to do is create a superuser for accessing Django’s built-in admin site. Luckily, we can run any management commands against our production server just by sshing right into the box:

$ flyctl ssh console
# From inside the ssh session, run:
> cd app
> poetry run python manage.py createsuperuser

Next Steps

And that’s it! You should have a fully functional Django app on Fly.io with a Postgres database, and hopefully you learned some useful flyctl commands along the way.

What I’ve presented here is a “minimum viable deployment”, but to make sure your deployment is safe for users and not vulnerable to attacks, it’s important to look through Django’s deployment checklist. Some things have already been addressed throughout this article, but items like the DEBUG setting are important to be aware of for production Django.

Hello, Solar System!

Mon, 03 Oct 2022 17:00:00 GMT

Hello World, again! I’ve rewritten this website using the Astro web framework. Astro’s very new — it had its 1.0 release just under two months ago. There’s three things that got me excited enough about it to rewrite my whole site.

JSX. JSX is best known as the templating syntax that powers React. While Astro can integrate with React, Vue, Svelte, or any other framework from a growing list , .astro templates are written with a mix of TypeScript and JSX and are rendered directly to vanilla HTML with no JavaScript at runtime. TypeScript is written in a preamble section that looks similar to Markdown frontmatter and sets up the scope for the rest of the template. I found Hugo’s templating system that is built directly on Go’s templating library to be very limiting by contrast. At this point, I definitely believe the adage that “any powerful templating language eventually grows into an awkward programming language” — I’d much rather use a full programming language as my templating language.
Scoped CSS. CSS written in .astro templates are scoped to that template only. I’m not a fan of dogmatic separation of concerns in CSS. I also don’t have much experience with Tailwind, and I enjoy writing SCSS. So scoped styles in all my templates is a happy medium where my styles live close to my templates while I can write in a familiar syntax.
Lightweight. With everything from archetypes to taxonomies, Hugo is really a static CMS more than a blog generator. It’s a much more general system than I need. Astro is a bit “closer to the metal” in that respect. When I’ve had to write some helpers that I took for granted in Hugo, writing custom helpers is much easier since it can be accomplished in a few lines of Typescript.

Astro is sponsored by Netlify, and in many ways it looks like their answer to Vercel’s extremely popular NextJS framework. NextJS is frontend-first and dynamic by default while it puts the option to export to static HTML off to the side. By contrast, Astro is server-first by default, making you opt-in to server-side runtime rendering and client-side reactive “Islands”. It’s an approach that was very attractive for building a static site, and I’m interested to see how they plan to evolve the framework from here.

The transition from Hugo to Astro wasn’t too bad since I was able to find a 1-1 mapping between my Hugo templates and partials with Astro components. I’d like to give a special shoutout to Bryce Wray, whose site repository and blog posts on Astro were a great reference when building out my own site. While the Astro docs are great, they are also new, and there’s a few things that I was able to find out about by snooping around Bryce’s repo.

I’ll miss Hugo’s support for rendering .org files – I’ve unfortunately had to convert posts from my time using emacs to Markdown using pandoc. For the time being, I’m stuck opening Chrome when working on the site because Astro’s dev server doesn’t hot refresh properly in Safari. Besides these two nitpicks, I’m overall happy with the move. I think it’ll help me make new additions to this site over time with less friction than before. We’ll see how that goes!

Renting in New York City

Sun, 08 May 2022 17:00:00 GMT

I just signed a lease for my second apartment since moving to New York City after college. Some friends asked me for advice on how to find an apartment here, and I didn’t realize at first how many opinions I have on the whole process. I’ve tried to distill my thoughts and approach here.

Renting an apartment in New York is all about tradeoffs, mostly between time and money. If you’re willing to spend more time seeing apartments in person, then you’re much more likely to find a great apartment that others are overlooking because the information online is incomplete or hard to find. If you’re willing to spend more money or give up something that most other renters are looking for, you can find a nicer apartment without searching for a diamond in the rough.

If you can put in the time to develop an intuition for the types of apartments on the market in your budget, you’ll be much more confident in your decision when you ultimately sign a lease.

NYC’s rental market is famously crazy, and the disruptions brought by COVID-19 haven’t made it any less so. Renters were often able to negotiate their rent downward or sign new leases with attractive deals during 2020 and the first few months of 2021, but since last summer, rents have bounced back and then some. A lot of people have since been priced out of their current apartments, which only adds to how much demand there is in the market right now.

New York is a huge city, and I’m an early-career software engineer who works in Midtown Manhattan looking for apartments in Lower Manhattan and adjacent parts of Brooklyn. I also happen to be a bit of a recovering HGTV junkie who enjoys looking at floorplans more than is probably healthy. I can’t expect this post to cover everyone’s rental experience, but hopefully there are some small bits of advice you can pull from it.

When to begin your search

The first listings for a given move-in date will start getting posted 1.5 months before, and new ones keep appearing up to as little as a week before move-in. It can be really useful to start looking at the apartments that are available in the month before your target move. This helps you understand what’s on the market in your price range and develop the intuition to spot a good deal when it presents itself.

Start on StreetEasy

How do you even find apartments that are available to rent? StreetEasy is your best friend. It’s a listing website exclusive to NYC where brokers generally post all their listings. You can create an account and save searches to come back to and get email updates for.

I recommend taking a visit to the “Email Preferences” section of your account settings. I set all my saved searches to send emails once per day, and also I checked the setting to combine all the searches into one email. This meant that every morning I’d get a digest of all the new apartments that were listed within my price range and desired locations.

I got into the habit of checking this email every morning before starting work, and clicking the “Request a Tour” button on every listing that was interesting to me. Brokers usually reach out with a day or two.

Beyond StreetEasy

StreetEasy makes it super simple to find listings and contact listing agents, but it isn’t the whole process. Forming too much of an opinion based off a StreetEasy listing can lead you to miss out on a great apartment.

The listing for my first apartment only had a photo for one out of its four bedrooms. The listing for my second apartment had no photos of the apartment itself, just photos of the shared rooftop and the mail room. In both cases, the listing agents weren’t being malicious. They were just working with old and incomplete photos and couldn’t take new ones with tenants still living in the apartment.

My second apartment was far-and-away better than most others I’d seen, but it had an order of magnitude fewer “likes” on StreetEasy. Only one other person showed up to the open house compared to the dozens of people I had seen at a very well-photographed apartment around the corner a few weeks before.

Don’t be afraid of going to listings without many photos! Sometimes agents just haven’t been able to take them yet, and you’ll be able to see a place before other renters start taking it seriously. You still need to do your due dilligence. Check out the building on Street View or walk by in person if it’s convenient for you. I also would NOT recommend signing for an apartment without any pictures at all if you’re not able to visit in person.

On the other side of the coin, it’s important to recognize that apartments with tons of photos aren’t always represented more accurately. Misleading camera angles and different lighting choices can make apartments look a lot better on your screen than they do in real life. Visiting apartments in person can help you learn how photos map to reality and set expectations more appropriately.

In a similar vein, I have a friend living in Boston who rented his apartment off of Craigslist, which lots of people consider a less legitimate website than StreetEasy or Zillow. He was able to find a great apartment that others were discounting for more superficial reasons.

Talk to listing agents

When you visit an apartment, make sure to talk to the agent and let them know what you’re looking for. It’s pretty likely they’ll be trying to rent out other apartments, and sometimes, they’ll show you places that haven’t been listed yet. Ultimately, real estate agents earn their income on commission, and so they’ll do what they can to help you find an apartment where they can get the broker’s fee. Getting sneak previews like this is a great way to get a head start on other renters.

Asking questions

It’s important to ask questions to the listing agent about the apartment while you’re touring. Here are some that I’ve found useful or illuminating.

What utilities are included, and which are my responsibility to pay for?
Does the super live in the building?
[If there isn’t central air conditioning] are A/C units included, or will I need to buy and bring my own?
[If you want to apply] What will the application process look like?

If the current tenants are present during the showing, don’t be shy:

How has your experience living here been?
Why are you moving out?
How responsive or helpful is the super and building management?
What has been your most/least favorite part of living here?

Lots of tenants give candid answers that can be super helpful in your decision process.

Applying

Once you find an apartment you like, the next step is to apply! Tenant rights are strong in New York, so building management companies do a lot of due dilligence to make sure you can really pay rent before drawing up a lease for you to sign.

My main advice through the application process is to be as responsive as possible and be honest with the leasing agent and building management. Before you sign the lease there’s nothing legally requiring them to continue working with you, so try to get requested documents to them as quickly as you can. It’s also a good way to make a positive first impression on your potential your future landlord.

The first type of application I’ve encountered has you pay a deposit that’s anywhere from $500 to a month’s worth of rent. This will be refunded if your application is rejected by management. While it might seem really burdensome to give up money before even having a lease to sign, the apartment will likely be taken off the market while your application is being evaluated. This is great when you find a place you really like – it means that you don’t need to worry about competing offers, or other renters applying who are more attractive tenants for whatever reason.

This most recent search, I ran into a lot of applications that didn’t require a fee. Instead, the listing agent collected as many applications as came in after an open house, and management picked the one they like the best. This seems nice because there’s no upfront cost, but there’s generally more waiting since building management has more applications to read, and there’s no guarantee that you’ll get the apartment even if you have a great application. I submitted and was rejected in three “no fee” applications at different apartments before eventually having an application accepted along with a deposit.

Signing the lease

Congrats! Your application’s been approved. Now it’s time to sign a lease. I don’t really have many opinions on this part of the process. I’m also certainly not a lawyer, but one thing I will say is make sure to actually read your lease! Even if you don’t want to negotiate anything inside it, it is always good to know what you’re signing.

Now it’s time to relax before packing up and moving. Good work, and good luck in the new place!

Task Management in Obsidian

Sun, 23 Jan 2022 01:20:05 GMT

Obsidian is first and foremost a Markdown editor with first-class support for internal links between notes. But just because it wasn’t built as a to-do app, doesn’t mean it can’t become one. Community plugins and external tools have made Obsidian work just as well for me as any task management app I’ve used in the past. As an added bonus, because I’m keeping notes about my day and what I’m reading, it’s easy for me to keep tasks in the context of the thoughts that spawned them.

Plugins:

External Tools:

Siri Shortcuts

Agenda View

Like I mentioned in Part Zero, the first real productivity workflow that clicked for me was Org Mode’s agenda system. My tasks could stay in the context of the notes and project that spawned them in the first place, but could be collected together into a global to-do list on command by calling up the agenda buffer. I’ve tried to replicate this quite closely in Obsidian.

By default, Obsidian doesn’t have any real organization for tasks beyond Markdown support for rendering checkboxes and a special filter in the core Search plugin. There’s a handful of plugins that can help you collect tasks in a more global list, but I chose towards Obsidian Tasks because of its custom query block. In any note, I can add a code block with the special type tasks and create an ad-hoc to-do list for tasks that satisfied a given query. Combining a few of these queries, I put together a central “Agenda Note” that contains my entire to-do list, at a glance. This is what it looks like right now:

And you can click here to reveal the full source code.

### Overdue

```tasks
not done
due before today
```

### Due today

```tasks
not done
due today
```

### Due in the next two weeks

```tasks
not done
due after today
due before in two weeks
```

### No due date

```tasks
not done
no due date
```

### Done today

```tasks
done today
```

All in all, pretty easy to understand! The queries look very close to natural language.

I also give myself a view of where I stand on tasks that are important for today in each daily note. I’ll dive deeper into my daily notes next time, but here’s a snippet from that template:

## To Do

```tasks
not done
due before {{date}}
```

```tasks
due on {{date}}
```

These two queries let me see everything I should handle today: overdue tasks “roll over” automatically from previous days. I also keep a record of tasks done on each specific day in my daily notes, as an automated part of my journal.

Capturing Tasks

Viewing and managing existing tasks is only half the picture. I want to be able to capture tasks when they come into my head so I don’t have to keep focusing on them as my day continues around me. My catch-all location for new tasks is in my daily note. Obsidian Tasks has a command shortcut for creating a new task, with a nice modal that lets me enter due dates similar to Todoist. As a lightweight replacement for org-habit, I keep repeating tasks in their own notes so I can track how often I complete them. I also will add tasks that relate to specific topics directly in-line to notes, relying on my agenda queries to slurp them up properly and link back to the context that I in which created them.

When I’m on my computer or actively using Obsidian on mobile, this works great. But I’m actively trying to spend less time idly on my devices. There’s lots of moments where I’ve realized I need to capture a task but I’m not actively using Obsidian. What then?

Siri Shortcuts

I’ve never used Siri Shortcuts before, but pawalt shared a shortcut with me that takes text input (or voice input from Siri) and appends it to a file in his vault. Here’s my slightly modified version:

It works beyond perfectly in tandem with Obsidian Tasks, since I can add mobile tasks to the same note and they’ll still be collected with everything else in the agenda view. “Everything is a file”, so the Obsidian developers didn’t need to write any particular integrations with Shortcuts. The shortcut just edits a file on disk, and Obsidian reads the file as it normally would, interoperating seamlessly without even knowing it.

This shortcut inspired me to go one step further. It’s a small hastle, but a hastle nonetheless to always have to navigate back to my Agenda note from another part of the app when I want to see my tasks on mobile. If I was using a dedicated to-do app like Reminders or Todoist, I would always be in the task-management view, since it’s the only thing those apps do.

Obsidian has a pretty easy way to create bookmarks and shortcuts to notes from outside of the app through Obsidian URI support. And using the Advanced URI plugin, I can create a bookmark that will always open my Agenda view in preview mode rather than edit mode. I’m sure there’s multiple ways to add bookmarks to the homescreen, but I just created a one-action Shortcut to open obsidian://advanced-uri?filename=Agenda&viewmode=preview, and iOS and Obsidian handle the rest. I have my two new shortcuts on my homescreen, and they look and work almost as seamlessly as if they were native apps.

These shortcuts were a real “holy shit” moment for me as I was building out my personal workflows. I certainly didn’t expect iOS, which is notoriously difficult to customize, to be able to support these kinds of custom actions and integrations with Obsidian. It’s opened my eyes to what Siri Shortcuts can do. iOS doesn’t let me edit the system’s crontab and set a Python script to run every four hours. However, Shortcuts does let me create a “Personal Automation” to run a shortcut that I write every weekday at 8am, or every time I enter or leave a geofenced area. I’ll certainly be thinking more about what I can do with Shortcuts going forward.

Kanban View

I wanted to give an honorable mention to the Obsidian Kanban plugin. Trello never worked for me as a general task management system, but I’ve found that kanban boards do have their place in my workflow for personal project management. I’m currently working on a video game in my spare time. Having a board to keep track of my ideas has been a great way to organize my thoughts and the direction I want to take the game. I created another board to track article ideas through the writing process for this blog, too. And like everything in Obsidian, the plugin stores all the data as readable, plaintext Markdown files.

Wrapping up

And that’s the first part of my Obsidian workflow! Even if none of this feels like it clicked for you personally, I hope you learned something about Shortcuts, automation, or a new Obsidian plugin. And I hope to catch you in the next part of the series!

Thinking with Obsidian

Fri, 21 Jan 2022 04:00:34 GMT

This is the first part of an in-progress series on the Obsidian knowledge base. You can find all the articles with the obsidian tag.

I’m not an organized person by nature. I tend to try to just commit important things to memory rather than write them down in any systematized way, which means I end up forgetting things about 20-30% of the time, and just hoping I remembered the most important things in some kind of implicit neural priority queue. I’ve tried a bunch of different task management and notetaking systems before, but nothing really stuck. At the end of the day it’s because I always struggled to organize my thoughts in a single hierarchy. Even a regular paper notebook is organized implicitly in a timeline, with older notes in the front and newer notes in the back. I always spend way more time thinking about where my note should fit in the hierarchy than I do actually capturing my thoughts.

The closest I got to a system that worked for me was Org Mode in Emacs. Org is a full notetaking and outlining system first and foremost, but Org’s agenda views were game-changing for me: I could write down tasks and events in any file, and my agenda view would aggregate and sort them all to show me what was important. The issue of up-front organization completely went away. Org was an extremely powerful system, but its only true implementation was the major mode written in Emacs LISP for the Emacs environment. Mobile apps exist that can parse and manipulate subsets of Org’s syntax,¹ but none of them handle all the features of Org I used, like recurring tasks and agenda view, as well as the original implementation. I loved Org for how easy it was to capture thoughts and tasks, but I still couldn’t effectively capture and process notes and tasks when I wasn’t in front of my own computer with Emacs open. Unfortunately, that was pretty limiting for me.

So I was open to alternatives. Personal Knowledge Management (PKM) has been a trend for a few years now, popularized by Roam and its interconnected web of small, atomic notes. Like Org Mode, there was no need for a strict hierarchy — the relationship between notes is implicit through bidirectional, internal links. There’s a lot of choices now in this space. Roam Research, Notion, Org-roam, Dendron, Logseq, Obsidian all come to mind. If I really had to be honest with myself, I chose Obsidian primarily because it was the only option besides Emacs with Org-roam that included vi keybindings to edit text. In addition, its feature-equal mobile app was an attractive improvement over my existing system.

While I chose Obsidian relatively superficially, I’ve been really happy with the product for two main reasons: extendability and interoperability.

Obsidian follows VSCode’s philosophy of having a rock-solid, minimalist core with most functionality built on top as plugins.² The Obsidian docs say that “Used in the most basic way, you can edit and preview Markdown files” within Obsidian. The one real addition to that is that internal links are “first class citizens,” and Obsidian’s core will index your internal links for autocomplete and note renaming. Almost everything else in Obsidian, from the file manager to the graph view to the word count in the text editor, is a plugin built on top of the base Markdown text editor. The fact that the developers have written so many core plugins using the plugin API (27, by my count) means that they’ve spent a lot of time thinking about the ergonomics of that API, and the quality of Obsidian’s third-party plugin ecosystem shows how this pays off. For a relatively new app, there’s almost a plugin for anything, from PDF annotation to an RSS reader fully inside the editor. And if what you’re looking for doesn’t exist, it’s possible to make it yourself, either as a standalone plugin or with some of Obsidian’s advanced scripting capabilities that some plugins expose.

Obsidian is fully based on local Markdown files. While there’s some non-standard syntax for internal links, its emphasis on such a universal format makes it extremely easy for plugins to work together and for Obsidian to interop with external tools. All plugins read and write to Markdown files. If you want to sync your Obsidian vault with some third-party service, just write a script to pull down data from an API and write it to Markdown files. It’ll show up in your Vault when you open it next. It’s that easy! I’ve been able to do it to great success in my own setup.

My Obsidian System

I’m pretty happy with the org-mode-inspired Obsidian workflow that I’ve built. Over the next week or two I’m planning on posting a series that goes into detail on parts of my workflow, explaining plugins and configuration as I go. These sorts of posts were really helpful to me when setting up my org-mode workflows originally, and I’m hoping to contribute to a similar ecosystem of workflows for Obsidian. Luckily, Obsidian is modular enough that you can grab only the parts that are exciting for yourself. I’ll see you in the next post!

BeOrg and Plain Org are two examples of polished apps which focus on different subsets of org-mode’s syntax. ↩
I’d be remiss not to note that this itself is an evolution of Emacs’s now 40-year-old configuration-over-convention philosophy that leads some to refer to Emacs more as an “editor toolkit”, or joke that Emacs is “a great operating system that just needs a good text editor.” (HN source, but you can find this sentiment all over.) ↩

Digital Mindfulness

Tue, 04 Jan 2022 22:19:42 GMT

I’m not normally one for new year’s resolutions. Mostly because I’ve found it hard to make lasting changes to my habits in the past. This year, though, I’ve got a pretty open-ended one: to be more mindful and intentional about how I use digital technology and the Internet.

Digital Minimalism

I read Digital Minimalism by Cal Newport over the summer, but the signifance behind the book didn’t sink in for me until the end of 2021.

The main prescription in Digital Minimalism is the “digital declutter”: thirty days during which you unplug from all digital technology not directly related your job. During and after the declutter, you’re supposedly able to more clearly see what kinds of technology are truly beneficial to your everyday life, and what you can live without. Originally I thought this was overly onerous. I thought that if the declutter was how you determined which activities were “critical”, then how could you set aside a full month when some of those activities were critical?

You might have noticed that I inserted the word “critical” into my interpretation of the declutter. I’ve been conditioned, or maybe I’ve conditioned myself, to think about a lot of my own habits as critical, when really they just facilitate a nice, local maximum of contentedness. It only took a few days’ declutter for me to begin to see beyond the rut, though.

Taking time off

In the last weeks of 2021, I took my first real vacation days since I started working at MongoDB in August. I was home with my parents, away from the busyness of New York City and without any pressing commitments. Even just not working was more of a shock than I’d felt during inter-semester breaks back in school — after two days off, I had a bit of a “I’m not sure what to do with my hands” feeling. That’s probably because I was generally burned out by the end of every semester-long sprint in college. Working life for me has been much more of a maintainable marathon, so I was able to recharge past a simple baseline over this break.

I decided to go a step further than disconnect from work and not even go on my personal laptop for a few days. I spent most of my time reading rather than watching TV, which I also tried to stay away from. I went on walks without listening to podcasts or music, and generally just let myself relax a muscle that I didn’t even know was under tension.

Stepping off the treadmill

Steve Jobs once called the personal computer “a bicycle for the mind”. He meant it in the context of how human-made tools can drastically increase our effeciency. The quote also brings to mind how a computer can be both fun and stimulating, like a bicycle. I’ve realized that modern digital technology and the internet gives even more layers of meaning to the metaphor.

Like riding a bike, using digital technology takes energy and focus. It’s clear why this is true on a mechanical level. Most of my personal media consumption is in the form of reading articles and comment threads. Focusing on text on a backlit screen requires a level of focus and concentration that I didn’t realize I was expending until I wasn’t doing it anymore.

Being “wired in” also causes other mental muscles to atrophy. Even two days into my declutter, I found myself free associating and daydreaming when I didn’t have anything else occupying my attention in a way that I hadn’t done for years.

When I’m at a computer or on my phone, I feel like I’m in a constant feedback loop with the machine. I perform some action, get some response, which might prompt some hit of dopamine, and regardless will most likely perform another action (or the same one again). The presence of a feedback loop itself wasn’t troubling — all of consciousness involves feedback loops. When I stepped off that machine-human-machine treadmill, the new feedback loop was completely internal to my mind. I was able to think much more freely and with less friction, because I was no longer beholden to the rigid limits of how the computer logically responds to my actions.

Fidget spinners

I started using Opal to physically block traffic to distracting websites, like Twitter, YouTube and Hacker News on my phone, and edited /etc/hosts on my laptop to achieve the same effect. I noticed that any time I had any spare moment, I felt like I needed to check on something on my phone. Lots of times it would be completely subconscious. I wouldn’t even realize that I’d unlocked my phone and gone to HN or lobste.rs until the blank page on Safari snapped me back into reality.

Given a free moment, I just wasn’t letting my mind wander. I was subconsciously craving that feedback-loop mental treadmill and the small chance for a hit of dopamine if there happened to be a new video or story or comment or tweet or whatever – and there usually wasn’t. I became frustrated that my phone felt useless with this new limitation. What was an iPhone good for if it couldn’t save me from being bored?

Slowly, I started stepping back and realizing that that’s a pretty idiotic thought. Smartphones can look up any fact imaginable on Wikipedia, show you how to drive anywhere you want with GPS, or order a taxi for you with Uber and Lyft if you can’t drive one yourself. As much as I used my phone as a mental fidget spinner, it was still a real tool that had real uses. More disturbingly, I clearly got along just fine before I got on social media and started using a smartphone in 2013. Why did I not feel fine now?

What changed?

What’s interesting about all of this is that I’d already cut out virtually all algorithmic social media (Instagram, Twitter, Facebook) years ago. My main muscle memory on a new tab takes me to news.ycombinator.com, which has a much gentler algorithm than most main-stream social media, no infinite scroll, and no ads.

I think an important mental shift for me happened at the start of COVID-19 lockdowns in March 2020. I felt like I had nowhere to be, and nothing to do, and so why not just sit in front of my computer, refreshing message boards and (doom-)scrolling through Twitter? Once formed, habits are hard to break. As lockdowns eased up, the country reopened, and life started to resemble the Before Times™ more and more, that muscle memory that I’d formed in the first half of 2020 didn’t go away on its own. It took me another 18 months for me to put a name to the feelings and identify what had really changed. As the US deals with another variant wave in Omicron, I want to make sure I don’t entrench the same habits even more.

One of the things I learned from Digital Minimalism is that “minimalism” as a philosophy doesn’t just mean “less is more”. It’s a recognition that some things in life are zero-sum and that not every output has a linear relationship with its inputs. I can keep increasing my short-term contentedness by spending more time online, albeit with quickly diminishing returns. The global maximum for my own happiness levels might, and probably does, lie in dialing down my time spent staring at screens to a much lower level than what I’ve been living with day-to-day over the past few years.

Where to go from here

A big goal for me is to reframe why I pick up my phone (and laptop) in the first place. Before this break, my primary use of technology was to stave off boredom. Even if I got work done on a personal project or caught up with friends, the vast majority of “pickups” were mental fidgets because I couldn’t find any other external stimuli at that precise moment in time.

Going forward I want to shift my objective to be to use technology to accomplish a goal. When I pick up my phone or laptop, I should already have a specific intention in mind beyond “keeping myself occupied”. Something like “I have an idea for a blog post and want to start writing” or “I wonder how so-and-so is doing? I should call them.”

Blocking habit-laden websites will help re-train my muscle memory as I go, but I also need to be conscious and mindful about my behavior. When I find myself with a free moment, I should just try and let my mind wander, and be content with internal stimuli. If I’m really craving external stimulation, I should pick up something long-form, like a book, or open a weekly newsletter I’ve subscribed to.

My digital declutter was much shorter than what Newport recommends in Digital Minimalism, but I was able to see the power in the approach. I would need to balance a comprehensive declutter with wanting to remain social as a young person whose friends wouldn’t also be doing a declutter. It’ll probably involve setting up auto-responses somehow and making sure friends know to call rather than text me. A few weeks ago I would’ve considered a month-long declutter completely out of the question, but now I might consider it down the line.

PiClock: Waking Up to the Unix Philosophy

Thu, 20 May 2021 17:00:00 GMT

I built a clock! It runs Linux. Sounds like quite the over-engineering project, but I learned a ton along the way and was able to flex some muscles that had been lying dormant for a while. I write a decent amount of code, but I hadn’t worked on an electronics project in years and had never really embarked on a 3D modeling or design project outside the classroom.

I discovered the Yayagram a few weeks ago, and my project was inspired by many of the same principles. Although I’m building this clock for myself rather than an older family member, I love the idea that objects can be smart and internet-connected in useful ways without interactive touchscreens. Making smart devices with lower-fidelity input and output can help slow down the feedback loops that we often encounter with the modern internet. I’m hoping my clock can be a step for me away from doomscrolling and towards a more measured relationship with content consumption on the internet.

The Problem

I’m a relatively deep sleeper. I sometimes sleep through alarms, and even when I wake up most of the time, I fall asleep again just as quickly after I hit “off” or “snooze.” I’ve resolved to just setting four or five alarms staggered five minutes apart around the time I want to wake up. It’s not a use-case that’s supported too well by Apple’s Clock app:

On top of all this, needing to grab my phone to snooze my alarm makes it easy for me to keep scrolling rather than get up and start my day. Off-the-shelf digital alarm clocks are a lot less distracting than the general-purpose internet communicators we carry around all day. Still, they generally can’t set more than one or two alarms and are difficult to program when schedules change day-to-day.

The Spec

I wanted a clock that would:

Let me set multiple alarms per wake-up.
Not be distracting in the morning when I want to start my routine.
Provide a bit of news to begin my day without sucking me into an algorithm-powered bottomless doom vortex.

Point #3 means that this clock needed to be internet-connected. But point #2 means that it couldn’t be distracting. This rules out writing a custom app for my phone or something like an Echo Show.

The Hardware

Electronics

My strange alarm staggering requirements meant that I also felt like I’d want to program in custom behavior. With the interaction of all these requirements, I decided to have the brains of the clock be a Raspberry Pi Zero W running a standard Raspbian distribution. For the splash of news in requirement #3, I wanted to have the choice between audio podcasts and small snippets from Twitter and newsletters, along with the weather and recent COVID statistics for my area, so I opted to add in a thermal receipt printer for a more low-screen digital output.

I sourced all the parts from Adafruit, which has a fantastic selection and quick delivery times from their warehouse in NYC to where I’ve been in Philadelphia. I’ve listed a more-or-less complete bill of materials in the Github repository.

Enclosure

The enclosure was a challenge — my first non-school project in SolidWorks since taking a mechanical design elective two years ago and my first 3D printing project since I was on my high school’s robotics team. If I’d had access to a laser cutter, then that would’ve been my go-to since the enclosure is just a simple box. Engineering facilities at Penn were closed down to most undergrads this year without pressing work to do, so my trusty Printrbot Simple Metal would have to cut it.

I had more issues with bed adhesion and warping than I’d ever encountered before. Once I ironed out the adhesion issues, I couldn’t pull the finished prints off the bed without applying some serious lever force. I ended up investing in a BuildTak FlexPlate to avoid damaging the printer more going forward.

I knew that the software was the most exciting part of this for me and what I was most comfortable with. So I didn’t let myself get started on the coding until after I’d finish the enclosure; otherwise, I’d be left with a mess of wires on my bedside table until the end of time. Ultimately, though, I’m happy with the finished result. It’s about as small as possible while still fitting all the components and looks clean by my bed.

The Software

Adafruit’s whole library stack for their peripherals, I’m pretty sure, is only available for Python on the Pi and Arduino for more basic microcontrollers. This basically made my programming language decision for me, and I’m more than comfortable enough in Python to be happy with that solution.

Before this, my other electronics/embedded computing experience has come from from Arduino, with its setup(){.verbatim} and loop(){.verbatim} functions that control the whole project. I was dreading writing all my code in a single Python while True{.verbatim} loop that would get initiated on system startup. When I worked on a similar digital watch project five years ago, the code ended up being extremely brittle, with different modes and state riddled all over the place. Fundamentally, I realized I was asking myself why I would try and write a robust mini-OS for a clock when the computer it’s running on is already running Linux?

The architecture I landed on is spread among different files and uses cron{.verbatim} to orchestrate all the behavior. Python scripts control various aspects of the clock’s functions and are called on one-minute intervals. Luckily, my display doesn’t have space for seconds since this is the smallest interval available for cron.

tick.py{.verbatim}: updates the seven-segment displays with the current time.
alarm.py{.verbatim}: checks alarm settings and rings an alarm if appropriate.
podcast.py{.verbatim}: depending on the command line argument, downloads a new podcast episode from an RSS feed or plays the most recent one.

It’s been my first real foray into building utilities that faithfully follow the Unix philosophy of doing one small thing well in a composable way. Settings like alarm schedules are saved in JSON files, and I’m planning on writing wrapper commands around jq{.verbatim} to make it easy to modify settings without dropping into an editor over SSH. It’s certainly been discussed how composable Unix utilities can lend themselves well to a productive developer workflow. It was really cool seeing this approach work well for an embedded system, too, albeit without actual realtime requirements.

Next Steps

There are certainly some more directions I want to take the clock in. On the software side, I still haven’t played around with the receipt printer much. It’ll be interesting to experiment with data sources and layouts that can work well for morning updates. I also think it could be cool to print out transit directions on-demand. I’m moving to New York City soon, and I always find myself making sure I don’t close out of Google Maps when I’m on the subway, since service can be spotty and I often forget what stop I need to get off on or what train to transfer to next. I’m also thinking about integrating to-dos for the day to track what I plan on getting done in the morning.

On the hardware end, there’s certainly room for expansion. The clock currently has a single pushbutton that I use to interact with alarms. It might be cool to add a time-of-flight sensor for gesture control. Or maybe a fingerprint sensor to allow for different controls based on what finger I press to the sensor.

Code exists in any number of copies. It’s been awesome tinkering and tweaking a physical thing that’s more-or-less one of a kind. And unlike an off-the-shelf product, it’ll be cool to see it evolve over time as I tweak whaat I’ve got add more features to it.

Top Classes in Undergrad

Sat, 10 Apr 2021 17:00:00 GMT

I’m graduating from college in May. Over my four years working towards my degree, I’ve taken about forty different courses. As I enter the last month of school, I thought it would be a good time to look back and reflect on the courses that I enjoyed the most. I’ve picked out five classes, and instead of trying to make any absolute ranking of them, I just decided to present them in the order they were taken, and try and express how each one impacted me.

CIS 240: Introduction to Computer Architecture (Fall 2018)

CIS 240 with C.J. Taylor was my favorite introductory CS class I took at Penn. I came into school with some programming experience, but I was certainly ignorant of the inner workings of how computers took the code I would write, execute it and ultimately give output back to me. 240 started with transistors and logic gates, built up to CMOS circuitry, before jumping the boundary between atoms and bits and introducing us to machine code, assembly, and C for an educational architecture known as LC4. I’ve heard that pointers in C are a notoriously difficult concept for beginners to grasp, but the evolution up the ladder of abstraction within a computer made C feel like a natural extension of assembly. The course had a very intentional flow to it, and every separate concept felt like it fit into the course well.

Outside of the material, Professor Taylor brought great enthusiasm and a sense of humor to the class. Along with an arsenal of cheesy jokes (my favorite kind) that I’m sure were honed over the semesters lecturing for the course, he came to our midterm exam, which happened to be during the final block of class on the afternoon of Halloween, dressed as Darth Vader, mask and all. He somehow managed to stay in character the entire exam, which I’ve always respected.

PSCI 232 / COMM 226: Introduction to Political Communication (Fall 2018)

This course was billed to me as one of the most difficult, but also most rewarding, political science courses at Penn. Professor Kathleen Hall Jamieson had been teaching the class since the late 90s, and her expertise shone through. 232 was all about presidential elections, the media, and the electorate. It took a deep dive into presidential campaign advertising in every presidential campaign since Eisenhower’s election in 1952. Each week we’d watch campaign ads from a certain campaign, and Professor Jamieson would dig into what made each one persuasive to its audience, or why a given ad flopped.

More important to me than the technicalities of advertising and public communication, we dug into the role that the media itself has in shaping the narrative around elections, and how campaigns can rise or fall in how they take advantage of the media’s agenda-setting powers. Jamieson has been an outspoken voice about deception and falsehoods in politics for decades. Standing in 2021, her warnings back then certainly seemed prescient, and the class helped teach me the very active role that social and traditional media play in helping decide how our politics unfolds in the United States.

PSCI 253: International Politics of the Middle East (Fall 2019)

The syllabus was so daunting I almost dropped 253 before the first lecture. Six books in fourteen weeks would add up to roughly two hundred pages of reading each week, and on top of an upperclassman engineering course load, it didn’t seem particularly feasible. But after a single lecture with Robert Vitalis, I knew I would stick it through. I feel like it’s pretty easy to determine which professors have enjoyed tenure for a good amount of time. While the observation is generally considered to be a nock against a professor’s dedication to their teaching responsibilities, I’ve come to think that there are two distinct sub-types of the “tenured professor” stereotype: there’s the stereotypical apathetic professor, but there’s also those who take risks with their pedagogy that let you know that they haven’t had to justify their teaching methods to a committee for some time. Professor Vitalis brought a particularly wacky attitude to his lectures, often starting class with comments and observations that were intentionally provocative to stir up discussion and disagreement among the class. Even the 200-page weekly readings were, in fact, a teachable moment. Our TA let it slip that Dr. Vitalis’s goal was to teach us how to skim effectively in a sort of “baptism by fire.”

The books we read in the course also tied into each other in thought-provoking ways. Each dealt with American intervention in the Middle East over the 20th century in a different light. Written assignments were comparative book reviews, a format I haven’t heard any other professor using in their classes. The fundamental question of the course, it seemed, was why? Why did the US maintain its interventions on the other side of the world? Why did it present those interventions to the public as it did? We never reached a real answer to any of these questions, but the books brought another lens through which to look at government and the responsibility of those that hold the levers of power in American society, over not just our lives, but the lives of people all over the world.

CIS 341: Compilers and Interpreters (Spring 2020)

I certainly didn’t expect 341 to be the class that perfectly capped off my computer science education when I first signed up for it. The course is taught in a bottom-up manner, starting with an assembler for a subset of X86 before moving up to an LLVM to x86 compiler backend. Only after understanding some of the nuances of LLVM does the class move to the frontend of the compiler and deal with lexing, parsing, and types.

Steve Zdancewic expertly connected all the disparate threads of computer science that I’d experienced throughout college and showed how mathematical theory and engineering discipline can be combined to enable the programming languages that developers take for granted today. I don’t think there’s a better way to explain this than by listing all the topics that 341 touched on, and the other courses that it built on top of:

CIS320 (Data Structures and Algorithms): Graph coloring for register allocation
CIS262 (Automata, Computability and Complexity): Parsing classes and context-free grammars
CIS240: Assembly language and machine code
CIS371 (Computer Architecture II): Optimizations around instruction ordering and processor pipelining

I was able to put some knowledge from this course to use right away. Penn Course Review needs to load in a SQL dump of new course reviews every semester. Up until now, we’d relied on loading all the data into a blank MySQL database that we spun up, and then querying it back out in the format we expected. It was a lot of moving parts and generally pretty slow, too. I was able to write a parser that pulled out that same data without having to run it through a full MySQL instance. It just goes to show how the skills that are involved in writing a compiler are useful in their own merits, in addition to being used for compilers.

341 was also where I got exposed to type theory as a subject for the first time, and it sparked my interest in the study of programming languages.

CIS 552: Advanced Programming (Fall 2020)

Stephanie Weirich bills CIS552 as a class that “take[s] good programmers and turn[s] them into excellent ones.” It’s Penn’s only full-credit class taught with Haskell and going into it, I knew next to nothing about the language besides that monads are something people find scary. This might be kind of cliché, but learning Haskell through 552 did change how I thought about programming across the board. Haskell evaluates expressions lazily when they’re needed, and not when they’re defined. This is an oddity in mainstream languages, but it makes you think more critically about when your code runs. It helped me realize that some expressions in my Python code were being evaluated at their definition when they should have been evaluated lazily.

552 shines in how it introduces monads and explains their usage. Monads aren’t not something that clicks for everyone, and they certainly didn’t for me right away. We started from the assumptioon that the general definition of monads is too abstract for programming, and the class worked more by example. Through the Maybe{.verbatim} monad, State monad, List monad, and a few other examples, the intuition slowly built up for me about how powerful the concept can really be. Haskell’s a fundamentally pure functional language that deals only with inputs and outputs to functions. What monads add in this context is a way to abstract out the glue code and plumbing behind a lot of programming patterns in a way that makes code easier to follow and allows Haskell, with a bit of syntactic sugar that the do{.verbatim} block provides, to take on some imperative-seeming features that one may find in a more traditional language, like exceptions and global contexts, while maintaining its underlying purity.

The back half of the class was a great tour of what you can do when strong typing is taken to its logical conclusion. I got to work with a friend on a awesome final project, building a typechecker for MongoDB aggregation queries. We were both impressed with the final state of the project, but it’s pretty crazy how much type-driven programming helped us out here. We started by defining our abstract syntax tree, and then split up to work on the schema and query parsers and the typechecker itself. Any changes we made in the types were checked by the compiler and we were able to modify our logic, and our monad stack, to add new features without worrying about breaking existing ones.

I don’t harbor any notions about how difficult it would be to find a job working with Haskell in industry, but it’s certainly a language that I’ll continue to play around with going forward. It’s got some awesome ideas that are starting to percolate down towards more mainstream languages like Rust.

Wrapping Up

All in all, I consider myself lucky to be able to study things that I find interesting in their own right. Penn’s been difficult at times, but it’s also afforded me the opportunity to learn from great professors, work on super awesome projects, and meet other students who are extremely driven and passionate about every topic under the sun.

I don’t know if there’s a single thread that runs through all five of these classes. If I had to pick something, they all certainly had an intentionality to their curriculum. The professors made sure that everything they taught had its place in the larger narrative, helped me understand how each lecture fit into the course topic, and how the course itself fit into the larger field of study. As is pretty apparent from me deciding to write this article, I like finding and understanding patterns — it makes sense that the courses I enjoyed the most were able to weave fabrics out of seemingly disparate threads.

It’s certainly cliché to say, but I don’t plan on my education stopping after I get my diploma. Who knows if I’ll ever go back to school, but I’ll take what I’ve learned about learning itself along with me as I start my career.

Hotwiring the Web: What does Basecamp's New Magic bring to the table?

Tue, 22 Dec 2020 23:38:55 GMT

Hotwire, which seems to be short for (H)tml (O)ver (T)he (Wire), is a collection of frameworks just announced by Basecamp that work together to help build “traditional” server-rendered web applications that look and feel to users like modern, Single-Page Applications (SPAs) built in React, Angular, Vue or other frontend frameworks. Basecamp’s CTO put out a blog post on why he believes in Hotwire, but most of the justification seems to be handwavy claims that JavaScript is inherently “complex,” never mind that Ruby’s syntax and dynamic type system can be just as head-scratching to a newcomer. I think that Basecamp’s built a really interesting tool, and a better argument for Hotwire can be made by fully engaging with the benefits that SPA “thick clients” bring to the table, their specific shortcomings, and all the different ways framework developers are trying to address those shortcomings today.

Problems with Web Application Development Today

I want to separate “web app” development from general “web” development, simply because most websites aren’t single-page applications, or web apps in the sense we think of gmail, or hey.com. It’s put most concisely in “How the Web is Really Built”. Here, though, I’ll be talking about that subset of the web which can be categorized more as an application than simply a web site.

Building a web app is always going to be a complex endeavour. SPA view libraries like React have emerged in the past five or so years as an awesome way to handle that complexity on the client side, allowing developers to build interactive user-interfaces in a composable way, re-using components easily in different parts of the site.

Duplication of Concerns

Generally, when building a React app, you get any dynamic data through a loosely coupled JSON API (in my personal preference, probably written with Django REST Framework). If you’re using a web framework like Django or Ruby on Rails, then you’re going to have a lot of complexity and business logic on both the frontend and backend. Whenever you add a feature, you need to ask yourself where it should be added. The loose coupling between parts that may very well be written in different languages means that it’s really difficult to share code, and even if you’re doing something as simple as validating a form, you’ll probably end up with a decent amount of duplicated business logic, a big no-no if you follow the DRY principle!

Data Fetching Boilerplate

Loosely-coupled APIs present another challenge when building web apps: the boilerplate that comes along with getting the data out of your database and displayed in a browser. The general process for an SPA follows this outline:

[User] Navigates to a page, submits a form, or clicks on a resource.
[Browser] Send a request, along with any route parameters and arguments, to a backend API endpoint.
[Server] API endpoint is called, fetches data from the database using the parameters in the request.
[Server] After retrieving data from the database, marshal that data into JSON to be sent back to the client.
[Browser] Receive JSON from request, un-marshal the JSON into data structures that the web app can understand. For a React app, this would include passing the data as props to each component that uses the data to for render itself.

And that’s only the happy path! In the case of a validation error, for example, the server would return a 400 response with some JSON details, and the client again would need to catch that error, and render it properly. It really is a lot of boilerplate for any given request.

There are frontend libraries like SWR which factor out as much boilerplate as possible and encourage declarative patterns for your own data fetching, but “thick client” web apps will always need to marshal data to JSON on the server, and un-marshal to some view-able format on the client.

Managing SPA Complexity in 2020

There’s a bunch of different solutions to these problems for those who still want to provide an SPA experience, and they fall broadly into two separate categories.

Frontend-First: The JAMStack

In recent years, there’s been a big push towards the JAMStack: pushing all your complexity and business logic onto your frontend, and relying mostly on commodified API services for your backend work. JSON is still the lingua franca here, but reliance on pre-built backends as a service for different functionalities, along with query languages like GraphQL, means that there isn’t any backend boilerplate handled by developers. All the complexity and business logic is pushed to the client-side, which alleviates worries about duplication of concerns.

JAMStack and associated “micro-backends” like Auth0, or “backends-as-a-service” like Supabase and Google Firebase allow people who haven’t done too much backend work in the past to build truly full-stack apps on their own. m3o is even building a constellation of JAMStack-oriented “micro-services” to provide the batteries to power most web apps. Hmm… “batteries included”? Where have we heard that before…

Backend-First: Re-discovering the batteries

MVC Frameworks like Django and Rails came about during the heyday of Web 2.0 to abstract away a lot of the boilerplate associated with building CRUD web applications, the exact issues that we spoke about above. Back in 2006, all those interactions were through normal browser HTTP requests. SPAs, built with anything from Backbone or Ember to React and Vue, were more responsive. These web frameworks became frameworks for JSON API servers, and for many web app developers, functionality like Django’s templates and forms and the battle-tested abstractions for linking them together became vestiges of an earlier age. Django’s Form{.verbatim} classes can render validation errors in templates with virtually no boilerplate written by developers. As soon as you want to put that form action over a JSON API, any responses from your server, which were previously just the HTML that the browser displayed, now have to be un-marshalled from JSON on the client and handled specifically. How much was Django really a “batteries included” framework if you needed to pull in REST Framework and OAuth Toolkit whenever you wanted to work with a “modern” frontend?

Many people, myself included, enjoy modeling business logic in the ways Django and its ilk allow for. Backend-first fullstack frameworks have begun to proliferate built on top of these existing frameworks. Phoenix LiveView and Laravel Livewire are two that come to mind immediately, and have been around for a year or more.

On Monday, even the React Core team at Facebook threw their hat in the ring, with their Server Components that have the opportunity to allow for React components to be rendered much like PHP templates, interspersing database calls and server-side JavaScript with the layout description inside a server component’s render{.verbatim} function.

These fullstack frameworks go a long way towards solving both of the concerns with traditional SPAs listed above. Separation of concerns is no longer an issue, since there is no separate, loosely coupled frontend codebase. Data fetching is drastically simpler in this paradigm. No longer does all your data need to be serialized to JSON before being converted into HTML; your data-fetching flow looks a lot more like this:

[User] Navigates to a page, submits a form, etc.
[Server] Backend route is called, fetches/stores appropriate data from the database based on the request.
[Server] Data is used to populate an HTML template, which is sent to the client and rendered with the help of the framework.

Your backend business logic renders HTML directly, completely replacing steps 3, 4 and 5 above with a single step: map your data into its visual representation in HTML. The two server steps remaining are the actual business logic in your application: the full-stack framework handles the smooth transitions, without the developer having to worry about serializing and de-serializing their own data. The logical flow of your application becomes a lot simpler for a single developer to follow and to handle.

Even React’s Server Components fit this new paradigm: Data fetching no longer happens in AJAX requests, but by declaring a child server component which fetches the data and displays it in its own DOM tree without the developer having to serialize to/from JSON themselves. After the component renders on the backend, its virtual DOM gets sent to the frontend by React itself for display. The developer’s interface into this whole process remains high-level and declarative.

As the Pendulum Swings

We started the decade with frameworks like Meteor.js with extremely tight couplings between the client and server, and after a long time wandering in the wilderness of duplicated compelxity across loosely-coupled frontend and backend, it seems like we’re entering the twenties with a renewed push towards a more monolithic approach to web development. When even a frontend framework like React is beginning to bridge the gap with the backend, you know it’s an interesting idea to explore right now.

So what is Hotwire?

The folks at Basecamp, the company behind Hotwire, have always been skeptical of thick clients with loads of JavaScript. Hotwire is Basecamp’s latest answer to the challenge of building modern, responsive, “snappy” single-page applications where the domain logic lives entirely on the server. They used it to build out their new email service, Hey.com.

At Hotwire’s core is Turbo, a new library that takes HTML from AJAX requests and dynamically modifies the currrent page. It comes out of an existing library called Turbolinks, now called “Turbo Drive” as of today, which is a utility that intercepts all click events on anchor tags, loads the resources over AJAX, and swaps out the <body>{.verbatim} tags, all while handling browser history.

Turbo Frames, one of the new components, is pretty intriguing. Turbo Drive will still AJAX-ify form submissions and link clicks behind the scenes, but instead of swapping out the entire webpage each time, Turbo will look for matching <turbo-frame>{.verbatim} tags on the current page and in the new page’s content. If there’s a match, it’ll dynamically replace that section of the page. Basically, you can compose webpages together, using <turbo-frame>{.verbatim} to delineate template partials as scoped components, similarly to how you’d think of components in a React app. The benefit here being that all the logic is handled on the server-side rather than split between two code-bases.

Trying out Hotwire with Django

What’s special about Turbo when compared to Phoenix LiveView and Laravel Livewire is that Turbo is completely backend-agnostic: Drop the JS bundle into a <script>{.verbatim} tag in your page’s <head>{.verbatim}, and Turbo Drive works its magic without any co-operation from the server. Turbo Frames can be adapted by wrapping <turbo-frame>{.verbatim} tags around template partials in any backend framework. Turbo Streams, the solution for incremental data updates, can also be used in the context of HTTP requests without any co-operation from the server beyond modifying your template partials. It’s only if you want to use Turbo Streams over WebSockets where you’ll need some custom code for your specific backend framework.

Since the push behind Hotwire came from DHH and Basecamp, it makes sense that their examples are with Ruby on Rails, and that’s where they’ve made their supporting libraries. I decided to take a shot at building a demo app similar to what’s shown in Hotwire’s demo video using Django rather than Rails. It really didn’t take long! I got my start in Web Development with a JQuery app with a REST API, and even after moving on to Django, I always used Django REST Framework. I never really took advantage of the templating functionality, or the super-useful built-in CRUD operations with Django forms. It was an interesting experience working with CreateView{.verbatim} and DetailView{.verbatim} rather than ModelViewSet{.verbatim}, and I’ll be excited to keep exploring this going forward.

After an hour or so more of experimentation and digging into the turbo-rails codebase, I got a working prototype of a Turbo Streams Broadcastable{.verbatim} mixin for Django! I’m working on something similar for Django REST Framework right now, which definitely helped in hitting the ground running. I’ll probably look to clean up the code and make sure it works for the other actions, and split it out into its own pypi package.

I’ll have to look at how the Rails integration handles authorization — right now, anyone would be able to subscribe to any stream for a given model, which is obviously not ideal for actual production applications.

I was really surprised at how easy it was to set up Turbo to work effectively with a Django backend. The chat app that I built was really simple, but it also was just not many lines of code: not having to worry about moving data around from the frontend to the backend really decreased the amount of time spent on the implementation. Turbo though is in a pretty early beta, and the one main thing I’d like to see be addressed would be a good fallback mechanism for Turbo Streams over websockets. Right now, if you want to broadcast updates over websockets, then you can’t also send Turbo Streams in HTTP responses to form actions without getting duplicate data appended. The solution in the Hotwire demo video is simply to not send updates over HTTP, and only stream over websockets. This doesn’t seem particularly robust, however, in the case that a websocket connection fails or a client simply doesn’t support it. In addition to the five actions, there should probably be an append-or-replace{.verbatim} action that looks for an element with a matching id{.verbatim}, performs a replace{.verbatim} action if one is found, and otherwise performs an append{.verbatim} action. The duplicate updates from the HTTP response and the websocket stream wouldn’t conflict in that case, since one will append, and the other will replace with identical data.

Closing Thoughts

This is definitely an exciting time for frontend development! I’m hoping to do some more experimentation in the coming weeks, and I’m glad that framework authors accross the board are putting effort into thinking about how to move the web app developer experience to the next level.

Lessons Learned from Leading Penn Labs

Mon, 25 May 2020 16:15:00 GMT

I just finished writing a year-in-review blog post for the Penn Labs blog. For some context, Penn Labs is a student-run software development group that builds apps and websites to help students navigate life on campus. For the past year, I’ve been one of the two co-directors of Labs, managing the operations and direction of the organization.

I wrote the year-in-review post as a way to highlight the awesome work the whole team had done this year, and hopefully start a tradition of outgoing directors doing the same in the future. While I was writing it, though, I started thinking about what my personal year-in-review would look like. I’ve written down my thoughts here, in the form of different lessons I hope to take with me going forward.

Delegate

Many clubs at Penn operate with a governing board structure. Labs currently doesn’t have any sort of student board outside of the two Co-Directors. Because of this, the Co-Directors are directly responsible for everything from managing finances to grabbing supplies for group work sessions to heading up recruiting efforts. All of these are critical to the club functioning properly, but it’s also a ton of extremely different things for only two people to juggle in their heads at once.

This year as director really made me realize the power of delegating responsibilities. We’re lucky enough in Labs to have a ton of team leads and members who are just as committed to the organization as the directors are. Taking advantage of members’ enthusiasm to spread the work around is a really good way to alleviate stress and take some work off my plate.

An example that I’ve caught myself with a lot is email. We get tons of messages regarding specific products that we run. The directors could worry about all of these, but it’s much more time efficient for us to delegate to team leads regarding their product. It may seem trivial and obvious to do, but I find if kind of difficult to stop myself from digging into research required to answer each email we get.

Communicate

One of the ways to view the director role is as a manager-of-managers. Directors are in charge of keeping up with team leads and making sure they have the resources they need to lead their teams. Something my co-director and I made standard this year was meeting with every team lead individually to get an update on how their team was working throughout the semester.

While these were certainly intended as avenues for us to get feedback, we were sometimes surprised by points that leads brought up in these on-on-ones which seemed relatively urgent, yet were not raised beforehand. It’s easy to wonder exactly what we were doing wrong to not appear as open to talk to. However, I think the answer lies more in human nature. When people have issues, especially those with leadership responsibilities, it’s natural to try and resolve it on your own. It’s up to us as managers to keep in sync with our direct reports and expect that we’ll sometimes need to “pull” information rather than get it “pushed” to us. Sometimes, the birds-eye view that Directors have in the organization can be really useful for working through an issue that a specific team is facing.

The need for communication also manifests itself during the recruitment cycle. Directors manage the recruitment process. But it’s important before the process starts for us to have a good idea of what individual team needs actually are. If it turns out no team really needs backend web developers, we should think about how and if we should recruit for that role.

Final deliberations and team assignments can also get contentious if talent for a particular role is scarce. It’s up to directors to mediate between leads and find a fair solution that everyone can live with.

Think Big Picture

At this point, Penn Labs runs about 10 services across six product teams. Before I was director, I was a team lead for Penn Courses, which solely manages products surrounding course registration. My largest change in thinking going from team lead to director was having to keep the priorities of all the different teams within the organization in mind whenever making a decision. This ties in well with both themes mentioned above. The only way to get the whole picture is to continuously communicate with team leads and general members about where they see their projects going. And the only way to lead a whole organization effectively is to advise leads while trusting them to make the right decisions.

It can be tempting to try and force projects in one direction or another. But I found it important to keep my eyes on the big picture and leave it up to leads to sweat the smaller things. The more minutiae directors manage and worry about, the less context individual leads have to deal with problems that inevitably come up when the directors don’t have the bandwidth to handle it.

Think Long

Maybe not as long as the Long Now Foundation, but thinking on a year-plus time horizon is an important responsibility for directors specifically. Not many software organizations have 100% turnover every three or four years. In industry, it’s common to be able to send a Slack message or walk over to the desk of the person who wrote the code that you’re now tasked with changing. As a student organization, all our members graduate and move on a very regular basis. It’s extremely important that we’re able to transfer knowledge effectively from older members down to younger members.

It’s up to directors to remind developers and leads of this reality. Documentation is always in the back of my mind. Written docs aren’t a panacea, but in my opinion, they can’t hurt when the alternative is trying to decipher code written years before. We’ve also started asking leads in their one-on-ones to make sure they have at least one younger member in mind to mentor and ultimately to replace them.

Recruit with Intention

In my opinion, recruitment is the most important job that directors have. Considering that the 25%/year turn-over rate is a reality that we have to deal with as a student organization, it’s important that we take recruitment seriously and look for prospective members who can contribute a ton to the club, both from a technical and community perspective.

Mentorship and Community

This year, we brought in somewhere around 25 new members in the Fall, doubling the size of the club to 50 members total. All these roles were filling needs that teams saw themselves having, especially with a sizeable senior class graduating in the Spring. While team leads definitely had work for all our new members to do, we didn’t necessarily foresee the reality that if there was a one-to-one ratio between newbies and oldies, then every oldie would need to be a mentor to a newbie.

There are tons of articles online about the difficulty of scaling company culture. Labs is no different here. We pride ourselves on our sense of community, and our rapid growth made it easier for new members to get lost in the fray.

When thinking big, it’s also important to think holistically. There’s obviously the need for technical bandwidth, but it was important for us to balance that with our goal to be an amazing and welcoming community on campus.

Prove your Concept

By the nature of the projects we’re working on, we meet pretty frequently with school administrators. To me, one of the most interesting parts of the job has been getting to see the inner workings of how the University makes decisions. It’s not always the most exciting process, but I’ve learned a lot through being director about how to get your point across to key stakeholders who don’t necessarily need your help as much as you need theirs.

One thing that I’ve found is that an MVP is a powerful tool. One of the team leads on the Penn Mobile project was able to give students the ability to book Wharton study rooms through our app by reverse-engineering the web-based API. This was super brittle: and any change to the first-party website’s flow would break our integration as well. We got in contact with the IT division in charge of developing the room booking system. After we showed them the demand that students had for using our mobile app, they agreed to work with us to build an official API integration. It’s not for certain, but I really believe that showing that our reverse-engineered solution was popular was a big part of us getting our foot in the door in those discussions. A cold email can certainly go a long way with the right recipient, but showing that you’ve already put in sweat to start solving a problem can really get your foot in the door.

Conclusion

Obviously, a lot of what I wrote about above is particularly relevant to leading Penn Labs, and maybe slightly more broadly, to leading a club at a university. But I think I’ve learned some more generally applicable things about leadership along the way.

If I could sum it up in one sentence, I’ve learned that a big part of being a leader is being proactive: reach out to reports to check in; take the initiative to prove out ideas before pitching them to decision makers; think ahead and anticipate challenges to confront them head-on.

I’ve loved being director. It’s helped me give back to an organization that has been a huge part of my college experience. I’ve still got a year left in the club, and I’m excited to see where it all goes from here.

Switching to Emacs

Tue, 31 Mar 2020 21:58:32 GMT

My personal editor journey has been kind of strange. I started off using Sublime Text for most things, and then switched over to JetBrains IDEs on a student license. JetBrains IDEs are pretty amazing — especially when it came to PyCharm’s Django support. One thing I started to worry about recently has been my reliance on a closed-source, and, once I’m out of college, pretty expensive editor. Sure, many employers pay for licenses, but it would be awesome if I could configure an editor that worked well for me and was also completely open-source.

A few months ago, I switched my Python development to VSCode, where I had already been doing frontend JS/TypeScript dev for a few years. It started up faster than PyCharm and had extensions for most things, but I never really felt comfortable. After learning vim keybindings this summer, I would’ve definitely liked to also be using the keyboard more often. Primarily, though, I’d heard really cool things about org-mode, and wanted to try it out. Full support of org-mode is only present in emacs, which in the midst of quarantine would be a bit of excitement to check out.

At the recommendation of a friend, I decided to use doom-emacs instead of starting a config from scratch. I knew nothing about emacs, and after my experience trying to start a vim setup from scratch over this summer, I knew I wouldn’t have the patience to learn everything I needed to about emacs lisp and package management before jumping in. Doom gives opinionated defaults for those comfortable with vim and its keybindings and lots of curated “modules” which wrap around existing emacs plugins. While the existing getting started docs are comprehensive for sure, this article was really a great introduction to get the ball rolling for me.

And honestly, it didn’t take long at all to get a setup where I feel like I can be productive. I make heavy use of goto definition when I’m coding, so I knew I’d need that functionality. Luckily, doom has easy support for the Language Server Protocol, same as VSCode does. All that’s required to enable it in Doom is to add the line (python +lsp) to the doom! block in your .doom.d/init.el file. The Python language server was a bit finnicky, but after setting it up, along with JavaScript, I’m already at a point where I can be productive. I enabled a few more Doom modules, like a terminal emulator (term) and a file explorer (treemacs). Here’s where I’m at, about a day and a half in:

You can see the VSCode inspiration pretty clearly here! Another thing that’s cool about Doom is that in a lot of ways, it’s an entire configuration platform. While it comes with a lot of modules curated by its author, you can easily add modules yourself with your own configuration. It’s something expanded on in this article comparing Doom and Spacemacs, another emacs “distro”. I’m not sure if I’ll continue using Doom long-term as I learn more about emacs and how to configure it. There’s a big possibility I jump to using a more “vanilla” experience, with only the packages I know I use. Only time will tell here!

But one thing I’m really happy about is how much less I’m relying on the mouse than I was in either VSCode or any JetBrains IDE. I’m able to navigate around the entire editor and go through file trees from the keyboard, thanks to great VIM keybindings provided by evil-mode. I haven’t even touched org-mode yet, and I’m already glad I made this transition. I’m looking forward to learning more in the coming weeks!

At Home

Wed, 18 Mar 2020 02:11:32 GMT

Last Wednesday, Penn extended Spring Break by a week and moved classes online for a semester in response to the threat of COVID-19. The same day, the WHO declared the virus’s global spread officially a pandemic. In the days since, I’ve packed up my dorm room and moved home. I’ve tried to see how I could use this time effectively — tackling projects I wouldn’t necessarily have time for in a busy college schedule, and learning new skills. Here’s a good a place as any to jot down some of my ideas, and hopefully it’ll keep me somewhat accountable.

Re-make When2Meet. It’s a super simple and well-liked tool on campus, and I think it would be a cool experience to see how I’d make it, and if I could find any ways to improve on it while keeping its simplicity. I’ve been interested in functional programming lately, so I think I’ll try to tackle this using ReasonML and Elixir.
Learn iOS development. Something I’ve been thinking about for a little bit. I’ve been enjoying frontend development more lately, and it might be cool to add another skill to my toolbox on this front, especially with how many product ideas are mobile-first.
Keep working on Penn Labs products. With students scattered around the globe, tools like Penn Course Alert and Penn Course Plan are going to be even more critical to students. We’re hoping to release a new version of Penn Course Alert with some new and helpful features before Penn’s course registration starts, and that deadline should help keep me motivated and moving.
Improving this site. This only just went up a week ago or so, and right now is still using a pretty vanilla theme. I’m planning on adding some features like a project showcase and spicing up the theme a bit, with some graphics and color.

My First Post

Fri, 28 Feb 2020 03:23:37 GMT

Hello, world! I just bought a domain name today (davi.sh) and am setting up the site. We’ll see how this goes! Hopefully it can get me to start blogging about cool stuff :)

Davis Haupt's Blog

Ideas for an Agent-Oriented Programming Language

Humans should be able to read and edit Markov code

Sum types for fast compiler feedback

Compiler errors as prompts

Token optimized syntax

Interoperability is a non-goal

Conclusion

Footnotes

Reviewing Project Hail Mary by Andy Weir

Reviewing Culpability by Bruce Holsinger

Reviewing Mistborn: The Final Empire by Brandon Sanderson

Reviewing A Tree Grows in Brooklyn by Betty Smith

Reviewing The Strength of the Few by James Islington

Luceum

Obiteum

Res

Splitting the Book

Conclusion

Thinking About Thinking With LLMs

Discourse on the Internet

Step-Functions in Abstractions

Footnotes

Reviewing The Will of the Many by James Islington

Reviewing The Telling by Ursula K. Le Guin

Reviewing The Farthest Shore by Ursula K. Le Guin

Reviewing The Tombs of Atuan by Ursula K. Le Guin

Reviewing A Wizard of Earthsea by Ursula K. Le Guin

Reviewing You Dreamed of Empires by Álvaro Enrigue

Reviewing A Fisherman of the Inland Sea by Ursula K. Le Guin

Reviewing Tress of the Emerald Sea by Brandon Sanderson

Reviewing Zen and the Art of Motorcycle Maintenance by Robert M. Pirsig

Reviewing Abundance by Ezra Klein and Derek Thompson

Reviewing Golden Son by Pierce Brown

Reviewing Red Rising by Pierce Brown

2024 in review

Blog Posts

The Newsletter

What I Learned

Reviewing Tomorrow, and Tomorrow, and Tomorrow by Gabrielle Zevin

[TIL] nix flake update: Updating dependencies of a Nix flake

Configuring VSCode with Nix on macOS

Installing VSCode

Editor Settings and Keybindings

Extensions

Cleaning up lists of attributes with the with clause

Playing nice with Spotlight

Stepping Back

Conclusion

Footnotes

The Spice Didn't Always Flow

Questions

The universe’s best-kept secret

When was spice discovered?

The Theory

Managing dotfiles on macOS with Nix

Install home-manager

Managing .vimrc

Paths

Setting attributes in Nix

Configuring zsh with a home-manager DSL

Configuring git

Installing packages with home-manager vs nix-darwin

Conclusion

Package management on macOS with nix-darwin

Installing Nix

Setting up nix-darwin

Activating our nix-darwin config

Installing your first Nix package

Searching Nixpkgs

Installing packages from Homebrew

Nixpkgs vs Homebrew

Additional configuration

Footnotes

What I Like About Nix

Long-term, shared memory

A complete picture of your environment

Synchronization between machines

Reducing manual toil

Conclusion

Cleaning up lists of attributes with the `with` clause

Install `home-manager`

Managing `.vimrc`

Configuring zsh with a `home-manager` DSL

Installing packages with `home-manager` vs `nix-darwin`

Setting up `nix-darwin`

Activating our `nix-darwin` config

The `FileBuilder`

The `DATABASES` dictionary