Anonymous recursion and the Y combinator

These are notes for myself to help me build intuition. Use at your own peril.

The theory

The identity function $id(x)$ evaluates to its input:

def id(x):
  return x

The function $f(x)$ evaluates not to $x$, but to $f(x)$:

def f(x):
  return f(x)

We say that an expression is self-replicating if it evaluates to itself For example $f(42)$ is a self-replicating expression because $f(42) = f(42)$ (by construction).

In lambda calculus, we cannot name functions, therefore we can't do this thing of evaluating the function f inside of its definition by name. The question is: does there exist a self-replicating expression under this constraint?

The answer is yes.

def f(x):
  return x(x)

$f(f)$ evaluates to $f(f)$ so $f(f)$ is a self-replicating expression, and $f$ does not invoke itself by name so it is a valid function in lambda calculus. $f(f)$ is known as the Ω combinator. Obviously it doesn't make sense to evaluate it in Python, since it's an expression that recurses infinitely. But it is interesting as an object of study, because it's an example of recursion where we haven't called a function from inside itself by name.

A fixed-point combinator is a function $Y(f)$ such that $Y(f) = f(Y(f))$. Again, we could define

def Y(f):
  return f(Y(f))

But this definition would be illegal in lambda calculus because we use $Y$ by name inside its definition. The question is: can we define a function $Y$ such that $Y(f) = f(Y(f))$, under the constraint?

Yes we can. Consider

def Y(f):
  def g(x):
    return f(x(x))
  return g(g)

Then $Y(f) = g(g)$. But evaluating the RHS using the definition of $g$ gives $g(g) = f(g(g))$, and $f(g(g)) = f(Y(f))$. Therefore $Y(f) = f(Y(f))$ (for now we are treating this as merely a mathematical statement, without considering its computational implications). $Y$ is known as the Y combinator.

The general principle at work in both of these examples is that we cannot self-reference a function inside its definition. However, we can work around this by passing a function to itself as a parameter – then, the function can invoke its parameter, i.e. can invoke itself. This gives us the recursion. This is what we do with both $f(f)$ and $g(g)$.

Anonymous recursion without the Y combinator

Consider the traditional recursive factorial definition:

>>> def fac(n):
...     return 1 if n == 0 else n * fac(n-1)
...
>>> fac(5)
120

What's the smallest change we can make to avoid explicit self-reference by name? The factorial function must invoke a function recursively, but it cannot explicitly invoke itself by name. Let's introduce a new parameter $f$ for this function. In order to use the factorial function, we must now pass it to itself so that it knows how to recurse:

>>> def fac(f, n):
...     return 1 if n == 0 else n * f(f, n-1)
...
>>> fac(fac, 5)
120

In lambda calculus all functions are at most one-parameter. We can use currying to express the logic above with only functions of one parameter:

>>> def fac(f):
...     def inner(n):
...         return 1 if n == 0 else n * f(f)(n-1)
...     return inner
...
>>> fac(fac)(5)
120

To demonstrate that this corresponds to a valid expression in lambda calculus, we can rewrite the exact same thing using only single-argument anonymous functions, with a modest loss of readability:

>>> (lambda f: lambda n: (lambda n: 1 if n == 0 else n * f(f)(n-1))(n))(
      lambda f: lambda n: (lambda n: 1 if n == 0 else n * f(f)(n-1))(n))(5)
120

The point here is that functionally, the Y combinator isn't strictly necessary to obtain recursion in lambda calculus. Instead, it's enough to use the principle of passing a function to itself so it can invoke itself.

...so why do we even need the Y combinator?

Our example works, but it’s ugly:

• Firstly, we need to explicitly include the function parameter $f$ and thread it into the recursive call to itself. This threading action is auxiliary to the actual recursive business logic we're encoding, yet it ends up conflated with it. This function-threading is a pattern that needs to happen for every recursive function we write, so if possible, we'd like to abstract it out.
• Secondly, we are forced to state the function fac twice: the first time is when we define it, and the second is the parameter that we pass into the invocation of the definition. This is especially evident in the lambda form, where the full function definition must be written twice. In lambda calculus, everything is written in its full expanded form, so this deficiency is especially bothersome.

The Y combinator, then, can be introduced as a convenient, reusable tool that encapsulates the pattern of passing functions as parameters into themselves, allowing us to write recursive functions more easily and cleanly.

We return to the curried prior example, but attempt to fix the deficiencies above: we'll remove the function threading and the repeated statement of the function.

>>> def fac(f):
...     def inner(n):
...         return 1 if n == 0 else n * f(n-1)
...     return inner
...
>>> fac(5)
<function fac.<locals>.inner at 0x10440dd30>

Now the code is free of the deficiencies above. Unfortunately we can't actually invoke the function in this form.

This is where the Y-combinator comes in.

>>> def Y(f):
...   def g(x):
...     return f(x(x))
...   return g(g)

>>> factorial = Y(fac)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in Y
  File "<stdin>", line 3, in g
  File "<stdin>", line 3, in g
  File "<stdin>", line 3, in g
  [Previous line repeated 995 more times]
RecursionError: maximum recursion depth exceeded

What's happening here is that $Y(f)$ evaluates to $f(Y(f))$ which evaluates to $f(f(Y(f))$ and so on, infinitely. If we were working in a language with lazy evaluation such as Haskell, it would work because it would only evaluate the argument to $Y$ if necessary. But Python evaluates eagerly, so it tries to do the infinite expansion.

We can bolt lazy evaluation onto the Python example by judiciously passing defensive lambdas as function arguments instead of actual values. Our functions can then choose to evaluate or not evaluate their parameters.

>>> def lazy_fac(f):
...     def inner(n):
...         val = 1 if n == 0 else n * f()(n-1)
...         return val
...     return inner
...
>>> def Y(f):
...     def g(x):
...         val = f(lambda: x()(x))
...         return val
...     return g(lambda: g)
...
>>>
>>> fact = Y(lazy_fac)
>>> fact(5)
120

And we're done!

Let's take one last look at how the Y combinator actually does its thing. We'll consider the non-lazy version because it's cleaner – even though it doesn't actually run in Python.

We have a function $fac$ which takes a recursive argument $f$, and returns a function $inner$ that invokes $f$. When we call $fac$ with a realised function $f$, the returned $inner$ is actually a closure with its $f$ bound.

$Y(f)$ evaluates to $f(Y(f))$. So when we call $Y(fac)$, the returned value is $fac(Y(fac))$. In other words, the returned value is the closure $inner$, with its recursive function $f$ bound to $Y(fac)$. But this $Y(fac)$ is also the closure $inner$ with the same bound function. And so on.

This means that the value $Y(fac)$ is the closure $inner$ with its free variable $f$ bound to the closure $inner$. In other words, when $inner$ calls $f(n-1)$, it is in effect just calling $inner(n-1)$.

The Y combinator allows the factorial function to function exactly as it would if it were using named recursion. But instead of encoding the named recursion into the code, it effectively binds the name into the function at runtime using the mechanics of closures, achieving the same thing. Neat!

Acknowledgement goes to this post for some of this intuition.

The Wikipedia page on anonymous recursion does a similar breakdown to here.

Traits of effective knowledge workers

Recently I've been thinking about what it means to be good at this job. In the 14 years I've worked in technology, I've mentored a handful of people and observed many more grow and succeed. Most were smart, knowledgeable, and (sufficiently) experienced for their role. But when it came to actually being effective – delivering consistently, navigating complexity, sustaining momentum – there was a broad range.

I've been trying to pinpoint some of the less obvious qualities that correlate to effectiveness at the job. These traits aren’t innate – they're skills that can be taught, practiced, and refined. Focusing on them directly, outside of general mentorship, can be one of the highest-leverage ways to grow as a developer and to help others grow.

Leverage natural curiosity

A bright new grad recently reflected that, although he was delivering results, he didn’t feel like he was building lasting knowledge. His focus was on completing tasks efficiently – resolving blockers quickly, shipping code, moving forward. When he encountered an issue, he’d find the first viable workaround. Often, this meant copying a solution from elsewhere in the codebase or applying the top Stack Overflow answer. These fixes worked in the moment and allowed him to maintain momentum, but they didn’t deepen his understanding of the systems he was working with.

To his credit, he recognized this. He saw that while he was progressing in his day-to-day tasks, he wasn’t learning much about how things actually worked. He wasn’t building a holistic picture of the systems or the broader technical context around his work.

This pattern is common, especially for early-career developers in fast-paced environments. And in urgent situations – production outages, critical deadlines – taking the fastest path is appropriate. But when this approach becomes the default, it can quietly undermine long-term growth. It prioritizes output over comprehension, speed over depth.

The alternative is to adopt a mindset of curiosity, and to treat each obstacle as a learning opportunity. When you solve a problem with a snippet from Stack Overflow, take a few extra minutes to understand the unfamiliar concepts mentioned in the post. When editing code, take time to examine the surrounding logic – why certain parameters exist, how they interact with the rest of the system. Develop the habit of tracing a problem back to its source, not just resolving the symptom.

The cartography analogy

A useful analogy is that of a cartographer. At the beginning of a career or a new position, your mental map of the system is blank. Each task you’re assigned is a destination: get here. You have two ways to approach that journey.

Option 1 is to head directly toward the target. When you encounter a barrier – a complex code path, a failing test, a mysterious dependency – you do just enough to get past it and keep moving. You arrive, but with little understanding of the landscape you crossed.

Option 2 is to slow down and map the terrain as you go. When you see a barrier, you take time to understand what it is, where it comes from, and how it connects to other parts of the system. You document what you learn. You may even build tools or abstractions to make future crossings easier. It’s slower in the short term – but over time, it pays off. You begin to anticipate obstacles. You navigate with confidence. And importantly, the knowledge you accumulate becomes shareable. You help others avoid the same pitfalls and take more efficient routes.

Option 1 emphasizes delivery. Option 2 builds systems knowledge, organizational context, and reusable understanding. Striking the right balance between the two is essential.

For most developers, a reasonable default is a 50/50 split between execution mode and exploration mode. You could probably go as far as 70/30 in either direction. Personally, even after nearly a decade at my company, I still lean toward exploration. I spend more of my time in a mode of curiosity than a mode of execution: understanding systems, following leads, documenting knowledge, and preparing for future complexity. That investment helps me to resolve issues that span multiple components, to identify subtle risks that lie beneath the surface, and to guide others through unfamiliar systems.

Most early-career developers focus far too much on immediate delivery. If that’s you, here's a heuristic: for every unit of work you deliver, pair it with a unit of exploration. When you add a method to an API, read and digest the rest of the API. When you find a bug, read the version control history and understand how it got there. When you're improving the performance of a system, learn how the upstream and downstream systems work, and map out their interactions with your system.

Effective developers have a natural curiosity and wield it with deliberation. They draw the map as they go. They invest intentionally into building a body of understanding, and the investment pays dividends down the line.

Learn to learn

Don't consume knowledge, build mental models. 

I think this is the single most useful piece of advice I have.

When faced with something you don’t understand, one approach is to start reading the code and docs and gradually build a picture from the ground up. A more effective approach is to first construct a mental model of how the system could or should work, then learn the system by cross-referencing reality against your imagined version. Each time your model is contradicted or clarified, refine it. By the end, your model should closely match the actual system.

This approach is unreasonably effective. Bear with me.

Everyone knows it’s easier to remember things you created yourself than things you merely consumed. Somehow, the ideas you generate yourself seem to lodge deeper in memory. This technique is a pedagogical hack: it engages the “origination” part of the brain during learning. Building the initial mental model is a creative act, and so is refining it in light of new information. Instead of passively absorbing facts, you’re actively shaping and adjusting a construct in your mind.

A useful side effect of this method is that, over time, it lets you absorb new systems and ideas very quickly. As you gain experience and internalise the idioms of your domain, your initial models will more often match the reality from the outset. Learning becomes an exercise in spotting the handful of differences. It’s like already having most of the jigsaw puzzle in your head – learning a new system becomes a matter of dropping in a few missing pieces, and maybe correcting a couple.

You’ll find yourself thinking, “OK, yep – this module handles query logic, this class manages the transaction log, and here’s the sequence of transactions – yep, makes sense.”

This technique works for understanding software systems, but it also applies to anything intellectual: mathematics, organisational structures, the stock market.

Use the Feynman technique to learn brand-new things.

What do you do if you want to learn something, but you don't even have the building blocks yet to build an initial mental model? Use the Feynman technique.

The Feynman technique, supposedly popularised by physicist Richard Feynman, is in my opinion the most effective technique for "bottom-up" learning. It essentially puts into practice the notion that "the best way to understand something is to teach it".

I think the Feynman technique is effective for precisely the same reason as the mental model-building technique: by "teaching" something to yourself, you transform a mental consumption activity into a mental construction activity.

The link above gives a far better outline of the technique than I ever could. All I’ll add is this: I spent six years completing a part-time pure mathematics degree while working a demanding full-time job, and I couldn’t have done it without the Feynman technique.

Diagnose effectively

Much of our job consists of diagnosing problems: debugging new code, tracking down the causes of user errors, figuring out why the organisation isn’t operating as effectively as it could. You almost certainly spend more time diagnosing problems than writing code.

Effective, fast diagnosis is a skill worth mastering. Here are some techniques.

Get the experiment time down. 

Once in a blue moon, you’ll pinpoint a bug just by reading code. Most of the time, though, diagnosis involves repetition and interaction: adding print statements or breakpoints, interacting with the system in a REPL, running a test suite. Each of these is an experiment.

The highest-leverage move is to get the experiment time down before you start. You’ll probably need to rerun the experiment 5 or 10 or 20 times. Each minute you shave off pays for itself. Fast iteration maintains momentum, keeps the problem details in working memory, and helps you stay focused on the goal.

Narrow the search space thoughtfully. 

At the start of diagnosis, you usually have no idea where the problem lies. Then, through a mix of thinking and testing, you gradually narrow the possibilities – to a system, to a module, to a block of code. Eventually, the bug is just sitting there in front of you.

This narrowing can be done naively or thoughtfully. Aim for the latter. The goal of each step is to rule out as much of the search space as possible, regardless of the experiment outcome. I recommend a kind of generalised binary search: if you know the bug is somewhere in a system, mentally partition it in two, and design a test that will tell you which half it’s in. Then repeat on that half.

It takes time up front to conceptualise the system and design the right experiment – but pays off by reducing the number of tests you need, and increasing the information you get from each one.

Each time you rerun the experiment, limit it to the smallest space that definitely contains the problem. If you’ve narrowed it to a subcomponent, run the next test just on that. This keeps your focus sharp and keeps experiment time down. Don’t rerun your whole app end-to-end to debug a bug in email address validation.

Think about the edges between

In most organisations, some of the most impactful work exists in the "edges" between systems. What do I mean by this?

In a technological organisation, you and your team will usually have some remit in the form of one or more systems. If you work on the database team, this remit is the database server and client. If you work on the ML team, the remit is the ML infrastructure. If you work on a research team, the remit might be the research tooling.

Systems in an organisation can be modelled as a graph. The systems themselves are the vertices, and the edges represent interactions or interfaces between systems.

Most of the time, you'll be focusing on these systems themselves. You'll consider what features they need, what are their deficiencies of their current features, what can you make faster, and so on. Organisational gravity pulls your attention to the vertices of this graph; the vertices are probably what your team is named after, and therefore what you think you should be working on. You could spend your whole career working on the vertices.

My thesis is that, in most organisations, some of the most impactful work exists in the edges between systems. Some reasons I believe this:

• Teams orient themselves around systems, so a disproportionate amount of organisational attention goes toward the vertices. Relatively little attention is paid to the interactions between systems.
• There are more edges than vertices in this graph, so the edges are more likely to be under-attended and under-explored.
• Each vertex evolves relatively independently of its adjacent vertices. A team builds a new feature in their system, but those working on adjacent systems don't notice the new feature, or notice it but don't have the time or inclination to leverage it.
• Thinking about edges gives you a higher-level view of what's going on. You're not thinking about one system, but about multiple systems and their interactions.

Let me be more concrete. In my organisation, if I consider any particular system in isolation – say, a database – it looks reasonable. Its features seem sensible. I can't find any low-hanging fruit for optimisation. Obvious bugs have been ironed out. It is essentially a self-contained, self-consistent design.

But then I pick an adjacent system that acts as a client for the database. When I consider that system in isolation, it also looks fine. But when I look at how the client uses the database, I often immediately notice some gaping issues. Maybe the client's query pattern doesn't make sense for the database's revamped implementation. Maybe the client is redundantly caching query results that the database is already caching. Maybe the client is missing out on a recent optimisation that was implemented within the database.

In my experience, almost every time I delve into the edge between two systems, I find glaring problems. And these problems often seem more severe than the problems within any particular system. I'd go as far as to say that, in sufficiently complex webs of interconnected systems, inefficiencies at interaction points tend to grow to dominate inefficiencies within any particular system.

This is such an easy way to have impact within an organisation. I encourage everyone around me to spend more time thinking about the edges.

I also suspect the same principle can apply to other disciplines of knowledge work. For example, in mathematics research, many significant results come from techniques in one area being effectively applied to an entirely disparate area. In this way, focusing on an "edge" between two areas yields results.

Move fast and break (some) things

As a culture, we've moved past "move fast and break things". But Zuckerberg's infamous words hold a degree of truth.

Any time you break something, whether it be a silly CI pipeline or your live production system, it has a cost – usually measurable in terms of money, person-hours wasted, or people woken up in the middle of the night. I make these claims: 

1. the cost of breaking something in a given system is often hard to predict. Calibrating this sense takes time and experience with the system;
2. as a result, many people (especially in their early career) err on the side of significantly over-estimating the cost of breakages, and place disproportionate weight on the perceived emotional consequences ("people will be angry with me");
3. this aversion to potentially breaking things can slow down one's progression, in terms of learning and delivery. 

Here are some disorganised thoughts:

• Breakings things can cost you in trust. You will lose more trust if you break something in a predictable way because you didn't follow a known process, e.g. you made a change without following the checklist. You won't much trust if you break something in a novel way, e.g. you were tinkering with a new system. You can even gain trust by breaking something in a really interesting way, because it demonstrates curiosity.
• If something breaks repeatedly in the same way, it's a process problem rather than a people problem, and you can regain the lost trust (and more!) by fixing the process.
• If you break something, immediately take responsibility and fix it. Generally, people care about breakages inasmuch as dealing with those breakages costs them time and energy (or money, of course). If you break something and fix it yourself, and it doesn't have a significant monetary impact, people won't mind.
• Breaking something fragile, then making it robust in the process of fixing it, is usually more valuable than not breaking it in the first place. 
• The cost of a breakage is sort of proportional to how long it goes unnoticed for. The worst sorts of breakages are where something breaks in a subtle way and costs the business money over a long time.
• The newer you are to a team or system, the more leeway is extended to you to break things. Take the opportunity of being new to experiment. If you break things, it's probably a process problem more than a people problem, and improving your team's processes will help others down the line.

Of course, all of this depends on the culture of your team and workplace. So take the temperature of that before you do anything else.

But why must anything break at all? As with many things in life, the strategy that optimises for success on a long time horizon may differ from the strategy on a short horizon. Let's take "success" to mean that you finish the work you're assigned, and you grow your knowledge and experience in an area.

Some newer developer err on the side of being too careful, and stepping gingerly in their work. The reality is that most successful projects reach their endpoints with a bunch of zig-zagging and experimentation along the way. At the outset of a project, one of your goals should be to discover the unknown unknowns. There will be things that don't work – you're better off finding them early by breaking things, rather than theorising really hard about how to do things as safely as possible.

On this subject one of the most valuable approaches you can take when building something new is to invest time upfront to build a safety net that will contain the damage of future breakages. For example, when building a new system from scratch, design at the very start a robust staging setup so that you can flail around wildly and break things in the test system without affecting prod data.

In terms of your own learning, becoming proficient in a new system or area is like learning to ride a bike. If you're terrified of ever falling over, it will take you a long time to get good. You learn the boundaries of a system by overstepping them.

As a final note,. Lessons that you learn through personal failure will stick with you much more than those you learn by reading the warning label on the bottle. 

Do "Do things properly" properly

Sometimes it feels like I spend most of my day deciding how well something is worth doing. By “well,” I mean: the right blend of simplicity, comprehensibility, efficiency, maintainability, scalability, and all the other -ilities.

We have finite time and too many things we want to do. After building features, fighting fires, deploying releases, and running interviews, we’re left with a narrow sliver of time and attention to invest in technical excellence. We can’t do everything perfectly. So we triage. Prioritising deliberately and aggressively is one of the highest-leverage things a developer can do.

An effective developer avoids aesthetic or “vibe-based” judgments when making these prioritisation calls. Instead, they assess trade-offs in terms of concrete benefits and risks.

When examined this way, some “obviously necessary” proposals often turn out to have marginal value. And some seemingly harmless hacks reveal themselves to be more damaging than they first appear.

Some engineers fall into the trap of doing everything “the right way.” In my view, very few things need to be done 100% right. In fact, an insistence on perfection can erode the trust placed in you – it can make it seem like you prioritise technical indulgence over the needs of the business. A better approach is to build goodwill by working efficiently and pragmatically most of the time, and then cash in that goodwill to do things properly when it really matters.

Prioritisation often involves talking to others. In these conversations, it’s important to speak in concrete, factual terms – not in emotional appeals. Developers are flawed and fickle, but we respond well to structured, objective reasoning. Leadership may tune out vague complaints about technical debt, but they understand structural risk.

It takes time to learn the language for these discussions. A few useful resources:

• Ben Rady’s CRUFT model (“complexity, risk, use, feedback, team”), which he covers on his blog and in a Two's Complement podcast episode.

• An earlier episode of the same podcast with Rady and Matt Godbolt, where they discuss other frameworks for framing technical debt.

• A piece I wrote recently on this blog, about modelling liabilities in software. The section on scale and longevity is especially relevant here – an effective developer understands that the severity of an imperfection depends on both its blast radius and how deeply embedded it is in the system.

Know how to use the computer

Being an effective developer isn’t just about writing good code. It’s about moving fluidly through your tools. Typing speed, keyboard shortcuts, terminal fluency, and precise navigation aren’t just nice-to-haves. They directly impact your ability to get work done. Do not underestimate this!

The common rebuttal is that “typing speed doesn’t matter – most developers only write a few lines of code per day.” That’s true. Raw typing output isn’t the bottleneck. But most of a developer’s time isn’t spent in flow-state, writing production-ready code. It’s spent interacting with the computer in a thousand small ways: jumping between files, searching through unfamiliar code, inspecting logs, testing hypotheses, debugging, running commands, pulling down changes, writing documentation, managing version control, sending messages, and searching the web. These tasks are interleaved with actual coding – and how efficiently you move through them has a real impact on how quickly and effectively you work.

If you're skeptical, spend some time pairing with someone who lacks computer fluency. A slow typist, someone unfamiliar with keyboard shortcuts, someone who struggles to navigate the terminal or doesn’t know basic command-line tools. You’ll notice that their pace drags – each action takes longer, context-switching is more cumbersome, errors take more time to recover from, and momentum is harder to sustain. The inefficiency adds up.

Most junior developers I meet are too slow. Not because they lack talent, but because they haven’t practiced intentionally. Learn to type properly. Learn the keyboard shortcuts. Learn the suite of tools at your disposal. An intern I mentored told me years later that this advice fundamentally changed how he worked for the better – and that now he’s the one frustrated when others fall behind.

Fluency compounds. When the friction between thought and action is low, you can think more clearly, move more freely, and work more effectively.

Modelling software liabilities via state, coupling, complexity, and code size

These are notes from a tech talk I gave at work in late 2024. I've removed some details specific to the company.

This talk is completely indebted to this this life-changing Hacker News comment. Also thanks to the author of this Medium post, who independently came across that comment and wrote their own interpretation.

What makes "bad design" bad?

Over years of working with code, developers develop intuitions about what makes for "good" or "bad" software design. These intuitions are aesthetic and often visceral – you might hear phrases like "elegant", "clean", "feels wrong", or "the genius who thought this was a good idea should be retroactively aborted". At the same time they can appear hazy and contradictory ("don't repeat yourself! Except here, where copy-pasting is clearly better"), and can be hard to articulate.

In this talk I'll describe a model for thinking about software design that tries to capture some of this intuition. The model proposes that most "bad design" can be attributed to the presence of four distinct liabilities, and it gives some simple, practical rules to juggle these liabilities to make your software more maintainable and evolvable. Plenty of examples throughout!

If it goes well, you'll understand when to ignore the "DRY" rule, when to hand-roll your own CSV parser, and when to use inheritance – and you'll be able to explain why.

This talk might be useful for those who'd like to develop their taste for design. It might also be useful for those who have already developed this taste, but struggle to articulate it during code reviews or mentorship.

---

• As a programmer, you learn from making mistakes. e.g.

• You learn to use functions because when you didn't, you ended up with spaghetti mess

• You learn to use global variables because when you didn't, it was impossible to reason about each  piece of code

• You learn to write backwards compatible code because you broke prod

• There are patterns in the mistakes you make, and patterns in how you solve them

• Over time you develop intuitions based on these patterns you notice.

• These intuitions are very powerful because they let you anticipate situations where you need to make an important choice, and they nudge you towards the correct choice

• These intuitions are also mostly universal, surprisingly - most experienced developers made similar mistakes, so they learned the same lessons and share most of the same intuitions.

• there are exceptions. e.g. functional programmers vs imperative will have some different intuitions.

• It's useful to give words to these intuitions for two main reasons:

• To be able to have meaningful discussions and justify design decisions, trade-offs, etc. to others

• To help shape the still-forming intuitions of less experienced developers

• Lots of examples of models/heuristics that try to capture this intuition: YAGNI, CUPID, DRY, SOLID, KISS, …

• But some of these are vague or easy to misunderstand

• This talk is about a model that I believe works better than all of these. It's more general/flexible, easier to use, more prescriptive, harder to misunderstand.

Four liabilities

• The only system with no liabilities is the one that doesn't exist.

• As soon as you start designing a system / writing code, you immediately start dealing with liabilities:

• You now have code, and the code takes time to read, understand, to learn (for others), to maintain.

• You've made some language choices, which impacts subsequent maintainability and expressibility and features etc.

• You pick a DB, which needs to support all your data and has to be backed up and versioned and sometimes migrated etc.

• You pick dependencies, which need to be managed.

• You need to version and release it, so have to think about compatibility.

• You make it a distributed system, with all the issues that come with that.

• Liabilities are not bad things. But if not properly controlled and managed, they will lead to problems.

• Software design is about taking on the right liabilities for your task, and reducing the remaining liabilities or trading them off each other.

The model

• The model I will present asserts that there are essentially 4 categories of liabilities that are most important to the software design process.

• The model asserts that there is a strict hierarchy of risk/danger to these liabilities. The first liability is much worse than the others, the second liability is worse than the third and fourth, and so on.

• Therefore you should prioritise reducing the first liability; only then should you reduce the second; and so on.

Liability 1: State

• State is when the behaviour of your code depends on what it's done in the past.

• You need to think about configurations of state to understand code.

• State is an incubator for bugs.

• State is enemy of concurrency

• Makes testing harder

• Makes debugging harder

• You learn early on that state is a bad thing. in CS101 you hopefully learned not to use mutable global variables. 

• Code smell suggesting that you have hidden state: methods on a class need to be called in a particular order, or you have a post-init "config" method.

• Another code smell: caching. Caching is a form of state maintanence that can affect performance as well as correctness.

Liability 2: Coupling

• Coupling is when you have two components such that changes to one induces changes to the other.

• If you have a big network of coupled components, it is very hard to make a change to a component – or even to reason about a change – because of the knock-on required changes. "Change amplification"

• Vicious cycle: highly-coupled systems are harder to refactor. So the more coupled a system is, the harder it is to uncouple. Eventually it becomes sort of frozen.

• Avoid coupling by having clear stable interfaces between components

• Coupling exists between functions, classes, modules, services, repositories

Liability 3: Code complexity

• Complexity here means anything that makes the process of reading code more difficult for the code reader.

• Including use of advanced language features: reflection, regular expressions, higher-order functions, templates, decorators, generators, call-backs, context managers, async, dependency injection...

• Complexity is cultural and dependent on the dominating programming paradigm. E.g. monads would probably be considered complex in C but not in Haskell.

• Incudes bad naming, lack of comments

• Cyclomatic complexity: number of paths through a piece of code. Functions with lots of boolean params, nested if-statements, long functions

Liability 4: Code length

• Code itself is a liability, regardless of its characteristics.

• All things being equal, prefer to solve your problem with a shorter piece of code than a longer one. All things being equal.

• BUT: you shouldn't prefer the shorter solution if the code is more complex, if the system is more coupled, or if it's stateful.

How to use the model

It wouldn't be interesting to say that you should reduce coupling, or reduce complexity by writing simpler code. 

The model is more opinionated than this.

The model makes 3 controversial statements:

1. You should be willing to write more verbose code if it reduces the code complexity, coupling, or statefulness.

2. You should be willing to add complexity at the code level through, say, advanced language features, or writing a long function with 20 parameters, if it decrease the coupling or statefulness in the design of the system.
3. You should be willing to increase the coupling in your system if it reduces the amount of state.

Cross-cutting factors

There are two cross-cutting factors that can make a liability much more tolerable or much riskier:

Factor 1: Scale

• The scale at which a liability manifests multiplies the impact of a liability.
• State might not be too bad if it's carefully contained to one component. But if your entire app runs on shared global state, you're gonna have a bad time.
• You might prefer to have a self-contained instance of state change, to something that adds complexity to every function of your app. 

Factor 2: Longevity

• The duration for which a liability is present multiplies the impact of a liability.
• It might be fine to introduce a disgusting bit of state in a midnight hotfix of production. You'll fix it properly the next day.
• But if state is at the core of the design of a new system, and will be hard to remove later, tread very carefully.
• Same goes for the other liabilities.

Examples

DRY

• Inexperienced programmers hear "don't repeat yourself". So as soon as they see two pieces of code that look the same, they factor out the repeated logic into a function.
• But two pieces of code can look the same without expressing the same infomation. They can look the same by coincidence, or because they encode logic that is the same right now but will differ soon.
• Code length has been reduced, but now there is coupling between the two callsites and the function.
• So what happens when the two pieces of code need to differ? The inexperienced developer introduces a boolean flag to the function! And then needs to update both callsites! The flag introduces code complexity to the function, and the extra coupling leads to annoying changes.
• An experienced developer knows that copy-pasting is frequently the right thing to do – it incurs an increase in code length to avoid increasing complexity, coupling, or state.

Inheritance

• The advice "composition over inheritance" is just a special case of this more general theory.
• Inheritance is usually justified as a way to reduce code size, but it leads to extra coupling (between the classes in the hierarchy) and extra complexity for the code reader.

Information-passing between processes

• Imagine you have some structured data to send from a sender process to a receiver process, and you must choose between two options: (1) the sender sends it as encoded plaintext in an environment variable, or (2) the sender writes it into Redis and the receiver reads it.
• Cons of using environment variable: it adds complexity. It's untyped, unstructured data, you need to decide what to name it and where to store the name, and you're exposed to OS-level constraints (e.g. limits on env variable length). But there's no state to worry about.
• Writing into Redis might feel more robust, but it exposes you to state-related issues:
• What if the sender runs twice in a row?
• What if there are two instances of the sender?
• What if the sender crashes and needs to be retried?
• What cleans up Redis?
• To me, these state-related consequences feel much more dangerous than the complexity-related consequences of the enviroment variable. So in the abscence of other information, I would pick the env var approach. What do you think?
• Obviously neither option is ideal. But in this job we often must pick between two evils. The model gives us a framework for this.

Testing command-line flags

• Suppose you're writing an integration test in Python's unittest framework to check that your app respects certain command-line flags.
• Option 1: in setUp(), set the global state of command-line flags. In tearDown(), undo it.
• Option 2: in each test function, set/unset the global flag state at the start/end respectively.
• Option 3: monkey-patch the flag state in the code under test using e.g. mock.patch.
• Option 4: rewrite the code-under-test so that it consumes a flags/argparse object, and then inject this dependency in the test code.
• The model would argue that these options are listed in order of worst to best. Can you see why? Do you agree?

List comprehensions

• The model is also useful for small-scale design nits.
• Do you prefer an explicit for-loop x=[]; for ...: x.append(...) or a list comprehension x = [...]? How would you analyse each one in terms of state, complexity, and code length?