“Thomas, who was the father of both Eleanor and Richard, had a grandson through Eleanor named William, who in turn had two daughters, Charlotte and Margaret, while Richard had a son named James, who was the father of both Beatrice and Arthur, the latter of whom married William’s daughter Charlotte.”

Most of us have a hard time to get such a sentence.
What about the diagram below?

Easier right? By switching the representation, we ends up with something easier to understand. Here relationship is easier to grasp with a graph rather than a sequence of words.

Following the same idea, which formulation do you prefer?

• “For any given value of x, the corresponding value of y is found by adding 2 to x.”

• “The value of y is always 2 more than the value of x.”

• y = x + 2

Kids often ask why they have to learn mathematics. What’s the point of learning times tables if the calculator can do it faster and better?
As we grow, we understand two things about mathematics:

• It brings mental models and patterns that help resolve anything

• Its semantics and syntax are top-notch. One of the best set of representation humans have ever created.

The mathematical expression isn’t just shorter - it’s transformative. It allows manipulation, extension, and application impossible with prose descriptions.

The powers of cognition come from abstraction and representation: the ability to represent perceptions, experiences, and thoughts in some medium other than that in which they have occured, abstracted away from irrelevant details. This is the essence of intelligence, for if the representation and the processes are just right, the new experiences, insights, and creations can emerge.
The important point is that we can make marks or symbols that represent something else and the do our reasoning by using those marks

Natural Language is a Terrible Spec Language

Natural language doesn’t seem so ambiguous in our daily life:

[...] the ambiguity almost always results from the analysis of single, isolated sentences: in real situations, where several interacting people deal with real events, the sentences usually have only one meaningful interpretation. Actually, even when communications are ambigous, they are usually not perceived as such by either speaker or listener, even though both may have different interpretations of the meaning.

It is the lack of perception of ambiguity that is important, and it derives from the communicative, social nature of language, something that is entirely missed when the language is studied as isolated, “simplified” printed sentences or utterances, completely abstracted from the real, social settings.1

Replace “printed sentences” with “tokens” and you have modern LLM training.

Also, when an LLM generates a response to an ambiguous prompt, it confidently selects one interpretation and proceeds - with no awareness that alternatives exist. It doesn’t “perceive” ambiguity because it has no mechanism for recognizing that its statistical resolution of a sentence is just one of several possible interpretations.

Humans don’t notice ambiguity because context resolves it most of the time.

And so while natural language still remains our main interface, the way we think and build creative solution is usually embedded into something deeper.

Math Driven Spec

I’ve been exploring lately how instead of creating a spec the size of a technical O’Reilly book and creating a Spec Observer skill, we could use a mathematical representation for defining our spec.

The intuition here is that the shift happening from writing code to writing specs will expose how bad natural language is at precision.

That to extract a consistent, lean, and so maintenable and evolutive software specification, math can play a better role than natural language.

And so I’ve created this math spec-driven skill. It treats mathematics as a specification method, using sets, functions, relations, invariants, and explicit edge cases to define what a system is and what it must do.
It enforces three principles:

• Explicitness: every assumption is stated

• Composability: parts can be specified independently, then combined safely

• Falsifiability: claims are written so they can be proven or disproven

For example building a TicTacToe game using this spec methodology looks like this:

Board := P = {1..9} → {X, O, ∅}; initially ∀p ∈ P: P(p) = ∅
Players := T = {X, O}; turn(n) = X if n is odd, O if n is even
Move := move(p, t) requires P(p) = ∅ ∧ turn(n) = t; sets P(p) ← t
Lines := L = {{1,2,3},{4,5,6},{7,8,9},{1,4,7},{2,5,8},{3,6,9},{1,5,9},{3,5,7}} Win := win(t) ⟺ ∃ℓ ∈ L : ∀p ∈ ℓ, P(p) = t
Draw := draw ⟺ (∀p ∈ P: P(p) ≠ ∅) ∧ ¬win(X) ∧ ¬win(O)
Terminal := game ends when win(X) ∨ win(O) ∨ draw
Invariant := |{p : P(p)=X}| − |{p : P(p)=O}| ∈ {0, 1}

It can be frightening for most of us I do agree. But the LLM can easily return a translation for this:

The board is a 3×3 grid of 9 cells, each either X, O, or empty. All cells start empty.

Two players — X and O — alternate turns, with X always going first. On your turn, you place your mark in any empty cell.

A player wins if they fill an entire row, column, or diagonal (there are 8 such lines). The game is a draw if every cell is filled and nobody has won. The game ends the moment someone wins or the board is full.

Invariant: at any point, the number of X’s on the board is either equal to or exactly one more than the number of O’s — this guarantees proper alternation.

I did try that skills on several projects, and even if it might be useless for agents2, it's valuable for me to iterate on the spec. I feel it easier to iterate on next steps when I have such rigorous foundation.

At the end of the day, the context we give to agents should be multi modal:

• text for intents, examples, problem-context

• diagram for relationship between entities

• and math for explicitness, composability, and falsifiability3

This way we can use the full value of different representation and make things easier to maintain for both agents and humans.

The Maxwell Equations For Spec?

If we push the idea further here we could imagine finding the ultimate substrat of mathematical representation to describe any SaaS, any software.
That’s sounds like an utopia by the time I write this - but what we have achieved with frameworks and libraries in code really seem similar to a common set of equation expressed into a single formalism.

To build complex, maintenable and malleable software - ultimately to build at a higher level of abstraction - we somehow need the power that come from that level of representation.

In that scheme, we can see math as the source of truth and code or natural language as a projections.

Math spec. come also with a crazy idea here: what if you apply the math themselves to create software and test it after the fact. We can imagine a similar build as SMT-solver or constraint optimization methodology, basically:

• math spec = the score function - the gradient field that defines what “valid” looks like

• code = a sample from the distribution of valid programs

• generation process = iterative refinement where an agent proposes code and the spec’s invariants pull it toward correctness, like a potential field where the math model spec. allow to create the whole object in one-shot - swarm of agent iterating toward the optimal solution.

I’m not an expert in LLM architecture and optimization process, but that also resonate with diffusion models where you start from noise and iteratively denoise toward a sample that satisfies the learned distribution.

The Gradient Points To People

If Yann LeCun is right - and language really is just “lossy compression” of reality - then every spec we write in prose is already degraded before an agent touches it. I think he’s right - and the implication for software is direct. Every time we write a spec in natural language, we lose information. Every time an agent reads that spec, it loses more. The losses compound. Math doesn’t fix everything, but it preserves structure in a way prose simply cannot.

“You want to write the least code as possible” has been the mantra for years. Now we’re getting there. But the bottleneck doesn’t disappear, it moves. It moves from code to spec. From spec to intent. From intent to understanding the actual problem.

Which means, paradoxically, the more formal and mathematical our specifications get, the more human the hard part becomes. Creating good software means having good specs. Having good specs means having context and taste. And context and taste come from one place only: talking to people, understanding what they need, sitting with their problems long enough to see the shape of them. Communication is the job.

Natural language has been grown by humans to share cognition and sentiments. These are ultimately the two things why people work with you. The math gives us precision. The conversation gives us direction. We need both - and knowing when to switch between them might be the real skill now4.

Over the last few years, a subject has come up: we don’t hire juniors or interns anymore1. LLM coding agents are the new juniors, they say.

You’ve probably heard the pitch by now. The sales guys and the tech evangelists - they all love to say it:

“AI is like having an army of interns at your disposal.”

But I don’t think we’ve ever had a problem where we thought, “Man, you know what we need right now? An army of interns. Just like, hundreds of people who don’t really know what they’re doing. That would REALLY help.”

The problem isn’t capability or capacity - it’s trust.

How do you trust a junior developer? Over time, through practice and teaching. We forget that trust is a resource you have to build. It’s slow. It’s expensive. It’s exactly like working with a junior developer. You don’t just hand them the keys to production on day one (or at least, I hope you don’t). You teach them. You review their PRs. You watch them break things, fix things, and eventually - slowly - you learn to trust their judgment.

I actually feel more and more the same way working with LLM coding agents for serious work. When it’s not a greenfield project, I cautiously give them the proper scope, take the time to review their work, give them only the easy stuff. The same work I’d give to an intern.

So why are we so convinced we should hire the artificial ones?

I mean, yes, the artificial ones are trained. It’s expensive to train them. Our whole economy could crash because of it. And yes, we trust them - at least to some extent.
Since trust is something you build over time, the more we use these new orbs, the more we’ll trust them. I already feel pretty confident run a Codex Agent, use Cursor for any kind of serious tasks or let my Comet Perplexity agent take control of my browser to move a Google Doc into a Google Slide.

But I still feel the need to train someone. Not just for empathy or self-ego, but to see where I don’t understand things, where I have holes in my reasoning - and ultimately, to pass on a culture.
The instinct with investing in an intern is to curate a culture, not to teach, not primarily.

Because if we don’t hire juniors anymore2, maybe we shouldn’t hire the artificial ones either... those don’t really get on the culture stuff.

The more mature teams have already figured out the new cultural shift: you still hire juniors and interns, but now you teach them to use AI as part of their work. As part of the craft itself. Of the culture we’re building right now. It really feels like we are building from the ground up - not just adding tech here.

Because if nobody teaches new people how to use AI properly, eventually nobody will know how to use it at all.

Let’s not shy away from developing pro tools, but encourage and teach them.

📡 Expected Contents

When natural language feels unnatural

Language is flexible, but flexible does not mean precise. Most people who use LLMs regularly have experienced a variation of the following:

• You ask for a small revision, and the assistant rewrites the entire document.

• You request a color change in a codebase, and the tool “helpfully” restructures half your components.

• You ask for a tighter paragraph, and it returns something in a completely different voice.

The problem isn’t that the model is incapable of understanding the request. The problem is that the interface is too ambiguous for the level of precision the user expects. Language carries nuance, but it also carries room for misinterpretation. The more specific the adjustment, the more unnatural it feels to communicate it through free-form text.

I hope (and I am sure) we will see not only a text box. We actually see the premise: we use /commands or skills more and more. We create graphs of prompts or agents. We speak to the computer.

At first I was expecting new components to appear, new things in the UI itself. But the more I see where we go so far, the more I see it’s more of a multimodal evolution. It resonates a lot with DynamicLand.

👉 Read the blog

A Human on a Bycicle is Among the Most Efficient Forms of Travel

Bikes allow us terrestrial folk to be more like fish. Wheels, a simple machine, let us coast without putting in power by pedaling, and the rigid frame supports the sitting rider against gravity. “They turn humans into this hyperefficient terrestrial locomotor because they make being on land more like swimming,” says Tyson Hedrick, a comparative physiologist at the University of North Carolina at Chapel Hill. The main drawback is our clunky human shape; bicyclists aren’t streamlined like bluefin tuna, so they must overcome more drag. Hedrick calculates that bicycles with an aerodynamic shell, called velomobiles, can let humans move with even more aquatic efficiency.

👉 Read the article

Using power tools

Knowing when to use which tool is a mark of maturity. The master woodworker isn’t defined by owning the most powerful equipment but by judgment. They know when to reach for the table saw, when to use a hand plane, and when to stop cutting altogether.

Proof of Concept

Power tools

In art school, we had a curriculum that forced us to have a well-rounded perspective of different crafts outside of our major—fostering the archetype of being a generalist. Even though I majored in Drawing and Painting, I studied several courses in Photography, Ceramics, Sculpture, Jewelry/Metalsmithing, and Wood Design…

Read more

4 months ago · 15 likes · 3 comments · David Hoang

The Programmer Identity Crisis

I think we’re wrong about vibe coding. It’s not a regression from expert developer. And maybe it’s a real win: flow state.

When I vibe code, I’m in flow. I talk to the computer or write energetically. When I am building an agent, and when that agent is working, I start a new one. I stay in the state. I don’t wait for CI or for the machine to build. I keep working and the agents work for me. I stay in flow state.

👉 Read the blog

📰 The Blog Post

No deep dive blog post this time; quite busy recently. I will see what can be put on papers later this month.

🎨 Beyond The Bracket

Before you ask AI another dumb question, watch this:

Lot of things happening here: end of the year, Kestra 1.2 release, moving in a new appartement, planning vacation, etc. It’s great. Designing and introducing Asset in Kestra is exciting. Moving in a new flat is even more.
I’ve been excited over Claude Code and Superwhisper lately. You see it at work when a product teammate successfully merges a PR in C++ for a DuckDB extension. When I skip asking my designer for UI mockups and just build high-fidelity prototypes directly. When I speak to my computer and watch my personal website rebuild itself.

We all know the shift is happening. And like most of you, I’m excited to see where we will land.

See you until next one!

Building is fun. At least for engineers like me. You have an idea, you ask your favorite orb to do it, you iterate, you test, you push to Git and deploy to Vercel. You call it a day.

Because we don’t control distribution as much as we do with our keyboards1, most of these projects fall into oblivion. More than we want to admit.

I personally always try to frame such project around a pain, a challenge I have myself. A potential market, as small as it can be.
And while wondering at the market size or the best pricing model is pretty enjoyable intellectually, distributing the actual thing to the world is not.
It’s just not fun.

Playing the game of the social networks, probing SEO algorithms2, or reaching to influencers: this feels like chores.
Yes marketing is partially automatable nowadays - but again, it’s not fun and most of the time it doesn’t feel great...

Back in the 17th century, Marin Mersenne was the guy to know if you wanted to pitch your idea and ultimately distribute it at – so-called – scale.
His network was broad, spreading from Descartes to Galileo, from Hobbes to Pascal, from Fermat to Gassendi.

Did he do it for fun?
He was actually a productive music theorist and engineer himself, so he must have found another kind of joy in all that work to keep his network linked.

Like the size of our friend relationships today, the scale of distribution was smaller back then. It was face-to-face, word of mouth only. The product was the brand. Because great things scale.

It was also fun. Mersenne wasn’t running ads or gaming algorithms. He was connecting people who’d genuinely geek out together.
If people are building off-grid networks, or simply writing newsletters, there might still be a bit of fun out there.

If you were Mersenne today, who would you connect? What network would you build for the joy of it?

📡 Expected Contents

New Maxime Beauchemin Project

Exciting project from the father of Airflow and Superset! This time it’s orchestration again, but agent orchestration.
Looking at the demo and preview, it seems very experimental. But I can also feel that’s where we need to move forward: new UI and UX around AI.
I haven’t tested it yet – to me, the costs are a bit too high for developers willing to use this kind of tool without any limits - but this is definitely a project to keep in mind.

👉 Check Agor

Branding & Good Design

While I was already a design geek before joining Kestra, I am even more so after working with Nicolas as Design Lead.
Having a dedicated expert on design in a software company brings something unique to the brand. Being able to ask, edit in peer sessions, and see the evolution month after month makes me think again about how important design is.

Recently, I discovered that Grammarly acquired Superhuman (the email client company) and requested a whole new branding design from the Smith Diction agency. The results and reflection are shown in the blog shared below – it’s very inspirational and it shows how much craft can be involved in such a journey. Really beautiful and well executed!

👉 Check the blog

When a Bug Saved the Company

This blog shows how much a business model can be off. And how a simple touch - or a bug in that case - can improve the sales.

By the way, if you are looking for a great resource on how to build a pricing model: here is GitHub repository regrouping many o the best ones on the web (thanks for managing this repository)

👉 Read the story

Delayed Innovation

Last time we talked about the long nose of innovation. While I am looking for the productization part for data, I am still looking around at the innovation part on other domains.

This has been highlighted by MKBHD: it seems like tech companies are developing and ultimately deploying products backwards. They are not delivering finished products but the promise of them. Look at this “telepathy device” or this “autonomous robot”. The promise is on par with the best science fiction books from the 1970s. The actual product is not.

While less controversial, it has been interesting to see how we are trying to reinvent the computer; moving from numeric to analogic. Look at this thermodynamic computing thing.

All of these will likely be delivered eventually. The argument for shipping this early is all about training data (and a bit about money too) and the early adopters will likely pave the way forward – like most of the time. It’s just delayed innovation. Because the future is not evenly distributed.

📰 The Blog Post

While dbt recently released support for UDF functions in Python, it doesn’t change the fact that we write SQL with Jinja templating. Like we do with Helm, in Terraform in another way, or in jQuery a decade ago.

Happy to release that one as it has been in the draft folder for a long time. It seems to resonate with fellow engineers. Hopefully, we won’t stay in that state for too long.

🎨 Beyond The Bracket

Are you making something heavy?

The next time you receive this newsletter: I will be 30 years old, and we will be in 2026.
I am sending these monthly updates and curated parts of the internet to more than 2000 of you, since 5 years now. Big thank you!

While I will continue this ritual, I am also exploring how to do something heavier there. While all these substackery are a nice playground, I feel the call to engage second gear on writing - not specifically on data stuff. I am not sure I will succeed, but the goal is to move the needle slightly toward this direction. Let’s see how it happens next year!

I wish you the best for the next convolution around the sun ☀️

We added Jinja to our dbt project to avoid copy-pasting SQL. Then we needed loops. Then conditionals. Then macros that call other macros. Now we’re debugging template rendering errors at 2am, and somewhere along the way, our “configuration” became a full programming language - just one without a debugger, type system, or any of the tooling we’d expect from actual code.

That’s basically the state of data engineering nowadays. Actually, the state of any ops-oriented engineering work.

When we move further in the configuration complexity clock, we often find ourselves templating configuration files.
We start with clean YAML or SQL, then gradually pollute it with Jinja or any similar templating language. The YAML/SQL becomes unreadable - neither a declarative configuration nor proper code. These templating languages emerge as expression languages to bridge gaps, then become the problem themselves.

Why This Matters More Than Ever?

There’s a reason this problem feels more acute now. Software engineers are experiencing dramatic productivity gains from AI coding assistants - but data engineers and similar ops-oriented positions largely aren’t. At least not in the same way nor at the same scale.

Data is stateful, code is stateless. While software engineers work with stateless code that can be easily mocked and tested, data engineers are maintaining large, stateful systems where the code is in service of ops work - configuration, orchestration, and data pipelines that can’t be casually experimented without risking expensive mistakes. The tooling reflects this reality: we reach for declarative configs, then template them when they get complex, then debug the templates, stuck in a cycle hard for humans to debug and nearly impossible for AI to safely refactor.

DOM & jQuery

In the early 2000s, browsers had inconsistent, verbose DOM APIs. jQuery emerged to provide a concise, uniform interface for element selection and manipulation, event handling, AJAX calls, etc.
jQuery could be dropped onto any page. A single <script> tag and developers immediately wrote:

$(’.btn’).on(’click’, function() { … });

It replaced dozens of lines of vanilla DOM code with one.
As applications grew, teams used jQuery for: templating HTML, state management, animation sequences, even rudimentary component systems.
The more jQuery was capable, the more it was used - even when its imperative, selector-driven approach wasn’t appropriate. Codebases became riddled with tangled event handlers, hidden state mutations, and performance bottlenecks.

Then React appeared. It basically introduced a two-layer pattern:

• JSX - a declarative markup - defines what the UI should look like.

• JavaScript - imperative, programmable - implements the logic.

By keeping UI description separate from application logic, React eliminated the need to intermix DOM queries and state updates across the codebase.

Components bundle markup, styling, and state. Instead of scattering jQuery selectors and mutations throughout hundreds of lines, React encapsulates logic in reusable, testable units:

function Button({ onClick, label }) { 
  return <button onClick={onClick}>{label}</button>; 
}

We can call this mismatch between what we want to express (our intent) and how verbose or awkward it is to express it in the available tools the “semantic impedance”.

jQuery’s story mirrors what’s happening in data engineering for years now. Just as jQuery bridged the gap between JavaScript and inconsistent DOM APIs, templating languages like Jinja bridge the gap between declarative - static - configuration (SQL, YAML) and dynamic runtime needs1. And just like jQuery, it creates a semantic impedance. Even more: our codebase metastasizes.

And so the natural questions coming next are: should configuration or query languages be powerful enough from the start that they don’t need expression or templating layers? Should we move back to a programming language?

The Runtime Predicate Problem

At Kestra2, I see the following systemic pattern across different kinds of user maturity:

1. “I just need to reference an output...”: {{ outputs.task.value }}​

2. “I need to handle missing data...”: {{ outputs.task.value ?? ‘default’ }}​

3. “I need date formatting...”: {{ trigger.date ?? execution.startDate | dateAdd(-1, ‘DAYS’) | date(’yyyy-MM-dd’) }}​

4. “I need conditional logic...”: {% if inputs.data.value | jq(”.[23]”) | first == 3%}...{% endif %}​

Such a pattern is also common in all declarative-based tools, i.e., dbt with Jinja, Kubernetes with Helm, Terraform HCL with templating, etc.

Globally speaking, we can articulate it as follows:

1. Expression or templating language is introduced for legitimate narrow use case

2. Users discover it’s more convenient than switching to proper programming

3. Expression usage metastasizes across the codebase

4. Configuration becomes unreadable and unmaintainable.

Long story short: expression languages emerge as a necessary evil when configuration languages lack composability. And now our YAML looks like this monstrosity3.

One can think we should go further with the DSL clock and build configuration and query languages powerful enough from the start that they don’t need expression layers.

In my opinion, this can be a good idea in some cases, but likely not enough on themselves.

For example, Cue brings a lot of things to YAML or JSON configuration by adding strong typing, inheritance, validation, and schema definition capabilities that automatically validate your data and reduce errors when scaling configuration-based codebase.

Same with Malloy which brings composability to SQL.

All these new DSL aims to fix the flaws of previous languages, usually by bringing composability, proper typing, and unification.

While they mainly succeed at this, we still need to evaluate predicates at runtime. If dev, then the target database shouldn’t be the same as if prod. At runtime, we need dynamic evaluation. The problem isn’t that expression or templating exists - it’s that there’s no boundary on where it should be used.

Linting Templating to Save Us From Cancer?

While drawing architectural boundaries is the primary prevention we should apply to maintain the inherent complexity of our codebase, there is a second point of intervention we should evaluate here.

The example below is a Cue file policy checking on bad patterns within dbt queries:

// dbt_policy.cue
// Simple policy to catch bad Jinja patterns in dbt models

package dbt

import “strings”

// Define what a dbt model should look like
#DbtModel: {
	// The SQL content of your dbt model
	sql: string
	
	// Rule 1: No nested macros (macro calling macro calling macro)
	_macroCount: strings.Count(sql, “{{”)
	if _macroCount > 5 {
		_warning: “Too many Jinja expressions (\(_macroCount)) - consider simplifying”
	}
	
	// Rule 2: No loops inside WHERE clauses
	if strings.Contains(sql, “WHERE”) && strings.Contains(sql, “{% for”) {
		_violation: “Loop detected in WHERE clause - move logic to CTE or upstream”
	}
	
	// Rule 3: No complex date math in Jinja
	if strings.Contains(sql, “dateadd”) && strings.Contains(sql, “{{”) {
		_violation: “Date arithmetic in Jinja - use SQL date functions instead”
	}
	
	// Rule 4: No string concatenation in Jinja for SQL generation
	if strings.Contains(sql, “~ ‘”) || strings.Contains(sql, “’ ~”) {
		_violation: “String concatenation in Jinja - build SQL properly in model”
	}
	
	// Rule 5: Limit macro nesting depth
	if strings.Contains(sql, “{{ ref(”) && strings.Count(sql, “{{”) > 3 {
		_warning: “Multiple nested refs - consider breaking into separate models”
	}
}

// Example: Good dbt model
good_model: #DbtModel & {
	sql: “”“
		-- ✅ GOOD: Simple ref, clear SQL
		SELECT
		    user_id,
		    created_at,
		    amount
		FROM {{ ref(’staging_orders’) }}
		WHERE created_at >= ‘{{ var(”start_date”) }}’
		“”“
}

// Example: Bad dbt model (this will fail validation)
bad_model: #DbtModel & {
	sql: “”“
		-- ❌ BAD: Loop in WHERE clause
		SELECT *
		FROM orders
		WHERE order_id IN (
		    {% for id in var(’order_ids’) %}
		        {{ id }}{% if not loop.last %},{% endif %}
		    {% endfor %}
		)
		“”“
	// This will trigger: “Loop detected in WHERE clause”
}

// Example: Another bad pattern
bad_date_model: #DbtModel & {
	sql: “”“
		-- ❌ BAD: Date math in Jinja
		SELECT *
		FROM orders
		WHERE created_at > ‘{{ dateadd(days, -7, execution_date) }}’
		“”“
	// This will trigger: “Date arithmetic in Jinja”
}

This is likely not a realistic example, but I’m sure you got it. Here, we check not only for pure code syntax, but also for bad templating patterns.

Like for classic lint4 checks, the goal is to find a way to secure our codebase - in a systemic way. Embedding such policies in a CI/CD pipeline sounds like a good start here.

The cost of a linting rule is measured in minutes. The cost of debugging template logic at 2 am is measured in hours - or careers. Choose wisely.

What Would the React Moment Look Like?

Linting is an intervention at the symptom level. But what would intervention at the cause look like?

In medicine, metastasis happens when cells lose their differentiation - they forget what type of cell they’re supposed to be. A liver cell that forgets it’s a liver cell doesn’t become healthy; it becomes cancer.

That’s exactly what’s happening in our codebases. A configuration that forgets it is a configuration doesn’t become flexible - it becomes Jinja templates with 8 levels of nesting. Code that forgets it is code doesn’t become declarative - it becomes YAML files that secretly execute arbitrary logic.

React didn’t solve the jQuery problem by making JavaScript better at DOM manipulation. It solved it by establishing a social contract enforced by tooling: JSX is for structure, JavaScript is for logic.

dbt compiles Jinja to SQL at build time, yes. But the source code you write, debug, review, and maintain is still the Jinja spaghetti. The AI trying to help you refactor sees Jinja spaghetti. Your colleague reviewing the PR sees Jinja spaghetti. The boundary between logic and configuration exists at build time, but not where it matters - at development time.

As explored earlier, we could try to fix this. Jinja but with better linting. Helm but with type checking. But we are likely solving the wrong problem.

React didn’t eliminate runtime evaluation. Components evaluate props and state at runtime constantly. What React eliminated was logic embedded inside the declarative structure.

Look at the difference:

// jQuery era - logic embedded in markup
$(’#orders’).html(`
  <div>
    ${env === ‘prod’ ? ‘<span>Production</span>’ : ‘<span>Dev</span>’}
    ${orders.map(o => `<div>${o.amount}</div>`).join(’‘)}
  </div>
`);

// React - logic happens before structure
function Orders({ env, orders }) {
  const label = env === ‘prod’ ? ‘Production’ : ‘Dev’;
  const items = orders
    .filter(o => o.amount > 100)
    .map(o => <div key={o.id}>{o.amount}</div>);
  
  return (
    <div>
      <span>{label}</span>
      {items}
    </div>
  );
}

Same runtime behavior. But in React, computation happens in a computation language, and then the structure is declared with computed values.

The structure itself contains no logic—only references to already-computed values.

Now look at our dbt models:

-- Logic embedded in structure
SELECT 
    user_id,
    {% if var(’env’) == ‘prod’ %}
        prod_column
    {% else %}
        dev_column  
    {% endif %} as column_name
FROM {{ ref(’staging_orders’) }}
WHERE created_at >= {{ 
    ‘CURRENT_DATE’ if var(’lookback’) == ‘today’ 
    else “’” + var(’lookback’) + “’” 
}}

The SQL structure contains conditional logic, string concatenation, and function calls. We’re doing computation inside the declarative query language using a different language.

What if we separated them?

# Computation happens in a computation language
def get_orders_query(env: str, lookback: str) -> Query:
    column = ‘prod_column’ if env == ‘prod’ else ‘dev_column’
    source = ref(’staging_orders’)
    where_clause = ‘CURRENT_DATE’ if lookback == ‘today’ else f”’{lookback}’”
    
    # Structure is declared with computed values
    return Query(f’‘’
        SELECT 
            user_id,
            {column} as column_name
        FROM {source}
        WHERE created_at >= {where_clause}
    ‘’‘)

Still using string interpolation here5 - not perfect, this is just a simple specification exploration. But the key difference is: all logic is in Python, and the SQL is data being constructed. No language-switching mid-file.

Or with Malloy:

# Computation happens in Python
def get_orders_query(env: str, lookback: str) -> MalloyQuery:
    column = ‘prod_column’ if env == ‘prod’ else ‘dev_column’
    date_filter = ‘today’ if lookback == ‘today’ else lookback
    
    # Structure is declared in Malloy with computed values
    return malloy.query(f’‘’
        query: orders -> {{
          select: user_id, {column}
          where: created_at >= @{date_filter}
        }}
    ‘’‘)

Still need runtime evaluation? Yes. Still need to check env? Yes. But the evaluation happens in a real programming language, then the result is passed to the declarative structure.
This is what React actually did to fix the impedance mismatch.

• Computation layer: Evaluate predicates, transform data, make decisions. Use a programming language

• Declaration layer: Describe structure with computed values. Use a declarative language

The cancer happens when we try to do computation inside the declaration layer. That’s when we reach for templates.
Jinja exists because we’re trying to compute inside SQL/YAML. Helm exists because we’re trying to compute inside YAML. HCL Templating exists because we’re trying to compute inside Terraform config.

Metastasis of templating languages in our codebases kills, not because it’s unstoppable, but because we intervene too late. The medical field learned this through decades of tragic outcomes. We don’t need to repeat that mistake in software. Software should be soft, and while we can have infinite debate over whether YAML or SQL is soft or not, we need better tooling around these to scale, help our AI handle them, and ultimately realize the promise of the declarative paradigm shift.

# Computation in Python
def get_orders_query(env: str, lookback: str) -> Query:
    column = ‘prod_column’ if env == ‘prod’ else ‘dev_column’
    source = ref(’staging_orders’)
    date = datetime.today() if lookback == ‘today’ else parse_date(lookback)
    
    # Declaration using a query builder (not string templates)
    return (
        Query
        .select(’user_id’, column.as_(’column_name’))
        .from_(source)
        .where(col(’created_at’) >= date)
    )# Computation in Python

Consultancy firms all have internal tools. They are usually built during inter-contract time, by fellow engineers tired of building the same thing over and over, from customer to customer. Ultimately, a library as simple as templating SQL with Jinja succeeds in gathering a community. In “building a product”.

It’s easy to say after the fact, but merging with a company that succeeded in providing value by externalizing data moves seems pretty natural. If we remove open source from the debate, it’s kinda surprising it didn’t come earlier.

Some of us will look for the alternatives. For both a community and a “real” open source project for transforming data with SQL.
As engineers, we continually search for something better, but a moment’s reflection reveals that it’s hard to ask for much than these templating and model dependencies engines.

I am. Either because I’m tired of seeing the same patterns. Or because I feel we are mature enough now. Probably both. Templating SQL query with Jinja sounds so much like a graduate project to me now. Don’t get me wrong, I’ve built such a thing myself before dbt. I used dbt. I use dbt. I will use dbt in the future. And I don’t have an off-the-shelf solution here; just saying we should think deeper about the basis. SQL wasn’t designed for analytics. Not a lot of things in our industry really were.

Now is a moment to pause, to reflect on our codebase, on our practices. Is templating a language not designed for analytics, with a mix of brackets, untestable macros, and markup files is the best we can do?
I don’t think so. For now, it indeed seems hard to ask for much than that. And upstairs, people really don’t care about how we provide the figures. The decision around this company merger probably involved more Excel files than dbt models1.

So while we should be at our desks bringing value to stakeholders, there’s still a gap to fill. For now, it will still involve a lot of brackets, of YAML, of templated SQL, and ultimately humans. And while the latter is not about to go, I really feel the former should be part of history.

📡 Expected Contents

The Long Nose Innovation

What technologies have been here for 10 years, yet not really in production at high scale?
In data, I can’t really find one. This suggests we are at a plateau.
Looking at 5-year timespan brings a bit more of novelties: accents to lakehouse, declarativness, or BI as code seems the next path for productization.

Here is a nice article reminding us that technological innovation occurs at a slower pace than we prefer to admit.

My belief is there is a mirror-image of the long tail that is equally important to those wanting to understand the process of innovation. It states that the bulk of innovation behind the latest “wow” moment is also low-amplitude and takes place over a long period—but well before the “new” idea has become generally known, fully refined, much less reached the tipping point where it becomes widely adopted. It is what I call The Long Nose of Innovation.

👉 Read here

What’s DevRel?

Pedram shares a good roundup of what developer’s relationship is after two years leading the initiative at Dagster.
Beyond the nice read, it’s also good to see that such a position brings the path toward landing a job at Anthropic. Congrats on that Pedram!

👉 Read the post

Docker-compose with AI models

Here is a nice blog post from Pradumna, a new teammate at Kestra!
Even if I’m moving away from Docker more and more, this one specification from Docker is nice to see.

👉 Read the blog

What Moves the Business Tick?

If it’s not the latest lakehouse architecture, what is it?

For me, the most painful part is BI. Yes, we have tools for building dashboards. But users of these dashboards still have a hard time making decisions. And if they make decisions, it is still not clear what the outcomes are (that’s often another subject...).
It’s been more than 7 years I’m working close to C-levels, either directly or in a second order. And what remains in the end is a complexity in making decisions. That’s their job. It’s a complex one.
Intuition is their most valuable asset, and they build it on top of their experiences. Data is here to provide facts - to bring signal - not decisions. It’s here to validate intuition. When done best, it’s building intuition.
The decision is mainly about an intuition, i.e. the result of the latent space built over data analysis discovery and their own experience.

Do you see the real bottom line here? Experience, discovery, intuition. Not sure a new architecture for data storage resonate with these.

📰 The Blog Post

New blog post coming in very soon!!

This one has been in the draft folder for over a year now. Writing is about patience, and seeing the words unfold almost naturally after such a long span of time provides great joy.

Here is a short spoiler - the blog post is likely to be titled “The Metastasize of Templating in Configuration Language”:

We added Jinja to our dbt project to avoid copy-pasting SQL. Then we needed loops. Then conditionals. Then macros that call other macros. Now we’re debugging template rendering errors at 2am, and somewhere along the way, our “configuration” became a full programming language - just one without a debugger, type system, or any of the tooling we’d expect from actual code.

That’s basically the state of data engineering nowadays. Actually, the state of any ops-oriented engineering work.

When we move further in the configuration complexity clock, we often find ourselves templating configuration files.
We start with clean YAML, then gradually pollute it with Jinja or any similar templating language. The YAML becomes unreadable - neither a declarative configuration nor proper code. These templating languages emerge as expression languages to bridge gaps, then become the problem themselves.

Be sure to subscribe to receive it in your inbox soon ;)

🎨 Beyond The Bracket

This is what’s behind the scenes. At giant hall, 12 hours before the party starts. People not having a clue of what’s ETL are building high quality booths for venue expecting more thousands of other kind of people.

I had such a great time in London: speaking with fellow data product builders, playing darts in English pubs, looking at smart marketing hack from Duck people, eating pies, walking in second hand book store, etc.

But I also saw the truth behind our industry: it’s serious money, propelled by marketing efforts. It was really surprising to be one of the few teams building a product based on an open-core strategy. To be here with engineers, not only sales people.
Half of the people crossing the booth were here for the stickers, the t-shirts and the iPad. At other booths, marketing teams brought free coffee, Lego game, and Playstations.
Because you know, it should be fun2.

Demos were mostly run by salespeople. I was expecting more from demo provided by a leader of the industry than just showing a notebook and run SQL queries.
AI was not really a thing. Making sense of all these logos was the real moat. Business analysts were lost, their data engineering fellows too focused on technical artifacts - as always.

I realize after writing this one how a pessimist call it can be. But don’t get me wrong. I want the fun too, but being serious first. I want open source, but being independent first. I want templating, but being free of brackets3 . I want goodies but without the branding.

Anyway, definitely looking forward to the nighttime here. I’ve been moving to Qwerty lately4, entering the mechanical keyboard fantasy, doubling down on good cake and tea, chess, and reading5. I’ve also just returned from Lake Como and the countryside in Italy - it was lovely!
Hope you had a good weekend! See you next time!

A complex problem has a real part and an imaginary part.
We tend to blab about the real part. It's obvious and measurable. It comes back in every informal chat.
The imaginary part, though, is the underlying emotional, psychological, or systemic factors that aren't immediately visible but fundamentally shape the problem's behavior.

In mathematics, every complex problem can be expressed as r·e^(iθ)—a magnitude and a phase angle.
Changing the frame like this usually helps in simplifying a complex subject into a complicated puzzle. And complexity is not complicated.

Along these lines, many problems can only be "solved" by acknowledging both z and its complex conjugate z*—we need to address not just what's happening, but also its reflection or opposite.

I sense it's the same with our own problems.

Many "impossible" real equations become straightforward once you allow complex solutions. Sometimes the answer involves embracing what feels "negative" or "imaginary".

To uncover such a new angle, my trick is to plot problems on a coordinate system where one axis is "what everyone talks about" and the other is "what no one mentions."

As the real part seems easy to solve, the imaginary part often freezes us.
To close the gap is to think "magnitude and direction".

📡 Expected Contents

Why Semantic Layers Matter — and How to Build One with DuckDB

Nice usage of the Boring Semantic Layer. Yes, the one we discussed with Julien in our last blog post together.

I'm glad to see this project going out. Again, shout-out to Julien here as I'm thrilled to be part of the early testers and contributors.

👉 Read the blog

The Joy of Under-Engineering

Under-engineering is a fun exercise. But it’s still just that: an exercise. Please don’t go to work next week and tear down your production Kubernetes cluster because some random guy on the internet said it’s fun to deploy software to your own VPC via ssh.

Funny enough, I try to stop using Docker. Not because I don't like it: I've been a huge fan of it, but because my professional laptop is only 8gb of memory. And I use Chrome a lot. Or Chrome is using my memory too much. I don't know anymore.

Joke aside, I feel a recent push toward simple CLIs and better environment managers that ultimately means less engineering for the end developer.
There is also a trend towards HTML-only websites that I found particularly on spot1.

Here is a great piece on of under-engineering systems is at least a good exercise to reflect on. And maybe to practice a bit more. And to add to our prompts.

👉 Read the blog

Python Documentary

They finally release it!
CultRepo (formerly Honeypot) release again a great documentary; this time on Python.
I binged watch it: and it's great. If you like it, I recommend to watch the Kubernetes one too.

Tabs vs Spaces

Tabs mean you see my code the way you choose to. Spaces mean you see it the way I want you to

You make more money, but this feels like a debate that never end. It seems we are either psychopathic megalomaniac or obsequious sycophant.

I'm more a spaces developer, but I'm changing my mind recently after a discussion with a senior engineer, highlighting the following:

[...], in the sense of lexical semantics. TABs were literally invented for indentation. Spaces are for spacing things. You wouldn't use a TAB between words in a sentence, because that's the wrong character. Similarly, you shouldn't use spaces to indent your code, because that's the wrong character.

In the end, do we still care? We don't write code anymore right? And text has more spaces, so maybe the the war over?

📰 The Blog Post

I have a billion ideas for story lines. Just not prioritizing the writing for now.
Will see what comes out of the recent read and exploration.
Sometimes it's just best to wait for the rush. The need that makes blank pages flush.
I'm waiting for that natural discharge. Serenely.

🎨 Beyond The Bracket

Done is better than perfect.
The image above is Google Maps 20 years ago.
You don't need to achieve a dream in one shot. Just release a first version, then iterate.

Back to school!
This time with Kestra 1.0 and BigData London in two weeks.
This summer ranks among the best of the eighty I'll get to live2. I wrote about it here.

What’s the name of your car?
I mean, you’ve named your car, right?
I actually haven’t, but if I had to choose, I’d call it “Tommy.”

It’s somehow a companion: I talk to it, it follows me on every trip, I take care of it, and I’m afraid something bad might happen to it.

Now, what about the names of your data tools?
We make decisions with them, we spend our days with them, we get frustrated with them, we take care of them.

Yet we don’t name^[1] them—beyond their actual boring names. They might have cute duck, octopus, or engine-like faces, but that doesn’t help us form a personal connection. Everyone uses the same tools.
Kids, too, play with the same toys. But they ultimately give them personal names.

There’s something intrinsic to our software tools: they don’t exist in our physical world. And so, it’s hard to feel the same attachment as we do with our cars or toys.

I’ve been rediscovering my old Sony PSP recently, and wow! The buttons, the lightness, the screen—everything is so well made. The same feeling goes with a vinyl record, a book, a nice chessboard. Chess players even bridge the gap between the real and the virtual with these incredible chessboards.

The keyboard is likely the closest physical object to our digital tools. It’s not surprising to find fans of mechanical keyboards and custom keycaps quite easily.
Should we have keycaps for dbt—a hotkey to open our dbt project in Cursor? The same for BigQuery, Metabase, or ChatGPT.
These are basically our CMD+OPTION+<the key you never remember because you only use the tool once a quarter> shortcuts embedded in our physical world.

I feel we’d make these tools more like companions if we could truly make them our own.
Yes, software should be soft. But the revival of BlackBerry, the fandom around mechanical keyboards, the Car Thing from Spotify, or the Claude Poetry camera all refresh the idea that the best instrument is a companion. It's sometimes weird. It can break. It needs care.
And now, it can even “talk” to you.

📡 Expected Contents

Betting Against Agents

Resonating with the intro: this post comes with very good arguments on why we shouldn't focus too much on agents.

Long story short: error rates compound exponentially in multi-step workflows. 95% reliability per step is equivalent to 36% success over 20 steps.
In reality, we usually need 99.9%...
Also, context windows create quadratic token costs making long autonomous conversations economically unsustainable.
It's not that agents don't work. It's that they don't work like we thinks. Don't get me wrong, I'm bullish on AI in general, but this read among others confirms that the current narrative suggesting 2025 is the breakthrough year is less and less a good conjecture.

Must read!

👉 Read the blog

Thoughts on DuckLake

I didn't follow that much DuckLake announcement a few month ago. Max gives good thoughts on this post.

I'm not sure if such a framework is really needed right now. But I kinda expect the Duck team to have a proper plan and deploy the project even more on the market shortly.

👉 Read the blog

dbt + AI follow up

AI isn’t just for modeling—here’s a sharp breakdown of how it can drive robust data testing and monitoring, too.

This second part of Mikkel's blog walks through pairing dbt with AI to automate checks, flag anomalies, and keep your data quality tight. Worth a read if you want to see what the future of analytics engineering looks like in practice.

Related: lately I thought about the sub-agent thing brought by Claude. I feel more and more eager to write these agents' "brain" prompts rather than writing code myself. Committed as markdown in the repo, and the entire team can use these robots to work way faster. Nothing really new, just wondering if it's one orthogonal path alongside semantic layering and more tooling for humans 🤔. To be continued.

👉 Read the blog

Using uvx

If like me you write more and more CLI for daily operations, you might be interested in uvx:

The most important aspect is defining entry points (console scripts) in your pyproject.toml. This tells uvx which commands are available when the package is installed:

[project]
name = "your-package"
version = "0.1.0"
description = "Your tool description"

[project.scripts]
your-command = "your_package.module:main_function"
another-command = "your_package.cli:cli_main"

When uvx is invoked, uv installs the corresponding package, which provides the command. The tool looks for executable commands defined in the package's entry points.

👉 Read the doc

📰 The Blog Post

No blog post this time. I'm keen to explore more around UX for data + LLM, so I hope to write some follow ups of the last post with Julien. Here is the link again I you didn't read it yet.

🎨 Beyond The Bracket

This is code vibing.

I've been playing piano on the side for a decade now^[2] and I'm always amazed by how technology helps to learn and develop new sounds, new vibe.
This time the instrument is code.

Playing around Strudel makes you see music in a totally different way. It gives new ideas.
It's again a call for me to draw a parallel with how literature, cognitive, and linguistic studies can help us use our new toys in novel ways and explore our own selves.

Just returned to France after a month in the US. Loved it! Felt like being a kid again - everything seemed fresh and waiting to be explored. Sure, there's probably novelty bias at play (only my second US trip). But after experiencing the wildlife and those massive open spaces, I feel cramped back in this tiny French box.

Already tons of things planned for the back to school - with a stop in London in September. I'll be in BigData London for those wanting to hang out there!

On September 23rd, 1846, Urbain Le Verrier (calculation) and Johann Gottfried Galle (observation) discovered Neptune.

The story begins with Uranus, which had been puzzling astronomers for years. Its orbit showed strange irregularities that didn't fit existing models.
Using Newton's law of universal gravitation, Le Verrier analyzed the perturbations in Uranus's orbit and worked backward to determine the hidden planet's likely mass, orbit, and position. The calculations were extraordinarily complex—all done by hand without modern computational tools.

Le Verrier published his preliminary findings in June 1845, then spent months refining his work throughout 1846. He didn't just predict the planet's existence; he pinpointed its precise location in the sky.

When Galle pointed his telescope to Le Verrier's predicted coordinates, Neptune was there waiting.

Most of the planets of our solar systems have been found by direct observation. Only Uranus where found with technology assisted obsvervation1. Neptune was the first planet found through pure mathematical modeling.

While the analogy will sound peculiar to some, I feel we have been through the same pattern recently.

Le Verrier worked backward from Uranus's orbital irregularities to find Neptune.
Context engineers work backward from inconsistent AI outputs to identify what contextual structures might stabilize performance.

Prompt engineering was the obvious. Then we enter context engineering which is more thoughtful and methodical.
A natural follow up question I'm sure you asked yourself is "how to build context then?".

I'm not talking about the first order context: the one we feed our synthetic PRDs ultimately digested by our best models. I'm talking about the one we build upfront.

I don't have a master's in cognitive science or linguistics, but my gut feeling here is we're using these new models to uncover just how powerful language really is—and how it contains computational structures we're only starting to grasp2.

Put it in another way: there's a call here to focus more on literature, linguistics, and cognition. The actual disciplines allowing us - humans - to develop larger, thorougher, newest, contexts.

📡 Expected Contents

Obituaries, a Way of Creativity

My summer - nerd - reading is Metaphors We Live By. Likely my favorite book of the year so far.
Metaphors are very powerful and most of the time underestimated. Exploring uncommon ones is good for creativity as highlighted in this post:

Gentner calls this property mapping — when people borrow attributes like texture or shape from one concept and apply them to another. It’s a kind of remote association, and clearly more creative than imagining a standard school desk. But Gentner identified something even more powerful: structure mapping. This happens when you transfer the relational structure of one concept to another. Say you combine “pony” and “chair” and picture a chair shaped like a pony — that’s still property mapping, just more elaborate. But if you imagine a small chair, you’ve made a bigger leap. That’s structure mapping: drawing on the idea that a pony is smaller than a horse, and applying that relationship to redefine the size of a chair. These kinds of mappings — especially when the underlying relations are abstract or non-obvious — tend to produce the most original and surprising combinations.

A big part of this newsletter is driven by exploring metaphors; this one post resonates a lot:

👉 Read the post

The Illusion of Causality in Charts

Again a call for "corellation isn't causation". Our industry is stuck somehow in a local optima: the ETL era. Into "observational data."

While causality cannot be directly inferred from observational data, different degrees of plausibility can be derived from it.
But we might also lift our new tools to move forward on proper A/B tests and methods designed since decades to run causality experiments.

👉 Read the post

Using AI for Data Modeling in dbt

I'm working with an analytics engineer at Kestra, and he surprised me very early in the onboarding process: he set dbt with some AI plugin in VS Code. This took less than 2 days and it drove the whole analytics motion.

dbt with an AI copilot will likely be the default setup when it comes to analytics engineering. Even niche code editor like Nao are emerging to support that project.

👉 Read the post

When to Design For Emergence

I can't recommend this post more. Likely promoted in a previous issue (shame on me if not). Reading it again in the plane right now and it highlight something we might need in the vibe-coding phase.
Yes, building very specific product that fit a niche problem is good. But to sustain a tool letting users find solutions by themselves is another game.
Again, I'm thinking about Malloy here, and globally speaking any tool or DSL that let users create new meaning their own way.

👉 Read the post

📰 The Blog Post

Very proud to release that one with Julien. We were very exicted to write and explore on the semantic layer and the future of analytics here.
Please read it quickly, you'll be surprised how far you can go with a simple semantic layer definition, a MCP, and proper prompt techniques.

👉 Read the post

🎨 Beyond The Bracket

Quitting programming as a career right now because of LLMs would be like quitting carpentry as a career thanks to the invention of the table saw.

Not that much to add. But it’s a good reminder for the current time. The new paradigm in set.

Our end goals as engineers will likely be the same. We will just use better tools.

And so again, the same loop… This is a call to focus on what matters.

Yes, you should learn to use the new tools.

But what are you doing with these tools? What are you focusing on right now?

Hope it’s not tool specific.

Hope you try to build a house, like it always been.

Not a wooden toy for kids you don’t have.

Writing this issue in the plane toward Portland. I'm quite proud of my reading setup here: my printed magazine of my favorite blog posts, a book, and Obsidian on my Macbook. I don't any WiFi, so the latter is very welcomed as I can access my own librairie of thoughts, and project I save and nurture since years now.

Can't wait to land on that new territory. The other side of the planet for me.
Hope the world on your side is doing good.

Tupperware filled bankruptcy in 20241. It stemmed not from inferior products but failure to reinvent distribution for digital-native consumers accustomed to Amazon Prime and social media influencers.

According to its bankruptcy filing, consumers now prefer making purchases in stores or online, and only a small share buy via the direct-selling model. Still, in 2023 some 90% of Tupperware’s sales were from direct sales.

At work, I recently ran a pre-sales call with a prospect. This one went really well. It was actually better than a year ago. Mainly because I digested my learning, but also read and thought about the sales motion lately.
When I pull the thread further, my engineer mindset always falls back to what makes sales easy: good distribution.
This resonates a lot with Sam Altman's call: "Most careers turn into a sales job when you get senior enough"2

Today, we are on the verge of a new distribution system for software. The home-cooked software and barefoot developers is getting serious. Like when online stores change the answers to the questions of how and where we buy goods, I can expect the same questions on software to have a totally different answer than in the current timeframe.

How do you buy software?
Where do you buy?
Do you still buy software?

There’s no reason to design software at an industrial scale. People don’t want to use a thousand to-do apps or a million messaging tools. We have car factories because cars are expensive to manufacture by hand; we have shirt factories because people want a million shirts. Neither of those is true for software (anymore).

Though we can see recent acquisitions as a great bundling in the higher market, it seems we're entering an unbundling phase in the consumer and developer markets.
And like any unbundling phase, we might need to use some glue.

📡 Expected Contents

The total derivative of a metric tree

Graph theory always enlightens me. Was boring when I was in engineering school, is now a great modeling approach most of the time.
Max comes here with a mathematical application of Metric Tree: computing the total derivative of a metric tree

I like the idea that a metric tree is essentially just a function. You can think of it as a multivariate function that takes a set of inputs and returns a single output. There may be intermediate outputs, depending on the metric’s complexity. The metric tree is just a way to visualize the function’s structure.
Any well-defined function can be differentiated – i.e. provided it is continuously differentiable. In particular, the total derivative gives us a way to understand the change in the output with respect to each input. It is a linear approximation of the function at a given point.

Go read that one: it's not that complex and it's short.

👉 Read the blog

Zip Bomb Protection

Protecting a website from malicious attacks, or lately LLM crawling, is now a complete industry.
There are proxies, load balancers, log analyzers, and other smart techniques. There is a simple dummy one:

A zip bomb is a relatively small compressed file that can expand into a very large file that can overwhelm a machine.

👉 Read the blog

I'd rather read the prompt

Revolving around AI prompts and the value of it. Very nice post.
Please show me your prompts.

The resulting output has less substance than the prompt and lacks any human vision in its creation. The whole point of making creative work is to share one’s own experience - if there’s no experience to share, why bother? If it’s not worth writing, it’s not worth reading.

👉 Read the blog

The Fairytale Structure

Strategic planning using the fairytale structure?

It’s going to look like this:

1. Objectives: What are we trying to get / what happens if we don’t?

2. Obstacles: What stands in our way?

3. Actions: How do we overcome the obstacles while advancing our objectives?

4. Measures: How will we know we’ve been successful?

5. Not Doing: What important things have we chosen not to do, so that we can achieve the above?

Great framework using humanity’s collective unconscious to guide strategic problem-analysis, and plans that everyone understands.

👉 Read the blog

📰 The Blog Post

No blog post this month; things have been busy. If you are curious to know what I've been doing at work, the biggest writing piece of the month on my side is the Kestra 0.23 Release Blog Post.

Yet, I've fresh ideas on the back burner. Have to sit and write.

🎨 Beyond The Bracket

All the parts of human anatomy that don't have clear evolutionary advantages or seem actively unhelpful and poorly designed – Source: Quirks of Human Anatomy (Held 2009)

As I opt out of Twitter and Instagram, my dopamine shots from the internet are coming from two sources: Substack and YouTube.
On the latter, I regularly watch content from startup founders... And it questions my ambition to create a company.
In one of these videos, one founder says that having a baby is the most complex and challenging project of life.
And I kinda agree. Even if I don't have a child yet, I can feel in myself that it will be the most exciting project of my life. Likely, the most challenging too.
And so I'm thinking about it. Instead of founding a startup, I'm probed to focus on founding a family. Two different businesses, but is it?

Out of this wondering, I remembered again the great Maggie Appleton, and her post on growing a human

This post is a masterpiece for at least two things:

• If you are a man like me, you should read it. Point.

• It's an entire journey explained, with the highs and lows. See it like an Andress Horowitz manifesto not about startup, but about pregnancy.

If you hadn’t guessed already, I’m not a woman of faith. At least not in the sense of temples or Allah or throwing blessed-by-Christ-holy-water on people. I don’t have a weekly standing appointment to reflect on my place in the universe and bow down before higher-order beings. But I do carry around a strong sense of reverence and awe for higher-order forces; the unreasonable effectiveness of evolution, the tectonic plates floating about on molten mantle sculpting our earth, the symbiotic relationship between clownfish and sea anemones, the fact trees warn each other of dangers via chemical signals and send reinforcements, the mystery of consciousness, and whatever the fuck is going on with black holes. These things all give me Carl Sagan’s energy marvelling at the pale blue dot.

Mid-year already; it's getting fast lately. My four-week trip to the US is in less than one month. Excited to be there. Excited to be in the plain summer.

I've been exploring several things lately:

• board games: I'm reconnecting with a joy from my childhood. Been playing Ticket to Ride (it's more complex than it seems), Jaipur, and looking for this Fleet, the dice game

• printing newspaper: I'm about to get 24+ hours in planes this summer, so I'm printing a special edition of this newsletter for myself 😅 Lately explored how to print a newspaper at home (tired of reading the internet on a screen...)

• vibe coding a project management tool: mainly a mix of Sunsama (the tool I'm using currently) and Basecamp tailored to my exact needs. You can have a look at the MVP here3.

• vibe coding boring analysis: the natural brother of Boring Data from Julien for data analysis specific. Mainly a CLI for rendering pre-made analysis and charts with Malloy

Need 48h in a day to get these done, so not moving forward that much, but it's here and I keep iterating, so...

See you in a bit, I might skip one edition during summer, we will see :)

"Improving your business is about more acquisition and less churn."

How can I improve the business?

“Among the February cohort of new users, 67% of them churned after 3 days"

What's the churn rate after 3 days in February?

"Users are churning because there's an issue when they try to invite a new user to their organization."

Why are users churning after 3 days of usage?

"Here is the code I deployed to production to fix the churn issue: </....>"

What code did you deployed to fix the churn issue in production?

In many applications, users can audit logs. I.e., admins can see what happened, revise, and restore changes.

That's one of the main reasons the declarative-dom emerged, and why it sticks.
We're not clicking here and there, we declare our actions. Ultimately, we store and we version control them.

The new trendy interface - MCP - echoes that concept.

Now actions are taken out of our writing or talking.
It's not only about the past anymore.
Manuscripts, printing, computing: it's all about acknowledgment from the present that we will retrieve in the future.
But MCP breaks that pattern now.
It's now about action. Present action.
I'm writing what I want the computer to do. Right now.

We could fool ourselves and call it a day.
But we still need language. Old ones, common ones, and new ones.

Recently, I've been playing with MCP. First with Kestra where I've been able to control my instance only with natural language, second with Malloy, where I got closer to the true "text-to-SQL" motion I've been talking lately.

And again, the choice of interface is probed.
For example, it wouldn’t be optimal to use natural language for keeping track of orders made by a customer.
Rows and columns are handy here.
However, asking my computer to store my last customer order rather than writing an INSERT INTO statement or clicking in some interface sounds way easier.

A graphical interface is built because we need to persist actions.
But now, as action can be declared, these "graphical" interfaces don't provide as much value for taking action.

Ultimately, our native language feels natural because we can easily say nonsense without realizing it. Natural language is very useful because it fades.

This is the real value of MCP. Writing logs of actions and decisions without realizing it.

I don't know if reverse MCP server is already a thing, but I can see a world where we would like to play Jeopardy for real.
To audit the action taken.
To imagine a world and play backward thinking.
To probe ourselves.

📡 Expected Contents

Mold and Rust Logic

I've recently been reflecting on how data is dust. On why software only moves forward, and why ephemeral software should decay and lose data.
In the physical world, our artifacts fade - we put them in the trash. In the digital, they kinda stick forever.
Here is a nice post reflecting on rust and mold logics:

Rust logic’s central tension is preventative maintenance vs. planned obsolescence.
Mold logic’s central tension is calculated sanitation vs. managed overgrowth.

👉 Read the blog post

Malloy again

Yes, I'm here again with Malloy. I'm not a prophet and don't know if it will stick in the long term. But this project keeps triggering me. It has a deep promise about building a semantic layer that looks like mathematics to me.

If you need a fresher on it, here is a nice discussion where Lloyd Tabb presents Malloy and goes deeper into it (the index part is thrilling!).

You probably didn't miss it: everyone is around MCP now. Malloy doesn't deviate the rule. With the great Malloy Publisher that takes the semantic models defined in Malloy and exposes them through a server interface, it was natural to see a dedicated MCP for it.

I put down some words on what could be the implications of Malloy on analytics in the blog post section later on this issue.

👉 Check Malloy Publisher

On the foolishness of "natural language programming".

We are sticking up for LLMs, and that's cool.

I wonder, what if you did the opposite? Take a project of moderate complexity and convert it from code back to natural language using your favorite LLM. Does it provide you with a reasonable description of the behavior and requirements encoded in the source code without losing enough detail to recreate the program? Do you find the resulting natural language description is easier to reason about?

I think there's a reason most of the vibe-coded applications we see people demonstrate are rather simple. There is a level of complexity and precision that is hard to manage. Sure, you can define it in plain english, but is the resulting description extensible, understandable, or more descriptive than a precise language? I think there is a reason why legalese is not plain English, and it goes beyond mere gatekeeping.

From one gut feeling I derive much consolation: I suspect that machines to be programmed in our native tongues —be it Dutch, English, American, French, German, or Swahili— are as damned difficult to make as they would be to use.

Same for maths and coding - once you reach a certain level of expertise, the complexity and redundancy of natural language is a greater cost than benefit. This seems to apply to all fields of expertise.
Here is a great post discussing natural language programming, from 2010... I think we can move on.

👉 Read the blog post

Killed by Google

A reminder that even Google throws away its projects. It's also a reminder that creating projects is like VC: only one of many will cover your investment. So, be wide, ready to throw away stuff, but ultimately reach your (time) investments thanks to only a few projects.

💀 Killed by Google

📰 The Blog Post

What Syntax for the Semantic Layer? I'm sticking to my conclusion for now, but recent developments in Malloy makes me think we're on the path for something great.
Here are some reflections on recent tests and vibe-coding sessions using Malloy.

🎨 Beyond The Bracket

In April, I went through the Channel Tunnel from Paris to Dover by car. From there, I drove northward, winding my way through the English countryside toward Matlock, nestled in the rolling hills of Derbyshire in the heart of the Midlands.

It was the first time I had to drive on the left. I anticipate the change to be more difficult than it actually proves to be in practice.
Maybe my neuroplastic is still in good shape 😅

The Roman thermal in Bath reminded me that the Romans actually went up to England from AD 43 to AD 410.
Archaeological evidence suggests the Romans drove their carts and wagons on the left side of the road, a practice they likely brought to Britain during their occupation1
This wasn't arbitrary: in medieval times, most people were right-handed, and traveling on the left side of the road allowed knights and travelers to keep their dominant sword-hand free to defend against potential attackers approaching from the opposite direction.

In France and North America, the shift toward right-side driving was influenced by the use of large freight wagons pulled by multiple horses. Unlike British carriages, these wagons had no driver's seat-instead, the driver sat on the left rear horse to keep their whip hand (right hand) free. From this position, drivers had better visibility when passing other wagons on the right, gradually establishing this as standard practice.

Before this, the custom was for the wealthier classes -  who travelled by carriages and horseback  -  to drive on the left side of the road. The poor people did not have these luxuries, so they walked, and these pedestrians were relegated to the right side.

During the French Revolution, the government under Maximilien Robespierre mandated driving on the right as an egalitarian measure, erasing the class distinction that had previously reserved the left side for the aristocracy and wealthy classes. Napoleon subsequently enforced this practice throughout territories under French control, spreading right-hand traffic across continental Europe

Back in France after eleven days, I didn't had any issue driving back to the right.
Sounds like we often anticipate the change to be more difficult than it actually proves to be in practice.
That also suggests that with proper preparation and awareness, adaptation can occur
relatively quickly...

Ohh there's sun out there! Truly, I can't make that stuff up: sun, good food, friends, love, family, book reading, walk, sleep...

Time off abroad, visits, walks, and talks got me off the current nihilism happening in tech.
But I'm still wondering how our work and life are evolving. Being focused on product management, i.e. engineering cycle, UX issues, customer feedback and marketing (and still a foot in data) I can see tons of things I would like to automate. At least, willing too.
But each time I’m on the verge to automate something worth my time, it hits a dead end... Like, it sounds rubbish, non-human, robotics. I also fear it's not good work.

This is because models aren't that smart yet; but also because some of my work is inherintly human. Probing users, coordinating human work, priorization, design, etc. The more I deepen in a subject, the more a very good probing session is the only true valuable asset. The answer unfold logically then, it's cheap. But because of upstream taste, time, and questions2.

What's your question?

In the 1960's, aviation organizations called for a uniform system to understand and communicate weather conditions. This led to the development of METAR, a basic code format that can be transmitted quickly and understood universally.

This codified form is still used today. For example, the weather in Paris, France now is:

METAR LFPG 120600Z 17003KT 8000 RA FEW006TCU SCT040 BKN100 13/12 Q1011 TEMPO BKN006

Nearly every novice pilot asks "why can't this just be in plain language?"

True enough, most flight planning applications will translate the code into readable text.

However, seasoned pilots (along with air traffic controllers and other aviation professionals) consistently prefer the coded format. It's concise (a single line versus an entire paragraph), structured in a standardized way (you know precisely where to find any specific information you need), and it leaves no room for misinterpretation with its clear definitions.

Asking second-order questions to an LLM and hoping it will translate to a working SQL query is naive. Same with any programming language actually. What we lack in the current analytics motion is a language expressing fully our semantics. SQL is the lingua-franca for databases. Natural language is good for instructions and probing. What's the in-between? The space where we can manipulate, extend and transform our analytics concepts?

Basically, where is our analytics mathematic?

Mathematic without formalism

Kids often ask why they have to learn mathematics. What's the point of learning times tables if the calculator can do it faster and better?

As we grow, we understand two things about mathematics:

• It brings mental models and patterns that help resolve anything.

• Its semantics and syntax are top-notch. One of the best semantic engine humans have created.

I mean, which formulation do you prefer1?

• "For any given value of x, the corresponding value of y is found by adding 2 to x."

• "The value of y is always 2 more than the value of x."

• “y = x + 2”

The mathematical expression isn't just shorter—it's transformative. It allows manipulation, extension, and application impossible with prose descriptions.

The theory behind today's AI revolution is fundamentally mathematical. Every breakthrough from Newton's calculus to Einstein's relativity to modern machine learning happened not just because of ideas, but because mathematical symbolism gave those ideas wings. Without formal mathematical language, these concepts would have remained trapped in verbose, imprecise descriptions.

Also, when we discover the perfect syntax to express complex semantics, we don't just describe the world differently - we see it differently.

Analytics with formalism

I recently played with MCP and Malloy2. The magic of open source has brought the work of the great Kyle Nesbit with its Malloy Publisher (software to serve Malloy model over API) and Nick Amabile's MCP build for Malloy.

[Malloy] Publisher takes the semantic models defined in Malloy – models rich with business context and meaning – and exposes them through a server interface. This allows applications, AI agents, tools, and users to query your data consistently and reliably, leveraging the shared, unambiguous understanding defined in the Malloy model.

Just as “y = x + 2” allows for transformations impossible with prose descriptions, a semantic modeling language like Malloy allows us to express “revenue = sum(order_amount) - sum(discount)” as a reusable concept that carries consistent meaning across all queries. This isn't just about brevity - it's about creating manipulable building blocks that can be composed into increasingly sophisticated analytical expressions.

As per the image below, an MCP is one of the possible interfaces to interact with this now proper semantic layer.

Here are some afterthoughts after playing with these:

First-order questions dealing with direct data retrieval and simple aggregations ('What were total sales in March?') are fine, but second-order questions that require understanding relationships between concepts, business logic interpretations, or multi-step analysis ('How did customer retention correlate with our pricing changes across different segments?') typically fail. The latter requires not just data access but a semantic understanding of what the data represents in a business context.

For example, as the semantics for users and orders are properly defined, such prompt works well

But complex prompts challenge the model.

This might be due to poor prompting on my part, but it still seems like a fairly basic question that a human could easily answer by using a Malloy or SQL query directly.

These mixed results are not surprising. The MCP server wasn't extensively fed with Malloy query examples or explanations
It's likely that if we properly educate LLMs on Malloy's syntax and semantics, it could function as an ideal "text-to-SQL" framework, with that proper semantic layer built in Malloy in-between.

After all, the effectiveness of an LLM often hinges on the quality and relevance of its training data. We could argue that this applies to any semantic source, including SQL. So what makes Malloy special?

First-order questions can be handled by an LLM using SQL. The “text-to-SQL”. But when we move to second-order complexity, asking good questions and getting reliable answers absolutely depends on having good underlying semantics.

Like, how do we address situations where the LLM misinterprets business logic or incorrectly assumes relationships within the data? We will again have to "teach" the system about our specific data semantics... And SQL isn't designed for such. We somehow need proper formalism for our business theory.

Context is the new frontend; backend?

In traditional systems, we separate frontend (what users see and interact with) from backend (where data lives and logic executes).

On our side, the semantic understanding of data is a coevolutionary relationship between the frontend we interact with through natural language and the backend that ensures query accuracy and consistency.

The frontend not really new: we’ve been asking dashboard with natural language for years now. But now, we are about to fit the backend.

What were the weather conditions in Paris regarding temperature, winds and visibility on daily-average over the last month?

source: weather is duckdb.table(“weather.parquet”) extend {
measure:
avg_temperature is temperature.avg()
avg_wind_speed is wind_speed.avg()
avg_visibility is visibility.avg()

view: weather_paris is {
group_by: date_.day
aggregate:
avg_temperature
avg_wind_speed
avg_visibility
where: location = ‘Paris’
}
}

1

by the way here is the first answer I get out of the latest model:

Last Sunday I vibe-coded. Jeez, that word... Sounds like an expression we will all forget in a few weeks.
For me, it was a simple single-page app to plan my remaining PTO over the coming months1. I built, deployed, and bought the domain in less than forty-five minutes.

Instead, I could have used the good old spreadsheet. But I'm tired of this interface. I prefer ones designed for the specific. And we can build anything, so...

As many have already discussed, a background in software engineering gives us an advantage over those who are only here for the vibe. I mean, my prompt explicitly asked to use React and Tailwind under a specific folder structure constraint. This way, I can understand what's going on, use Git properly, and potentially maintain this longer than just for a vibe.

I acknowledge I vibrated when I saw the result of the first prompt. It felt like reading a great article from a niche blog.
Software might be content now. But is it good? What's good content anyway?

As for good content, good software makes us vibrate. And in the end, I don't think my app gives me the same buzz after the rush and excitement I felt from that coding session.

Those last years reduced the value of content - blogs - but it didn't reduce the value of content makers - writers. And it’s the same for software and software engineers:

[...] reducing the value of software is not the same as reducing the value of software engineers. Vibe coded software, when built by a skilled engineer, isn’t really vibe coded; it begins to feel like where an EM has overseen software built by a team of junior to mid-level engineers.2

The vibe-coding charm strikes our sight. But the merit of thoughtful design and engineering contents our soul. As it has always been.

📡 Expected Contents

The creators of SQL

All information must be represented explicitly at the logical level, and in exactly on way: by values in tables.

That's one of the twelve rules of E.F. (Ted) Codd - creator of the relational model in database management - whose research ultimately led to SQL. It does resonate with the limitations exposed by Lloyd Tabb.

I recently came across this awesome conference from Don Chamberlain, one of the co-creator of SQL.
Beyond the great story of SQL and interesting historical facts, I'll keep two things:

• "SQL Philosophy: provide flexible tools. Trust the user". I think we shouldn't trust the user. Especially when it try to map non-linear, business relationship, with a tool not designed for such. I trust that users will find innovative ways to use the tool beyond what I could have imagined though…

• SQL++. With AsterixDB as the open-source project, partially sponsored by CouchBase. It's 10 years old, and it's interesting that even with one of the creators of SQL in the background, this project didn't grow - as far as I know - that much.

👉 Watch the conference

👉 Read the paper

How to build an Activity Layer

I recently been involved in some activity modeling and I can feel that classic OBT or Kimball-like modeling doesn't fit perfectly. It works, but thinking and reading more about Activity modeling, my opinion starts to shift.
Here is a great post looking at most of the question I faced and unfolding the value of choosing such pattern.

👉 Read the post

Vibe Coding with Cursor

That will be the first and last link around vibe-coding in this newsletter. If like me you want to try Cursor within an afternoon, these ones are great:

👉 Build an iOS app with Cursor

👉 Interesting workflow

Data Stack Templates

I'm quite excited about Julien's news: he just announced the public opening of BoringData, a set of Terraform templates designed to help you kickstart your data stack quickly.

As an early beta-tester and following my discussion with Julien on this subject, I can't wait to see it out there!
I start to think of the same pattern for data analysis. Imagine templates for standard analysis: LTV, churn, cohort based analysis, etc.
With BI as Code and new tools around the corner, I feel we're close to better automation and faster data-to-insight-to-action.
More on this subject later, but for now:

👉 Checkout Boring Data

📰 The Blog Post

No new blog post this month. Have been busy by trips and work recently. But definitely coming back with more soon.
Instead a funny one (am I the only one having issues with Google Meet recently?)

🎨 Beyond The Bracket

I currently can't stop to read Recursion by Blake Crouch.

Our ancestors lived in the oceans. Because of how light travels through water versus air, their sensory volume-the region in which they could scan for prey-was limited to their motor volume—the region they could actually reach and interact with. What do you think the result of that might be?"

She considers the question. "They could only react to immediate stimuli."

"OK. Now, what do you think happened when those fish finally crawled out of the ocean four hundred million years ago?*

"Their sensory volume increased, since light travels farther in air than in seawater."

"Some evolutionary biologists believe this terrestrial disparity between motor and sensory volume set the stage for the evolution of consciousness. If we can see ahead, then we can think ahead; we can plan. And then we can envision the future, even if it doesn't exist."

"So what's your point?"

"That consciousness is a result of environment. Our cognitions— our idea of reality-are shaped by what we can perceive, by the limitations of our senses. We think were seeing the world as it really is, but you of all people know...it's all just shadows on the cave's wall.

I won't spoil it, but the story explores time travel and memory alteration in classic sci-fi fashion3.
This weirdly resonates with the unexpected loss of my favorite artist Kartell, taking a piece of me with him as his music deeply shaped my identity. I find comfort in knowing his music remains—both as nostalgia and as part of my future.

And April hits! It has been a sturdy month but for the best. I'm glad to have coordinated work around the Kestra 0.22 release, messed around with Cursor and MCPs, wrote this newsletter, taken time with relatives, and had the chance to see the sun shining again ☀️

Next month is mainly about new horizons and good times: a weekend to celebrate my love relationship, heading to Germany for another weekend, then the UK for 10 days.
I'm also planning my upcoming trip to the US this summer: if you're in the Seattle/Portland area, please reach out! 🇺🇸

2

davidj.substack

Vibe coder

I have mentioned before how I have continued using Windsurf to build more things, since my success using it with sqlmesh and dlt…

Read more

a year ago · 3 likes · 1 comment · David Jayatillake

There was a wall. It did not look important. It was built of uncut rocks roughly mortared; an adult could look right over it, and even a child could climb it. Where it crossed the roadway, instead of having a gate it degenerated into mere geometry, a line, an idea of boundary. But the idea was real. It was important.
For seven generations there had been nothing in the world more important than that wall.
Like all walls it was ambiguous, two-faced. What was inside it and what was outside it
Ursula K. Le Guin's "The Dispossessed"

I've been building analytics motion recently. Thinking about exposing data to non-technical people while ensuring the technical is done properly.
That depending on the side of the wall you stand by, you get what you expect.
Each side presents things differently. But they are ultimately the same under the hood. Dashboards or rectangle tables. They are two faces of the same coin.

Working with both populations, I find myself on that wall. Always battling between the duality. Business. Engineering.
Walls are nice. They bring strength which is essential to maintain identifiable character. Focus.
But they ultimately create ghettos.

In a city made up of ghettos, people have the support of the most basic [...] People in the ghetto are usually forced to live there, isolated from the rest of society, unable to evolve their way of life, and often intolerant of ways of lie different from their own.
Christopher Alexander, “A Pattern Language”

On one side we have the CLI. The MDC file into a fresh new code editor.
On the other the cluttered CRM. The 2 months old - already dead - spreadsheet.

Analytics engineering is one of the clutches standing on that wall. But its vocation is only looking at one side.

The other side is crappy too. Gold paved, with ambiguous foundations. Because of that wall.

We design pipeline roads to ultimately break that wall. Bringing the stone foundation for (expected) infinite streets.

The only way to break that wall is to get messy. To look at the other side. To understand the incentive. To truly understand why this spreadsheet exists.

Sometimes it feels like only one side is trying to break it. But it's because the other is under never-ending pressure. It has to pave the street.

Standing on that wall sometimes, I encourage you to look at the other side. To even try to pave the street too. Then you will understand. The spreadsheet. The deal closed. The optimism. The value.
You can come back into your ghetto. It's needed.
But out of LLM recents breakthroughs, now, we will soon expect everyone to meet on that wall.

📡 Expected Contents

Data Union

Union is not something I've been interested too, but the recent market and geopolitics trajectory, plus whats happening with AI gives me a sense of need for better counter-power.

Maggies comes back with a short post showing what a consumer-data-union could be. It's more just an idea than something real right now, but with more and more paywalls popping up (which lets us move away from ads all over the internet; get your wallet ready), it looks like a good path for controlling our own data in the future.

Data Unions

SQL Noir

I never really learned SQL at school or my job. It was mostly on the spot, to get things done.
Most of the time, I learn new knowledge and tricks via projects.
But here comes something I would love to have back then when I was bored in the classroom.
This sandbox allows you to learn SQL by playing the role of a detective. Even knowing a bunch of SQL myself, I couldn't stop to resolve the case.

👉 Play SQLNoir

Making a spreadsheet

Spreadsheet is now a common good. Too common sometimes. But did you try to build one yourself?

This is what David does in this post: creating a spreadsheet. It's not that complex but a savant mix of algorithm and UX thinking.

👉 I made a spreadsheet

The single most important SaaS metric

Very good post on how choosing a good metric is key in business. TL;DR There is not only one; it depends on your business.
What's your current one? What's your next one?

The most important SaaS metric

📰 The Blog Post

Speaking about walls. Here is one with some windows.
Finally put down my thoughts on the semantic layer concept. It's my digest of last year discussions, talk and reading on the subject.

🎨 Beyond The Bracket

Friendly reminder: this is a tiny space.
You're reading this inside a way smaller part of that space.

Close your laptop, smartphone, or whatever screen you're glaring at.
Enjoy birds, a Moroccan tea, a Sade’s song, whatever makes you happy.

I recently succeeded in reducing my smartphone screen-time - which was already less than 3 hours per day - to less than 1 hour.

I quit Twitter. With quite a nostalgic feeling as it's where I started to write, blog, and grow a kind of network.

Though I have mixed feelings about the leader of this platform, taking some time there doesn't bring any value anymore. I have to admit that on Bluesky neither…

I feel like I'm entering the post-era of social networks. Reading about switching to dumb phones. Only getting contact with the news through Substack newsletters, this French show sometimes, and my friends or family.

FOMO isn't really here anymore - as the pace of things is too fast, there's no real value in keeping up to date with everything. Call this pessimism or seniority. Or maybe, optimism and wisdom, for things that actually matter.

Had a very nice February here. I went to the UK: I love this country. As a French I’m not ashamed to say that English people are nice and they somehow have good food (yes you're reading well). Going back there in few weeks to eat more jelly beans!

March's already in good shape! Tons of trips are planned this year, and I have a blurry sense of new projects to come. The time out there will probe it.

Hope you're doing well; see you in a month 👋

For a few years now, we have seen the emergence of a concept called the Semantic Layer. It's not a new fresh idea, but it kinda resonates with the current state of analytics maturity we're reaching.

But while the abstract idea of a semantic layer can be defined pretty easily, it's still hard to navigate around the syntax, the implementation of this layer.

Why do we need a semantic layer in the first place?

We have customer relationship management (CRM), we have project management, we have HR, and we have finance software. The core value of these tools is to play as interfaces. These are the layers where faces get to interact with each other. Inter-faces.

But what's the interface for data?

The obvious answer is the database - especially the data warehouse now. These now have role-based access control (RBAC) and teams (data engineers) managing what's in and out. Hence, it's the main interface to move forward on the data-to-insight motion.

However, this interface is kinda flawed. Firstly because the main syntax to play here hasn't been designed for such, but also because the faces who need interaction with data are plenty, and from very different backgrounds.

Out of these realities, we talk nowadays about the Semantic Layer:

A centralized control panel for company's data – a semantic layer manages data modeling, data access, and sends consistent data and metrics to every BI software, data app, and AI/LLM tool.

This is indeed a layer, an interface, that allows any users to access and interpret data using familiar terms without needing deep technical (data) knowledge and in a consistent way. The best-suited use cases for a semantic layer involve centralized metrics definition, unified data modeling, query optimization with a cache layer, and security and access control governance1.

As I'm writing these lines in 2025, the current state of semantic layers is not as gorgeous: nowadays, a semantic layer is basically a team of data engineers and data analysts that play the ETL game and get ad-hoc analysis done.

Don't get me wrong: like CRM, HR, finance, product, etc. there will always be ad-hoc things and Excel spreadsheets. But like our cousins, we will somehow have the proper software to get our interactions fixed and guidlined. Basically, shifted left.

SQL is not designed for analytics

We have the chance to know our semantics quite well. At least we have proper (sometimes endless) discussions about it. It's how we define ARR, what's a user, what's the user journey, etc. All of these elements are different from company to company, but ultimately we are able to sit down and put the equation on paper.

The next question is how to move from paper to proper sheets?

Here, the database and its main syntax, SQL, is the first thing that comes to mind.

SQL has actually shaped how we think about data. It's all about rectangular pieces. But that brought us limitations.

SQL is not a functional programming language. It's not possible to build up easily queries on top of queries. Put another way: it's not possible to build up our semantics. There's no polymorphism, no functions. It's hard to factor our work.

Semantics often trail behind syntaxes: our world is moving fast and so our ideas and needs. Our syntax is more than 30 years old. It has been designed for OLTP applications, for relational models. But business models aren't relational models.

To bridge this natural gap we have built tools, frameworks, and platforms.

And it works. It's now common to see medallion architectures built on top of dbt and cloud platforms all connected to BI tools and data applications.

This propels a kind of data-to-insight-to-action motion. That makes us, somehow, data-driven.

But in the end, we have yet to upgrade our steering wheel. Managing all the data work through hundreds of lines of SQL, templated with Jinja to fix the lack of functional capabilities, limits us to move to the next step2.

At some point, we will need new syntaxes. Leaders in that space already agree since years:

I have to admit, I didn't think about dbt and Malloy working together... And all of that makes sense in the longer term. I think it'll be really interesting to see how far Jinja gets stretched by what people do with metrics. It feels like entire mini-languages could get built within it, to the point where a lot of what you're describing—an imperative language that compiles to SQL—almost falls out by accident.3

But it's not easy. It's actually "lots and LOTS of work to do".

The LLM Fallacy

Nowadays we are expecting to primarily interact with software in natural language. That's lazy. I mean, good engineers are lazy. But they are lazy in a good way. Because they have interfaces that let them explore the problem and solutions space in a variety of ways. The interface offers nudges. Guidelines.

The bottleneck is in the data-interpretation, the code-comprehension. The bottleneck is always the problem at hand, it's never the boilerplate, it's never the patterns. Our expertise isn't in knowing how to write a case statement in SQL, or anything else you could get an LLM to write - it's in knowing how to deal with the specificities and intricacies of the problem we're working on. The work does happen to be code composition.

We could fall in the trap that natural language is the ultimate interface for the computer. But in the end, the code isn't for the computer—it's for us. It's for humans. If the coding language was meant for the computer, we would all be writing in pure binary instead of these abstracted and symbolic languages.

The code is the interface that we designed to be able to program the computer. It's what we need. It's objective, explicit, unambiguous, (relatively) static, internally consistent, and robust. English has none of these properties—it's subjective, meaning is often implicit, and ambiguous, it's always changing, contradictions appear, and its structure does not hold up to analysis.

And so, the text to SQL motion lacks an essential value: semantic composition. Yes, LLMs are one of the best partners to probe our semantics. It's like discussing with a human: you bring context sentence after sentence, question after question. Good prompting engineering can be a key here to getting good probing questions and moving forward with our meaning-making of the data.

When we create a well-defined and bounded semantic layer, we can provide LLMs with context. While LLMs don't truly understand semantics, they're similar to an intern who would be overwhelmed if given database schemas and asked to perform detailed analysis without proper context. By working within this structured semantic layer, it becomes much easier to combine our internal knowledge, business relationships, and desired insights in a meaningful way. We can probe and build up the semantics in natural language, as we would do naturally in the meetings and at the coffee machine.

As natural language comes with ambiguity and context-dependency making precise composition becomes a challenge.

That's the main reason we write code. We want to write the least possible, but still write it. Because it often brings composition and a set of routines hopefully close to our semantics.

Interface & Composition

I like to think of this whole problem space as a problem of interfaces and composition.

The first interface here is humans - natural language. The second is the database - SQL.

These two interfaces are very very good. While it's hard to build up our semantics with natural language, it's still very useful to understand and probe it. Especially with LLM and good context. There's a world - not so far - where you can talk to the best analyst and expert with your day-to-day language.

Same for SQL: it's probably the best language we had created to interact with the database. However, we probably went too far with it. It was originally designed for OLTP applications 30 years ago. There are no functions in it. We have brought tools around to fix it (templating language like Jinja, frameworks, etc.). But dbt is jQuery, not React. It's ultimately tough to implement the semantics we want with it.

And so comes the semantic layer. An intermediary interface allowing to compose our semantics.

Here the technology space is in bloom.

The leader here is probably Cube. It comes with a declarative DSL allowing to expose a semantic to any common interfaces (dashboards, apps, etc.)

It embarks cache, pre-aggregation, and access control. It also brings some polymorphisms and real composability when well designed, but I agree with Carl that YAML (or any markup language) is probably not the way forward here. At least not only.

Just like programming, data modeling and exploration is a very cognitively demanding process, and cognitively demanding processes benefit tremendously from fast feedback loops. At the Malloy team, we're trying as hard as we can to ensure that data modelers and analysts can achieve "flow state" while doing their work. This means doing hard things, and sweating every tiny detail that causes friction in the experience.

From my experience, I can say that YAML pitfalls don't really apply when it sits in a proper user interface. Also, Cube approach looks similar to Terraform or Kubernetes deployment in the way that it's easy to adapt, to fix and maintain without getting "in the flow". It's much more of an exposition layer.

On the other side we have new languages like Malloy. It changes the way we are looking at data and brings us way closer to the semantics we want to build up. It comes with routines and a defined syntax that just fits naturally. Just look at this.

It gets us in the flow state and allows us to compose our semantics with ease.

For now Malloy doesn't make it easy to expose this semantic. BI tools are waiting for rectangular inputs (tables). For a SQL API call (like in Cube).

However, it does come with a charting library.

Declarative BI is a thing now and there is probably a space where the final interface for insight-to-action motion is not propelled by a pure dashboard built by dragging and dropping around but in natural language bounded to a semantic layer that knows (or where you choose) what's the best way to represent data. Could be a chart, a pivot table, a text, a slide deck, a voice memo or an Excel file.

Recently, I also have been amazed by how dbt and Lightdash integrate well together. Using this duo with a bit of AI to fill dbt documentation: it allows to build a full semantic layer within dbt while taking advantage of pre-built aggregation in Lightdash. It's still SQL interlaced with crappy Jinja. But at least everything is as code and it's a realistic and pragmatic solution - especially now with dbt being a standard. I think it's a good first step to implement a semantic layer (plus in bonus, Lightdash has a metric tree builder - more a toy than anything, but still, we are making progress).

What's the syntax for the semantic layer then?

To me, it's YAML for exposing and maintaining the glue for downstream applications. And it's a new language that allows us to build that semantic upstream. The latter is probably built on top of SQL as it's the best database interface we have.

Depending on the maturity of our business and level of technicality, it could be a bounded chat application, only YAML with Cube, Malloy for cooking analysts, a mix of both, and sometimes it will be another thing. Don't worry we will have a buyer guide.

Like always, the syntax will trail behind our semantics. We should be flexible and be ready to move it to the trash. This is sometimes the best way to move as fast as possible on our ultimate goal: shorten the data-to-insight-to-action motion.

Why isn't a free Reforge for data analysis?
Why isn't a free Reforge for data stack?
If there is any, please point them out to me.

Yes, each company has a different set of data and contexts that makes one analysis hard to reproduce in another set.

Building a cohort analysis, a bar chart with time on the x-axis and a metric on the axis, or a pie chart that nobody will look at1: all these always have a common ground.

The core issue on this is two-fold: we build our charts without reusability in mind. And we build them with tools that make it hard to reproduce them.

"BI as Code" isn't a new trend out of thin air. It's again the realization that code is a necessary need to build up on our work. To do reusable work.

Yes, it's most of the time harder than drag and drop in the first place. But recent advances in tools, languages, and support, make our stall spaces starting to move forward again.

I'm craving for the common ground and "first go to" analysis every company is doing. Can we automate it? Make a template out of it?
If the inputs are an activity schema model or a classic dim & fact architecture, could I write a Malloy code, a Cube specification, or a Rill declarative dashboard that will fit many cases?
Where is this library of analysis reusability and the community that goes with it?

I'm exploring such a possibility and launching an MVP. Feel free to contact me if this means anything to you!

Leave a comment

📡 Expected Contents

Lessons Learned Implementing Metric Trees

I recently had a nice call with a Head of Data: he struggled with shadow data practices and tons of data to expose.
It's hard to prioritize the next dashboards, the next north star metric, the next SQL queries.
This nice interview traces back to the implementation of the metric tree in a startup. As for the activity schema, it's great to see these implementations of new data frameworks taking life in real companies.
I can't wait to read similar feedback2.

👉 Read the interview

The Icerberg Scam?

I recently wanted to deep dive into S3 tables and how to use DuckDB as the query engine here.
I quickly stopped as per the screenshot below and this great post it's only possible to access these S3 tables with Apache Spark as an open-source query engine.
Yes, it's not related to Apache Iceberg per se. Still, from discussions with some pairs in the field, it sounds like Databricks or Snowflake are the only "serious" (not too painful) way to go. But then I feel the promises of "open" table format and cost reduction aren't satisfied.

Again, it's a very new field, but my main point here is why we care so much right now?
I mean, we have great data warehouses. It's not that expensive. Especially knowing we don't have big data and the promise is to create value out of the data. Not reducing costs as the main focus.
To me, it sounds like a very engineering subject. Trendy because the macro-economic trend is about cost reduction and efficiency. But having our engineers focus on implementing a lakehouse while our data models are still crappy and hard to maintain sounds like a suboptimal path to me. Ahead of phase.

I might be wrong and too opinionated on this. What's your view?
Sounds like our busyness isn't that much focus on the business recently.

dbt, sdf, sqlmesh, quary

We're seeing the SQL for analytics world converging. dbt aquired sdf. sqlmesh acquired quary.
I recently built a full data model using only BigQuery views and corresponding Terraform declarations. Don't get me wrong, I started by using SQLMesh. But in the end, these tools are only necessary when you have a strong team of analysts and you want to enforce some best practices. In my case, I was alone, and without tons of data sources to gather.
It's ok to schedule queries for partition creation, to create views of views, to template queries in plain Python. To simply make things done, without creating a monster of dependancies.

These new frameworks are new and are mostly glue allowing to template SQL because we lack a proper lingua for the stuff we want to do.
Something I learned in the past 5 years: don't use a framework because it's trendy. Because it's what everyone is doing.
Use it when you know your incentives. When you explicitly need a frame to scale your work. That's the goal of a framework.

About Super-Intelligence

I still don't know how much I should deep into the AI subject3. I sometimes think about all these engineers, all these prompts, all these new models, etc. As we are probably approaching the end of the "early adopter" curve, it might be the time isn't?

If like me you're looking for a great resource for getting the big picture, I can't recommend more Leopold Aschenbrenner's blog. He is an ex-Open AI and has written a huge series of posts regarding AI and the superintelligence singularity.

Take a big cup of coffee before pulling these threads.

🧵 Threads about AI

📰 The Blog Post

Last month I wrote about a more realistic call regarding my "SQL is not designed for analytics" views. I still stand by this opinion, but to move forward we need small steps. We need to be pragmatic. That's what I'm exploring in this blog, with BigFunctions as one solution in this complex equation.

🎨 Beyond The Bracket

It's amazing how quickly Midjourney blog header images went from a "cool way to stand out" to a "sign of low-effort content".
Long-time readers know how much I care about aesthetics, and especially about photography. And my profound connection to identity.

I craft most of this newsletter and blog post manually, without AI intervention. I remain an optimist and technology enthusiast, so I primarily use AI as an editorial assistant. It performs admirably, allowing me to preserve my voice while benefiting from precise proofreading and grammatical checks.

On the same spot, my most compelling reads emerge from authentic writers. My favorites are those with unmistakable, distinctive styles. Pieces that create an immediate sense of intimacy and comfort. As if we're long-time friends sharing an unspoken understanding.

In an era where AI usage will become as commonplace as using Google Docs, I might take advantage of this new paradigm. Yet, ultimately, the core of my writing—the essence of my voice—resides in my daily experiences, in spontaneous conversations, in those crumpled notebook pages, and scattered iPhone notes. Not in the sterile output of a generic AI prompt interface.

At work, I've been doing similar things. These past weeks, I focused on building a data model. I used AI a bit: to clean up my database schema and check my metrics. The real work was connecting ideas, testing assumptions, and designing solutions. The final result matches a specific case that would be hard to create through simple prompting.

I feel like I got a life upgrade recently. Though work. Heart loaded. It feels like my meaning-making delta isn't in sync with my current version of self.

Yet, I know that everything will ultimately clock together. As usual, in the months to come.

The writing I'm putting out right now is probably the last bit of a me.
And so, the reading I'm discovering right now is probably the first bit of a new me.

See you in March ☀️

Reminder: code is technical debt. The optimal spot for engineers is to write the least code possible but still write code.

And SQL makes this particularly painful. Like every data person, I've written my share of 300+ line queries - cluttered with nested CASE WHEN statements, complex window functions, and endless ORDER BY clauses. We've all been there.

It…

Read more

Code is technical debt. The optimal spot for engineers is to write the least code possible but still write code1.

Nowadays we might write our code with Cursor, Windsurf, etc. We even design new code editors. But we still want to write code. Not pure text in natural language.

Expecting users to primarily interact with software in natural language is lazy.
It puts all the burden on the user to articulate good questions. What to ask, when to ask it, how to ask it, to make sense of the response, and then to repeat that many times.
But a user may not know what they don't know.
A good user interface let's me iteratively and incrementally explore the problem and solution space in a variety of ways.
A great user interface guides me and offers nudges.
Couldn't a natural language interface help with that?
Certainly.
But not as the only option. Probably not even the main interface.2

Writing code is basically rendering semantics in a given syntax. The semantic is the problem to solve, the syntax is our 300 lines of SQL. The sweet spot of this journey is when our syntax is neat and close to our semantic.

And this is hard to get.
Especially since our semantics are getting blurry as everything is getting faster and wider in our problem space. And our syntaxes - interfaces - are old.

Also, our data stacks - interfaces - are planned through tools and silos. So we get tools and silos. But I come to realize that if you plan for insights and activation, you get insights and activation.
The data-to-insight-to-action motion doesn't come out of thin air. We still need interfaces. We still want to write code (again, the least possible).
But for now, the code we write is far from our semantics. The insight-to-action motion propelled by our tools sounds to be on a good dynamic: it's the dashboard, the Slack alerts, the o3-based RAG.
But the first part, the data-to-insight, doesn't feel well designed. It's too long codebases. It's not functional. It misses proper interfaces.

I long to see how new languages, new layers, and ultimately new UI, will move us closer to that sweet spot.

📡 Expected Contents

Code Fast

And so you want to write code. Welcome!
I recently wanted to move forward on my web development learning. These last quarters I ultimately failed on this: learning Next.js, HTTP serving in Go and basic authentication, etc. I learned tons of things but didn't ship anything serious.

Following solopreneurs out there, the recent course from Marc Lou hits at the perfect time. It doesn't go through days of lessons to learn all the details of React frameworks: it's 12 hours of videos and tips to learn how to ship a project - serving actual customers.

If like me you're not looking to only sharpen your coding skills but also ship something: I urge you to follow Marc and look through his course.

👉 Marc's Vlog

👉 Learn with Codefast

Ikea Oriented Development

If you have a hammer and screwdriver, you can build Ikea furniture. Everything else comes in the box. Nobody wants to hunt for a 6.2mm allen key when what you really need is a bookshelf.
[...]
In the computing world, screws are made of plaintext, HTTP, etc. Today's shells and standard libraries offer ubiquitous screwdrivers like Regex manipulation, HTTP processing, and JSON parsing.
If you can't bundle allen keys for your hex fasteners, stick to screws. Likewise, if you lack the engineering resources to support multiple SDKs, make damn sure your web API is easy enough to access with curl.
My MarioKart 64 cartridge probably won't inform me that Python2.7 was deprecated. If your program isn't designed to work 20 years from now, it won't.

Again a great post on why software should be soft.

👉 Ikea Oriented-Development

Make it ephemeral

In the physical world, much of what we create has a natural tendency to decay and that is really useful information. A sticky note on a monitor gathers dust and fades. A notebook fills with notes and random scribbles, becomes worn, and eventually ends up in a cabinet to finally end its life discarded in a bin. We probably all clear out our desk every couple of months, tossing outdated items to keep the space manageable. When I do that, a key part of this is quickly judging how “old” some paper looks. But even without regular cleaning, things are naturally lost or discarded over time on my desk. Yet software rarely behaves this way. I think that’s a problem.

Great call on how "skeuomorphism" can be an important concept to consider when designing software and pipelines. In our digital world, legacy stuff is such a headache while the physical world just seems to roll with it naturally.

👉 Read the post

Why software only moves forward

Once you save state, your code will need to understand that state forever. This is double true for state that leaves your system and becomes distributed.

Kinda echo the previous link above - nice write-up on how data makes things only move forward in the way we design states and corresponding software.

👉 Read the post

📰 The Blog Post

Ok, I'm writing five blogs at the same time. It's probably not the best way to ship one each month. So, here we go again with no proper blog post...

Since I enabled paid subscriptions on this newsletter, I only sent posts under paywall to corresponding subscribers - I didn't want to spam your inbox too much.

Starting next year I'll try to send them to everyone here. I'm expecting some drops in subscriptions. That's fair enough :)

If you want to give it a try, here is a discount for the new year 😉

Get 50% off for 1 year

You can get a taste with this last post I wrote a few months ago:

🎨 Beyond The Bracket

As I delve deep into our daily syntax and semantics, I find myself stepping back to contemplate how natural language has evolved. Here again: semantics evolve constantly - as our world change - but syntaxes root themselves in different timeframes.

Just look at how the Latin alphabet roots itself from older syntaxes. That's a good reminder not to get caught up chasing new syntaxes just to match the semantics we want to push forward. Syntaxes move slower, so we need to find those subtle tweaks that'll actually give us something workable.
Again, SQL isn't designed for analytics, but it's still doing a great job at it. We have new frameworks and tools that improve it along the way. Same with JavaScript - yes, new frameworks pop up constantly, but the core JavaScript that renders HTML and CSS dynamically in our browser hasn't really changed.

So here we are again; using a 30 years old alphabet. A civilization try to make it move forward. A prophet3 asks good questions.
The only thing I can take action on now is to write it thoughtfully, as little as possible, but still write with it.

Here we go for 2025!

I'm personally so grateful for the things I accomplished in 2024: giving a course to final-year students, learning tons of things about product and sales motions at work, speaking in front of 300 people, etc.

2025 already looks promising!

I wish you the best for the next convolution around the sun ☀️

We often get questions like “What’s the difference between a BI engineer and a data engineer”? “Is the analytics engineer the new BI engineer”?

Those questions are definitely relevant. Beyond buzz words, they catch the same feeling that data in general, isn’t really a new field. There are new words around, but the real background behind them is the same as ten years ago…

Back to Business Intelligence

Let’s step back 20 years ago.

Companies are already using data to make decisions. The paradigm already has a name: Business Intelligence. The main roles regarding data are Database Administrator, Reporting Analyst, and Business Intelligence Manager.

At this time the data scientist is more of a PhD Researcher and the data engineer doesn’t really exist.
Business Intelligence people deliver insights and rely on proper tools to do so. The end user doesn’t play with data directly, it asks the BI team for reports.

No self-service, no Big Data, no neural network, no semantic layer, no cloud provider, no machine learning, etc.

So what has changed?

Everything: business plans, economies, compute costs, etc.. But the real thing is “Big Data”.
Long story short, we live in a world creating way more data than 20 years ago. The velocity, volume, variety, and veracity of data are now huge.

This new shift brought new opportunities, but it also created new problems that can’t be solved within the Business Intelligence era.
Tools were very restrictive. Compute capacity was not able to handle billions of rows at a daily rate. Data processing couldn’t be done on high scales.
Corollary, people were trained to analyze tabular data and create basic charts. Not to deal with raw images, terabytes of unstructured tables, ingest real-time events, or run predictive analysis…

All the tooling has quickly moved to an obsolete state. And to a certain extent, BI people too.

Migration to data engineering

Then a new era appeared. Data moved to an even more engineer field as it was. Hence the data engineer.

There are several driving elements behind the shift towards cloud computing and the democratization of big data :

• Hardware capability increased by a big factor.

• Storage and compute costs decreased (allowing to store and try things non-fundamental for business at the time, but that would be key for innovation).

• Storage and compute could scale exponentially while the investment in hardware was quite linear (for the sake of simplicity: two 8CPU machines are cheaper than a 16CPU one).

• Networking improved with higher bandwidth and better technology — allowing easier data transfer on many nodes.

Those improvements were led by researchers and inside growing startups like Google, Yahoo!, Amazon, and Facebook. All of those companies faced the same challenge of indexing and storing Internet data at scale.

Quite naturally, some of them saw a disruptive opportunity :

“Hey, we have a lot of computers here ! What if we leverage this computer power to provide cloud-based services to businesses and individuals !?”

Cloud providers were born, several frameworks and code languages have been created, making it possible for businesses of all sizes to access powerful computing resources at a fraction of the cost of building and maintaining their own infrastructure.

🖇️ For more details on the whole history behind “Big Data”, read this great article

What we often miss in this new world is how data engineering is essentially a means to keep Business Intelligence practices up to date with the shift toward “big data”.

We build pipelines with new frameworks and program languages. BI did too with Talend, Pentaho, Datapine, MicroStrategy, etc.
We build dashboards with self-service tools. BI made dashboards too with SAP Business Objects, SAS Business Intelligence, PowerBI, etc.
We schedule jobs. BI did too with Cron, Rundeck, Control-M, and other similar tools.

Data engineers are to a certain extent a replacement for traditional BI tools.

When we look closely at those BI tools, we can wonder why we don’t use them that much nowadays? Why they are not part of the “Modern Data Stack” as they still have great features and bundle common services?

For example, many BI tools offer features like data governance, data quality management, and data lineage tracking: critical components of any modern data stack.

While data engineers have developed new tools and techniques to extract insights from big data, it’s essential to remember that many of the core features of BI tools are still relevant today and should not be overlooked.

The reason that traditional BI tools are not as commonly used today is that they lack the scalability and proper user experience required to handle the amount of data that organizations need to process. With data arriving at a faster pace than ever before, traditional BI tools simply were not designed to handle these constraints.

Hence the data engineers, are able to process any kind of data, create big data warehouse, and provision self-service data tools.
The emergence of a “modern data stack” is a direct response to the challenges faced by data engineers. Any new tools in the data industry now are here to enhance and automate the work of data engineers in some way.

While the technical solutions for managing big data have been available for some time, companies have been hesitant to adopt full automation due to the perception that it was cheaper to hire humans to develop and maintain projects rather than to invest in machines or start automation projects.

However, with the demand for skilled data engineers increasing and salaries rising in recent years, companies are now seeking new tools and technologies to fill this gap and help automate their data engineering processes. Putting money where there is more value.

Driving greater value

It’s interesting to note that the data industry often lags behind the software world by 5 to 10 years.

We data folks are starting to catch up though, and we’re adopting some of the same tools and techniques that software engineers have been using for years.

We end up writing Dockerfile, setting up CI/CD blocks, and using cloud services to host data-intensive applications. If it’s not DevOps what is it?
The data mesh is strongly inspired by software teams. New roles, such as machine learning engineers or analytics engineers are modeled by software needs.

Just as the front-end world moved from jQuery to React, and operations shifted from bash scripts to tools like Terraform and Kubernetes, the data industry is also moving towards a more declarative paradigm.

Read more

I'm lucky enough to have a recent shower, which flows hot water from a large shower head. It's a no-brainer, and like many people, it gives my brain a space to wander.

Last week I washed myself into a fairly old shower, with two taps: hot and cold.

This one was quite hard to control: the pressure is controlled by the sum of the (position of the) two taps, while their difference controls the temperature.

It took me several attempts to find the good spot, and then my brain couldn't think anything else than "how broken this interface is"...

Indeed, the interface we are given here is:

• Hot = (1,0)

• Cold = (0,1)

while what we want is:

• Pressure = (1,1)

• Temperature = (1,-1)

Unknowingly we apply the transformation in our head: “I'll adjust both taps to increase water pressure while keeping the temperature steady. Oops, now it's too hot. I'll lower the hot tap, but that'll reduce pressure. So I'll turn up the cold tap to maintain the right water flow…”

Beyond the importance of linear algebra to the art of taking showers in older bathrooms, it make me think (again) of how data interfaces are designed.

Is Tableau an optimal interface for driving decisions?

Is Excel an optimal interface for drilling down into data?

Is SQL an optimal interface for doing business analysis?

Is Python an optimal interface for writing data pipelines?

My take is that we should be way more intentional in choosing these interfaces.

Sometimes I feel we are using two-tap interfaces for things that should be brainless, so our actual brain can wander elsewhere.

📡 Expected Contents

Hex Explore Feature

I only had Hex in the back of my mind, as one out of the many players in the "notebook for analysts" field. The recent Explore feature raised my attention as it seems nicely engineered. I'm about to give it a try soon.
Do you already use Hex? I’m looking for reviews and use cases 👀

A take on Medallion Architecture

I would prefer naming like staging, model and presentation. This is much closer to what people already know and expresses what actually happens in the layers. Bronze, silver and gold make it easier to explain to non-technical users. That’s the only reason why the product marketers have gone for Medallion architecture, although whether non-technical users really need to understand “how the sausage is made” is another question.

I had the chance to meet David at the Forward Data Conference this week. He wrote this great piece around the Medallion Architecture: it's probably my go-to whenever I'm doing data modeling but like him, I'm starting to think that it's actually not "modeling"...

👉 Modellion

External Intelligence

I want to explore an idea I call externalized intelligence as an answer to this question.

Externalized intelligence is when a certain set of knowledge, or a certain type of thinking, is not integrated into the daily activity of the organism and is instead treated as “importable” or “referenceable” on demand.

For example, I know the concept of multiplication, and like everyone else, I know how to multiply numbers in my head up to, say, 47 squared. But what sort of “intelligence” does it take to multiply 153 by 238?

Not the kind I care about. I can defer this knowledge to a calculator or to ChatGPT. I can treat it simply as something external to me that can be summoned at will.

In fact, I would be disturbed if someone were to boast that they spent the weekend learning multiplication tables up to 300 by heart. At a certain point, knowing this begins to make you seem different — alien.

This position as an alien intelligence is exactly the position I think many data professionals find themselves falling into at organizations. While his peers may respect or even admire his skill with the numbers, he is nonetheless isolated as someone who is not quite on the same team. He is other.

Exploring an old legendary post from Stephen. Nothing to add to the quote above.

👉 Data, Despair, and the FLOWBEE

📰 The Blog Post

Things have been busy these last weeks, so no blog post for this time 😢.

However, got nice discussions and drafts going on! So expect some reading from me in the coming weeks 😉

Especially a follow-up of this one.

🎨 Beyond The Bracket

Last week, I got the chance to get on stage at the Forward Data Conference.1

Getting on stage with my "SQL is not designed for analytics" hot take in front of 300 people was stressful 😅. But I felt on the spot when several C-levels, data influencers, and industry veterans came by to congratulate me, relate to my story, and share that they learned something new.

The after-talk discussions were amazing. There's a palpable sense that we're all experiencing FOMO around a new paradigm: Text to SQL is fading, and we're moving toward the "Text to Semantic Layer" era, here are some references:

👉 Arnaud de Turckheim (CastorDoc) highlighted: "If you dump tons of tables and queries on a new recruit, he won't understand anything without context. The same applies to LLM".
👉 David Jayatillake (Cube) gave a terrific talk about "Text to Semantic Layer". It was interesting to see our talks sharing a common ground in that semantic layer space.
👉 Ethan Ding recent TextQL pivot, see this post.
👉 Looker's spiritual successors are emerging: Malloy (open-source project at Meta led by Lloyd Tabb), Omni (developed by ex-Looker team)

We're still in the "evangelism" phase, and the solution landscape remains blurry. But the conversations from last week convinced me that things would unfold soon 👀.

I might deep dive into this subject more in the coming weeks. I'll keep you posted 😉

This is the last issue of the year 🎄

I'm so grateful for these past months and how writing is now entirely part of my routines.
I launched recently Liminal Duality as a space to extend this passion. I'm even getting some ghostwriting contracting, and I'm eager to continue developing my skills here.
If you're looking to showcase your work, co-write a blog post, or put a vision on paper, do not hesitate to reach out!

Next year is already calling for great things on my side, and I hope the same for you :)
Have a good end-of-year!

See you soon 👋

What's the smartest people do on the weekend is what everyone else will do during the week in ten years

Removing the "smartest people" part, it resonates with how I operated on my weekends during the last few years.

Back then, I used to work on geeky projects over Sunday. It was all about football analytics. The backend dream was to reach a position in a football club, but you know, it was just a dream. I actually never thought that it would happen four years after the first line of code. I was enjoying my learning and applying them using football analytics as an excuse to do nerd stuff.

In that sense, the weekend version of myself was doing what I would do during the week at Olympique De Marseille.

Today, my nerd stuff over the weekend is about writing, exploring how product distribution works, and coding some CLI in Go1.
What's the back dream of this? I'm not sure. Probably that trendy solopreneur, CEO-inspired stuff.

What I'm sure, is that I love what I'm doing right now. I don't have any plans. I just craft. For the pleasure of it. I can't imagine where I would be in 3 years from now. But my gut feeling urges me to continue.

So let's craft 😉

📡 Expected Contents

The Configuration Complexity Clock

I’m not saying that it’s never appropriate to implement complex configuration, a rules-engine or a DSL, Indeed I would jump at the chance of building a DSL given the right requirements, but I am saying that you should understand the implications and recognise where you are on the clock before you go down that route.

Nice thinking about when we need a DSL, and when just a simple boiler code might be enough...

In some senses, it resonates with my recent writing calling for what's next after SQL.2
I do feel we're not using the best language for analytics. It might be the time to move to the next quarter...

👉 Read the post

The Rise of the Declarative Data Stack

I should stop writing on this subject 😅 But this one from Simon couldn't be in this newsletter edition.

A declarative data stack is a set of tools and, precisely, its configs can be thought of as a single function such as run_stack(serve(transform(ingest))) that can recreate the entire data stack.‍

Instead of having one framework for one piece, we want a combination of multiple tools combined into a single declarative data stack. Like the Modern Data Stack, but integrated the way Kubernetes integrates all infrastructure into a single deployment, like YAML.

Simon has again the good words to explain where we aim at (personally speaking, it's my day-to-day at Kestra), and especially why we need to move on.

👉 Read the blog post

The Revolving Door of BI

Since data modeling issues are relatively invisible to the business, it’s nearly impossible to convince business teams to prioritize improvements until the system starts breaking down. Many engineers can relate if they’ve ever tried to advocate for refactoring their code. It’s an invisible problem, that no one cares about until they need to care. And if you’re at the point where you need to care, then you’re already in hot water.

In the context of data, problems often manifest at the BI layer, even though the root cause lies deeper in the stack. This is where the business feels the impact.

As data teams come under pressure to address the problems, there’s a natural tendency to focus on the BI platform. Replace the BI tool, solve the problem. It’s a win-win for data teams. Not only can they avoid the assumption of responsibility, but a migration to a new BI tool also presents an opportunity to refactor some of the messy data models as part of the effort to implement a new BI platform — a clean slate. There’s opportunity to deprecate legacy metrics (like “total desks” in the WeWork example), and start fresh with up-to-date requirements and cleaned up data models to match.

The problem is solved for the short term, the data team is the hero, and the cycle begins all over again

Nice post explaining how much BI is trailing when it comes to engineering and especially sustainability. I identify my past self a lot, doing dashboard and asking for more modeling upfront 😅

👉 Read the blog post

Clock speed

Great analogy about CPU clock speed and the pace of innovation in our companies.

👉 Read these few lines

📰 The Blog Post

Last month I cross-posted a blog with Julien. We explored the escaping strategies for vendor lock-in.

Like everywhere, we should find balance. It's actually not realistic and optimal to escape a vendor lock-in. Sometimes (most of the time?) we just want to get things done. Not using a framework that will create more legacy than outputs.

We actually want to write as little code as possible.

Ju Data Engineering Newsletter

Escaping Vendor Lock-In

Bonjour…

Read more

a year ago · 21 likes · 5 comments · Julien Hurault and Benoit Pimpaud

🎨 Beyond The Bracket

Last month I played golf for the first time. I loved it!
The sensation of a good drive while walking in nature is the liminal space I'm constantly looking for.
It also reminds me of how I enjoyed mini-golf when I was a kid. And how much I found joy in my daily work. Even on the bad days.

✨ From Mini-Golf to Golf

November hits hard. Christmas time is calling. The night is here.

I hope you're doing great! You're more than 1400 reading these lines - all over the world.
I would love to hear from you. Your recent story. Data or not :)