Issue 33 - Complex vs Complicated
Ops: the realization
We often talk about complexity. In our systems, in our projects, in our relationships.
But most of the time, it's only complicated.
There is actually a hair's breadth between these two words.
Complexity is about a set of intricate elements. Solving a complex problem is often about statistics. For example: climate change, sociologic issues, diseases, etc.
A complicated problem is hard to understand but is often solvable thanks to some computing machinery. Examples: Maxwell's equations, managing the temperature in a nuclear power plant, controlling a plane, etc.
As you might feel, data and software are about both.
It's about complexity because of the problem we want to solve: optimizing business decisions, creating market fit, curating content, computing embeddings, etc.
All these problems need to manage different elements and we try to find some magic to make value out of them.
It's about complicated issues: how to keep track of all code changes? What is the last metric? How to spread compute on multiple servers? How to answer requests fast?
Though initially difficult to grasp, these issues are ultimately predictable and computable.
This is where we often end in tooling debates. Does SQLMesh is better than dbt? Does DuckDB can overcome Spark? Does Rust is better than Python?
These are not complex, but complicated. Ultimately, our engineers and scientific minds are well-equipped to navigate these issues.
Things get challenging when complexity is facing us.
It's when politics enter the debates. It's when emotions are involved. It's when things lie in the statistics rather than the precision of our billions of transistors.
With this newsletter, I try to find the bridge between those two words. I'm an engineer by background but find the real value of our will in the complexity we try to overcome. Does that resonate with you?1
📡 Expected Contents
Is it time to move from dbt to SQLMesh?
I wrote this blog post regarding the last discussions I had with Kestra users. I used and still love dbt. Like I loved jQuery.
But I have got the feeling that the end story is not quite there and that SQLMesh (like Malloy probably in the future) is bringing into data analytics what React has brought to web development.
Observable 2.0
Today we’re launching Observable 2.0 with a bold new vision: an open-source static site generator for building fast, beautiful data apps, dashboards, and reports.
Very nice move from Observable. It definitely goes the same way as Evidence, Malloy, or Rill.
BI as code is the next thing in the data stack, and we're not so far from building full-as-code stacks.
Generates fake data from a YAML schema
My friend and his team working at Soma released a great tool to generate fake data from YAML schemas.
It resonates a lot with Vianney's recent writing and my own too. Configuration is just another interface. But it makes things very simple and helps with reproducibility.
I'll keep this project in my toolbox the next time I'll need synthetic data to run tests.
Casual programming
End users are not “casual,” “novice,” or “naive” users; they are people such as chemists, librarians, teachers, architects, and accountants, who have computational needs and want to make serious use of computers, but who are not interested in becoming professional programmers.
Great post about what's programming and how everyone on this end is a "programmer". Just maybe "a casual one".
📰 The Blog Post
Everything Ops: Beyond the Hype
An essay around what I feel "Ops" is, what's behind the trendy job titles, and why it matters.
🎨 Beyond The Bracket
While I don't know if I will have kids or not, I'm already interested in what I would like to teach to my children - at least to the young generation.
I recently went through this great resource converting Naval thinking and observation to the kids level.
Naval's ideas, though some might find them idealistic or overly broad, resonate deeply with some of my own core principles.
Having this cheatsheet to quickly refresh my own mind or share it with family (young or not) is very handy :)
February has been nice here!
The weather is Manchester terrible, but hopefully, I love to walk in the rain. Paris is quite a nice place to be when it's all gray out there.
March already sounds like a good month, with a speech at the first DuckDB meetup in Paris, some new music from my favorite artists, and great relationships arising.
If the sun can shine a bit it would cherry on the cake ☀️
See you there 👋
hopefully it will, you won't be reading this newsletter otherwise 😅