Issue 24 - Declarative Everywhere
Discover more from From An Engineer Sight
Issue 24 - Declarative Everywhere
An open-source declarative data orchestrator
A special intro today as I recently joined an exciting adventure : Kestra.
Kestra is a new open-source declarative data orchestrator. It comes with a comprehensive, user-friendly UI, workflow as YAML definition. Crafted by such a talented team
.There I will be responsible for improving the product, bringing my vision and develop use cases !
Hence the new title : Product Owner & Data Engineer 😅.
There are a lot of things to do, but I truly believe that Kestra brings something new and fresh to the data stack.
Like ReactJS taking over jQuery in web development, Terraform replacing custom scripts in infrastructure operations, Kestra aims to moves data orchestration to the declarative paradigm and allow a broader audience to create data workflows.
However, no concession on data engineering practices and specific customization. It's open to everyone, whatever their skills.
No more marketing here, but if you're wanting to schedule tasks easily, feeling stuck in your Airflow legacy, or just looking for a great tool : let's give a try to Kestra, join the Slack, and give a star on GitHub.
📡 Expected Contents
English is a Terrible Programming Language
Do you need reasons AI won't displace programmers ? That's probably the best article I read since the AI hype recently !
Trying to specify a piece of software in English would be a proper nightmare. I can say so with confidence because this is the first part of every software project—you do your best to describe it in natural language first so that everyone is on the same page and you have a good idea of what you’re going to do.
Inevitably, the natural language specification falls short (and this is a serious understatement). There are all these considerations, technical details, compatibilities, versions, integrations, real-world data, and so many other things you have to worry about.
This is the job of a programmer, not so much the programming.
dbt-duckdb
I recently played with dbt-duckdb (in Kestra...), a library aiming to make both of the two buzzy data technology work together 😅.
Beyond the hype, it's very nice to be able to run my dbt project locally, with zero costs and still be able to deploy afterward on a bigger warehouse whenever needed.
Moreover, it seems it’s even more powerful than Spark 👀 Finally !
Serverless Postgres
We don't talk enough about Postgres. Recently, there are many new tools available that build on its strengths and modernize its capabilities.
Neon: a fully managed serverless Postgres. Love that one, serverless database is definitely something to look. During my journey in professional football, we ingested data from several APIs into AWS Aurora (serverless Postgres too): it was working like a charm.
FerretDB: open source MongoDB build on top of Postgres. This project got great traction recently (6500 starts on GitHub) 👀
TimeScale: Postgres for time-series data.
Citus: distributed database that scales out Postgres horizontally across multiple nodes.
ZomboDB: Postgres extension that provides native full-text search capabilities using the powerful ElasticSearch search engine.
Common geographic mental misplacements
Let looks at something less technical : a great storytelling piece about geographic misconceptions.
John shows us the little details we often lose about geographic - nicely arranged !
He also write a second and third post :
🖇️ Enclaves & Exclaves: A tour of the world's geographically engulfed and orphaned places
🖇️ Bruised Borders: A tour of the world's disputed boundaries and territories
Definitely for geography lovers ❤️ 🌎
📰 The Blog Post
Toward Declarative Data Orchestration with Kestra
My vision about the declarative shift happening in the data industry and how Kestra the new open-source declarative orchestrator is driving the move !
🎨 Beyond The Bracket
France quintessence - drinking and strike at the same time.
Down here in France, we got several strikes and important discussions along the unpopular vote to raise retirement age.
Beyond the politic discussions, I love how John Oliver on his "Last Night Week" show made a really funny out of the situation. As always, English versus French jokes 😅
Having worked in a journalism newsroom in the past, I have a deep appreciation for the importance of journalism, especially in today's age of social media.
While the profession is often criticised, its impact on public decisions and mindset cannot be understated. Journalism is a crucial source of discussions, learning, and decision-making, and its true power should not be underrated.
Similarly, in the tech industry, writing plays a vital role in driving innovation. Without blog posts or newsletters, many groundbreaking ideas may never have come to fruition.
Writing helps to develop and refine ideas, allowing for better argument thinking and identifying inconsistencies in our thought process. By putting ideas down on paper, it provides distance to reflect and think more critically about the argument.
In essence, both journalism and tech writing share the same core value of effective communication. They both help to drive ideas, spark conversations, and educate people about the world around them. It's essential to recognise the significance of good writing in these fields and the positive impact it can have on society.
Changing job is always something fulfilling for me. Joining Kestra just two weeks ago and I'm in a kind of constant "wow effect". New things to learn, new responsibilities, new projects, etc... It's really refreshing my mind. Joining a young startup working on a leading data tool is definitely a good move for me :)
Even if I'm really convinced Kestra can help you whatever you're doing, I will try to not make that much Kestra marketing here - you didn't subscribe for that.
Anyway, hope you're doing good, thanks for reading this always too long newsletter ! See you next month - sun is just uphill ☀️
Really, it the first time I see so much talents in the same boat. It's really motivating.