Cuttings

DuckDB 2.0.0 LTS “Cinnamon Teal” released. Includes “agent mode”, DuckDB as a server (quack protocol), triggers, and more

Linus Torvalds with some wise words, in his own inimitable style :D

Hey, we launched a new blog :) It’s even got an RSS feed!

100% technical content, covering Kafka, Flink, and much more.

“The LLM grinds everything into the same inherently untrustable confidently-expressed blandness”

Guess what? Even the cool kids know that sometimes you don’t reinvent the wheel, you just model your data properly :)

> the definition of active user depends on which dimensions you anchor to.

seeing how far I can get Claude to build a Kafka pipeline by prompting alone, without touching the Confluent Cloud UI or CLI myself. So far, pretty darn good. Claude making heavy use of

jfc it’s now a joy to simply read any blog post that’s not AI-generated.

if you actually have a good thing to write about, for the love of your readers and the good of the internet, please just write about it. claude might shit out a completed article but it’ll stink.

(1/4)

Chris Gambill nails it in this discussion of why holding data at multiple levels of refinement makes a ton of sense, and is not just “duplication”

🪝 I love a good hook at the beginning of a blog post, and Elena Verna nails this one

This rant from @Brent Ozar is my kind of rant. Love it. Is it ironic posting it on LI? Don’t care. People should read it.

last week I made r/dataengineering cross, this week it’s the AI-sceptics on lobst.rs :D

Brittany Ellich absolutely nails it in this excellent article: “But here’s the thing: no one actually knows what the future is going to be.”

Excellent blog post from Nicoleta Lazar comparing query federation between Trino and StarRocks with sources including Iceberg

Think I just realised why some software devs are still so voraciously stubborn in their anti-AI stance.

How was December 2013 both only yesterday and over twelve years ago?!

Either way, some articles’ core messages hold true over time—and this is one of those articles.

Rather chuffed to get featured on @c_a_dunlop’s blog (alongside many names who I’ve fanboi’d for years, giving me serious imposter syndrome 🙃)

I enjoyed this Hard Fork interview; very accessible for a layperson like me :)

> Amanda Askell, Anthropic’s in-house philosopher in charge of shaping Claude’s personality, discusses the company’s newly released “Claude Constitution”

I put together a quick list of TUIs for ApacheKafka. Any others to add to this list? Which ones are people using?

What one conference talk do you keep coming back to, again and again, over the years?

I’ll start: @Gwen (Chen) Shapira’s “Streaming Microservices: Contracts & Compatibility” talk from QCon 2017:

(Related to this, a tweet from Gwen a year later but it still makes me chuckle: “Don’t make me go out there and unfuck your data pipelines for you.” https://x.com/gwenshap/status/967118574014799872)

OK, this is exactly what I was looking for: “Lessons Learned from Building LinkedIn’s First Agent: Hiring Assistant”.

Anyone got more good examples like this? https://twitter.com/rmoff/status/1999426776037044420

Doesn’t everyone have a favourite RSS reader?! This one is definitely mine 👇

A step-by-step guide to building a key-value database from scratch: - love the explainer/interactive animation in this

One for my DBA friends! A very good talk from @andypavlo.bsky.social, looking at how LLMs can actually be used to help improve performance. It’s measured and reasonable, with plenty of caveats and real-world considerations from someone who really gets RDBMS.

I missed this release last week - ApacheFluss now supports writing to both ApacheIceberg and Lance (as well as the original ApachePaimon.

P99 CONF nails it in terms of both content, and UX for attending. Super-simple registration. Free registration. Videos available on-demand afterwards with no gating or games. Clear and usable website. These folk know how to do a developer conference. A+++

🎥 Videos & slides

Good post from Sam Harris about public speaking, including some good lessons for conference speakers specifically

Check it out - I was on a podcast :) @gamussa.dev was kind enough to spend some time chatting with me about my time working with Oracle, Kafka, and more.

🎥 https://www.youtube.com/watch?v=U0t5cCl9BWM

One of the things I love about @vanlightly.bsky.social’s posts is that he is both opinionated and highly informed and articulate in his argument. This post is a great example: Why I’m not a fan of zero-copy Apache Kafka-Apache Iceberg

Have you ever found yourself thinking…Flink Watermarks…WTF? 🔥 This new tool might be for you :)

It’s a hands-on, scrollytelling walkthrough of what watermarks in ApacheFlink are, why they matter, and how to use them.

Try it out

Trying out RevealJS with Asciidoc and liking it a lot so far. Some good resources/examples I’ve used:

https://zenika.github.io/adoc-presentation-model/reveal-my-asciidoc.html#source-18

https://github.com/martsec/asciidoc-slides?tab=readme-ov-file

https://marketplace.visualstudio.com/items?itemName=flobilosaurus.vscode-asciidoc-slides

Good article from @pedramnavid.com about building a successful DevRel team.

Measured and considered, and lots of wisdom. A++++ would read again.

I really enjoyed this talk from Paul Iusztin at QCon London. Covers lots of concepts around like LLMs, RAG, and Agents and explains them well.

🌟 Going to Current25 next month? Here are some of the top DataEngineering talks that I’d recommend 👇

🔥 Not already registered? 🎟️ You can get a 50% discount using code PRM-DEVREL at

• @Steven Wu - An ounce of prevention is worth a pound of cure - Fix data clustering in streaming write to Iceberg https://current.confluent.io/new-orleans/sessions#session-951063 • @Alex Merced - Tuning the Iceberg: Practical Strategies for Optimizing Tables and Queries https://current.confluent.io/new-orleans/sessions#session-956093 • @Russell Spitzer - What the Spec?!: New Features in Apache Iceberg™ Table Format V3 https://current.confluent.io/new-orleans/sessions#session-957876 • @David Anderson - Unlocking the Mysteries of Apache Flink https://current.confluent.io/new-orleans/sessions#session-955364 • @Tom Thornton Change Data Capture at Scale: Insights from Slack’s Streaming Pipeline https://current.confluent.io/new-orleans/sessions#session-932237 • @Sujay Jain & @Bhargavi Reddy Dokuru - Press Play on Data: Netflix’s Journey from Streams to Gaming Insights https://current.confluent.io/new-orleans/sessions#session-956506 • @Weiqing Yang - Scaling Streaming Computation at LinkedIn: A Multi-Year Journey with Apache Flink https://current.confluent.io/new-orleans/sessions#session-956562 • @Adi Polak - Stream All the Things — Patterns of Effective Data Stream Processing https://current.confluent.io/new-orleans/sessions#session-956997 • @Gunnar Morling - Transactional Change Stream Processing With Apache Flink https://current.confluent.io/new-orleans/sessions#session-956911 • @Zander Matheson - The Zen of PyFlink: The evolution towards a truly “pythonic” Flink https://current.confluent.io/new-orleans/sessions#session-954451 • @Allison Chang - Scaling the Past: Productionalizing the Flink History Server at LinkedIn for Stream and Batch https://current.confluent.io/new-orleans/sessions#session-928165 • @Michael Peacock & @Andrew Hahn - Empowering the Disconnected Edge: Shifting Far Left with Predictive Analytics for Naval Ships https://current.confluent.io/new-orleans/sessions#session-956295 • @Rupesh More & Guruguha Marur Sreenivasa - Powering Real-Time Vehicle Intelligence at Rivian with Apache Flink https://current.confluent.io/new-orleans/sessions#session-954053

Now I wish I’d watched this before I tried figuring it out for myself 😅 Tim Berglund nails it in this crystal-clear explanation of MCP

Details of how DataDog built a new time series storage in Rust, replacing five previous generations of platform, giving 60x ingest speed and 5x faster queries

Ostensibly about the adoption of StarRocks at Fresha, this blog post goes into a bunch of interesting detail about data platform architecture, migration strategy, data serving tiers, and more - recommended! https://freedium.cfd/https://medium.com/fresha-data-engineering/how-we-accidentally-became-one-of-uks-first-starrocks-production-pioneers-7db249f10010

srsly, probably 50% of the articles with interesting titles are just AI-generated crap. Smells include 🫵all✅the💽emojis • bullet • lists and then crappy ascii-art diagrams.

(I guess this list’ll get fed back into LLMs so they learn how to produce this shit better…)

I should go on vacation more often—whilst I was away ApacheFlink 2.1 was been released!

Ignore the clickbaity listicle title - there are some really good points in this article from Bernd Wessely https://freedium.cfd/https://medium.com/data-engineer-things/unlearning-data-architecture-10-myths-worth-killing-946e728f1acf

If you run ApacheKafka, this page is worth keeping an eye on. There were three CVEs published earlier this month impacting various versions including up to 3.9.0

Some details of SFDC’s lakehouse, with 4 million tables and 50PB of data, built on ApacheIceberg

If you’re interested in how new features are introduced for open source projects governed by the ASF, this thread about Apache Flink is a good example of the kinds of discussions that happen, and all in the open

✨✨ Want some tips on writing a winning abstract? ✨✨ 👇 Check out these resources 👇

💥 @adipolak.bsky.social:

💥 @tlberglund.bsky.social: https://www.youtube.com/watch?v=N0g3QoCuqH4&t=13s

💥 A few blog posts from me too :) https://rmoff.net/categories/abstracts/

Got a cool story about something in the data engineering space? You should 💯 submit it as a talk to Current 2025 in New Orleans 😁

Do it! Now! CfP is open until 15th June.

(Pro-tip: you only need an abstract at this point; writing the talk can be later 😅)