/explore
|
You are here 
|
quix.io | ||
| | | | |
www.ssp.sh
|
|
| | | | | Today, there are 6,500 people on LinkedIn who call themselves data engineers. The number of data engineers has doubled in the past year. So is it really the future of data warehousing? What is data engineering? These questions and much more I want to answer in this blog post. | |
| | | | |
tech.scribd.com
|
|
| | | | | Streaming data from Apache Kafka into Delta Lake is an integral part of Scribd's data platform, but has been challenging to manage and scale. We use Spark Structured Streaming jobs to read data from Kafka topics and write that data into Delta Lake tables. This approach gets the job done but in production our experience has convinced us that a different approach is necessary to efficiently bring data from Kafka to Delta Lake. To serve this need, we created kafka-delta-ingest. | |
| | | | |
www.morling.dev
|
|
| | | | | Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink's capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1. | |
| | | | |
www.kai-waehner.de
|
|
| | | Blog about architectures, best practices and use cases for data streaming, analytics, hybrid cloud infrastructure, internet of things, crypto, and more | ||
