Reset Kafka offsets without stopping your consumer apps. AutoMQ 1.6.5 adds --force to the bundled consumer-group CLI. Consumers automatically rejoin and resume from the new offsets. A rebalance still occurs. github.com/AutoMQ/automq/rel… #ApacheKafka

Excited to share that AutoMQ was featured on the AWS Storage Blog. For teams running Kafka on AWS, the classic trade-off has been difficult: keep low latency with local disks and pay for replication and cross-AZ traffic, or move toward object storage and accept higher write latency. This new AWS blog shows a better path: Diskless Kafka with AutoMQ, Amazon S3, and Amazon FSx for NetApp ONTAP as the low-latency WAL layer. The results are compelling: - Average write latency under 10 ms - Multi-AZ resilience - Near-zero cross-AZ data plane traffic - S3-level storage economics - 94% cloud infrastructure cost savings in the cost analysis What I like most about this architecture is that it keeps Kafka protocol compatibility while changing the storage foundation: brokers become stateless compute, hot writes land on FSx for ONTAP, and historical data is flushed to S3. This makes Diskless Kafka practical not only for log ingestion, but also for latency-sensitive workloads such as microservices, risk control, and trading systems. Read the AWS blog here: aws.amazon.com/blogs/storage… #Kafka #AutoMQ #AWS #AmazonS3 #FSx #StreamingData #CloudNative #EventStreaming

Is AutoMQ the first vendor to integrate with the latest AWS S3 Tables?

(1/7) 🚀 Excited to officially introduce AutoMQ: a cloud-native replacement for Apache Kafka! 🌐 The evolution of the streaming world can be summarized in three stages: 1️⃣ Kafka is the beginning (2011~): Kafka started it all in 2011, quickly becoming the de facto standard. 2️⃣ Commercial leads innovation (2014-2023): @confluentinc and @redpandadata led continuous technological advancements. 3️⃣ Cloud reshapes architecture (2023~): Enter @AutoMQ_Lab, a new generation built entirely on the cloud. Truly shared storage, 10x cost-effective. Stay tuned as we deep dive into AutoMQ! 🌟 #CloudNative #Streaming #Kafka #AutoMQ

🔄 Dive into the revolution of #AutoMQ, a cloud-native alternative to Apache Kafka, reshaping the future of data architectures: dev.to/automq/introducing-au… 1️⃣ Shared Nothing: Kafka's initial approach, marking the dawn of stream storage. 2️⃣ Shared Storage: AutoMQ's current state, leveraging cloud-native tech for cost-efficiency, operational agility, and full compatibility. 3️⃣ Shared Data: AutoMQ's future, integrating stream data into data lakes, breaking down data silos, and enabling comprehensive access to data.

Share a crazy idea: Construct a brokerless Kafka system—no brokers, no agents, just Kafka clients. All data ultimately needs to flow to S3. So can Kafka read and write to S3 directly via the client? This is the original idea. But, there is a big gap between Kafka partitions and s3 objects. Fortunately, we have s3stream(a shared streaming storage library, github.com/AutoMQ/automq/tre…) to bridge this gap. However, a metadata service is still absent here, essential for managing everything, much like an iceberg requires its catalog service. Given that both Brokerless and Broker architectures depend on KRaft, is it possible to integrate the two approaches? Are you interested in the brokerless Kafka architecture?

EBS(Elastic Block Store) is frequently misconceived as a Local Disk. However, EBS not only incorporates a multi-replica system but can also be utilized as fully-fledged shared storage via Multi-Attach and NVMe technologies. See github.com/AutoMQ/automq for how to leverage the performance of shared EBS. #AutoMQ #Kafka #SharedStorage #S3 #EBS #CloudStorage

Like the name of the `freight` cluster, Kafka is undoubtedly a data-intensive software. AutoMQ offers a unique solution utilizing object storage. We regard EBS as shared storage, similar to S3. However, we need the multi-attach and NVMe reservations of EBS to achieve shareability. In the AutoMQ architecture, EBS serves as the WAL while S3 acts as the primary storage. Each broker requires only a small EBS volume, which is cost-effective. This design encompasses all the benefits touted by the 'Freight' cluster or WarpStream, while maintaining crucial low latency for stream processing.

Last year, we explored new data trends utilizing S3 in @YingjunWu's tweet(xitter.cf/yingjunwu/status/17249…). Recently, I found a discussion started by a Reddit user on selecting a modern Kafka solution, reddit.com/r/dataengineering…. You may want to participate in it?@richardartoul @emaxerrno