Apache Iggy, a message streaming platform in Rust, graduates to an Apache TLP

spetz•about 3 hours ago

Apache Iggy™ has officially graduated from the Apache Incubator and is now an Apache Software Foundation Top-Level Project (TLP)

It’s been quite a journey, from a small Rust message-streaming experiment in 2023 to an independent Apache project with a growing community. And its biggest feature yet, being the Viewstamped Replication Revisited (VSR) clustering, is coming soon.

junon•about 3 hours ago

I believe spetz is on the project, so I'd like to ask:

Is this a MQ implementation, or can this be used in place of e.g. something like protobufs and iroh/tonic? Does it have something akin to grants/capabilities/authorization? Does it have RPC? Can I have services speak a more privileged set of instructions/commands and clients needing auth to speak dumber/more high level administrative commands just like gRPC?

Also, is the performance due to the (what I assume is custom) zero copy serialization?

The landing page simply states it's essentially "fast messaging" but beyond that it's not entirely clear when I'd use this or what its features/limitations are.

spetz•about 2 hours ago

It's more like Kafka or Pulsar in terms of being the message streaming infrastructure (so an append-only log, not the message broker like, e.g., raw RabbitMQ). I think that the main docs page should give you a good understanding of how the data is stored/organized https://iggy.apache.org/docs/ - for example, on top of the topics, we also have "stream" which is just an extra hierarchy that can be used for something like multi-tenancy isolation or anything else depending on the use case. And yes, there's a built-in RBAC (read/manage particular streams, topics, servers, etc.). As for protocols, we have native support for TCP, QUIC, WebSocket (these 3 are stateful binary protocols), and HTTP as well.

There are a bunch of things that sum up to the overall performance gains - using Rust, thread-per-core architecture + shared-nothing (think of Seastar), custom zero-copy serialization, io_uring for disk * network I/O, and VSR-based consensus (inspired by TigerBeetle) - we simply build all this stuff from the ground up to make the most out of modern hardware and Kernel features, with a custom protocol on top (but the Kafka proxy/gateway is also WiP). And there are also connectors (sink, source), benchmarking runtime, CLI and so on.

languagehacker•about 2 hours ago

I love my Italian Greyhounds so much, so this is great news to me.

spetz•about 2 hours ago

Awesome! You can find mine too in one of the older blog posts, hence the name :)

oulipo•about 1 hour ago

Will it integrate directly with MQTT? Or should I use a MQTT broker and pipe into Iggy?

0x457•2 minutes ago

Iggy is different kind of message queue. This is what you would use instead of Kafka. There is no sink or source for MQTT out of the box.

anony-123•about 1 hour ago

Why the website is looking AI slop?

anony-123•9 minutes ago

Why this comment is being flagged?

ramraj07•about 3 hours ago

I always thought Im just not good enough to understand, hey Im just too dumb, surely there's people out there who understand what differentiates all these apache products! Nope never found one.

kitd•about 2 hours ago

Apache is just an umbrella organisation for managing open source software projects. There's nothing that links those projects otherwise.

hofo•about 2 hours ago

Differentiation from each other or from competitor projects?

theplumber•about 2 hours ago

Do you understand the difference between computer devices?

otabdeveloper4•about 1 hour ago

Don't worry about it, Apache is the hospice where software projects go to die.

skeeter2020•27 minutes ago

What a crock of shite. Of course there's going to be some dead one when you have this many, but there are also projects that are foundational and many exciting new ones too:

https://projects.apache.org/projects.html

hasanpiker•about 1 hour ago

Apache Pop when

Apache Iggy, a message streaming platform in Rust, graduates to an Apache TLP

⚡ Community Insights

Discussion (18 Comments)Read Original on HackerNews