Idempotence layer on bloom filters

TLDR: we run several applications consuming multiple real-time streams. These applications implement idempotence using Redis sets. In this post I describe our journey moving to bloom filters (using the ReBloom module), which brought down our memory usage by almost 10x. Let’s briefly define idempotence: An operation is said to be idempotent when applying it multiple times has the same effect. There are some reasons why applications consuming a real-time stream must have an idempotence layer, e. ... Read more