Does pub/sub affect the throughput?

Dragonfly Technical
1

Hello!

On this blog post here:
https://www.dragonflydb.io/blog/scaling-performance-redis-vs-dragonfly

I see that dragonfly can have throughput in the millions.
But does having hundreds of thousands of subscribed channels, drastically affect this?
Have there been any benchmarks with this?

2

? ^^

3

having thousands of subscribed channels will affect the throughput

4

Is there any data about the relationship between the number of channels and the degradation? I am thinking of using pubsub in a SaaS, with separate channels for each tenant.

5

I do not think they have done any benchmarks with it

6

I do however know that you can use pub/sub with redis, and scale it horizontally with a cluster.
And pub/sub throughput should scale linearly with the amount of servers, due to the smart way pub/sub is handled with sharding.

7

I am going to try and benchmark how much through put i can achieve with dragonfly and a large amount of channels

8

That’s not really the case. Redis cluster can not send efficiently each message to 1000 channels. Clustering does not help here.

9

https://redis.io/docs/interact/pubsub/#:~:text=From%20Redis%207.0%2C%20sharded%20Pub,shard%20channel%20is%20hashed%20to.

From redis 7.0, it uses a hashing shard algorithm to only sent the message, to the node in the cluster which is responsible for the subscriptions

10

This means that just because you have 10 shards in the cluster, it does not mean that when you broadcast it sends the message to all nodes

11

This was previously an issue with redis

12

https://www.youtube.com/watch?v=6G22a5Iooqk

13

Here is a video on the issue that was previously present

14

I know this. I am saying that it’s either all your subscribers and publishers are collocated on the same shard and then you might have the single shard performance problem or, alternatively, you must use cluster bus to pass messages between shards. There is nothing magical here. You said “hundreds of thousands of subscribers”. if it’s indeed the case you are better off with the horizontal scale. However, you must ask yourself if you can partition all your publishers and subscribers into disjoint sets of independent groups.

15

I am intending to horizontally scale my redis cluster

16

I have done bench marks on redis vs dragonfly now, and dragonly is significantly faster than redis

17

I just want to go with the redis cluster solution, since it offers horizontal scaling