Having problems with large pipeline commands using the py-redis client

Migration to Dragonfly
1

I’m working on porting an app from a Redis cluster to Dragonfly. I’m running into a problem where I use the py-redis library to run pipeline commands to Dragonfly. I setup my client using existing code, using redis.RedisCluster and have Dragonfly running in cluster emulation. The I build a pipeline and start adding smember commands to it. However, if I go above somewhere around 150,000 smember commands in the pipeline, Dragonfly doesn’t respond, and doesn’t seem to issue an error either. At 150k, it works fine. At 160k, radio silence from Dragonfly.

The code works fine when I’m connecting to my old Redis cluster running in AWS elasticache. I’m sure there’s a configuration setting somewhere that’s causing this, but I can’t seem to find any guidance on what needs to change to get this to work. Any ideas?

2

This issue is discussed in a Dragonfly Discord thread instead.

We will try to summarize it later.