metrics to monitor latency at dragonfly server

Dragonfly Technical
1

Do we have any way to monitor latency(p99, p99.9) metrics at dragonfly server.

Need:
Executing MSET command with lettuce Redis client, and got significantly high p99 and p99.9 (greater than 30, 40ms), when application and dragonfly were running at ~20% CPU uses.

Jmeter (load generation tool) -> Load Balancer -> Application -> Dragonfly

I addressed this with connection pool, and achieved 1.5ms p99 and 4ms p99.9. (still getting 80ms p99 spike in every 15mins) | Application CPU, and GC are in control.
so I am curious about
Q1: How to find that this latency was introduced by the dragonfly or because of inefficiently using the client?
Because with Redis we didn’t get this problem(connection pool was required mostly when executing blocking commands etc).
Q2: How to identify that we are using dragonfly resources efficiently(only CPU%)?
Q3: How to mitigate frequent p99 spikes?

Sharing the code used to execute mset command in the same thread.

Dragonfly: c3-highmem-44 | /home/ubuntu/dragonfly --logtostderr --cache_mode=true -dbnum 1 --port 6379 --logbuflevel=-1 --conn_use_incoming_cpu=true --maxmemory=330G --dir=/
MSET: 20 Keys
Command executed per sec: 182K
Memory Uses: 16%
Clients: 302
Dragonfly CPU: 83%
p99: 1.5ms (80ms spike in every 15mins)

2

Lettuce Redis Client:
<dependency> <groupId>io.lettuce</groupId> <artifactId>lettuce-core</artifactId> <version>6.3.2.RELEASE</version> </dependency>

Code to run MGET command on Dragonflydb:

    private final GenericObjectPool<StatefulRedisConnection<String, String>> connectionPool;
        
    RedisStandaloneKVRepository(RedisStandaloneProperties redisStandalone, ZkProperties zkProperties) {
        this.redisClient = ConnectionFactory.getRedisStandaloneClient(redisStandalone);
        GenericObjectPoolConfig<StatefulRedisConnection<String, String>> config = new GenericObjectPoolConfig<>();
        config.setMinIdle(zkProperties.getMinIdle());
        config.setMaxIdle(zkProperties.getMaxIdle());
        config.setMaxTotal(zkProperties.getMaxTotal());

        this.connectionPool = createGenericObjectPool(() -> {
            StatefulRedisConnection<String, String> conn = ConnectionFactory.getRedisConnection(this.redisClient, redisStandalone);
            conn.setTimeout(Duration.ofMillis(100));
            return conn;
        }, config);
    }


    @SneakyThrows
    @Override
    public void mset(Map<String, String> map) {
        try (StatefulRedisConnection<String, String> connection = connectionPool.borrowObject()) {
            RedisCommands<String, String> redisCommands = connection.sync();
            redisCommands.mset(map);
        }
    }```
3

you can play with slowlog settings to filter out slow queries with latency higher than …

4

can you please share the output of info all command?

5

@inclusive-numbat Ran perf again on c3-highmem-22 machine, MSET command with 20 keys, Dragonfly CPU: 93%, Memory Uses: 33%, got similar spikes again at 15-20min intervals.

6

Info and slowlog, slowlogs shows that dragonflyDB took ~20ms to execute the some MSET commands (All MSET commands are same with different keys}
info_and_slowlog (29.8 KB)

7

@positive-camel emailed you