Always happy to help test the process 
Testing DragonflyDB with Bull
Seems this was partially my fault in my program based on how I was distributing status updates, but it still didn’t have this issue on Redis. There’s probably something missing for staggering jobs.
Currently testing the new XTRIM support
@inclusive-numbat It seems that as of https://github.com/dragonflydb/dragonfly/commit/cde6adb34d1dd016a03cdb875bf14ba2311d9244 , every job is being pushed into waiting and nothing is being processed with this error:
0|Calckey_social | WARN 6 [queue deliver] failed(Error: job stalled more than allowable limit) id=16664 attempts=0/10 age=1632m to=https://xxx.xxx/inbox
(where xxx.xxx is each instance)
I’m noticing some interesting graph patterns. Normally, Redis would have jobs constantly in delayed/processing and have a steady stream of active jobs.
Can you show the output of 'info all’s command, please?
127.0.0.1:6379> info all
# Server
redis_version:6.2.11
dfly_version:df-dev
redis_mode:standalone
arch_bits:64
multiplexing_api:iouring
tcp_port:6379
uptime_in_seconds:71
uptime_in_days:0
# Clients
connected_clients:166
client_read_buf_capacity:2039808
blocked_clients:30
# Memory
used_memory:19162048
used_memory_human:18.27MiB
used_memory_peak:19162992
comitted_memory:379453440
used_memory_rss:324980736
used_memory_rss_human:309.93MiB
object_used_memory:10577544
table_used_memory:4228560
num_buckets:100800
num_entries:22441
inline_keys:0
strval_bytes:660288
updateval_amount:4944779
listpack_blobs:0
listpack_bytes:0
small_string_bytes:854000
pipeline_cache_bytes:29987625
maxmemory:13420671795
maxmemory_human:12.50GiB
cache_mode:store
maxmemory_policy:noeviction
# Stats
total_connections_received:166
total_commands_processed:483034
instantaneous_ops_per_sec:7035
total_pipelined_commands:141365
total_net_input_bytes:82385877
total_net_output_bytes:13298040
instantaneous_input_kbps:-1
instantaneous_output_kbps:-1
rejected_connections:-1
expired_keys:9011
evicted_keys:0
hard_evictions:0
garbage_checked:0
garbage_collected:0
bump_ups:0
stash_unloaded:0
oom_rejections:0
traverse_ttl_sec:0
delete_ttl_sec:0
keyspace_hits:320103
keyspace_misses:32474
total_reads_processed:77878
total_writes_processed:46177
async_writes_count:58600
parser_err_count:0
defrag_attempt_total:0
defrag_realloc_total:0
defrag_task_invocation_total:0
# TIERED
tiered_entries:0
tiered_bytes:0
tiered_reads:0
tiered_writes:0
tiered_reserved:0
tiered_capacity:0
tiered_aborted_write_total:0
tiered_flush_skip_total:0
# PERSISTENCE
last_save:1685809990
last_save_duration_sec:0
last_save_file:
# Replication
role:master
connected_slaves:0
master_replid:a6b95962cf72f48db53ac12f8fa477eab032336d
# Commandstats
cmd_BRPOPLPUSH:487
cmd_COMMAND:1
cmd_DEL:2316
cmd_EVAL:189
cmd_EVALSHA:98152
cmd_EXEC:666
cmd_EXISTS:279051
cmd_GET:16231
cmd_HGET:8
cmd_HGETALL:1317
cmd_HINCRBY:248
cmd_HMSET:2556
cmd_HSET:725
cmd_INCR:2518
cmd_INFO:22
cmd_LLEN:42
cmd_LPUSH:1922
cmd_LRANGE:8291
cmd_LREM:425
cmd_MULTI:666
cmd_PEXPIRE:15
cmd_PSETEX:82
cmd_PSUBSCRIBE:42
cmd_PUBLISH:9582
cmd_RPOPLPUSH:697
cmd_RPUSH:223
cmd_SADD:1976
cmd_SCARD:671
cmd_SET:42850
cmd_SISMEMBER:671
cmd_SMEMBERS:8292
cmd_SREM:421
cmd_SUBSCRIBE:55
cmd_ZADD:60
cmd_ZCARD:57
cmd_ZRANGE:399
cmd_ZRANGEBYSCORE:369
cmd_ZREM:694
cmd_ZREMRANGEBYRANK:15
cmd_ZREMRANGEBYSCORE:15
cmd_ZSCORE:15
# Errorstats
Dragonfly does not allow execution of a server-side Lua script inside MULTI/EXEC block:637
# Keyspace
db0:keys=22441,expires=1479,avg_ttl=-1
# CPU
used_cpu_sys:176.23134
used_cpu_user:139.794328
used_cpu_sys_children:0.0
used_cpu_user_children:0.0
used_cpu_sys_main_thread:11.422739
used_cpu_user_main_thread:9.95562
# Cluster
cluster_enabled:0
as of latest commit
And this is from my Calckey config file for relevant Redis/Bull settings:
# Redis server config
redis:
host: [redacted]
port: 6379
# Number of worker processes
clusterLimit: 8
# Job concurrency per worker
deliverJobConcurrency: 128
inboxJobConcurrency: 64
# Job rate limiter
deliverJobPerSec: 1024
inboxJobPerSec: 512
# Job attempts
deliverJobMaxAttempts: 10
inboxJobMaxAttempts: 4
Which are referenced here: https://codeberg.org/calckey/calckey/src/branch/develop/packages/backend/src/queue/queues.ts , https://codeberg.org/calckey/calckey/src/branch/develop/packages/backend/src/queue/initialize.ts
Cc @inclusive-numbat (I think I forgot to tag you)
Thank you ! Seems the only obvious issue left to handle is execution of lua scripts from multi/exec blocks.
Ah, the lack of that is probably why I’ve been noticing these things: https://github.com/dragonflydb/dragonfly/issues/782#issuecomment-1575426054