So far dragonfly has been amazing (using single instance, no replication.) For some reason our app keeps randomly getting a server error from dragonfly that says “ERR Can not execute during LOADING”. I have been crawling the web but haven’t been able to find anything. I searched for the error with redis as well and couldn’t find anything, so i think its specific to dragonfly.
It happens during the start phase when dragonfly loads the snapshot. It should not happen once the snapshot is loaded. You can check what happens by running “info” when you get “loading” response and i will check if something is unexpected. Just paste it here
at what points would dragonfly load snapshots? i know it makes them on interval, but if it has been steadily running and i’m not restarting it, i’m not sure what would trigger loading a snapshot
please send me the output of “info all” command. You can also DM me. It does not include PII, and I will be able to see if there is anything weird there
message.txt (5.96 KB)
I see that the process is indeed live for many days. I do not see anything weird there. Can you send the output of redis-cli info errorstats again?
I want to see whether it replied with “LOADING” errors after you last message
error finally happened again, here is a log right after (looks about the same)
Thanks. Looks weird, i will open a bug. Btw, why not upgrading to the latest version? Maybe the bug disappears there?
i will upgrade to see
okay. i have upgraded the version. will let you know if that helped
Thanks!
got the error even after the update
# Server
redis_version:6.2.11
dfly_version:df-v1.2.1
redis_mode:standalone
arch_bits:64
multiplexing_api:iouring
tcp_port:6379
uptime_in_seconds:18393
uptime_in_days:0
# Clients
connected_clients:72
client_read_buf_capacity:78080
blocked_clients:0
# Memory
used_memory:1479448
used_memory_human:1.41MiB
used_memory_peak:1757752
comitted_memory:43663360
used_memory_rss:44593152
used_memory_rss_human:42.53MiB
object_used_memory:18446744073709542440
table_used_memory:563808
num_buckets:13440
num_entries:1023
inline_keys:0
strval_bytes:43552
updateval_amount:8448771
listpack_blobs:160
listpack_bytes:7961
small_string_bytes:120432
pipeline_cache_bytes:71013
maxmemory:6584798412
maxmemory_human:6.13GiB
cache_mode:store
maxmemory_policy:noeviction
# Stats
total_connections_received:16306
total_commands_processed:2401146
instantaneous_ops_per_sec:120
total_pipelined_commands:421119
total_net_input_bytes:583788223
total_net_output_bytes:29670492
instantaneous_input_kbps:-1
instantaneous_output_kbps:-1
rejected_connections:-1
expired_keys:9588
evicted_keys:0
hard_evictions:0
garbage_checked:0
garbage_collected:0
bump_ups:0
stash_unloaded:0
oom_rejections:0
traverse_ttl_sec:1755
delete_ttl_sec:0
keyspace_hits:666798
keyspace_misses:797691
total_reads_processed:1638524
total_writes_processed:1882270
async_writes_count:0
parser_err_count:0
defrag_attempt_total:0
defrag_realloc_total:0
defrag_task_invocation_total:0
# Replication
role:master
connected_slaves:0
master_replid:f26d8c80dd03d6f0cbfc802260b4ba82daf69cfe
# Keyspace
db0:keys=587,expires=561,avg_ttl=-1
db1:keys=436,expires=433,avg_ttl=-1
# CPU
used_cpu_sys:185.857186
used_cpu_user:259.495594
used_cpu_sys_children:0.0
used_cpu_user_children:0.0
used_cpu_sys_main_thread:79.961879
used_cpu_user_main_thread:82.85348
# Cluster
cluster_enabled:0
oh you know what. i just dug into some stack traces to see what command/operation was firing that returned this error
its using the atomic lock functionality on the cache driver for laravel
so i’m getting the error when it goes to release a lock