# Unresponsive during full\_sync **URL:** \ **Category:** Dragonfly Technical\ **Created:** [February 6, 2024, 2:11pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135 "2024-02-06T14:11:08Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![excellent-wolf](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@excellent-wolf](https://dragonfly.discourse.group/u/excellent-wolf)\ **Post date:** [February 6, 2024, 2:11pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/1 "2024-02-06T14:11:08Z")
Hi. We are running Dragonfly 1.12.0 with 2 nodes set up as master-replica. Works great in general, but found that whenever there’s a full-sync happening, the master node would go unresponsive during the entire full-sync duration. This appears to be the case for both to the clients (mostly doing MGET and SET) and the Sentinel nodes. 1. Is this unresponsiveness expected? 2. Let’s say we are willing to accept some data loss, is there any way to skip or turn off full-sync? Does this even make sense? ---
**Author:** ![realistic-ferret](https://avatars.discourse-cdn.com/v4/letter/r/eb8c5e/32.png) [@realistic-ferret](https://dragonfly.discourse.group/u/realistic-ferret)\ **Post date:** [February 6, 2024, 2:48pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/2 "2024-02-06T14:48:51Z")
hi @excellent-wolf thanks for reporting the issue ---
**Author:** ![realistic-ferret](https://avatars.discourse-cdn.com/v4/letter/r/eb8c5e/32.png) [@realistic-ferret](https://dragonfly.discourse.group/u/realistic-ferret)\ **Post date:** [February 6, 2024, 2:49pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/3 "2024-02-06T14:49:08Z")
by unresponsiveness you mean executing MGET and SET will get stuck? ---
**Author:** ![realistic-ferret](https://avatars.discourse-cdn.com/v4/letter/r/eb8c5e/32.png) [@realistic-ferret](https://dragonfly.discourse.group/u/realistic-ferret)\ **Post date:** [February 6, 2024, 2:49pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/4 "2024-02-06T14:49:21Z")
it never returns? ---
**Author:** ![realistic-ferret](https://avatars.discourse-cdn.com/v4/letter/r/eb8c5e/32.png) [@realistic-ferret](https://dragonfly.discourse.group/u/realistic-ferret)\ **Post date:** [February 6, 2024, 2:50pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/5 "2024-02-06T14:50:21Z")
i chatted with our team, and unresponsiveness should not be expected ---
**Author:** ![temperate-echidna](https://avatars.discourse-cdn.com/v4/letter/t/3ec8ea/32.png) [@temperate-echidna](https://dragonfly.discourse.group/u/temperate-echidna)\ **Post date:** [February 6, 2024, 2:54pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/6 "2024-02-06T14:54:30Z")
@excellent-wolf how do you determine the stage in the replication? is your replica unstable for some reason, so that it keeps reconnecting? because full sync is supposed to happen once in the beginning of the replication, and that’s it (that is of course not to say that the server will be unresponsive during that process) ---
**Author:** ![excellent-wolf](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@excellent-wolf](https://dragonfly.discourse.group/u/excellent-wolf)\ **Post date:** [February 6, 2024, 3:35pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/7 "2024-02-06T15:35:34Z")
- We took the replica node out to do some upgrade, full sync started upon rejoining. Lasted for about 40 seconds. - During those 40 seconds it took long enough for most (if not all) of the MGET and SET requests timed out. We set the client side time out to 200 ms, usually it only takes 1-2 ms. - Also Sentinel logs keep reporting sdown/odown on the master node during the full-sync period. ---
**Author:** ![temperate-echidna](https://avatars.discourse-cdn.com/v4/letter/t/3ec8ea/32.png) [@temperate-echidna](https://dragonfly.discourse.group/u/temperate-echidna)\ **Post date:** [February 6, 2024, 5:44pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/8 "2024-02-06T17:44:55Z")
do you have scheduled snapshot saves? if so, do you experience similar behavior during save? ---
**Author:** ![excellent-wolf](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@excellent-wolf](https://dragonfly.discourse.group/u/excellent-wolf)\ **Post date:** [February 7, 2024, 1:59pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/9 "2024-02-07T13:59:53Z")
@temperate-echidna we only do snapshots on shutdown. Can’t really say if we have the same behavior or not. ---
**Author:** ![excellent-wolf](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@excellent-wolf](https://dragonfly.discourse.group/u/excellent-wolf)\ **Post date:** [February 7, 2024, 2:00pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/10 "2024-02-07T14:00:12Z")
Now that we are sure this behavior is not expected we’ll try to come up with some reproducible example and get back to you. ---
**Author:** ![inclusive-numbat](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@inclusive-numbat](https://dragonfly.discourse.group/u/inclusive-numbat)\ **Post date:** [February 7, 2024, 3:13pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/11 "2024-02-07T15:13:33Z")
@excellent-wolf can you please run `redis-cli -3 --raw DEBUG OBJHIST` ---
**Author:** ![excellent-wolf](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@excellent-wolf](https://dragonfly.discourse.group/u/excellent-wolf)\ **Post date:** [February 7, 2024, 4:06pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/12 "2024-02-07T16:06:27Z")
[histogram.txt](https://dragonfly.discourse.group/uploads/short-url/hnN66QBDgNr2i5m2EbuOeZWjIrN.txt) (2.95 KB) ---
**Author:** ![dignified-nautilus](https://avatars.discourse-cdn.com/v4/letter/d/34f0e0/32.png) [@dignified-nautilus](https://dragonfly.discourse.group/u/dignified-nautilus)\ **Post date:** [February 8, 2024, 5:32pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/13 "2024-02-08T17:32:40Z")
Hi @excellent-wolf can you please share INFO ALL output? ---
**Author:** ![dignified-nautilus](https://avatars.discourse-cdn.com/v4/letter/d/34f0e0/32.png) [@dignified-nautilus](https://dragonfly.discourse.group/u/dignified-nautilus)\ **Post date:** [February 8, 2024, 5:35pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/14 "2024-02-08T17:35:37Z")
We do not expect the server to be unresponsive and would like to investigate it. Therefor need to understand your server state and configuration which I can see in info all ---
**Author:** ![excellent-wolf](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@excellent-wolf](https://dragonfly.discourse.group/u/excellent-wolf)\ **Post date:** [March 5, 2024, 9:15am UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/15 "2024-03-05T09:15:32Z")
Hi. Just to put some closure on this… so we spent some time trying to reproduce the issue, and found that it happens reliably when we have enough SETs mixed in with the MGETs. I was going to report back to you all on how exactly can we reproduce this, but just ran the test against the latest v1.15.0 and the problem seems to be gone already! ---
**Author:** ![inclusive-numbat](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@inclusive-numbat](https://dragonfly.discourse.group/u/inclusive-numbat)\ **Post date:** [March 5, 2024, 2:22pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/16 "2024-03-05T14:22:33Z")
Glad to hear this but I would like still see if you guys use swap disk and maybe this is what caused it ---
**Author:** ![inclusive-numbat](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@inclusive-numbat](https://dragonfly.discourse.group/u/inclusive-numbat)\ **Post date:** [March 5, 2024, 2:32pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/17 "2024-03-05T14:32:34Z")
please let us know if it happens again, we have not done any specific fixes AFAIK to solve this issue in 1.15 ---
**Author:** ![excellent-wolf](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@excellent-wolf](https://dragonfly.discourse.group/u/excellent-wolf)\ **Post date:** [March 5, 2024, 5:05pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/18 "2024-03-05T17:05:56Z")
@inclusive-numbat No I don’t see swap being used. Some more information I can share: - We were still able to reproduce this as of 1.14.5. - If we only do MGETs without SETs then there’s no problem. - We have one test setup with two clients, and SETs came from only one of them. The client with only MGET still experienced the issue. ---
**Author:** ![excellent-wolf](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@excellent-wolf](https://dragonfly.discourse.group/u/excellent-wolf)\ **Post date:** [March 5, 2024, 5:19pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/19 "2024-03-05T17:19:32Z")
The minimal setup is to run two dragonfly as containers, can be in the same of different hosts, with “–cache\_mode=true” flag. Put in some data (we used a snapshot with 30M keys, which should have the same distribution as the histogram above), run a load test client on one node, and about halfway through the load test run, trigger full sync by doing the replicaof command. ---
**Author:** ![inclusive-numbat](https://avatars.discourse-cdn.com/v4/letter/i/9f8e36/32.png) [@inclusive-numbat](https://dragonfly.discourse.group/u/inclusive-numbat)\ **Post date:** [March 5, 2024, 6:26pm UTC](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135/20 "2024-03-05T18:26:21Z")
thank you @excellent-wolf this is useful. What kind of loadtest client have you run? [Next page](https://dragonfly.discourse.group/t/unresponsive-during-full-sync/135.md?page=2)