Running Dragonfly in Memcached mode doesn't support value >~ 64 Kbytes?

Dragonfly Technical
1

I have set up Dragonfly with the default configuration, specifying only the Memcached port to enable the Memcached mode. In my attempts to store values, I discovered that I am unable to write any value that exceeds ~64 KB. Values at or below this size are stored successfully, but anything above this threshold results in a write error.

Given that the default configuration for Memcached supports a minimum value size limit of 1 MB, I was expecting Dragonfly in Memcached mode to exhibit similar behavior. I have searched through the Dragonfly documentation and various configuration options but could not find any setting related to this particular behavior.

2

Hi @egalitarian-kangaroo! Could you plz post an example and the error you are getting ?

3

I am unable to write any value that exceeds ~64 KB.
Oh I think this will suffice to reproduce

4

I will get back to you

5

@egalitarian-kangaroo can you please use --max_client_iobuf_len=1000000 and see if it helps?

6 
$mem = new Memcached();
$mem->addServer("127.0.0.1", 11211);

$mem->setOption(Memcached::OPT_SERIALIZER, Memcached::SERIALIZER_IGBINARY);
$mem->setOption(Memcached::OPT_COMPRESSION, false); 

$key = "testKey";
$size = 1;

function getRandomString($length = 1) {
    $characters = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
    $charactersLength = strlen($characters);
    $randomString = '';
    for ($i = 0; $i < $length; $i++) {
        $randomString .= $characters[rand(0, $charactersLength - 1)];
    }
    return $randomString;
}

while (true) {
    $randomString = getRandomString($size);
    $result = $mem->set($key, $randomString);

    if ($result) {
        echo "Successfully written: " . $size . " bytes\n";
        $size += 1;
    } else {
        echo "Write error at size: " . $size . " bytes\n";
        echo "Maximum successfully written size: " . ($size - 1) . " bytes\n";
        break;
    }
}
?>
7

Thank you! That helped!

8

…and I have encountered a couple of limitations that I am seeking clarification on.

Key Length Limitation: During my usage, I noticed that when I attempt to retrieve values with keys longer than 107 bytes, I encounter an error. I was under the impression that Memcached generally supports key lengths up to 250 bytes. Is there a specific configuration setting or limitation within Dragonfly that results in this reduced key length capability? If so, could you guide me on how to adjust this limit or provide some insight into why this limitation exists in Dragonfly?

Total Keys Limitation in a Single Request: Additionally, I have observed that when using the getMulti function to retrieve multiple keys in a single request, I face an error when attempting to retrieve more than 102 keys at once (with each key being 10 bytes long). Is there a particular setting or limitation in Dragonfly that dictates the maximum number of keys that can be retrieved in a single getMulti request? I would appreciate any guidance on how to increase this limit if possible.

Understanding these limitations is crucial for optimizing the performance of our application, and any information or assistance you can provide would be immensely valuable. Thank you in advance for your time and support.

9 
$memcached = new Memcached();
$memcached->addServer('localhost', 11211);

function getRandomString($length = 10) {
    return substr(str_shuffle("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"), 0, $length);
}

function setKeys($memcached, $keys, $value) {
    foreach ($keys as $key) {
        $memcached->set($key, $value);
    }
}

function getKeys($memcached, $keys) {
    return $memcached->getMulti($keys);
}

$value = 'test_value';

// Testing with increasing key length
$maxKeyLength = 1;
$fixedTotalKeys = 10;
$keys = array_fill(0, $fixedTotalKeys, getRandomString($maxKeyLength));
while (true) {
    setKeys($memcached, $keys, $value);
    $results = getKeys($memcached, $keys);

    if ($results === false) {
        echo "Error at key length: $maxKeyLength with fixed total keys: $fixedTotalKeys\n";
        break;
    }

    echo "Successfully retrieved with key length: $maxKeyLength, fixed total keys: $fixedTotalKeys\n";
    $maxKeyLength++;
    $keys = array_fill(0, $fixedTotalKeys, getRandomString($maxKeyLength));
}

// Testing with increasing number of keys
$fixedKeyLength = 10;
$maxTotalKeys = 1;
$keys = array_fill(0, $maxTotalKeys, getRandomString($fixedKeyLength));
while (true) {
    setKeys($memcached, $keys, $value);
    $results = getKeys($memcached, $keys);

    if ($results === false) {
        echo "Error at total keys: $maxTotalKeys with fixed key length: $fixedKeyLength\n";
        break;
    }

    echo "Successfully retrieved with total keys: $maxTotalKeys, fixed key length: $fixedKeyLength\n";
    $maxTotalKeys++;
    $keys = array_fill(0, $maxTotalKeys, getRandomString($fixedKeyLength));
}
?>
10

Key Length Limitation:
Not one I know of. I tried with a huge key string with 200+ characters and it worked. This is the one I used:

mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1mylongkey1

and then via telnet I called get. It returned the value (44) without any issue. So maybe there is an issue with your logic above?

Total Keys Limitation in a Single Request
Looking on https://pecl.php.net/package/memcached/3.0.0 https://www.php.net/manual/en/memcached.getmulti.php https://github.com/memcached/memcached/wiki/Commands#retrieval-commands

getMulti() is a php extension (that I assume) calls internally the get command multiple times. Maybe you are passing the wrong string name?

  1. What are the errors you are getting?
  2. Have you run the above script with memcached instead of DF without any issue ?
11

As a result:

/usr/bin/memcached -m 8 -u root -d

.....
Successfully retrieved with key length: 22097657, fixed total keys: 10
Successfully retrieved with key length: 22097658, fixed total keys: 10
.....
**I got tired of waiting and aborted the script after a few hours**

/usr/local/bin/dragonfly --logtostderr --maxmemory 256m --proactor_threads=1 --max_client_iobuf_len=1000000 --cache_mode --memcached_port 11211

.....
Successfully retrieved with key length: 106, fixed total keys: 10
Successfully retrieved with key length: 107, fixed total keys: 10
Error at key length: 108 with fixed total keys: 10
.....
Successfully retrieved with total keys: 101, fixed key length: 10
Successfully retrieved with total keys: 102, fixed key length: 10
Error at total keys: 103 with fixed key length: 10
12

let me run those and I will get back to you

13

also could you plz tell me, what the output of dragonfly is ? Can you copy pasted here? Are you sure you are not out of memorying the system (because I see a limit of 256m). Could you also remove this, --maxmemory 256m and let me know if this still happens?

In the example above, there is a problem:

function getRandomString($length = 10) {
    return substr(str_shuffle("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"), 0, $length);
}

will never return a string greater than the size of the string passsed to str_shuffle. So you are not actually creating keys with large strings.

see for yourself https://onlinephp.io?s=XY7JCoJQGEb3gu_wEy0UXGhzVETzPM9twuzqtexqd2ikd8-iTR34lofv5IsBDmRJlmxBLO76BBzExybZ-ccJpy5xlKiHiMMxFMDQVXjIEoRQxAUlwMSWcaqE2zAsbNtDSkQ3YvFEMpXOZM2ttUO2g939wTsSPzhRxsX5cr3dS-VKtVZvNFvtTrfXHwxH48l0Nl8sV-uIqoGuwfdUzcnS810XpYjxMOE_ztB_eQvIwj58hNwL&v=8.2.10

14

getMulti is not a part of PHP, it calls memcached_mget from libmemcached which takes an array of keys

15

ohh you are right, get is a multi key operation

16

thank you for pointing this out

17 
/usr/local/bin/dragonfly --logtostderr --proactor_threads=1 --max_client_iobuf_len=1000000 --cache_mode --memcached_port 11211

Still same… Here is the final output of the script:

......
Successfully retrieved with total keys: 46, fixed key length: 10
......
Key: i5aqSsgvP7, Value: test_value set successfully
Key: i5aqSsgvP7, Value: test_value set successfully
Key: i5aqSsgvP7, Value: test_value set successfully
Key: i5aqSsgvP7, Value: test_value set successfully
Key: i5aqSsgvP7, Value: test_value set successfully

Error at total keys: 47 with fixed key length: 10

memcached with a limit of 8 megabytes:

/usr/bin/memcached -m 8 -u root -d

Successfully retrieved with total keys: 598, fixed key length: 10
Key: rRBPhK7qwC, Value: test_value set successfully
Key: rRBPhK7qwC, Value: test_value set successfully
Key: rRBPhK7qwC, Value: test_value set successfully
Key: rRBPhK7qwC, Value: test_value set successfully

…and You can increase the size many times more

18

let me run those and I will get back to you, won’t be today but I will do it tomorrow

19

@egalitarian-kangaroo Hi again, I run the tests but hey don’t seem to fail for me?

Successfully retrieved with key length: 10775, fixed total keys: 10

and keeps going. Could you tell me:

  1. What version of DF are you using
  2. Can you send me the logs when DF crashes?
20

@egalitarian-kangaroo also I found out that an assertion fails on debug build which probably is a bug (and thus I kinda was able to reproduce)