Enabling Data Persistence

Hi folks,

What is the best way to enable data persistence when deploying a 3 replic setup of dragonfly onto a Kuberenetes cluster?

Today we have deployed a dragonfly with the operator with the below resource.

apiVersion: dragonflydb.io/v1alpha1
kind: Dragonfly
metadata:
  labels:
    app.kubernetes.io/created-by: dragonfly-operator
    app.kubernetes.io/instance: dragonfly-mover-integration-adapter
    app.kubernetes.io/name: dragonfly
    app.kubernetes.io/part-of: dragonfly-operator
    argocd.argoproj.io/instance: mover-integration-adapter-prod
  name: dragonfly-mover-integration-adapter
  namespace: delivery-prod
spec:
  args:
    - '--cluster_mode=emulated'
  authentication:
    passwordFromSecret:
      key: cache-password
      name: dragonfly-cache-secret-mover-integration-adapter
  image: 'docker.dragonflydb.io/dragonflydb/dragonfly:v1.24.0'
  replicas: 3
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 200m
      memory: 256Mi
  topologySpreadConstraints:
    - labelSelector:
        matchLabels:
          app.kubernetes.io/instance: dragonfly-mover-integration-adapter
      maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule

We regularly roll our cluster nodes for mainenance and upgrades and need data to survive this and for the key/values to expire on their own. We seem to have lost the key/value pairs when we rolled our cluster yesterday…

What is the best way to achieve this using the operator?

Hi @thedukedk,

Note that Redis supports two ways of data persistence: snapshot (RDB) and append-only file (AOF). Simply put, a snapshot is used to capture the whole data store at a point in time, and AOF captures individual commands. More details here.

Currently, Dragonfly supports only snapshot for data persistence. Thus, when used in Kubernetes, the configuration can be done via the snapshot spec.

For instance, you can easily choose to periodically save snapshots to PVC or directly to S3 by following our documentation:

In terms of your use case… Taking a snapshot is a rather heavy (but still fast, stable, and safe) process in Dragonfly. It doesn’t happen every second; it doesn’t happen for each individual command. Instead, it happens based on a defined period of time. Thus, any data writes that happen in between two snapshots are volatile, as they only exist in memory.

Note that Dragonfly is a stateful system, probably unlike your service application, which can be largely stateless. From a database or data store user’s perspective, I wouldn’t expect a system like this to frequently get restarted or replaced. :grinning: Let me know if I am missing anything, though. It’s always delightful to discuss.

Thanks. Yes, we were looking at this and came to a couple of conclusions.

  1. We ensure replicas(3) are spread across nodes.
  2. A PDB with a maxUnavailable=1 will help but will not ensure we don’t lose data. We would need either the readiness probe to wait until a replica is synced or the operator to ensure we always have a synced replica. Doesn’t look like either do so.
  3. A snapshot seems to be done when a pod is killed/terminated. This along with a schedule can help avoid data loss.

If we combine number three(3) with number two(2) we should be pretty safe as we ensure number one(1) and we roll our nodes one by one.

It would be a major improvement, we think, if this was implemented in the operator as the comment on the below issue suggests.

I think it would be nice if the PDB is dynamically changed by the operator, so that a disruption of the master is only allowed if there is at least 1 replica in full sync with the master. Otherwise it could happen that the master is evicted too early by k8s and the cache is gone.

2 Likes

I’m a colleague of @thedukedk

I whole wholeheartedly agree with @thedukedk that the PDB should be part of the Dragonfly operator itself. As there’s some internals that should be reflected in the logical implementation of the PDB. In our case that the Dragonfly instances are fully in-sync before an instance is removed.

The scenario where we saw the loss of data was because we upgraded the Kubernetes cluster whereon the Dragonfly setup is running. The way we do it is by interchanging Kubernetes nodes one or more at a time ( the amount depends on a set of rules. E.g. available compute resources and the like ). So let’s say worker a, whereon dragonfly instance x is running, is removed … as there’s no “insurance policy” between dragonfly instances that the data is in-sync accross the distributed dragonfly setup … we have the trouble.

Makes sense? If not @joezhou_df feel free to ask questions.

Thank you very much.

1 Like

I see. Yeah, I agree - it makes a lot of sense after reading more about PodDisruptionBudget.

Although from Dragonfly’s (or Redis/Valkey’s) replication perspective, it is a best effort in terms of data consistency. The failover is expected to be lossy due to the async nature of replication, especially when the primary instance is heavy on writes. Anyway, I suppose the GitHub issue above captures the needs, and let’s see how the team and community go about the implementation.

Thank you for the response.

There’s more to it I think. There’s this site on the dragonfly docs: Replication | Dragonfly - so Dragonfly knows the current synchronization lag level. So the PDB logic could be ( if implemented in dragonfly ):

  1. Block a long as the log is not below X sane value or event block until the lag is 0
    1. One might want to be able to tune the value on lag that the PDB uses … as I would assume that high churn systems would have a hard time getting to 0 lag. So on such systems the end-user would have to get to some compromise.
  2. When lag is 0 >> stop blocking and allow a dragonfly instance to be deleted

Thoughts @joezhou_df

This can be a good suggestion to add to the operator failover process. Although I am not sure if there’s a delivery timeline for this and PDB given the current workload we have for cloud and upcoming Dragonfly features.