Backing Up and Restoring Dash Enterprise

This documentation is for Dash Enterprise.
Dash Enterprise is the fastest way to write & deploy Dash apps and
Jupyter notebooks.
10% of the Fortune 500 uses Dash Enterprise to productionize AI and
data science apps. Find out if your company is using Dash Enterprise.

This page applies to Dash Enterprise 5. If your organization uses Dash Enterprise 4, go to the Dash Enterprise 4 version.

This guide can help you create a backup of your Dash Enterprise instance and use it to restore Dash Enterprise to a previous state on a different cluster.

You’ll back up everything inside your dedicated Dash Enterprise cluster—including Dash apps deployed by members of your organization, user information, your Dash Enterprise license, and more—so that Dash Enterprise is ready shortly after a restore.

The officially supported way to back up Dash Enterprise is with Velero, an open-source set of backup and restore tools designed for Kubernetes clusters. You’ll store your backups using your cloud provider’s storage service.

In this guide, we refer to the cluster you are currently using as your source cluster. In the event that you need to restore from a backup of the source cluster, you perform the restore on a destination cluster. Restoring Dash Enterprise on the same cluster where you backed it up from is not currently supported.

Prerequisites

Setting Up Velero

In order to use Velero, you’ll need to install the Velero CLI on your workstation and install the Velero server components on your source cluster.

The server components need to be configured with a location to store the backups, so you’ll first create this storage. To allow Velero to store data in this location, you’ll create an entity for Velero in your cloud provider and attach the appropriate policy. You can consider this entity a service account—that is, it exists for the Velero service and doesn’t belong to any particular individual in your organization.

Creating a Storage Location for Your Backups

Creating and Configuring a Service Account for Velero

Installing the Velero CLI

Install the Velero CLI on your workstation so that you can run the Velero commands in later steps.

There are a few ways to install the CLI depending on your workstation’s operating system. We recommend choosing your preferred method from the official Velero docs.

Installing the Velero Server Components on the Source Cluster

In this step, you’ll use the Velero CLI to install the Velero server components on your cluster.

To verify that the Velero installation was successful, check for running pods in the velero namespace: kubectl get pods -n velero.

Managing Backups

With Velero, you can create single backups and configure backup creation on a schedule. All your backups are stored in your dedicated storage location.

By default, backups expire after 30 days and Velero deletes all resources associated with expired backups, including the files in your storage location. You can change when backups expire by setting their TTL (time to live)
with the flag --ttl <duration> in a backup creation command. Replace <duration> with the TTL that you want to set, using the format 24h0m0s.

Creating a Backup

To create a backup for Dash Enterprise:

Connect to your source cluster if you aren’t already, and then verify that your backup location is available:
sh velero backup-location get

If PHASE is Available, your storage location is ready for new backups.

Create the backup:
sh velero backup create <backup-name> --include-namespaces dash-apps,dash-services,plotly-system --exclude-resources validatingwebhookconfiguration,mutatingwebhookconfiguration
where <backup-name> is the name you want to give to this backup.

Scheduling Backups

To schedule Dash Enterprise backups:

Connect to your source cluster if you aren’t already, and then verify that your backup location is available:
sh velero backup-location get

If PHASE is Available, your storage location is ready for new backups.

Create the backup schedule:
sh velero schedule create <backup-name> --schedule="* * * * *" --include-namespaces dash-apps,dash-services,plotly-system --exclude-resources validatingwebhookconfiguration,mutatingwebhookconfiguration
where <backup-name> is the name you want to give to the scheduled backups and * * * * * is a cron expression that defines the schedule.
For example, 0 0 * * * creates a backup every day at midnight.

For more information, refer to the Velero backup reference.

Velero does not alert you if a scheduled backup fails to create. We recommend setting up monitoring or alerts for your storage location using your cloud provider’s official services.

Examining Your Backups

To list your Dash Enterprise backups: * sh velero backup get

The output looks similar to:
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR backup1 Completed 0 0 2022-04-21 15:39:00 -0400 EDT 24d default <none> backup2 Completed 0 0 2022-04-21 15:47:48 -0400 EDT 24d default <none> backup3 Completed 0 0 2022-04-20 16:02:52 -0400 EDT 23d default <none>

To view the logs for a specific backup: * sh velero backup logs <backup-name>
where <backup-name> is the name of the backup whose logs you want to view.

To view more information about a specific backup: * sh velero describe backup <backup-name>
where <backup-name> is the name of the backup you want to inspect.

The output looks similar to:
```
Name: dev-2022-08-25-1381
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.23.8
velero.io/source-cluster-k8s-major-version=1
vvelero.io/source-cluster-k8s-minor-version=23

Phase: Completed

Errors: 0
Warnings: 0

Namespaces:
Included: dash-apps, dash-services, plotly-system
Excluded: <none>

Resources:
Included: * Excluded: validatingwebhookconfiguration, mutatingwebhookconfiguration
Cluster-scoped: auto

Label selector: <none>

Storage Location: default

Velero-Native Snapshot PVs: auto

TTL: 720h0m0s

Hooks: <none>

Backup Format Version: 1.1.0

Started: 2022-08-25 16:38:34 -0400 EDT
Completed: 2022-08-25 16:39:13 -0400 EDT

Expiration: 2022-09-24 16:38:34 -0400 EDT

Total items to be backed up: 1428
Items backed up: 1428

Velero-Native Snapshots: 16 of 16 snapshots completed successfully (specify –details for more information)

Restic Backups (specify –details for more information):
Completed: 7
```

If the Phase is Completed and there are no errors, it is safe to restore Dash Enterprise using this backup.

When you’re ready to perform a restore, you can choose from any available backup.

Preparing Your Destination Cluster

Dash Enterprise cannot be restored on the same cluster that you backed it up from. In this step, you’ll provision a new cluster,
called the destination cluster, to use for the restore. You’ll install the Velero server components on it so that you can
run the restore command.

The destination cluster has the following requirements:

It must use the same cloud provider as the source cluster.
It must be in the same region as the source cluster.
It must use the same Kubernetes version as the source cluster.
It must have the same amount of nodes, or more nodes, than the source cluster.
Its nodes must use instance types that are equal to or larger than the nodes on the source cluster.

Provisioning a New Cluster

Installing the Velero Server Components on the Destination Cluster

In this step, you’ll install the Velero server components on your destination cluster and configure it to have access to the
storage location where your backups are stored.

To verify that Velero the installation was successful, check for running pods in the velero namespace: kubectl get pods -n velero.

Restoring Dash Enterprise on the Destination Cluster

During a restore, you’ll need to update the DNS entry for Dash Enterprise. Plan your restore accordingly.

Creating a Restore

To create a restore on the destination cluster:

Change your backup storage to read-only mode (this prevents backup objects from being created or deleted in the backup storage location during the restore process):
sh kubectl patch backupstoragelocation default --namespace velero --type merge --patch '{"spec":{"accessMode":"ReadOnly"}}'
Create the restore:
sh velero restore create <restore-name> --from-backup <backup-name>
where <restore-name> is the name you want to give to the restore and <backup-name> is the name of the backup you want to use.

The output is similar to
txt Restore request "restore1" submitted successfully. Run velero restore describe restore1 or velero restore logs restore1 for more details

If the Phase is Completed when describing your restore with velero restore describe, the restore was successful. Note that some warnings similar to “could not restore… already exists” and
“the in-cluster version is different than the backed-up version” are expected.

Return your backup storage to read-write mode:
sh kubectl patch backupstoragelocation default --namespace velero --type merge --patch '{"spec":{"accessMode":"ReadWrite"}}'

Updating Your DNS

In this step, you’ll update your DNS in order for https://<your-dash-enterprise-server> to serve your restored instance of Dash Enterprise instead of your current one.

Wait for the change to propagate.

Rotating Resources and Redeploying

In this step, you’ll reapply metadata that Velero didn’t persist with the restore and rotate parts of the Dash Enterprise core system. Then, you’ll use the admin console to redeploy Dash Enterprise. The admin console is not available at its usual URL
due to changes to the DNS and core system, so you’ll access it via port-forwarding.

To rotate resources and redeploy Dash Enterprise:

Reapply metadata and rotate core system resources:

Note: The commands below use environment variables with Unix syntax. If your workstation uses Windows, run these commands in a terminal like Git Bash or adapt the environment variable syntax.

```shell
# Extract owner reference UID

DE_UID=$(kubectl get dashenterprise dash-enterprise -n plotly-system -ojsonpath=’{.metadata.uid}’)
BUILDSTACK_UID=$(kubectl get buildstack de-buildstack -n plotly-system -ojsonpath=’{.metadata.uid}’)
HARBOR_UID=$(kubectl get harborcluster de -n plotly-system -ojsonpath=’{.metadata.uid}’)
REDPANDA_UID=$(kubectl get cluster redpanda -n plotly-system -ojsonpath=’{.metadata.uid}’)

# Apply owner reference

kubectl patch appstack de-appstack -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “dash.plotly.com/v1alpha1”, “kind”: “DashEnterprise”, “blockOwnerDeletion”: true, “controller”: true, “name”: “dash-enterprise”, “uid”: ‘"“$DE_UID”"‘}]}}’
kubectl patch buildstack de-buildstack -n plotly-system –type merge –patch ‘{“metadata”: {“finalizers”: [“csc.dash.plotly.com/finalizer”], “ownerReferences”: [{“apiVersion”: “dash.plotly.com/v1alpha1”, “kind”: “DashEnterprise”, “blockOwnerDeletion”: true, “controller”: true, “name”: “dash-enterprise”, “uid”: ‘"“$DE_UID”"‘}]}}’
kubectl patch gitea gitea-cluster -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “dash.plotly.com/v1alpha1”, “kind”: “Buildstack”, “blockOwnerDeletion”: true, “controller”: true, “name”: “de-buildstack”, “uid”: ‘"“$BUILDSTACK_UID”"‘}]}}’
kubectl patch harborcluster de -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “dash.plotly.com/v1alpha1”, “kind”: “Buildstack”, “blockOwnerDeletion”: true, “controller”: true, “name”: “de-buildstack”, “uid”: ‘"“$BUILDSTACK_UID”"‘}]}}’
kubectl patch harbor de-harbor -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “goharbor.io/v1beta1”, “kind”: “HarborCluster”, “blockOwnerDeletion”: true, “controller”: true, “name”: “de”, “uid”: ‘"“$HARBOR_UID”"‘}]}}’
kubectl patch redisfailovers de-redis -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “goharbor.io/v1beta1”, “kind”: “HarborCluster”, “blockOwnerDeletion”: true, “controller”: true, “name”: “de”, “uid”: ‘"“$HARBOR_UID”"‘}]}}’
kubectl patch statefulset redpanda -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “redpanda.vectorized.io/v1alpha1”, “kind”: “Cluster”, “blockOwnerDeletion”: true, “controller”: true, “name”: “redpanda”, “uid”: ‘"“$REDPANDA_UID”"‘}]}}’

# Clean up resources

kubectl delete builder dash-app-builder -n plotly-system
kubectl delete appstack de-appstack -n plotly-system
kubectl delete deployment ingress-nginx-controller -n plotly-system
```

Port-forward the admin console:
sh kubectl port-forward -n plotly-system svc/kotsadm --address 0.0.0.0 8800:3000
Go to http://localhost:8800
Enter the admin console password that was stored as part of your Dash Enterprise installation.
Next to the Dash Enterprise version labelled “Currently deployed version,” select Redeploy.

<img>

Wait for the status to change to Ready.

Known issue: If the status does not change to Ready after 10 minutes, the dash-app-builder may have been unable to resolve the registry domain name after the DNS was updated.

To check the status of the dash-app-builder, run kubectl get builder -n plotly-system in a new terminal. If READY is False, run kubectl delete builder dash-app-builder -n plotly-system
to trigger a reconnection to the registry.

Go to https://<your-dash-enterprise-server> to access your restored instance of Dash Enterprise.
Verify that your restored instance behaves the way you’d expect before deleting your source cluster. We recommend checking the health of the core system with:
sh kubectl get dashenterprise -n plotly-system

If both Appstack and Buildstack are healthy, this indicates that Dash Enterprise is running normally.

A provisioning status indicates that the appstack or buildstack is in the process of provisioning what it needs to be in a healthy state.

A pending status indicates that the appstack or buildstack is waiting for another resource before it can change to a provisioning or healthy state.

Uninstalling Velero

We recommend keeping the Velero server components installed, but if you need to uninstall them from your source or destination cluster, use:

kubectl delete namespace/velero clusterrolebinding/velero
kubectl delete crds -l component=velero