This documentation is for Dash Enterprise.
Dash Enterprise is the fastest way to write & deploy Dash apps and
Jupyter notebooks.
10% of the Fortune 500 uses Dash Enterprise to productionize AI and
data science apps. Find out if your company is using Dash Enterprise.
This page applies to Dash Enterprise 5. If your organization uses Dash Enterprise 4, go to the Dash Enterprise 4 version.
This guide can help you create a backup of your Dash Enterprise instance and use it to restore Dash Enterprise to a previous state on a different cluster.
You’ll back up everything inside your dedicated Dash Enterprise cluster—including Dash apps deployed by members of your organization, user information, your Dash Enterprise license, and more—so that Dash Enterprise is ready shortly after a restore.
The officially supported way to back up Dash Enterprise is with Velero, an open-source set of backup and restore tools designed for Kubernetes clusters. You’ll store your backups using your cloud provider’s storage service.
In this guide, we refer to the cluster you are currently using as your source cluster. In the event that you need to restore from a backup of the source cluster, you perform the restore on a destination cluster. Restoring Dash Enterprise on the same cluster where you backed it up from is not currently supported.
In order to use Velero, you’ll need to install the Velero CLI on your workstation and install the Velero server components on your source cluster.
The server components need to be configured with a location to store the backups, so you’ll first create this storage. To allow Velero to store data in this location, you’ll create an entity for Velero in your cloud provider and attach the appropriate policy. You can consider this entity a service account—that is, it exists for the Velero service and doesn’t belong to any particular individual in your organization.
Install the Velero CLI on your workstation so that you can run the Velero commands in later steps.
There are a few ways to install the CLI depending on your workstation’s operating system. We recommend choosing your preferred method from the official Velero docs.
In this step, you’ll use the Velero CLI to install the Velero server components on your cluster.
To verify that the Velero installation was successful, check for running pods in the velero
namespace: kubectl get pods -n velero
.
With Velero, you can create single backups and configure backup creation on a schedule. All your backups are stored in your dedicated storage location.
By default, backups expire after 30 days and Velero deletes all resources associated with expired backups, including the files in your storage location. You can change when backups expire by setting their TTL (time to live)
with the flag --ttl <duration>
in a backup creation command. Replace <duration>
with the TTL that you want to set, using the format 24h0m0s
.
To create a backup for Dash Enterprise:
sh
velero backup-location get
If PHASE
is Available
, your storage location is ready for new backups.
sh
velero backup create <backup-name> --include-namespaces dash-apps,dash-services,plotly-system --exclude-resources validatingwebhookconfiguration,mutatingwebhookconfiguration
<backup-name>
is the name you want to give to this backup.To schedule Dash Enterprise backups:
sh
velero backup-location get
If PHASE
is Available
, your storage location is ready for new backups.
sh
velero schedule create <backup-name> --schedule="* * * * *" --include-namespaces dash-apps,dash-services,plotly-system --exclude-resources validatingwebhookconfiguration,mutatingwebhookconfiguration
<backup-name>
is the name you want to give to the scheduled backups and * * * * *
is a cron expression that defines the schedule.0 0 * * *
creates a backup every day at midnight.For more information, refer to the Velero backup reference.
Velero does not alert you if a scheduled backup fails to create. We recommend setting up monitoring or alerts for your storage location using your cloud provider’s official services.
To list your Dash Enterprise backups:
* sh
velero backup get
The output looks similar to:
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
backup1 Completed 0 0 2022-04-21 15:39:00 -0400 EDT 24d default <none>
backup2 Completed 0 0 2022-04-21 15:47:48 -0400 EDT 24d default <none>
backup3 Completed 0 0 2022-04-20 16:02:52 -0400 EDT 23d default <none>
To view the logs for a specific backup:
* sh
velero backup logs <backup-name>
where <backup-name>
is the name of the backup whose logs you want to view.
To view more information about a specific backup:
* sh
velero describe backup <backup-name>
where <backup-name>
is the name of the backup you want to inspect.
The output looks similar to:
```
Name: dev-2022-08-25-1381
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.23.8
velero.io/source-cluster-k8s-major-version=1
vvelero.io/source-cluster-k8s-minor-version=23
Phase: Completed
Errors: 0
Warnings: 0
Namespaces:
Included: dash-apps, dash-services, plotly-system
Excluded: <none>
Resources:
Included: *
Excluded: validatingwebhookconfiguration, mutatingwebhookconfiguration
Cluster-scoped: auto
Label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2022-08-25 16:38:34 -0400 EDT
Completed: 2022-08-25 16:39:13 -0400 EDT
Expiration: 2022-09-24 16:38:34 -0400 EDT
Total items to be backed up: 1428
Items backed up: 1428
Velero-Native Snapshots: 16 of 16 snapshots completed successfully (specify –details for more information)
Restic Backups (specify –details for more information):
Completed: 7
```
If the Phase
is Completed
and there are no errors, it is safe to restore Dash Enterprise using this backup.
When you’re ready to perform a restore, you can choose from any available backup.
Dash Enterprise cannot be restored on the same cluster that you backed it up from. In this step, you’ll provision a new cluster,
called the destination cluster, to use for the restore. You’ll install the Velero server components on it so that you can
run the restore command.
The destination cluster has the following requirements:
In this step, you’ll install the Velero server components on your destination cluster and configure it to have access to the
storage location where your backups are stored.
To verify that Velero the installation was successful, check for running pods in the velero
namespace: kubectl get pods -n velero
.
During a restore, you’ll need to update the DNS entry for Dash Enterprise. Plan your restore accordingly.
To create a restore on the destination cluster:
Change your backup storage to read-only mode (this prevents backup objects from being created or deleted in the backup storage location during the restore process):
sh
kubectl patch backupstoragelocation default --namespace velero --type merge --patch '{"spec":{"accessMode":"ReadOnly"}}'
Create the restore:
sh
velero restore create <restore-name> --from-backup <backup-name>
where <restore-name>
is the name you want to give to the restore and <backup-name>
is the name of the backup you want to use.
The output is similar to
txt
Restore request "restore1" submitted successfully.
Run velero restore describe restore1 or velero restore logs restore1 for more details
If the Phase
is Completed
when describing your restore with velero restore describe
, the restore was successful. Note that some warnings similar to “could not restore… already exists” and
“the in-cluster version is different than the backed-up version” are expected.
sh
kubectl patch backupstoragelocation default --namespace velero --type merge --patch '{"spec":{"accessMode":"ReadWrite"}}'
In this step, you’ll update your DNS in order for https://<your-dash-enterprise-server>
to serve your restored instance of Dash Enterprise instead of your current one.
In this step, you’ll reapply metadata that Velero didn’t persist with the restore and rotate parts of the Dash Enterprise core system. Then, you’ll use the admin console to redeploy Dash Enterprise. The admin console is not available at its usual URL
due to changes to the DNS and core system, so you’ll access it via port-forwarding.
To rotate resources and redeploy Dash Enterprise:
Note: The commands below use environment variables with Unix syntax. If your workstation uses Windows, run these commands in a terminal like Git Bash or adapt the environment variable syntax.
```shell
# Extract owner reference UID
DE_UID=$(kubectl get dashenterprise dash-enterprise -n plotly-system -ojsonpath=’{.metadata.uid}’)
BUILDSTACK_UID=$(kubectl get buildstack de-buildstack -n plotly-system -ojsonpath=’{.metadata.uid}’)
HARBOR_UID=$(kubectl get harborcluster de -n plotly-system -ojsonpath=’{.metadata.uid}’)
REDPANDA_UID=$(kubectl get cluster redpanda -n plotly-system -ojsonpath=’{.metadata.uid}’)
# Apply owner reference
kubectl patch appstack de-appstack -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “dash.plotly.com/v1alpha1”, “kind”: “DashEnterprise”, “blockOwnerDeletion”: true, “controller”: true, “name”: “dash-enterprise”, “uid”: ‘"“$DE_UID”"‘}]}}’
kubectl patch buildstack de-buildstack -n plotly-system –type merge –patch ‘{“metadata”: {“finalizers”: [“csc.dash.plotly.com/finalizer”], “ownerReferences”: [{“apiVersion”: “dash.plotly.com/v1alpha1”, “kind”: “DashEnterprise”, “blockOwnerDeletion”: true, “controller”: true, “name”: “dash-enterprise”, “uid”: ‘"“$DE_UID”"‘}]}}’
kubectl patch gitea gitea-cluster -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “dash.plotly.com/v1alpha1”, “kind”: “Buildstack”, “blockOwnerDeletion”: true, “controller”: true, “name”: “de-buildstack”, “uid”: ‘"“$BUILDSTACK_UID”"‘}]}}’
kubectl patch harborcluster de -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “dash.plotly.com/v1alpha1”, “kind”: “Buildstack”, “blockOwnerDeletion”: true, “controller”: true, “name”: “de-buildstack”, “uid”: ‘"“$BUILDSTACK_UID”"‘}]}}’
kubectl patch harbor de-harbor -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “goharbor.io/v1beta1”, “kind”: “HarborCluster”, “blockOwnerDeletion”: true, “controller”: true, “name”: “de”, “uid”: ‘"“$HARBOR_UID”"‘}]}}’
kubectl patch redisfailovers de-redis -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “goharbor.io/v1beta1”, “kind”: “HarborCluster”, “blockOwnerDeletion”: true, “controller”: true, “name”: “de”, “uid”: ‘"“$HARBOR_UID”"‘}]}}’
kubectl patch statefulset redpanda -n plotly-system –type merge –patch ‘{“metadata”: {“ownerReferences”: [{“apiVersion”: “redpanda.vectorized.io/v1alpha1”, “kind”: “Cluster”, “blockOwnerDeletion”: true, “controller”: true, “name”: “redpanda”, “uid”: ‘"“$REDPANDA_UID”"‘}]}}’
# Clean up resources
kubectl delete builder dash-app-builder -n plotly-system
kubectl delete appstack de-appstack -n plotly-system
kubectl delete deployment ingress-nginx-controller -n plotly-system
```
Port-forward the admin console:
sh
kubectl port-forward -n plotly-system svc/kotsadm --address 0.0.0.0 8800:3000
Go to http://localhost:8800
Enter the admin console password that was stored as part of your Dash Enterprise installation.
Next to the Dash Enterprise version labelled “Currently deployed version,” select Redeploy.
<img>
Known issue: If the status does not change to Ready after 10 minutes, the
dash-app-builder
may have been unable to resolve the registry domain name after the DNS was updated.To check the status of the
dash-app-builder
, runkubectl get builder -n plotly-system
in a new terminal. IfREADY
isFalse
, runkubectl delete builder dash-app-builder -n plotly-system
to trigger a reconnection to the registry.
Go to https://<your-dash-enterprise-server>
to access your restored instance of Dash Enterprise.
Verify that your restored instance behaves the way you’d expect before deleting your source cluster. We recommend checking the health of the core system with:
sh
kubectl get dashenterprise -n plotly-system
If both Appstack
and Buildstack
are healthy, this indicates that Dash Enterprise is running normally.
A provisioning
status indicates that the appstack or buildstack is in the process of provisioning what it needs to be in a healthy state.
A pending
status indicates that the appstack or buildstack is waiting for another resource before it can change to a provisioning
or healthy
state.
We recommend keeping the Velero server components installed, but if you need to uninstall them from your source or destination cluster, use:
kubectl delete namespace/velero clusterrolebinding/velero
kubectl delete crds -l component=velero