Backing Up and Restoring Dash Enterprise

This documentation is for Dash Enterprise.
Dash Enterprise is the fastest way to write & deploy Dash apps and
Jupyter notebooks.
10% of the Fortune 500 uses Dash Enterprise to productionize AI and
data science apps. Find out if your company is using Dash Enterprise.

This guide can help you create a backup of your Dash Enterprise instance and use it to restore Dash Enterprise to a previous state in the same namespace or a new namespace.

The officially supported way to back up Dash Enterprise is to use the solution that comes natively with Dash Enterprise.

Using backup and restore for Dash Enterprise, you can manage backups and perform restores directly from the Dash Enterprise interface.

Overview

All core system components, including their configurations, are included in Dash Enterprise backups. For example, Keycloak realm settings, identity provider integration (SSO), groups, and users are included.

In its current implementation, backup and restore for Dash Enterprise does not include user-created resources like Dash apps and workspaces. It is possible to recover Dash apps and workspaces from the Dash Enterprise App Manager after a restore, but if any Dash apps had data inside a managed Redis or Postgres database, or a persistent filesystem, then this data will be lost. In a future version of backup and restore for Dash Enterprise, user-created resources will be backed up.

When backups are created, they are stored in a cloud object storage service that is external to Dash Enterprise and that you manage. Supported storage service providers are AWS, Azure, and GCP.

The solution uses a combination of Restic (for data movement), pgBackRest (for the system Postgres backup), as well as Plotly-developed APIs.

Prerequisites

• A cloud object storage service (AWS, Azure, or GCP).
• The cluster that Dash Enterprise is running in must be able to reach the cloud storage service.
• kubectl access to the cluster that Dash Enterprise is running in.
• The admin role in Dash Enterprise.

Configuration

Creating the Kubernetes Secret for Storage Access

In this step, you’ll create a namespace-scoped Kubernetes secret that authorizes connection to the storage service. The command for creating this secret differs depending on the storage service provider.

To create the Kubernetes secret, you’ll need the access token or key for the storage account. If you don’t already have it on hand, obtain it from your service provider.

You can encrypt Dash Enterprise backups using a secure passphrase. When you set a passphrase, Restic encrypts your backups before sending them to your cloud storage. Setting a passphrase is recommended, but not strictly required if you prefer to rely solely on your bucket at-rest encryption.

Important information about passphrase encryption

If using a passphrase, store it securely according to your organization’s best practices. If the passphrase is changed or deleted, the system will detect a mismatch and it will not be possible to restore.

Create the Kubernetes secret using one of the following example commands.

Setting Up the Config in Dash Enterprise

In this step, you’ll define the location of your bucket and use the secret from the previous step to allow Dash Enterprise to connect to it.

Dash Enterprise can only connect to one bucket at a time. However, the same bucket can be synced to more than one Dash Enterprise instance (and will need to if restoring to a different namespace).

The config can be changed after initially saving it, except during moments when backups are being taken.

Creating Backups

You can create backups manually and/or on a schedule.

If you have already defined a backup schedule while setting up the storage config, then backups will be created automatically according to the schedule.

To define a backup schedule if you haven’t already, select Edit Configuration and then Schedule Backups.

Tip: Backup and restore for Dash Enterprise protects data on a best-effort basis, but to further help protect the data integrity, we recommend that you schedule and create backups when there is minimal activity on Dash Enterprise.

It is not currently possible to cancel backup operations.

You can create manual backups in addition to any automated backups created via a schedule.

To create a manual backup:

1. In the Dash Enterprise App Manager, go to Settings.
2. Go to Backup and Restore.
3. Select Backup Now.
4. (Optional) Enter a description for the backup.
5. Select Start Backup.

When complete, your new manual backup is added to the Backups list.

Restoring from a Backup

The time it takes to restore Dash Enterprise varies depending on the amount of data, but in most cases, a restore takes about 10 to 15 minutes.

You can restore Dash Enterprise to the same namespace or to a new namespace.

Restore Requirements

• The current Dash Enterprise version must match the Dash Enterprise version that was running when the backup was taken.
• If restoring Dash Enterprise into a new namespace, and the namespace is in a different Kubernetes cluster, the Kubernetes cluster must meet the minimum cluster requirements for Dash Enterprise.
• If restoring Dash Enterprise into a new namespace, Dash Enterprise must be installed in the new namespace with the same system configurations used when the backup was taken. These configurations include, but are not limited to:
• The hostname
• Features enabled or disabled at the system level (make sure that your installation uses the same helm install command and the values override file match the original installation).
• If restoring Dash Enterprise into a new namespace, the Kubernetes secret you defined in “Creating a Kubernetes Secret for Bucket Access” must exist in the new namespace―as well as its passphrase, if using. This is because the bucket config is part of the system components included in backups. Dash Enterprise will attempt to sync with the configured storage after the restore in order to display available backups, and the sync operation can only succeed if the secret is present in the new namespace.

Performing a Restore

To perform a restore:

1. In the Dash Enterprise App Manager, go to Settings.
2. Go to Backup and Restore. Your backups are listed in the Backups table.
3. Next to the backup from which you want to restore, select Restore.
4. Enter restore backup.
5. Select Yes, restore now to confirm. You are redirected to a page displaying the restore progress.
6. Wait for the restore to complete. In most cases, this takes 10 to 15 minutes, but can vary according to the amount of data.
7. Log back in to Dash Enterprise. The App Manager may indicate that user-created resources are available, but trying to access them results in an error. This is expected.
8. Recover user-created resources:
9. For each deployed Dash app, go to its Overview and select Rebuild and Deploy.
10. For each Dash app that has a workspace, delete the workspace and recreate it from the latest commit. We do not recommend rebuilding the workspace, as rebuilding after a restore does not always preserve the latest committed files.

Known Issues and Limitations

• Rebuilding a workspace after a restore does not always preserve the latest committed files. We recommend deleting the workspaces and recreating it from the latest commit.
• If restoring from a backup that contained Dash apps with attached managed databases (Redis or Postgres) or persistent filesystems, the data within the databases or filesystems will be lost.
• When a backup is created, new short-lived pods are spun up for handling backups of the core system components. These pods are currently scheduled on the same node as the component pods they are responsible for. You can expect to see temporarily increased CPU and memory usage on this node.