Dash Enterprise 4 to 5 Migration

This guide can help you if your organization has upgraded from Dash Enterprise 4.X to Dash Enterprise 5.X and you want to migrate your users and apps.

You have two options to migrate:
- Perform an automated migration. An automated migration involves running Plotly-provided migration scripts to recreate your Dash Enterprise 4 apps and users on Dash Enterprise 5.
- Perform a manual migration. A manual migration involves creating users, updating your apps, and deploying them to Dash Enterprise 5.

General Limitations

The following limitations apply to both migration methods:
- Due to API incompatibility, apps that rely on the Dash Enterprise 4.X GraphQL API will probably not work when migrated to Dash Enterprise 5.X. The Platform Analytics sample app (and likely any apps based on it) is known to not work.

Automated Migration

The Dash Enterprise migration scripts migrate user and app data from Dash Enterprise 4 to Dash Enterprise 5. After running them, you’ll have your Dash Enterprise 4 users (including user SSH keys), apps, app viewer access settings, and environment variables in Dash Enterprise 5.

The scripts also create any services required by apps and update app files to convert runtime.txt, apt, conda-requirements.txt and app.json to the file types required by Dash Enterprise 5. Note that the scripts do not update app.py.

If you plan on using an IdP with Dash Enterprise 5, use the migration scripts before setting up the IdP.

Automated Migration Limitations

The following are not handled by the migration scripts:
- Data from mapped directories is not migrated. Mapped directories are not available in Dash Enterprise 5. Create a persistent filesystem for an app’s data. See the Directory Mappings section below for more details.
- Postgres and Redis data is not migrated.

Prerequisites

• You have administrator access to Dash Enterprise 4 and Dash Enterprise 5.
• You have Python 3.8 installed.
• You have not manually migrated any users or apps. If you manually migrate users or apps before running the scripts, the scripts will fail.

Running the Migration Scripts

To run the Dash Enterprise migration scripts:

1. Download the scripts here and extract them. There are several folders and files that you will use during the migration. The key ones are:
   * migration_scripts: Folder containing the migration scripts.
   * migration: Folder to which migration data is saved as you run the scripts. This is initially empty.
   * .env.sample: A sample file showing how to declare the environment variables and values required by the migration scripts.
2. Using the same format as the .env.sample, create a .env file in the root of the scripts folder. This file must contain the following:
   * DE4_ADMIN_USERNAME
   * DE4_ADMIN_PASSWORD
   * DE4_ADMIN_TOKEN
   * DE4_HOST
   * DE4_SSH_USER
   * DE4_SSH_PORT
   * DE4_SSH_PRIVATE_KEY_PATH
   * DE5_HOST
   * DE5_KEYCLOAK_ADMIN_USERNAME
   * DE5_KEYCLOAK_ADMIN_PASSWORD
   * DE5_INTERNAL_PYPI_SERVICE (only required if Dash Enterprise 5.X is airgapped)

• To migrate from a Dash Enterprise 4 instance with a portal at https://example-dash-enterprise-4-hostname.com/Portal to a Dash Enterprise 5 instance with a portal at https://example-dash-enterprise-5-hostname.com/Portal, you’d set DE4_HOST=example-dash-enterprise-4-hostname.com and DE5_HOST=example-dash-enterprise-5-hostname.com

• The DE4_ADMIN_TOKEN is your Dash Enterprise 4 API key. You’ll find it in Dash Enterprise 4 by selecting your username and then Manage API key:

  <img>

• DE4_SSH_USER is the username used to create the VM where Dash Enterprise 4 is deployed to. If Dash Enterprise 4 is deployed on an Azure VM, this is often azureuser, and on AWS, it is often ubuntu.

• DE4_SSH_PORT is the port that the migration scripts will use to SSH into your Dash Enterprise 4 VM. Usually port 22.
• DE4_SSH_PRIVATE_KEY_PATH is the path to the private key that corresponds to the public key added to your Dash Enterprise 4 VM.

3. Create a virtual environment:
   bash python -m venv venv

And activate it:

Windows:
bash venv\Scripts\activate

Mac/Linux:
bash source venv/bin/activate

4. Run pip install -r requirements.txt to install the requirements for the scripts.
5. Go to the migration_scripts folder.
6. Run the scripts one at a time in the order indicated by their numbered prefixes. Here is an overview of what each script does and any options you’ll be presented with when you run them:
   - 00_credentials.py: Generates a new RSA-SSH key pair to use for connecting to Dash Enterprise 4.
   - 01_de4_download_data.py: Downloads information about Dash Enterprise 4 apps, users, and portal to the migration/de4 folder. If it can’t download a particular app or user, it will display a reason for the failure. If you want to inspect the output of the script, go to migration/de4, where you’ll find a users_de4.json file with user info, an apps_de4.json with app info, a portal.json file with portal info, and a failed_apps_de4.json if there were any issues downloading apps from Dash Enterprise 4.
   - 02_de4_clone_repos.py: Clones the app code and project files for each of your Dash Enterprise 4 apps to the migration/de4/apps folder. When you run the script, you’ll be given the option to clone with SSH or HTTPS. When running, it gives details of the app currently being processed and how many remain to be processed.
   - 03_convert_data.py: Converts the Dash Enterprise 4 data fetched by 01_de4_download_data.py to a format that is compatible with Dash Enterprise 5. It saves the converted app data to migration/de5/apps_de5.json and the converted user data to /migration/de5/users_de5.json.
   - 04_convert_apps.py: Converts the app code and project files to file types required by Dash Enterprise 5 and saves them to migration/de5/apps. It:
   • Converts Dash Enterprise 4 app.json and runtime.txt files to a project.toml file, capturing any predeploy and postdeploy scripts and a Python version if specified. The Python version pinned in project.toml corresponds to the major and minor version specified in runtime.txt. For example, if the runtime.txt specifies Python 3.7.5, it changes to 3.7.* in project.toml to minimize compatibility errors.
   • Builds an Aptfile based on Apt files from Dash Enterprise 4.
   • For Conda apps, creates an environment.yml file based on the Dash Enterprise 4 conda-requirements.txt file.
   • Makes changes required to the requirements.txt file: Adding the Dash Enterprise 5 packages endpoint
   • 05_de5_create_users.py: Creates users and groups from Dash Enterprise 4 in Dash Enterprise 5. Details on any failures will be found after the script runs in migration/de5/failed_users_de5.json.
   • 06_de5_create_apps.py: Creates apps in Dash Enterprise 5 that the app code will be deployed to.
   • 07_de5_attach_ssh.py: Attaches each user’s SSH key from Dash Enterprise 4 to the corresponding migrated user in Dash Enterprise 5.
   • 08_de5_deploy_apps.py: Deploys the app code to Dash Enterprise 5.
   • 09_de5_check_deploy_status.py: Validates that apps have been deployed. You’ll find details of any failures in migration/de5/status.json. In this file, successfully deployed apps show as success, apps that failed to build show as null, and apps that built but failed to deploy show as failed.
   • 10_de5_redeploy_failed_apps_with_old_pip.py: (Optional) Attempts to redeploy failed apps in the migration/de5/status.json file using an older version of pip.
   • 11_set_temp_passwords_for_users.py: Sets temporary passwords for users.
   • 12_clean.py: Removes the contents of the directories and JSON files created during migration. If you’re partway through a migration and need to restart, run 12_clean.py before restarting.

Additional Steps with an External IdP

After you’ve run the migration scripts, set up your external IdP using the Keycloak IdP setup wizard and then ask each user to complete the following steps to link their IdP user with the native auth user created by the migration scripts:

1. Log in to Dash Enterprise with your IdP username and password.
   You’ll receive a notification that an “Account already exists”.
2. Select Add to existing account.
3. Enter your native auth username and temporary password (Password123!) when prompted.
4. You’ll be asked to update your password. Enter your IdP password. You’ll now be able to log in to Dash Enterprise with your IdP credentials.

Known Issues

• There is a known issue where SAML groups aren’t successfully migrated to Dash Enterprise 5, but no error is displayed.
• There is a known issue where 09_de5_check_deploy_status.py shows an incorrect number for successfully deployed apps on Dash Enterprise 5. See the migration/de5/status.json file for the correct numbers.

Troubleshooting

• If the scripts fail to get apps from Dash Enterprise 4, you’ll find details in the migration/de4/failed_apps_de4.json file. If the scripts fail to deploy apps to Dash Enterprise 5, you’ll find details in migration/de5/status.json.
• If you experience issues with apps running after the migration, you may need to make manual adjustments to app requirements to resolve any issues. See the Apps section below for more details.

Manual Migration

This section shows you how to manually recreate your Dash Enterprise 4 users, apps, and app configurations on Dash Enterprise 5.

Users

If you are an admin migrating apps on behalf of a developer, create an account on Dash Enterprise 5 for them and use that when creating that user’s apps.

To create an account for a developer:

1. Create a new user with a license seat.

   If your organization is planning to use an IdP with Dash Enterprise,
   ensure the username of each account matches the value in the IdP that will be mapped to the username.

2. Log in with the user’s credentials and initialize, migrate, and deploy the apps.

3. When you’re done, share the credentials with the developer and set their password to Temporary so that they’ll be prompted to change it. If the user logs in via an IdP, they’ll be asked to link the IdP account to the account you created.

Apps

Some Dash Enterprise 4 features are not supported in Dash Enterprise 5 and other features work differently,
meaning in many cases you’ll need to make some updates to your apps for them to run on Dash Enterprise 5.
For example, although both Dash Enterprise 4 and 5 support pip and Conda buildpacks, the buildpacks use some different files in Dash Enterprise 5.

Key differences in Dash Enterprise 5
- Different default Python, pip, and Miniconda versions.
- For Conda apps, dependencies are now specified in an environment.yml file,
instead of with conda-requirements.txt and requirements.txt. A requirements.txt file prevents Conda-based dependencies from being installed.
- Changing the Python version is now done via a project.toml file instead of runtime.txt. In Conda,
the Python version can be set in environment.yml.
- Different URL for Dash Enterprise packages (you’ll need to change any --extra-index-urls).
- APT packages are now installed using one Aptfile.
- Pre-deploy and post-deploy scripts are configured in project.toml instead of app.json.

See also: Dash Enterprise 5 Breaking Changes.

Here’s an overview of manual app migration:

1. Initialize a new app on Dash Enterprise 5.
2. Update the app’s project structure and files to make the app compatible with Dash Enterprise 5.
3. Add any services that the app requires.
4. Add any environment variables that the app requires.
5. If your Dash Enterprise 4 app uses local directory mappings, update the app to instead use Dash Enterprise 5’s persistent filesystem.
6. Important: If deploying over SSH, remove the custom port in your SSH config. Dash Enterprise 5 uses the standard port 22 for SSH deploys, not a custom port (3022 by default) like Dash Enterprise 4. No change is required if you are deploying using the Dash Enterprise CLI (Dash Enterprise 5.2.X and later) because it uses HTTPS.
7. Deploy the app to Dash Enterprise 5.

Project Structure and File Updates

To change your app files and structure, you can either:
- Upload your existing app files to Workspaces and try running the app. If it doesn’t run, follow this guide to update it.
- Make the updates outlined here locally and then deploy the updated app to Dash Enterprise 5.

Dash Enterprise Package Compatibility

We recommend updating all Dash Enterprise packages to their latest versions.

At a minimum, the following changes are required:

• Versions of dash-enterprise-auth prior to 0.0.6 are not supported in Dash Enterprise 5.
  If you are using dash-enterprise-auth and have it pinned to a specific version,
  you’ll need to update it to be dash-enterprise-auth>=0.0.6.
• Versions of dash-snapshots prior to 2.2.1 are not supported in Dash Enterprise 5.
  If you are using dash-snapshots and have it pinned to a specific version,
  you’ll need to update it to be dash-snapshots>=2.2.1.

Resolving Dependency Issues

In your local environment or workspace, try the following:

• Reduce your requirements.txt file (or environment.yml file if using Conda) to the minimal dependencies your app needs (excluding subdependencies).

• If reducing your dependencies doesn’t work, remove version numbers from your requirements.txt or environment.yml file.

• If you continue to experience issues after trying both of the above, reinstall requirements one by one in the terminal.

Adding Services

Both Dash Enterprise 4 and 5 provide managed Postgres and Redis services. A key difference between services in Dash Enterprise 4 and Dash Enterprise 5 is
that in Dash Enterprise 5 services are app-specific, meaning a Redis or Postgres service you add to an app can only be used
by that app.

For details on how to add Redis and Postgres to an app, see:

• Adding Redis to an App
• Adding Postgres to an App

Adding Environment Variables

If your Dash Enterprise 4 apps relied on environment variables, you’ll need to add those environment variables in Dash Enterprise 5.
The process is similar in Dash Enterprise 5. See Adding Environment Variables
for full details on adding environment variables.

Directory Mappings

If you use local directory mappings for files for your Dash Enterprise 4 app,
save these files to your Dash Enterprise 5 app’s persistent filesystem.
You can reference files in an app’s persistent filesystem in your code with '../mount/<file-name>'.
See the Persistent Filesystem chapter for more details and examples.