This guide can help you if you are a new Dash Enterprise customer looking to start with a Dash Enterprise 5 installation, or if you are upgrading from Dash Enterprise 4.X.
For Dash Enterprise 4 installation instructions, see the Dash Enterprise 4 version.
Dash Enterprise 5 runs on Kubernetes, an open-source system that automates application lifecycles. Several managed services allow you to get started with Kubernetes-based software quickly. In this guide, you’ll learn how to install Dash Enterprise on Azure Kubernetes Service (AKS), available with an Azure subscription.
In AKS, you work with an AKS cluster, which consists of multiple worker machines that are also called nodes. You host your AKS cluster inside an Azure virtual network (VNet), a virtual network dedicated to your Azure subscription.
View diagram
<img>
Installing Dash Enterprise is an automated process. You use a bootstrap node to run Plotly-provided scripts that provision the infrastructure and install Dash Enterprise on it. A bootstrap node is a virtual machine (VM) whose only purpose is to run the scripts. After Dash Enterprise is installed, you can decommission it. Using fresh VMs is the best practice because the scripts are unlikely to run into errors caused by other installed software. This guide describes how to provision an Azure VM to serve as your bootstrap node (please reach out to our Customer Success team if you’re unable to use Azure VMs).
As part of the automated infrastructure provisioning, the AKS cluster is provisioned with the Azure CLI az aks create
command. A VNet is created if you don’t already have one, and a load balancer is created as the main point of entry for all traffic directed towards Dash Enterprise.
Remember that these resources count towards your Azure quotas. Review your Azure billing and quotas if necessary.
The cluster nodes belong to a single availability zone (AZ). The Dash Enterprise core system isn’t currently configured for high availability; however, as long as the core system is available, Dash app developers can take advantage of features like app replicas to increase the availability of deployed apps. High availability for the core system may be supported in the future.
You’ll be installing Dash Enterprise as the single tenant on the AKS cluster—that is, no other software is installed on the cluster (except mandatory supporting software). Single-tenancy is well-suited for Dash Enterprise because it is a complex platform: Dash Enterprise interacts with the Kubernetes API to organize resources on the fly when developers perform tasks like deploying Dash apps and creating databases. Multi-tenancy is not currently supported.
Plotly uses Replicated to package and deliver Dash Enterprise. You’ll be interacting with the KOTS Admin Console, part of the Replicated toolset, in the configuration step of this installation. After the installation, you’ll continue to use the KOTS Admin Console for system administration such as performing Dash Enterprise upgrades.
Here’s what you’ll need before you can start your Dash Enterprise installation:
.pem
file in the following format:txt
-----BEGIN CERTIFICATE-----
<Your>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<Your>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<Your>
-----END CERTIFICATE-----
Self-signed certificates, internally signed certificates, and using multiple certificates are not supported. If you obtained your certificate as multiple files, you need to combine them into a single .pem
file. You can do this with cat server.pem intermediate.pem trustedroot.pem > fullchain.pem
on Linux or copy server.pem+intermediate.pem+trustedroot.pem fullchain.pem
on Windows, replacing the file names if yours are different.
You’ll upload the full certificate chain and unencrypted private key during the configuration, and they will be used to terminate TLS/SSL. If you need to preview the required DNS entries, go to Creating DNS Entries, but note that you’ll only be able to create them after the installation.
* The following Azure resources:
* An Azure subscription. You’ll need to know the ID of this subscription. Obtain the ID with:
az account list --query '[?isDefault].id' -o tsv
A user with sufficient permissions to manage Azure VMs (required to provision a bootstrap node). One way to achieve this is to use the subscription-wide Contributor
role assignment: Go to your subscription > Access Control (IAM) > Add Role Assignment > Contributor > select your user > select Review & Assign.
A quota of 48 vCPUs (Standard Dash Enterprise) or 96 vCPUs (Premium Dash Enterprise) for Standard DDv5 Family vCPUs in the region you want to use for your AKS cluster.
(Each node in your cluster will use 8 vCPUs, and this requirement factors in an additional two nodes’ worth of vCPU quotas to ensure smooth node rotations).
A custom role with AssignableScopes
set to your subscription and the following permissions:
"Microsoft.Resources/subscriptions/resourcegroups/read",
"Microsoft.Resources/subscriptions/resourcegroups/write",
"Microsoft.Resources/subscriptions/resourcegroups/delete",
"Microsoft.ContainerService/managedClusters/write",
"Microsoft.ContainerService/managedClusters/read",
"Microsoft.ContainerService/managedClusters/agentPools/write",
"Microsoft.ContainerService/managedClusters/agentPools/read",
"Microsoft.ContainerService/managedClusters/listClusterUserCredential/action"
If you don’t already have a custom role that meets this prerequisite, create a role named “install-dash-enterprise” with:
az role definition create --role-definition '{"Name": "install-dash-enterprise","Description": "Role for Dash Enterprise installation","Actions": ["Microsoft.Resources/subscriptions/resourcegroups/read","Microsoft.Resources/subscriptions/resourcegroups/write","Microsoft.Resources/subscriptions/resourcegroups/delete","Microsoft.ContainerService/managedClusters/write","Microsoft.ContainerService/managedClusters/read","Microsoft.ContainerService/managedClusters/agentPools/write","Microsoft.ContainerService/managedClusters/agentPools/read","Microsoft.ContainerService/managedClusters/listClusterUserCredential/action"],"AssignableScopes": ["/subscriptions/'<subscription-id>'"]}'
where <subscription-id>
is the ID of your Azure subscription. Change install-dash-enterprise
if you want to use a different role name.
az ad sp create-for-rbac --name dash-enterprise-sp --role install-dash-enterprise --scopes /subscriptions/<subscription-id> --create-cert
<subscription-id>
is the ID of your Azure subscription. Change dash-enterprise-sp
if you want to use a different service principal name and install-dash-enterprise
if you used a different role name.Note: If you’re running this command in Git Bash on Windows, you’ll need to disable POSIX-to-Windows path conversion by adding
MSYS_NO_PATHCONV=1
to the command.
A service principal cert is generated. Note the appId
and tenant
in the output.
Domain/Port | Purpose | When | I/O |
---|---|---|---|
replicated.com 443 |
Validate license | Installation, upgrade, license validation | Outbound |
proxy.replicated.com 443 |
Download Plotly Docker image stored privately on quay.io | Installation, upgrade | Outbound |
registry.opensource.zalan.do 443 |
Download Docker image for Postgres operator | Installation, upgrade | Outbound |
ghcr.io 443 |
Download Docker image for Harbor | Installation, upgrade | Outbound |
gcr.io 443 |
Download Docker image for Kpack | Installation, upgrade | Outbound |
docker.io 443 |
Download Docker image for Kpack | Installation, upgrade | Outbound |
quay.io 443 |
Download inta images | Installation, upgrade | Outbound |
pypi.org 443 |
Download public Python packages when building Dash app images | Runtime | Outbound |
anaconda.org 443 |
Download Conda packages when building Dash app images | Runtime | Outbound |
*.ubuntu.com 443 |
Download APT packages when building Dash app images | Runtime | Outbound |
*.launchpad.net 443 |
Download APT packages when building Dash app images | Runtime | Outbound |
*.<base-domain> |
Access Harbor (registry) when building Dash app images | Runtime | Outbound |
where <base-domain>
is the base domain you chose for Dash Enterprise.
Calculating a more accurate IP requirement: Kubernetes needs one IP address for each service or pod. The Dash Enterprise core system uses 37 services and 97 (Standard) or 140 (Premium) pods. In addition, pods are created when Dash app developers perform certain actions. If you know about your organization’s intended usage of Dash Enterprise, such as how many apps and workspaces developers plan to create, you can calculate a more accurate IP requirement than the rule of thumb above.
Contact our Customer Success team to get started. We’ll ask you:
az aks create
command, which defines the AKS cluster to be provisioned. Note that not all options for this command are supported by Plotly.When we have all the information we need, we’ll send you a zipped folder called your Installation Plan. Your Installation Plan is tailor-made based on your conversation with Customer Success and contains everything you need to install Dash Enterprise for your organization.
Your Installation Plan contains:
provision_infra_az.sh
, containing the commands for provisioning the AKS cluster. The az aks create
command looks similar to the examples below:install_de_az.sh
, containing the commands for installing Dash Enterprise and supporting software.restore.sh
, containing commands for provisioning a new AKS cluster and installing supporting software on it. This script is provided in the event that you back up Dash Enterprise and need to restore your data. It is not used for the installation.Open the infrastructure provisioning script. At the top, edit the following variable values:
APP_ID
: The appId
of the service principal.TENANT_ID
: The tenant
of the service principal.PATH_TO_CERT
: The path to the service principal cert on the bootstrap node. You’ll move this key file to the home directory of the bootstrap node in a later step, so set this to ~/<file-name>
, where <file-name>
is the full name of the key file, including the .pem
extension.Next, open the installation script. At the top, edit the following variable values:
ADMIN_PASSWORD
: The password you want to set for the KOTS Admin Console.About storing and resetting this password: We recommend storing this password in your organization’s password manager, and giving access to any other members of your team who will be managing the Dash Enterprise system (notably performing upgrades and obtaining support bundles). This password is not retrievable with a
kubectl
command. It can be changed in the Admin Console UI by anyone who is able to log in with the current password. If lost, reset it by downloading the KOTS CLI and runningkubectl kots reset-password plotly-system
.
In this step, you’ll create an Azure VM that runs on Ubuntu 20. This VM will serve as your bootstrap node.
To create an Azure VM:
In the Azure Portal, go to the Virtual machines service.
Select Create > Azure virtual machine.
<img>
Under Project details, select the subscription and resource group that you want to use.
Configure the instance details:
<img>
<img>
Add a data disk:
Go to the Disks tab.
<img>
Use the default settings for everything else or adjust them to your preference. We recommend keeping the default inbound port rules, which opens port 22. If you delete this rule, you’ll need to add it back in a later
step in order to be able to SSH into the VM.
Select Review + create.
Select Create. You are prompted to generate the new key pair.
Select Download private key and create resource. The private key is downloaded as a .pem
file.
When the deployment is complete, select Go to resource.
Under Networking, note the Public IP address. You’ll need this to SSH into the VM in a later step.
Select Networking to go to the networking settings. Add an inbound port rule for port 8800 (required to port-forward the KOTS Admin Console) and, if not already present, an inbound port rule for port 22 (required to SSH into the VM).
In the example below, we’ve named our rule for port 8800 “PortForwardAdminConsole”.
<img>
If your organization uses outbound rules, then you’ll also need to add the following rules in the Outbound port rules tab.
Domain | Purpose | I/O |
---|---|---|
aka.ms | Download the Azure CLI | Outbound |
dl.k8s.io | Download kubectl |
Outbound |
github.com | Download Cert Manager | Outbound |
install.istio.io | Download Istio | Outbound |
kots.io | Download the KOTS plug-in | Outbound |
ubuntu.com | apt-get packages from Ubuntu |
Outbound |
launchpad.net | apt-get packages from Debian |
Outbound |
In this step, you’ll move your infrastructure provisioning script, Dash Enterprise install script, and service principal cert to the bootstrap node you’ve prepared. One way to do this is to use secure copy protocol (SCP).
To transfer your infrastructure provisioning script, Dash Enterprise install script, and service principal cert from your workstation to your bootstrap node using SCP:
Move the downloaded SSH private key to your .ssh
directory:
sh
mv /path/to/downloaded/private/key ~/.ssh/
where /path/to/downloaded/private/key
is the path to the current location of the private key.
Ensure you have read-only access to the private key (note this command has no output):
sh
chmod 0400 ~/.ssh/dash-enterprise-bootstrap_key.pem
changing dash-enterprise-bootstrap_key.pem
if your key has a different name.
On your workstation, unzip your Installation Plan if it isn’t already.
sh
scp -i ~/.ssh/dash-enterprise-bootstrap_key.pem path/to/installation/plan/{provision_infra_az,install_de_az}.sh path/to/service/principal/cert azureuser@<bootstrap-ip>:
where path/to/installation/plan
is the path to your Installation Plan folder containing the provision_infra_az.sh
and install_de_az.sh
scripts, path/to/service/principal/cert
is the path to the service principal cert, and <bootstrap-ip>
is the public IP address of your bootstrap node. Change dash-enterprise-bootstrap_key.pem
if your private key has a different name.
To provision the AKS cluster and install Dash Enterprise on it:
SSH into your bootstrap node:
sh
ssh -i ~/.ssh/dash-enterprise-bootstrap_key.pem azureuser@<bootstrap-ip>
where <bootstrap-ip>
is the public IP address of your bootstrap node. Change dash-enterprise-bootstrap_key.pem
if your private key has a different name.
In the home directory of your bootstrap node, run the infrastructure provisioning script:
bash provision_infra_az.sh
The script takes several minutes to complete. Continue when you are returned to the command prompt.
In the home directory of your bootstrap node, run the Dash Enterprise installation script:
bash install_de_az.sh
When you are prompted for the kots
install location by Enter installation path (leave blank for /usr/local/bin)
, press Enter
to accept the default.
When you are prompted to grant write permissions to /usr/local/bin
, press y
(you will not be prompted for a password).
The script takes several minutes to complete. Continue when you see the message Forwarding from 0.0.0.0:8800 -> 3000
(do not exit yet).
If you exit by mistake, restart the port-forward with
kubectl port-forward -n plotly-system svc/kotsadm --address 0.0.0.0 8800:3000
.
Now that your AKS cluster is provisioned and Dash Enterprise is installed on it, you’re ready for configuration. The KOTS Admin Console will take you through uploading your Dash Enterprise license as well as your TLS/SSL certificate and key.
To access the KOTS Admin Console and configure Dash Enterprise:
http://<bootstrap-ip>:8800
, where <bootstrap-ip>
is the public IP address of your bootstrap node.ADMIN_PASSWORD
in Defining Variables in the Scripts; then select Log in. You are prompted to upload your license.Ctrl+C
to disconnect from the Admin Console for now (you can reconnect with kubectl port-forward -n plotly-system svc/kotsadm --address 0.0.0.0 8800:3000
).In this step, you’ll create the DNS entries according to your organization’s best practices.
To create the DNS entries for Dash Enterprise:
sh
kubectl get service -n plotly-system ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' && echo
Name | Type | Value |
---|---|---|
<base-domain> |
A Record | <load-balancer-ip> |
api-<base-domain> |
CNAME | <base-domain> |
ws-<base-domain> |
CNAME | <base-domain> |
git-<base-domain> |
CNAME | <base-domain> |
registry-<base-domain> |
CNAME | <base-domain> |
auth-<base-domain> |
CNAME | <base-domain> |
admin-<base-domain> |
CNAME | <base-domain> |
Your base domain is an A record whose value is the IP of the load balancer that you obtained in step 1. The sub-domains required for Dash Enterprise are CNAMES whose values are your base domain.
https://admin-<your-dash-enterprise-server>
.Continue when the status in the Admin Console is Ready.
<img>
Before you can log in to Dash Enterprise at https://<your-dash-enterprise-server>
, you’ll need to create a Dash Enterprise user in Keycloak. Keycloak is the identity and access management solution for Dash Enterprise.
You’ll be returning to Keycloak when you’re ready to configure authentication for Dash Enterprise. To learn about important settings and choose between different identity provider modes, go to Using Keycloak.
In this step, you’ll retrieve the Keycloak password that is stored as a secret in your cluster and save it according to your organization’s best practices.
To obtain and store the Keycloak password:
sh
kubectl get secret keycloak-secrets -n plotly-system -o jsonpath='{.data.KEYCLOAK_PASSWORD}' | base64 -d && echo
Note about recovering the Keycloak password: If you change this password via the Keycloak interface, it will no longer correspond to what is
stored in your cluster. We recommend keeping it as is so that you can always recover it with thiskubectl get secret
command.
In this step, you’ll log in to Keycloak using the stored credentials and create a new user with the admin
role. The admin
role grants access to the Admin section of the Dash Enterprise App Manager, which you’ll use to configure system limits
in a later step. Learn more about the admin role.
To access Keycloak and create your admin user:
https://auth-<your-dash-enterprise-server>
<img>
Make sure Dash is selected in the realm list in the top left corner.
Select Users > Add User.
admin
role:admin
; then select Add selected. Note that if you intend on deploying Dash apps, you’ll also need the licensed_user
role, and assigning this role consumes a license seat.To log into Dash Enterprise with this user, go to https://<your-dash-enterprise-server>
and enter the credentials that you saved in Keycloak. Dash Enterprise opens to the Portal. Go to the App Manager by selecting Apps > App Manager.
<img>
You can now safely delete the Azure VM that you used as your bootstrap node and delete its SSH private key from your workstation.
In this step, you’ll safeguard Dash Enterprise against usage that would cause the Kubernetes cluster to exceed the resources it can support. Specifically, you’ll add limits to the amount of pods and volumes (PVC) that can exist, temporarily preventing Dash app developers from performing actions that would create more pods and volumes on the cluster when the limit is reached. To do so, you’ll use the System Limits setting in the Admin section of the App Manager. To learn how to calculate and set limits that are appropriate for your cluster, go to Pod and Volume Limits.