Install Dash Enterprise on Amazon EKS (Multi-node)

This guide can help you if you are a new Dash Enterprise customer looking to start with a Dash Enterprise 5 installation, or if you are upgrading from Dash Enterprise 4.X.

For Dash Enterprise 4 installation instructions, see the Dash Enterprise 4 version.

About the Installation

Dash Enterprise 5 runs on Kubernetes, an open-source system that automates application lifecycles. Several managed services allow you to get started with Kubernetes-based software quickly. In this guide, you’ll learn how to install Dash Enterprise on Amazon Elastic Kubernetes Service (Amazon EKS), available with an Amazon Web Services (AWS) account.

In Amazon EKS, you work with an EKS cluster, which consists of multiple worker machines that are also called nodes. You host your EKS cluster inside an Amazon Virtual Private Cloud (Amazon VPC), a virtual network dedicated to your AWS account.

Installing Dash Enterprise is an automated process. You use a bootstrap node to run Plotly-provided scripts that provision the infrastructure and install Dash Enterprise on it. A bootstrap node is a virtual machine (VM) whose only purpose is to run the scripts. After Dash Enterprise is installed, you can decommission it. Using fresh VMs is the best practice because the scripts are unlikely to run into errors caused by other installed software. This guide describes how to use Amazon Elastic Compute Cloud (EC2) to provision the bootstrap node (please reach out to our Customer Success team if you’re unable to use EC2).

As part of the automated infrastructure provisioning, the EKS cluster is created with an eksctl create cluster command. eksctl is an official command line tool that automates a lot of EKS cluster management tasks. A VPC is created if you don’t already have one, and an AWS network load balancer is created as the main point of entry for all traffic directed towards Dash Enterprise.

Remember that these resources count towards your AWS quotas. Review your AWS account billing and quotas if necessary.

Amazon EKS spreads out the cluster nodes across multiple availability zones (AZs). Note that the Dash Enterprise core system isn’t currently configured for high availability; however, as long as the core system is available, Dash app developers can take advantage of features like app replicas to increase the availability of deployed apps. High availability for the core system may be supported in the future.

You’ll be installing Dash Enterprise as the single tenant on the EKS cluster—that is, no other software is installed on the cluster (except mandatory supporting software). Single-tenancy is well-suited for Dash Enterprise because it is a complex platform: Dash Enterprise interacts with the Kubernetes API to organize resources on the fly when developers perform tasks like deploying Dash apps and creating databases. Multi-tenancy is not currently supported.

Plotly uses Replicated to package and deliver Dash Enterprise. You’ll be interacting with the KOTS Admin Console, part of the Replicated toolset, in the configuration step of this installation. After the installation, you’ll continue to use the KOTS Admin Console for system administration such as performing Dash Enterprise upgrades.

Prerequisites

Here’s what you’ll need before you can start your Dash Enterprise installation:

• A workstation with internet access and a web browser.
• A wildcard or multi-domain TLS/SSL full certificate chain and unencrypted private key obtained from a globally trusted certificate authority (CA). The full certificate chain must be a .pem file in the following format:
  txt -----BEGIN CERTIFICATE----- <Your> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <Your> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <Your> -----END CERTIFICATE-----

Self-signed certificates, internally signed certificates, and using multiple certificates are not supported. If you obtained your certificate as multiple files, you need to combine them into a single .pem file. You can do this with cat server.pem intermediate.pem trustedroot.pem > fullchain.pem on Linux or copy server.pem+intermediate.pem+trustedroot.pem fullchain.pem on Windows, replacing the file names if yours are different.

You’ll upload the full certificate chain and unencrypted private key during the configuration, and they will be used to terminate TLS/SSL. If you need to preview the required DNS entries, go to Creating DNS Entries, but note that you’ll only be able to create them after the installation. * The following AWS resources:
* An AWS account.
* An IAM user inside that account whose credentials you’ll use to provision the EKS cluster. Learn how to create a new IAM user. We recommend following the AWS best practice of not using a root user for this task. The IAM user you choose can be one that you consider a service account.
* Active access keys for that IAM user in order to interact with AWS from the AWS Command Line Interface (AWS CLI). Learn how to generate a key pair.
* Permissions for that IAM user:

json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor5", "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::*:role/AmazonEKS_EBS_CSI_DriverRole" }, { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "cloudwatch:*", "kms:DescribeKey", "cloudformation:*", "ec2:*", "elasticloadbalancing:*", "autoscaling:*", "logs:PutRetentionPolicy", "eks:*", "kms:CreateGrant" ], "Resource": "*" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*", "Condition": { "StringEquals": { "iam:AWSServiceName": [ "autoscaling.amazonaws.com", "ec2scheduled.amazonaws.com", "elasticloadbalancing.amazonaws.com", "spot.amazonaws.com", "spotfleet.amazonaws.com", "transitgateway.amazonaws.com" ] } } }, { "Sid": "VisualEditor2", "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*", "Condition": { "StringEquals": { "iam:AWSServiceName": [ "eks.amazonaws.com", "eks-nodegroup.amazonaws.com", "eks-fargate.amazonaws.com" ] } } }, { "Sid": "VisualEditor3", "Effect": "Allow", "Action": "iam:GetRole", "Resource": "arn:aws:iam::<your-aws-account-id>:role/*" }, { "Sid": "VisualEditor4", "Effect": "Allow", "Action": [ "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile", "iam:GetRole", "iam:GetInstanceProfile", "iam:TagRole", "iam:RemoveRoleFromInstanceProfile", "iam:CreateRole", "iam:DeleteRole", "iam:AttachRolePolicy", "iam:PutRolePolicy", "ssm:GetParameters", "iam:ListInstanceProfiles", "ssm:GetParameter", "iam:AddRoleToInstanceProfile", "iam:CreateOpenIDConnectProvider", "iam:ListInstanceProfilesForRole", "iam:PassRole", "iam:DetachRolePolicy", "iam:ListAttachedRolePolicies", "iam:DeleteRolePolicy", "iam:GetOpenIDConnectProvider", "iam:DeleteOpenIDConnectProvider", "iam:TagOpenIDConnectProvider", "iam:GetRolePolicy" ], "Resource": [ "arn:aws:iam::<your-aws-account-id>:oidc-provider/*", "arn:aws:iam::<your-aws-account-id>:instance-profile/eksctl-*", "arn:aws:iam::<your-aws-account-id>:role/eksctl-*", "arn:aws:iam::<your-aws-account-id>:role/aws-service-role/eks-nodegroup.amazonaws.com/AWSServiceRoleForAmazonEKSNodegroup", "arn:aws:iam::<your-aws-account-id>:role/eksctl-managed-*", "arn:aws:ssm:*:<your-aws-account-id>:parameter/aws/*", "arn:aws:ssm:*::parameter/aws/*" ] } ] }
replacing <your-aws-account-id> with your AWS account ID. You can obtain this ID with aws sts get-caller-identity --query Account.

• (If specifying an existing VPC) Your VPC’s security groups adhere to these outbound rules at a minimum:

where <base-domain> is the base domain you chose for Dash Enterprise.

• (If specifying an existing VPC) You have sufficient IP addresses available for Kubernetes to use internally. As a rule of thumb, we recommend 60 IP addresses times the number of nodes in your cluster. This corresponds to 240 IP addresses if your organization purchased the Standard offering of Dash Enterprise, or 600 IP addresses for Premium.

Calculating a more accurate IP requirement: Kubernetes needs one IP address for each service or pod. The Dash Enterprise core system uses 35 services and 79 (Standard) or 105 (Premium) pods. In addition, pods are created when Dash app developers perform certain actions. If you know about your organization’s intended usage of Dash Enterprise, such as how many apps and workspaces developers plan to create, you can calculate a more accurate IP requirement than the rule of thumb above.

• (If specifying an existing VPC) Your VPC contains a minimum of two subnets that are in different availability zones. You can specify up to six subnets (three public, three private) for Dash Enterprise to use.

• (If specifying an existing VPC with public subnets) Make sure that Auto-assign public IPv4 address is set to Yes on each subnet.

• (If specifying an existing VPC) The public subnets that you want to use are tagged with:

• (If specifying an existing VPC) The private subnets that you want to use are tagged with:

Preparing Your Installation

Contact our Customer Success team to get started. We’ll ask you:

• The base domain you want for your Dash Enterprise instance. It must be a fully qualified domain name (FQDN).
• Whether your environment has any requirements (if your environment requires advanced configuration, an additional Support and Services Engagement may be required).
• How your organization plans to use Dash Enterprise and whether you need changes to the default eksctl create cluster command, which defines the EKS cluster to be provisioned. Note that not all options for this command are supported by Plotly.
• The AWS region you want for the EKS cluster.
• The name you want for the EKS cluster.
• The scheme you want for your load balancer (internal or internet-facing).

Obtaining Your Installation Plan

When we have all the information we need, we’ll send you a zipped folder called your Installation Plan. Your Installation Plan is tailor-made based on your conversation with Customer Success and contains everything you need to install Dash Enterprise for your organization.

Your Installation Plan contains:

• Your infrastructure provisioning script, provision_infra_aws.sh, which provisions the EKS cluster. The eksctl create cluster command looks similar to the examples below:

• Your installation script, install_de_aws.sh, containing the commands for installing Dash Enterprise and supporting software.
• Your Dash Enterprise license file.
• Your restore script, restore.sh, containing commands for provisioning a new EKS cluster and installing supporting software on it. This script is provided in the event that you back up Dash Enterprise and need to restore your data. It is not used for the installation.

Defining Variables in the Scripts

Open the infrastructure provisioning script. At the top, edit the following variable values:

• AWS_ACCESS_KEY_ID: Your IAM user’s access key.
• AWS_SECRET_ACCESS_KEY: Your IAM user’s secret access key.
• AWS_ACCOUNT: Your AWS account ID. You can obtain this ID with aws sts get-caller-identity --query Account.

Next, open the installation script. At the top, edit the following variable values:

• ADMIN_PASSWORD: The password you want to set for the KOTS Admin Console.

About storing and resetting this password: We recommend storing this password in your organization’s password manager, and giving access to any other members of your team who will be managing the Dash Enterprise system (notably performing upgrades and obtaining support bundles). This password is not retrievable with a kubectl command. It can be changed in the Admin Console UI by anyone who is able to log in with the current password. If lost, reset it by downloading the KOTS CLI and running kubectl kots reset-password plotly-system.

Preparing Your Bootstrap Node

In this step, you’ll use Amazon EC2 to provision a virtual machine (called an EC2 instance in AWS) that runs on Ubuntu 20. This EC2 instance will serve as your bootstrap node.

To create an EC2 instance:

1. In the AWS Management Console, go to the EC2 service.
2. Select Launch instance.

<img>

3. In Name, enter a name for your EC2 instance, such as dash-enterprise-bootstrap.
4. Configure the Application and OS Images:
5. Under Quick Start, select Ubuntu.
6. For Amazon Machine Image (AMI), select Ubuntu Server 20.04 LTS (HVM), SSD Volume Type.
7. For Architecture, use the default 64-bit (x86).

<img>

5. For Instance type, use the default t2.micro.

<img>

6. Create a new SSH key pair that you’ll use to SSH into this EC2 instance:

7. In Key pair (login), select Create new key pair.

8. In Key pair name, enter a name for your key pair. We recommend dash-enterprise-bootstrap_key to more easily follow commands that reference the private key later in this guide.
9. Use the default RSA key pair type and .pem format.

10. Select Create key pair.

11. Configure the network settings:

12. In Network settings, select Edit.

13. Select the VPC and Subnet that you want to use.
14. Set Auto-assign public IP to Enable.
15. For Firewall (security group), select Create security group.
16. Enter a Security group name and Description.
17. In Inbound security groups rules, add a security group rule for port 8800 (required to port-forward the KOTS Admin Console) and, if not already present, an inbound port rule for port 22 (required to SSH into the EC2 instance).
    In the example below, we’ve given our rule for port 8800 the description “PortForwardAdminConsole”.

<img>

8. Use the default settings for everything else or adjust them to your preference.
9. Select Launch instance.
10. Once the EC2 instance is ready, go to its summary.
11. Copy the Public IPv4 DNS. You’ll need this to SSH into the EC2 instance in a later step.
12. If your organization uses outbound rules, go to your EC2 instance’s security group and add the following rules:

Moving Files to Your Bootstrap Node

In this step, you’ll move your infrastructure provisioning script and Dash Enterprise install script to the bootstrap node you’ve prepared. One way to do this is to use secure copy protocol (SCP).

To transfer your infrastructure provisioning script and Dash Enterprise install script from your workstation to your bootstrap node using SCP:

1. Move the downloaded SSH private key to your .ssh directory:
   sh mv /path/to/downloaded/private/key ~/.ssh/
   where /path/to/downloaded/private/key is the path to the current location of the private key.

2. Ensure you have read-only access to the private key (note this command has no output):
   sh chmod 0400 ~/.ssh/dash-enterprise-bootstrap_key.pem
   changing dash-enterprise-bootstrap_key.pem if your key has a different name.

3. On your workstation, unzip your Installation Plan if it isn’t already.

4. Transfer your infrastructure provisioning script and Dash Enterprise install script to your bootstrap node’s home directory:

sh scp -i ~/.ssh/dash-enterprise-bootstrap_key.pem path/to/installation/plan/{provision_infra_aws,install_de_aws}.sh ubuntu@<bootstrap-dns>:~
where path/to/installation/plan is the path to your Installation Plan folder containing the provision_infra_aws.sh and install_de_aws.sh scripts and <bootstrap-dns> is the Public IPv4 DNS of your bootstrap node. Change dash-enterprise-bootstrap_key.pem if your private key has a different name.

Provisioning and Installation

To provision the EKS cluster and install Dash Enterprise on it:

1. SSH into your bootstrap node:
   sh ssh -i ~/.ssh/dash-enterprise-bootstrap_key.pem ubuntu@<bootstrap-dns>
   where <bootstrap-dns> is the Public IPv4 DNS of your bootstrap node. Change dash-enterprise-bootstrap_key.pem if your private key has a different name.

2. In the home directory of your bootstrap node, run the infrastructure provisioning script:
   bash provision_infra_aws.sh

The script takes several minutes to complete. Continue when you are returned to the command prompt.

3. In the home directory of your bootstrap node, run the Dash Enterprise installation script:
   bash install_de_aws.sh

4. When you are prompted for the kots install location by Enter installation path (leave blank for /usr/local/bin), press Enter to accept the default.

5. When you are prompted to grant write permissions to /usr/local/bin, press y (you will not be prompted for a password).

The script takes several minutes to complete. Continue when you see the message Forwarding from 0.0.0.0:8800 -> 3000 (do not exit yet).

If you exit by mistake, restart the port-forward with kubectl port-forward -n plotly-system svc/kotsadm --address 0.0.0.0 8800:3000.

Configuration

Now that your EKS cluster is provisioned and Dash Enterprise is installed on it, you’re ready for configuration. The KOTS Admin Console will take you through uploading your Dash Enterprise license as well as your TLS/SSL certificate and key.

To access the KOTS Admin Console and configure Dash Enterprise:

1. On your workstation, go to http://<bootstrap-dns>:8800, where <bootstrap-dns> is the Public IPv4 DNS of your bootstrap node.
2. Enter the password that you set for ADMIN_PASSWORD in Defining Variables in the Scripts; then select Log in. You are prompted to upload your license.
3. Drag or browse to the license file in your Installation Plan; then select Upload license. The Admin Console opens to the Configure Dash Enterprise page.
4. Upload your TLS/SSL certificate and key.
5. Select Continue. The Admin Console runs preflight checks, which can take up to a few minutes.
6. Wait for the preflight checks to complete. If the results are all successful, select Continue. If you encounter an error, contact Customer Success.
   The Admin Console opens to the dashboard, where the status of the system is displayed. The system is not ready until DNS entries are created, so it is normal for the status to display “Missing” or “Unavailable”.
7. On your bootstrap node, press Ctrl+C to disconnect from the Admin Console for now (you can reconnect with kubectl port-forward -n plotly-system svc/kotsadm --address 0.0.0.0 8800:3000).

Creating DNS Entries

In this step, you’ll create the DNS entries according to your organization’s best practices.

To create the DNS entries for Dash Enterprise:

1. On your bootstrap node, get the load balancer hostname:

sh kubectl get service ingress-nginx-controller -n plotly-system -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' && echo

The output looks like ad52abddab74e44c98bcf129f5d0a230-d7906cbe482b8075.elb.us-east-2.amazonaws.com

2. Create the DNS entries according to the table below.

   Name	Type	Value
   <base-domain>	CNAME	<load-balancer-hostname>
   api-<base-domain>	CNAME	<base-domain>
   ws-<base-domain>	CNAME	<base-domain>
   git-<base-domain>	CNAME	<base-domain>
   registry-<base-domain>	CNAME	<base-domain>
   auth-<base-domain>	CNAME	<base-domain>
   admin-<base-domain>	CNAME	<base-domain>

Your base domain is a CNAME whose value is the hostname of the load balancer that you obtained in step 1. If a . is not supplied, add it after the hostname. For example, ad52abddab74e44c98bcf129f5d0a230-d7906cbe482b8075.elb.us-east-2.amazonaws.com.

The sub-domains required for Dash Enterprise are CNAMES whose values are your base domain.

3. Once the DNS entries are created and propagated, go to the Admin Console using its sub-domain: https://admin-<your-dash-enterprise-server>.

Continue when the status in the Admin Console is Ready.

<img>

Accessing Dash Enterprise

Before you can log in to Dash Enterprise at https://<your-dash-enterprise-server>, you’ll need to create a Dash Enterprise user in Keycloak. Keycloak is the identity and access management solution for Dash Enterprise.

You’ll be returning to Keycloak when you’re ready to configure authentication for Dash Enterprise. To learn about important settings and choose between different identity provider modes, go to Using Keycloak.

Obtaining and Storing the Keycloak Password

In this step, you’ll retrieve the Keycloak password that is stored as a secret in your cluster and save it according to your organization’s best practices.

To obtain and store the Keycloak password:

1. On your bootstrap node, retrieve the password to Keycloak (this displays the password in plain text):

sh kubectl get secret keycloak-secrets -n plotly-system -o jsonpath='{.data.KEYCLOAK_PASSWORD}' | base64 -d && echo

Note about recovering the Keycloak password: If you change this password via the Keycloak interface, it will no longer correspond to what is
stored in your cluster. We recommend keeping it as is so that you can always recover it with this kubectl get secret command.

2. Copy the password.
3. Add the password to your organization’s password manager or other secure storage, along with the username admin. You can share these credentials with other members in your organization who need to access Keycloak.

Creating Your Dash Enterprise Admin User

In this step, you’ll log in to Keycloak using the stored credentials and create a new user with the admin role. The admin role grants access to the Admin section of the Dash Enterprise App Manager, which you’ll use to configure system limits
in a later step. Learn more about the admin role.

To access Keycloak and create your admin user:

1. Go to https://auth-<your-dash-enterprise-server>
2. Select Administration Console.
3. Enter the Keycloak credentials that you obtained and stored.

<img>

4. Select Sign In.

5. Make sure Dash is selected in the realm list in the top left corner.

   

6. Select Users > Add User.

7. In Username, enter the username you want to use.
8. Select Save. Additional settings become available.
9. Go to Credentials.
10. In Password and Password Confirmation, enter the password you want to use.
11. Select Set Password; then set password again to confirm.
12. Assign the admin role:
    1. Go to Role Mappings.
    2. In Client Roles, select dash.
    3. In Available Roles, select admin; then select Add selected. Note that if you intend on deploying Dash apps, you’ll also need the licensed_user role, and assigning this role consumes a license seat.

To log into Dash Enterprise with this user, go to https://<your-dash-enterprise-server> and enter the credentials that you saved in Keycloak. Dash Enterprise opens to the Portal. Go to the App Manager by selecting Apps > App Manager.

<img>

You can now safely terminate the EC2 instance that you used as your bootstrap node and delete its SSH private key from your workstation.

Setting System Limits

In this step, you’ll safeguard Dash Enterprise against usage that would cause the Kubernetes cluster to exceed the resources it can support. Specifically, you’ll add limits to the amount of pods and volumes (PVC) that can exist, temporarily preventing Dash app developers from performing actions that would create more pods and volumes on the cluster when the limit is reached. To do so, you’ll use the System Limits setting in the Admin section of the App Manager. To learn how to calculate and set limits that are appropriate for your cluster, go to Pod and Volume Limits.