It’s often more efficient to include a copy of your dataset alongside your app rather than to have your app code download it on-the-fly.
When datasets are large, it isn’t always possible to store them in memory or version control them with Git.
Dash Enterprise’s persistent filesystem feature provides persistent, high-performance storage for data files.
The persistent filesystem allows you to store large datasets in a filesystem that is shared
between your deployed app (including every process declared in your Procfile) and its workspace, if one exists.
The persistent filesystem is a good fit for:
- Large datasets that are too large to store in Git and/or in memory. In particular, HDF5 or Parquet files analyzed with Vaex (a file-based, out-of-memory dataframe library).
- Dynamic datasets that are updated by periodic background processes and read by your app.
- Datasets that you want to manually update in your workspace without redeploying your app.
- Cached files that don’t change across deploys.
The persistent filesystem has a storage limit of 25 GB by default, but your administrator may have increased it.
How the persistent filesystem works internally
Enabling the persistent filesystem before creating a workspace adds a persistent volume (PV) to the Kubernetes cluster that Dash Enterprise runs in. (Workspaces also require a PV, so no additional PV is created if enabling the persistent filesystem when a workspace exists).
The PV uses the default StorageClass for the cluster. Or, if your cluster administrator has configured a custom StorageClass, it uses that one instead.
By default, the persistent filesystem is disabled.
To enable the persistent filesystem for an app:
Find and select the app in the App Manager.
Go to the Persistent Filesystem tab.
Select Edit Persistent Filesystem.
<img>
Select Enabled and then Save.
<img>
The persistent filesystem becomes available to your app (and its workspace, if one exists).
It is not available to other apps and workspaces on Dash Enterprise.
The persistent filesystem is located at ../mount relative to your app folder.
You can manage the contents of the persistent filesystem manually within the app’s workspace,
dynamically on deployment, or in a background task in your app code.
Workspaces provide a handy UI for interacting with the persistent filesystem folder.
To add files to the persistent filesystem from the workspace UI:
Select the persistent filesystem from the menu:
<img>
Drag files from your device to the file explorer (you’ll see a README.md file already there).
You can also add files by going to File > Upload Files and selecting files to upload.
To add files to the persistent filesystem from the workspace terminal, you can run commands like curl
that retrieve data from a URL and save it to a CSV file in the ../mount folder.
curl -o ../mount/1962_2006_walmart_store_openings.csv <a href="https://raw.githubusercontent.com/plotly/datasets/master/1962_2006_walmart_store_openings.csv">https://raw.githubusercontent.com/plotly/datasets/master/1962_2006_walmart_store_openings.csv</a>
You can also open a Python or IPython console within the workspace and write code that downloads data and writes it to a file:
$ python
>>> import pandas as pd
>>> pd.DataFrame({'a': [1, 2, 3], 'b': [3, 1, 2]}).to_csv('../mount/data.csv')
Or run a script or Jupyter notebook within the workspace.
You can configure your app to update the contents of the persistent filesystem each time you deploy your app.
On App Boot
In your app code, your app can write or read files to the ../mount folder on boot:
app.py
from dash import Dash, html
import pandas as pd
# Write and/or read files from mount on app start
pd.DataFrame({'a': [1, 2, 3], 'b': [3, 1, 2]}).to_csv('../mount/data.csv')
df = pd.read_csv('../mount/data.csv')
app = Dash()
# ...
In a Predeploy Script
Alternatively, you can run a Bash script before the web command is run by creating
a predeploy script that contains commands to fetch the data
and reference the script in a project.toml file.
In this example, we add the command we ran in the terminal to a file fetchdata.sh and define that file
as a predeploy script in a project.toml file. The system runs fetchdata.sh as a Bash script
before the app is deployed.
fetchdata.sh
curl -o ../mount/gapminderDataFiveYear.csv <a href="https://raw.githubusercontent.com/plotly/datasets/master/gapminderDataFiveYear.csv">https://raw.githubusercontent.com/plotly/datasets/master/gapminderDataFiveYear.csv</a>
project.toml
[scripts]
predeploy = "fetchdata.sh"
Update files while the app is running if your datasets change over time.
To add or update a file when an app is running:
Write a script that creates or updates a file in the persistent filesystem.
Add the script in your Procfile.
In this example, we write data periodically to a CSV file in the persistent filesystem in task.py.
The app, app.py, reads the data when an app user selects the Get Data button.
The Procfile has a line worker: python task.py that runs the task.py script in the background.
app.py
from dash import Dash, html, dcc, Input, Output, callback
import plotly.express as px
import pandas as pd
from pathlib import Path
app = Dash()
server = app.server
app.layout = html.Div(
children=[
html.Button("Get Data", id="get-data", n_clicks=0),
dcc.Graph(
id="graph",
)
]
)
@callback(
Output("graph", "figure"),
Input("get-data", "n_clicks"),
)
def update_output(n_clicks):
data_path = Path.cwd().parent / 'mount' / 'data.csv'
data = pd.read_csv(data_path)
figure = px.scatter(data, x="x", y="y")
return figure
if __name__ == "__main__":
app.run(debug=True)
task.py
## Writes random numbers to data.csv every 2 seconds
import random
import time
from pathlib import Path
data_path = Path.cwd().parent / 'mount' / 'data.csv'
with open(data_path, 'w+') as f:
f.write(('x,y\n'))
while True:
with open(data_path, 'a') as f:
x = (random.randint(0,9))
y = (random.randint(0,100))
f.write((str(x) + ',' + str(y) + '\n'))
time.sleep(2)
Procfile
web: gunicorn app:server --workers 4
worker: python task.py
Note that your app code will need to read the file on-the-fly in callback functions or a layout function
when it needs to use the data; if you load the data in advance on app boot, the data will only be read
into memory when the app is deployed and will not read the updated file until the app is restarted or redeployed.
To access a file in the ../mount folder within your app, you can use any of Python’s built-in modules and functions for handling files.
These are both valid ways to read files from the persistent filesystem:
import pandas as pd
df = pd.read_csv('../mount/gapminderDataFiveYear.csv')
or using pathlib to construct the path of the file:
from pathlib import Path
import pandas as pd
gapminder_path = Path.cwd().parent / 'mount' / 'gapminderDataFiveYear.csv'
df = pd.read_csv(gapminder_path)
Unlike your regular app files, the contents of the persistent filesystem are not meant to be version controlled.
This means that they should not be part of your app project folder. To run your app locally,
mimic the folder structure of Dash Enterprise by creating a mount folder
that is a sibling to your app project folder.
Without a ../mount folder, your project code might look like this:
└── my-project
├── Procfile
├── app.py
├── data.csv
└── requirements.txt
where the root of my-project is where you run de deploy or direct git commands (the entire folder is version controlled).
To use the ../mount folder, move the contents of my-project into a folder called app/ that
is on the same level as the new mount/ folder:
└── my-project
├── app
│ ├── Procfile
│ ├── app.py
│ └── requirements.txt
└── mount
└── data.csv
With this structure, you run de deploy or direct git commands at app/, ensuring that the app files
remain version controlled but mount/ is not.
Alternatively, you can keep the my-project/ folder name and create a new parent folder like
my-parent-project:
└── my-parent-project
├── my-project
| ├── Procfile
| ├── app.py
| └── requirements.txt
└── mount
└── data.csv
With this structure, you run de deploy or direct git commands at my-project/, ensuring that the app files
remain version controlled but mount/ is not.
The folder containing the app code (app/ in the first example, my-project/ in the second example)
should not be referenced in your code, so you can name it anything.
The mount/ folder’s name is referenced in code, so it needs to be called mount.
Its location is also referenced (app.py accesses it at ../mount), so it always needs to a sibling of your app folder.
Important: When you disable a persistent filesystem, all files stored in it are removed and are not recoverable.
To disable the persistent filesystem:
Find and select the app in the App Manager.
Go to the Persistent Filesystem tab.
Select Edit Persistent Filesystem.
<img>
Select Disabled and then Save.
<img>