Part 4. Sharing Data Between Callbacks | Dash for R Documentation

This is the final chapter of the essential Dash Fundamentals.
The previous chapter covered how to use callbacks
with the dccGraph component. The rest of the Dash
documentation covers other topics like multi-page apps and component
libraries. Just getting started? Make sure to install the necessary
dependencies.

One of the core Dash principles explained in the Getting Started Guide on Callbacks is that Dash Callbacks must never modify variables outside of their scope. It is not safe to modify any global variables. This chapter explains why and provides some alternative patterns for
sharing state between callbacks.

In some apps, you may have multiple callbacks that depend on expensive data
processing tasks like making database queries, running simulations, or downloading data.

Rather than have each callback run the same expensive task,
you can have one callback run the task and then share the results to the other callbacks.

One way to achieve this is by having multiple outputs
for one callback: the expensive task can be done once and immediately used in all the
outputs. For example, if some data needs to be queried from a database and then displayed in
both a graph and a table, then you can have one callback that calculates the data and creates
both the graph and the table outputs.

But sometimes having multiple outputs in one callback isn’t a good solution. For example, suppose
your Dash app allows a user to select a date and a temperature unit (Fahrenheit or Celcius), and
then displays the temperature for that day. You could have one callback that outputs the temperature
by taking both the date and the temperature unit as inputs, but this means that if the user
merely changes from Fahrenheit to Celcius then the weather data would have to be re-downloaded, which
can be time consuming. Instead, it can be more efficient to have two callbacks: one callback that
fetches the weather data, and another callback that outputs the temperature based on the downloaded data.
This way, when only the unit is changed, the data does not have to be downloaded again. This is an
example of sharing a variable, or state, between callbacks.

Dash is Stateless

Dash was designed to be a stateless framework.

Stateless frameworks are more scalable and robust than stateful ones. Most websites that you visit are
running on stateless servers.

They are more scalable because it’s trivial to add more compute power to the application.
In order to scale the application to serve more users or run more computations,
run more “copies” of the app in separate processes.

In production, this can be done by running the app in multiple Docker containers or servers and load balancing between them.

Stateless frameworks are more robust because even if one process fails, other processes can continue
serving requests.
In Dash Enterprise Kubernetes, these containers can run on separate servers or even
separate regions, providing resiliency against server failure.

With a stateless framework, user sessions are not mapped 1-1 with server processes.
Each callback request can be executed on any of the available processes.

Why Global Variables Will Break Your App

Dash is designed to work in multi-user environments where multiple people view the application at the
same time and have independent sessions.

If your app uses and modifies a global variable, then one user’s session could set the variable to some value
which would affect the next user’s session.

Dash is also designed to be able to run with multiple workers so that callbacks can be executed in parallel.

When Dash apps run across multiple workers, their memory
is not shared. This means that if you modify a global
variable in one callback, that modification will not be
applied to the other workers / processes.

Here is a sketch of an app that will not work reliably because the callback modifies a global variable, which is outside of its scope.

library(dash)

df <- data.frame(
  student_id = 1:10,
  score = c(1, 5, 2, 5, 2, 3, 1, 5, 1, 5)
)

dash_app() %>%
  set_layout(
    dccDropdown('num', lapply(1:5, function(x) list(label = x, value = x)), 1),
    "was scored by this many students:",
    div(id = 'output')
  ) %>%
  add_callback(
    output('output', 'children'),
    input('num', 'value'),
    function(value) {
      df <<- subset(df, score == value)
      nrow(df)
    }
  ) %>%
  run_app()

The callback returns the correct output the very first time it is called, but once the global df variable is modified, any subsequent callback
that uses that dataframe is not using the original data anymore.

To improve this app, reassign the filtered dataframe to a new variable inside the callback as shown below, or follow one of the strategies outlined in the next parts of this guide.

library(dash)

df <- data.frame(
  student_id = 1:10,
  score = c(1, 5, 2, 5, 2, 3, 1, 5, 1, 5)
)

dash_app() %>%
  set_layout(
    dccDropdown('score', lapply(1:5, function(x) list(label = x, value = x)), 1),
    "was scored by this many students:",
    div(id = 'output')
  ) %>%
  add_callback(
    output('output', 'children'),
    input('score', 'value'),
    function(value) {
      filtered_df <- subset(df, score == value)
      nrow(filtered_df)
    }
  ) %>%
  run_app()

Storing Shared Data

To share data safely across multiple
processes or servers, we need to store the data somewhere that is accessible to
each of the processes.

There are three places you can store this data:

In the user’s browser session, using dccStore
On the disk (e.g. in a file or database)
In server-side memory (RAM) shared across processes and servers such as a Redis database. Dash Enterprise includes onboard, one-click Redis databases for this purpose.

The following examples illustrate some of these approaches.

Example 1 - Storing Data in the Browser with `dccStore`

To save data in the user’s browser’s session:

The data has to be converted to a string like JSON or base64 encoded binary data for storage
Data that is cached in this way will only be available in the user’s current session.
If you open up a new browser window, the app’s callbacks will always
re-compute the data. The data is only cached between
callbacks within the same session.
This method doesn’t increase the
memory footprint of the app.
There could be a cost in network traffic. If you’re sharing 10MB
of data between callbacks, then that data will be transported over
the network between each callback.
If the network cost is too high, then compute the aggregations
upfront and transport those.
Your app likely won’t be displaying 10MB of data,
it will just be displaying a subset or an aggregation of it.

The example below shows one of the common ways you can leverage dccStore: if processing a dataset takes a long time and different outputs use this dataset, dccStore can be used to store the processed data as an intermediate value that can then be used as an input in multiple callbacks to generate different outputs. This way, the expensive data processing step is only performed once in one callback instead of repeating the same expensive computation multiple times in each callback.

app %>% set_layout(
  dccGraph('graph'),
  html$table(id = 'table'),
  dccDropdown('dropdown'),

  # dccStore stores the intermediate value
  dccStore('intermediate-value')
)

app %>% add_callback(
  output('intermediate-value', 'data'),
  input('dropdown', 'value'),
  function(value) {
    # some expensive data processing step
    cleaned_df <- slow_processing_step(value)
    # Note that the data must be stored as a JSON string
    jsonlite::toJSON(cleaned_df)
  }
)

app %>% add_callback(
  output('graph', 'figure'),
  input('intermediate-value', 'data'),
  function(json_clean_data) {
    df <- jsonlite::fromJSON(json_clean_data)
    figure <- create_figure(df)
    figure
  }
)

app %>% add_callback(
  output('table', 'children'),
  input('intermediate-value', 'data'),
  function(json_clean_data) {
    df <- jsonlite::fromJSON(json_clean_data)
    table <- create_table(df)
    table
  }
)

Notice that the data needs to be serialized into a JSON string before being placed in storage. Also note how the processed data gets stored in dccStore by assigning the data as its output, and then the same data gets used by multiple callbacks by using the same dccStore as an input.

Note about a previous version of this example

This example used to be implemented with a “hidden div”.
We no longer recommend using the hidden div approach, and instead recommend using
dccStore, which stores the data in the user’s browser’s memory instead
of the browser’s DOM and makes the intent more clear.

Example 2 - Computing Aggregations Upfront

Sending the computed data over the network can be expensive if
the data is large. In some cases, serializing this data to JSON
can also be expensive.

In many cases, your app will only display a subset or an aggregation
of the processed data. In these cases, you could precompute
the aggregations in your data processing callback and transport these
aggregations to the remaining callbacks.

Here’s a simple example of how you might transport filtered or aggregated data to multiple callbacks,
again using the same dccStore.

app %>% set_layout(
  dccGraph('graph1'),
  dccGraph('graph2'),
  dccGraph('graph3'),
  dccDropdown('dropdown'),
  dccStore('intermediate-value')
)

app %>% add_callback(
  output('intermediate-value', 'data'),
  input('dropdown', 'value'),
  function(value) {
    cleaned_df <- slow_processing_step(value)

    # a few filter steps that compute the data as it's needed in future callbacks
    df_1 <- subset(cleaned_df, fruit == 'apple')
    df_2 <- subset(cleaned_df, fruit == 'orange')
    df_3 <- subset(cleaned_df, fruit == 'fig')
    datasets <- list(df_1 = df_1, df_2 = df_2, df_3 = df_3)
    jsonlite::toJSON(datasets)
  }
)

app %>% add_callback(
  output('graph1', 'figure'),
  input('intermediate-value', 'data'),
  function(json_clean_data) {
    datasets <- jsonlite::fromJSON(json_clean_data)
    df <- datasets$df_1
    figure <- create_figure1(df)
    figure
  }
)

app %>% add_callback(
  output('graph1', 'figure'),
  input('intermediate-value', 'data'),
  function(json_clean_data) {
    datasets <- jsonlite::fromJSON(json_clean_data)
    df <- datasets$df_2
    figure <- create_figure2(df)
    figure
  }
)

app %>% add_callback(
  output('graph1', 'figure'),
  input('intermediate-value', 'data'),
  function(json_clean_data) {
    datasets <- jsonlite::fromJSON(json_clean_data)
    df <- datasets$df_3
    figure <- create_figure3(df)
    figure
  }
)

Example 3 - Caching and Signaling

This example:
- Uses the memoise package for storing “global variables” in an in-memory cache on the server.
This data is accessed through a function (global_store()), the output of which is cached and keyed by its input arguments.
- Uses the dccStore solution to send a signal to the other
callbacks when the expensive computation is complete.
- Note that instead of an in-memory cache, you could also save this to the file
system. See
the documentation for the cache argument in memoise()
for more details.
- This “signaling” is performant because it allows the expensive
computation to only take up one process and be performed once.
Without this type of signaling, each callback could end up
computing the expensive computation in parallel,
locking four processes instead of one.

Another benefit of this approach is that future sessions can
use the pre-computed value.
This will work well for apps that have a small number of inputs.

Here’s what this example looks like.

Example of a Dash App that uses Caching

Here’s what this example looks like in code:

library(dash)
library(memoise)

app <- dash_app()

app %>% add_stylesheet(list(
  'https://codepen.io/chriddyp/pen/bWLwgP.css',
  'https://codepen.io/chriddyp/pen/brPBPO.css'
))

N <- 100
df <- data.frame(
  category = c(rep("apples", 5*N), rep("oranges", 10*N), rep("figs", 20*N), rep("pineapples", 15*N))
)
df$x <- rnorm(nrow(df))
df$y <- rnorm(nrow(df))

app %>% set_layout(
  dccDropdown('dropdown', lapply(unique(df$category), function(x) list(label=x, value=x)), "apples"),
  div(
    className = "row",
    div(dccGraph('graph-1'), className="six columns"),
    div(dccGraph('graph-2'), className="six columns")
  ),
  div(
    className = "row",
    div(dccGraph('graph-3'), className="six columns"),
    div(dccGraph('graph-4'), className="six columns")
  ),

  # signal value to trigger callbacks
  dccStore('signal')
)

# perform expensive computations in this "global store"
# these computations are cached in a globally available
# cache that's available across processes for all time.
global_store <- memoise(
  function(value) {
    print(paste('Computing value with', value))
    # simulate expensive query
    Sys.sleep(3)
    subset(df, category == value)
  }
)

generate_figure <- function(value, figure) {
  filtered_df <- global_store(value)
  figure$data[[1]]$x <- filtered_df$x
  figure$data[[1]]$y <- filtered_df$y
  figure$layout <- list(margin = list(l =  20, r = 10, b = 20, t = 10))
  figure
}

app %>% add_callback(
  output('signal', 'data'),
  input('dropdown', 'value'),
  function(value) {
    # compute value and send a signal when done
    global_store(value)
    value
  }
)

app %>% add_callback(
  output('graph-1', 'figure'),
  input('signal', 'data'),
  function(value) {
    data <- list(
      list(
        type = "scatter",
        mode = "markers",
        marker = list(
          opacity = 0.5,
          size = 14,
          line = list(border = "thin darkgrey solid")
        )
      )
    )
    # generate_figure gets data from `global_store`.
    # the data in `global_store` has already been computed
    # by the `compute_value` callback and the result is stored
    # in the global cache
    generate_figure(value, list(data = data))
  }
)

app %>% add_callback(
  output('graph-2', 'figure'),
  input('signal', 'data'),
  function(value) {
    data <- list(
      list(
        type = "scatter",
        mode = "lines",
        line = list(
          shape = "spline",
          width = 0.5
        )
      )
    )
    generate_figure(value, list(data = data))
  }
)

app %>% add_callback(
  output('graph-3', 'figure'),
  input('signal', 'data'),
  function(value) {
    data <- list(
      list(
        type = "histogram2d"
      )
    )
    generate_figure(value, list(data = data))
  }
)

app %>% add_callback(
  output('graph-4', 'figure'),
  input('signal', 'data'),
  function(value) {
    data <- list(
      list(
        type = "histogram2dcontour"
      )
    )
    generate_figure(value, list(data = data))
  }
)

app %>% run_app()

Some things to note:

We’ve simulated an expensive process by using a system sleep of 3 seconds.
When the app loads, it takes three seconds to render all four graphs.
The initial computation only blocks one process.
Once the computation is complete, the signal is sent and four callbacks
are executed in parallel to render the graphs.
Each of these callbacks retrieves the data from the
“global server-side store”: the in-memory cache or filesystem cache.
As we are running the server with multiple processes, we set threaded to False. A Flask server can’t be be both multi-process and multi-threaded.
Selecting a value in the dropdown will take less than three seconds
if it has already been selected in the past.
This is because the value is being pulled from the cache.
Similarly, reloading the page or opening the app in a new window
is also fast because the initial state and the initial expensive
computation has already been computed.

Example 4 - User-Based Session Data on the Server

The previous example cached computations in a way that was accessible for all users.

Sometimes you may want to keep the data isolated to user sessions:
one user’s derived data shouldn’t update the next user’s derived data.
One way to do this is to save the data in a dccStore,
as demonstrated in the first example.

Another way to do this is to save the data in a cache along
with a session ID and then reference the data
using that session ID. Because data is saved on the server
instead of transported over the network, this method is generally faster than the
dccStore method.

This method was originally discussed in a
Dash Community Forum thread.

This example:
- Caches data using memoise(cache = cachem::cache_disk("df_user_cache")) into the a df_user_cache directory. You can also save to an in-memory cache or database such as Redis instead.
- Serializes the data as JSON.

Creates unique session IDs for each session and stores it as the data
of dccStore on every page load. This means that every user
session has unique data in the dccStore on their page.

Note: As with all examples that send data to the client, be aware
that these sessions aren’t necessarily secure or encrypted.
These session IDs may be vulnerable to
Session Fixation
style attacks.

Here’s what this example looks like in code:

library(dash)
library(memoise)

app <- dash_app()

app %>% add_stylesheet(list(
  'https://codepen.io/chriddyp/pen/bWLwgP.css',
  'https://codepen.io/chriddyp/pen/brPBPO.css'
))

app %>% set_layout(
  function() {
    session_id <- uuid::UUIDgenerate()

    div(
      dccStore(id = 'session-id', data = session_id),
      button('Get data', id = 'get-data-button'),
      div(id = 'output-1'),
      div(id = 'output-2')
    )
  }
)

get_dataframe <- function(session_id) {
  query_and_serialize_data <- memoise(
    cache = cachem::cache_disk("df_user_cache"),

    function(session_id) {
      # expensive session-specific data processing step goes here
      # simulate a session-unique data processing step by generating
      # data that is dependent on time
      now <- Sys.time()
      df <- data.frame(
        time = c(
          as.character(now - 15),
          as.character(now - 10),
          as.character(now - 5),
          as.character(now)
        ),
        values = c('a', 'b', 'a', 'c')
      )

      # simulate expensive query
      Sys.sleep(3)
      jsonlite::toJSON(df)
    }
  )

  jsonlite::fromJSON(query_and_serialize_data(session_id))
}

app %>% add_callback(
  output('output-1', 'children'),
  list(
    input('get-data-button', 'n_clicks'),
    input('session-id', 'data')
  ),
  function(value, session_id) {
    df <- get_dataframe(session_id)
    div(
      paste('Output 1 - Button has been clicked', value, 'times'),
      simple_table(df)
    )
  }
)

app %>% add_callback(
  output('output-2', 'children'),
  list(
    input('get-data-button', 'n_clicks'),
    input('session-id', 'data')
  ),
  function(value, session_id) {
    df <- get_dataframe(session_id)
    div(
      paste('Output 2 - Button has been clicked', value, 'times'),
      simple_table(df)
    )
  }
)

app %>% run_app()

Example of a Dash App that uses User Session Caching

There are three things to notice in this example:

The timestamps of the dataframe don’t update when we retrieve
the data. This data is cached as part of the user’s session.
Retrieving the data initially takes three seconds but successive queries
are instant, as the data has been cached.
The second session displays different data than the first session:
the data that is shared between callbacks is isolated to individual
user sessions.

Questions? Discuss these examples on the
Dash Community Forum.

Sharing Data Between Callbacks

Why Share State?