In Databricks notebooks, using the display() function triggers an action that forces Spark to execute the code and produce a result. However, Spark operations are generally divided into transformations and actions.
Transformations create a new dataset from an existing one and are lazy, meaning they are not computed immediately but added to a logical plan. Actions, like display(), trigger the execution of this logical plan.
Repeatedly running the same code cell can lead to misleading performance measurements due to caching.
When a dataset is used multiple times, Spark's optimization mechanism caches it in memory, making subsequent executions faster. This behavior does not accurately represent the first-time execution performance in a production environment where data might not be cached yet.
To get a more realistic measure of performance, it is recommended to:
* Clear the cache or restart the cluster to avoid the effects of caching.
* Test the entire workflow end-to-end rather than cell-by-cell to understand the cumulative performance.
* Consider using a representative sample of the production data, ensuring it includes various cases the code will encounter in production.
References:
* Databricks Documentation on Performance Optimization: Databricks Performance Tuning
* Apache Spark Documentation: RDD Programming Guide - Understanding transformations and actions