Converting the data to Delta Lake may not improve query performance on free text fields with high cardinality, such as the review column. This is because Delta Lake collects statistics on the minimum and maximum values of each column, which are not very useful for filtering or skipping data on free text fields.
Moreover, Delta Lake collectsstatistics on the first 32 columns by default, which may not include the review column if the table has more columns. Therefore, the junior data engineer's suggestion is not correct. A better approach would be to use a full-text search engine, such as Elasticsearch, to index and query the review column. Alternatively, you can use natural language processing techniques, such as tokenization, stemming, and lemmatization, to preprocess the review column and create a new column with normalized terms that can be used for filtering or skipping data. References:
* Optimizations: https://docs.delta.io/latest/optimizations-oss.html
* Full-text search with Elasticsearch: https://docs.databricks.com/data/data-sources/elasticsearch.html
* Natural language processing: https://docs.databricks.com/applications/nlp/index.html