Explanation
Correct code block:
itemsDf.filter(size("itemNameElements")>3)
Output of code block:
+------+----------------------------------+-------------------+------------------------------------------+
|itemId|itemName |supplier |itemNameElements |
+------+----------------------------------+-------------------+------------------------------------------+
|1 |Thick Coat for Walking in the Snow|Sports Company Inc.|[Thick, Coat, for, Walking, in, the, Snow]|
|2 |Elegant Outdoors Summer Dress |YetiX |[Elegant, Outdoors, Summer, Dress] |
+------+----------------------------------+-------------------+------------------------------------------+ The big difficulty with this question is in knowing the difference between count and size (refer to documentation below). size is the correct function to choose here since it returns the number of elements in an array on a per-row basis.
The other consideration for solving this question is the difference between select and filter. Since we want to return the rows in the original DataFrame, filter is the right choice. If we would use select, we would simply get a single-column DataFrame showing which rows match the criteria, like so:
+----------------------------+
|(size(itemNameElements) > 3)|
+----------------------------+
|true |
|true |
|false |
+----------------------------+
More info:
Count documentation: pyspark.sql.functions.count - PySpark 3.1.1 documentation Size documentation: pyspark.sql.functions.size - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1