The requirement is that both valid (good) and invalid (bad) records must be captured and available separately for downstream processing. Invalid records should not simply be dropped; they must be quarantined in a dedicated table.
In Databricks Lakeflow Declarative Pipelines (DLT) , this is achieved by creating separate output tables :
* One table for valid records (Silver table) that pass the expectations.
* Another quarantine table that explicitly captures records failing the expectations.
Option A correctly implements this by:
* Declaring a DLT table trips_data_quarantine.
* Using .filter(expr(quarantine_rules)) to isolate invalid records (records where latitude or longitude is NULL).
* This ensures analysts can query both good records (from the main Silver pipeline table) and bad records (from the quarantine table).
Why not the others?
* B : Uses @dlt.expect_or_drop, which drops invalid records instead of quarantining them. This violates the requirement that quarantined data should be available.
* C : Same as B, but applies expectations in bulk with expect_all_or_drop. Again, bad data is dropped, not quarantined.
* D : Adds an is_quarantined flag in the same table. While it marks bad records, it does not separate them into a distinct quarantine table as required by the business use case.
Therefore, Option A is the only solution aligned with Databricks documentation for quarantining invalid data into a dedicated table while keeping valid data in the main pipeline.