Databricks Databricks-Certified-Data-Engineer-Professional Exam Questions : Databricks Certified Data Engineer Professional Exam

The data engineer is using Spark's MEMORY_ONLY storage level. Which indicators should the data engineer look for in the spark UI's Storage tab to signal that a cached table is not performing optimally?

• A. On Heap Memory Usage is within 75% of off Heap Memory usage
• B. Size on Disk is < Size in Memory
• C. The number of Cached Partitions> the number of Spark Partitions
• D. Size on Disk is> 0
• E. The RDD Block Name included the '' annotation signaling failure to cache

A data engineer is using Lakeflow Declarative Pipelines Expectations feature to track the data quality of their incoming sensor data. Periodically, sensors send bad readings that are out of range, and they are currently flagging those rows with a warning and writing them to the silver table along with the good data. They've been given a new requirement ?the bad rows need to be quarantined in a separate quarantine table and no longer included in the silver table.
This is the existing code for their silver table:
@dlt.table
@dlt.expect("valid_sensor_reading", "reading < 120")
def silver_sensor_readings():
return spark.readStream.table("bronze_sensor_readings")
What code will satisfy the requirements?

• A. @dlt.table
  @dlt.expect_or_drop("valid_sensor_reading", "reading < 120")
  def silver_sensor_readings():
  return spark.readStream.table("bronze_sensor_readings")
  @dlt.table
  @dlt.expect_or_drop("invalid_sensor_reading", "reading >= 120")
  def quarantine_sensor_readings():
  return spark.readStream.table("bronze_sensor_readings")
• B. @dlt.table
  @dlt.expect_or_drop("valid_sensor_reading", "reading < 120")
  def silver_sensor_readings():
  return spark.readStream.table("bronze_sensor_readings")
  @dlt.table
  @dlt.expect("invalid_sensor_reading", "reading < 120")
  def quarantine_sensor_readings():
  return spark.readStream.table("bronze_sensor_readings")
• C. @dlt.table
  @dlt.expect("valid_sensor_reading", "reading < 120")
  def silver_sensor_readings():
  return spark.readStream.table("bronze_sensor_readings")
  @dlt.table
  @dlt.expect("invalid_sensor_reading", "reading >= 120")
  def quarantine_sensor_readings():
  return spark.readStream.table("bronze_sensor_readings")
• D. @dlt.table
  @dlt.expect_or_drop("valid_sensor_reading", "reading < 120")
  def silver_sensor_readings():
  return spark.readStream.table("bronze_sensor_readings")
  @dlt.table
  @dlt.expect("invalid_sensor_reading", "reading >= 120")
  def quarantine_sensor_readings():
  return spark.readStream.table("bronze_sensor_readings")

A data engineer is designing a pipeline in Databricks that processes records from a Kafka stream where late-arriving data is common. Which approach should the data engineer use?

• A. Use batch processing and overwrite the entire output table each time to ensure late data is incorporated correctly.
• B. Use a watermark to specify the allowed lateness to accommodate records that arrive after their expected window, ensuring correct aggregation and state management.
• C. Implement a custom solution using Databricks Jobs to periodically reprocess all historical data.
• D. Use an Auto CDC pipeline with batch tables to simplify late data handling.

A data engineer is implementing Unity Catalog governance for a multi-team environment. Data scientists need interactive clusters for basic data exploration tasks, while automated ETL jobs require dedicated processing. How should the data engineer configure cluster isolation policies to enforce least privilege and ensure Unity Catalog compliance?

• A. Allow all users to create any cluster type and rely on manual configuration to enable Unity Catalog access modes.
• B. Create compute policies with STANDARD access mode for interactive workloads and DEDICATED access mode for automated jobs.
• C. Configure all clusters with NO ISOLATION_SHARED access mode since Unity Catalog works with any cluster configuration.
• D. Use only DEDICATED access mode for both interactive workloads and automated jobs to maximize security isolation.

Why are Pandas UDFs often preferred over traditional PySpark UDFs in performance-critical applications involving large datasets?

• A. They allow row-level execution of functions in Python with native Spark optimization, removing the need for columnar execution.
• B. They minimize memory usage by streaming each row individually through a lightweight Python wrapper, avoiding batch processing overhead.
• C. They eliminate the JVM-Python boundary by bypassing serialization entirely, thereby avoiding data conversion overhead.
• D. They leverage Apache Arrow to enable vectorized operations between the JVM and Python runtimes, reducing serialization costs and improving computational efficiency.