最も有効な Certified-Data-Engineer-Professional 試験参考書

A data engineer created a daily batch ingestion pipeline using a cluster with the latest DBR version to store banking transaction data, and persisted it in a MANAGED DELTA table called prod.gold.all_banking_transactions_daily. The data engineer is constantly receiving complaints from business users who query this table ad hoc through a SQL Serverless Warehouse about poor query performance. Upon analysis, the data engineer identified that these users frequently use high- cardinality columns as filters. The engineer now seeks to implement a data layout optimization technique that is incremental, easy to maintain, and can evolve over time. Which command should the data engineer implement?

• A. Alter the table to use Z-ORDER and implement a periodic OPTIMIZE command.
• B. Alter the table to use Liquid Clustering and implement a periodic OPTIMIZE command.
• C. Alter the table to use Hive-Style Partitions and implement a periodic OPTIMIZE command.
• D. Alter the table to use Hive-Style Partitions + Z-ORDER and implement a periodic OPTIMIZE command.

A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records.
In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?

• A. Perform an insert-only merge with a matching condition on a unique key.
• B. Rely on Delta Lake schema enforcement to prevent duplicate records.
• C. VACUUM the Delta table after each batch completes.
• D. Set the configuration delta.deduplicate = true.
• E. Perform a full outer join on a unique key and overwrite existing data.

A data engineer is building a Lakeflow Declarative Pipelines pipeline to process healthcare claims data. A metadata JSON file defines data quality rules for multiple tables, including:
{
"claims": [
{"name": "valid_patient_id", "constraint": "patient_id IS NOT NULL"},
{"name": "non_negative_amount", "constraint": "claim_amount >= 0"}
]
}
The pipeline must dynamically apply these rules to the claims table without hardcoding the rules.
How should the data engineer achieve this?

• A. Invoke an external API to validate records against the metadata rules.
• B. Use a SQL CONSTRAINT block referencing the JSON file path.
• C. Load the JSON metadata, loop through its entries, and apply expectations using dlt.expect_all.
• D. Reference each expectation with @dlt.expect decorators in the table declaration.

The data architect has mandated that all tables in the Lakehouse should be configured as external Delta Lake tables.
Which approach will ensure that this requirement is met?

• A. When the workspace is being configured, make sure that external cloud object storage has been mounted.
• B. When tables are created, make sure that the external keyword is used in the create table statement.
• C. When configuring an external data warehouse for all table storage. leverage Databricks for all ELT.
• D. Whenever a database is being created, make sure that the location keyword is used
• E. Whenever a table is being created, make sure that the location keyword is used.

A nightly batch job is configured to ingest all data files from a cloud object storage container where records are stored in a nested directory structure YYYY/MM/DD. The data for each date represents all records that were processed by the source system on that date, noting that some records may be delayed as they await moderator approval. Each entry represents a user review of a product and has the following schema:
user_id STRING, review_id BIGINT, product_id BIGINT, review_timestamp TIMESTAMP, review_text STRING The ingestion job is configured to append all data for the previous date to a target table reviews_raw with an identical schema to the source system. The next step in the pipeline is a batch write to propagate all new records inserted into reviews_raw to a table where data is fully deduplicated, validated, and enriched.
Which solution minimizes the compute costs to propagate this batch of data?

• A. Perform a batch read on the reviews_raw table and perform an insert-only merge using the natural composite key user_id, review_id, product_id, review_timestamp.
• B. Use Delta Lake version history to get the difference between the latest version of reviews_raw and one version prior, then write these records to the next table.
• C. Configure a Structured Streaming read against the reviews_raw table using the trigger once execution mode to process new records as a batch job.
• D. Reprocess all records in reviews_raw and overwrite the next table in the pipeline.
• E. Filter all records in the reviews_raw table based on the review_timestamp; batch append those records produced in the last 48 hours.