Databricks Databricks-Certified-Data-Analyst-Associate Databricks Certified Data Analyst Associate Exam Exam Practice Test

 Option A uses the SELECT DISTINCT statement to remove duplicate rows from the table_bronze and create a new table table_silver with the deduplicated data. This is the correct way to deduplicate data using Spark SQL12. Option B simply inserts all the rows from table_bronze into table_silver, without removing any duplicates. Option C is not a valid syntax for Spark SQL, as there is no MERGE DEDUPLICATE statement. Option D appends all the rows from table_bronze into table_silver, without removing any duplicates. Option E overwrites the existing data in table_silver with the data from table_bronze, without removing any duplicates. References: Delete Duplicate using SPARK SQL, Spark SQL - How to Remove Duplicate Rows

A Structured Streaming pipeline that processes data in micro-batches and populates gold-level tables every minute requires a high level of compute resources to handle the frequent data ingestion, processing, and writing. This could result in a significant cost for the organization, especially if the data volume and velocity are large. Therefore, the data analyst should share this caution with the project stakeholders before setting up the dashboard and evaluate the trade-offs between the desired refresh rate and the available budget. The other options are not valid cautions because:

B. The gold-level tables are assumed to be appropriately clean for business reporting, as they are the final output of the data engineering pipeline. If the data quality is not satisfactory, the issue should be addressed at the source or silver level, not at the gold level.

C. The streaming data is an appropriate data source for a dashboard, as it can provide near real-time insights and analytics for the business users. Structured Streaming supports various sources and sinks for streaming data, including Delta Lake, which can enable both batch and streaming queries on the same data.

D. The streaming cluster is fault tolerant, as Structured Streaming provides end-to-end exactly-once fault-tolerance guarantees through checkpointing and write-ahead logs. If a query fails, it can be restarted from the last checkpoint and resume processing.

E. The dashboard can be refreshed within one minute or less of new data becoming available in the gold-level tables, as Structured Streaming can trigger micro-batches as fast as possible (every few seconds) and update the results incrementally. However, this may not be necessary or optimal for the business use case, as it could cause frequent changes in the dashboard and consume more resources. References: Streaming on Databricks, Monitoring Structured Streaming queries on Databricks, A look at the new Structured Streaming UI in Apache Spark 3.0, Run your first Structured Streaming workload

 The SQL code provided in Option D is the correct way to create a view named sales_by_employee that will always stay up-to-date with the sales and employees tables. The code uses the CREATE OR REPLACE VIEW statement to define a new view that joins the sales and employees tables on the employee_id column. It selects the employee_name as sales_person and all sales for each employee, ensuring that the data entity/object is always up-to-date when new data are added to these tables.

The result set provided shows a combination of grouping by two columns (group_1andgroup_2) with subtotals for each level of grouping and a grand total. This pattern is typical of aGROUP BY ... WITH ROLLUPoperation in SQL, which provides subtotal rows and a grand total row in the result set.

Considering the query options:

A)Option A:GROUP BY group_1, group_2 INCLUDING NULL- This is not a standard SQL clause and would not result in subtotals and a grand total.

B)Option B:GROUP BY group_1, group_2 WITH ROLLUP- This would create subtotals for each uniquegroup_1, each combination ofgroup_1andgroup_2, and a grand total, which matches the result set provided.

C)Option C:GROUP BY group_1, group 2- This is a simpleGROUP BYand would not include subtotals or a grand total.

D)Option D:GROUP BY group_1, group_2, (group_1, group_2)- This syntax is not standard and would likely result in an error or be interpreted as a simpleGROUP BY, not providing the subtotals and grand total.

E)Option E:GROUP BY group_1, group_2 WITH CUBE- TheWITH CUBEoperation produces subtotals for all combinations of the selected columns and a grand total, which is more than what is shown in the result set.

The correct answer isOption B, which usesWITH ROLLUPto generate the subtotals for each level of grouping as well as a grand total. This matches the result set where we have subtotals for eachgroup_1, each combination ofgroup_1andgroup_2, and the grand total where bothgroup_1andgroup_2areNULL.

In Databricks, to determine the owner of a managed table, you can utilize the Catalog Explorer feature. The steps are as follows:​

Access Catalog Explorer:

In your Databricks workspace, click on theCatalogicon in the sidebar to open Catalog Explorer.​

Navigate to the Table:

Within Catalog Explorer, browse through the catalog and schema to locate the specific managed table whose ownership you wish to verify.​

View Table Details:

Click on the table name to open its details page.​

Identify the Owner:

On the table's details page, review theOwnerfield, which displays the principal (user, service principal, or group) that owns the table.​

This method provides a straightforward way to ascertain the ownership of managed tables within the Databricks environment. Understanding table ownership is essential for managing permissions and ensuring proper access control.​

In Databricks SQL, a subquery is a nested query within a larger SQL query that allows for the retrieval of data without the necessity of creating a table or view. This is particularly useful for simplifying complex queries by breaking them down into more manageable parts. Subqueries can be employed in various clauses such as SELECT, FROM, and WHERE to perform operations like filtering, transforming, and aggregating data on-the-fly. This flexibility enhances query efficiency and readability without the overhead of persisting intermediate results as separate tables or views.

External tables are tables that are defined in the Databricks metastore using the information stored in a cloud object storage location. External tables do not manage the data, but provide a schema and a table name to query the data. To create an external table, you can use the CREATE EXTERNAL TABLE statement and specify the object storage path to the LOCATION clause. For example, to create an external table named ext_table on a Parquet file stored in S3, you can use the following statement:

SQL

CREATE EXTERNAL TABLE ext_table (

col1 INT,

col2 STRING

)

STORED AS PARQUET

LOCATION 's3://bucket/path/file.parquet'

AI-generated code. Review and use carefully. More info on FAQ.

The approach that cannot be used to share the dashboard and meet the security requirement is D. Generating a Personal Access Token that is good for 1 day and sharing it with the client. This approach would give the client access to the Databricks workspace using the token owner’s identity and permissions, which could expose other information, resources, or artifacts in the database1. The other approaches can be used to share the dashboard and meet the security requirement because:

A. Downloading the Dashboard as a PDF and sharing it with the client would only provide a static snapshot of the dashboard without any interactive features or access to the underlying data2.

B. Setting a refresh schedule for the dashboard and entering the client’s email address in the “Subscribers” box would send the client an email with the latest dashboard results as an attachment or a link to a secure web page3. The client would not be able to access the Databricks workspace or the dashboard itself.

C. Taking a screenshot of the dashboard and sharing it with the client would also only provide a static snapshot of the dashboard without any interactive features or access to the underlying data4.

E. Downloading a PNG file of the visualizations in the dashboard and sharing them with the client would also only provide a static snapshot of the visualizations without any interactive features or access to the underlying data5. References:

1: Personal access tokens

2: Download as PDF

3: Automatically refresh a dashboard

4: Take a screenshot

5: Download a PNG file

To get the number of orders per customer, you need to join the customers and orders tables on the customer_id, count the order_id, and group the results by customer_name. The correct SQL syntax, as outlined in Databricks SQL documentation, is to use GROUP BY on the selected customer field and use COUNT for aggregation. Only option A does this correctly, while the other options contain syntax errors or incorrect field names.

Descriptive statistics refer to statistical methods used to describe and summarize the basic features of data in a study. They provide simple summaries about the sample and the measures, often including metrics such as mean, median, mode, range, and standard deviation. Databricks learning materials highlight that descriptive statistics use summary statistics to quantitatively describe and summarize data, providing insight into data distributions without making inferences or predictions.

In Databricks SQL, to configure a refresh schedule for a query that is not attached to a dashboard or alert, a data analyst should use the Query Editor. Within the Query Editor, there is an option to set up scheduled executions for queries. This feature enables the query to run at specified intervals, ensuring that the results are updated regularly. By scheduling queries in this manner, analysts can automate data refreshes and maintain up-to-date query results without manual intervention.​

Data Explorer is a user interface that allows you to discover and manage data, schemas, tables, models, and permissions in Databricks SQL. You can use Data Explorer to view schema details, preview sample data, and see table and model details and properties. Administrators can view and change owners, and admins and data object owners can grant and revoke permissions1. References: Discover and manage data using Data Explorer

A user-defined function (UDF) is a function defined by a user, allowing custom logic to be reused in the user environment1. To apply a UDF to a table, the syntax is SELECT udf_name(column_name) AS alias FROM table_name2. Therefore, option E is the correct way to use the UDF price to create a new column customer_price based on the existing columns customer_spend and customer_units from the table customer_summary. References:

What are user-defined functions (UDFs)?

User-defined scalar functions - SQL

V

 The mean value of a variable is the average of all the values in a data set, calculated by dividing the sum of the values by the number of values. The median value of a variable is the middle value of the ordered data set, or the average of the middle two values if the data set has an even number of values. The mean value is sensitive to outliers, which are values that are very different from the rest of the data. Outliers can skew the mean value and make it less representative of the central tendency of the data. The median value is more robust to outliers, as it only depends on the middle values of the data. Therefore, when the variable contains a lot of extreme outliers, the mean value and the median value will be meaningfully different, as the mean value will be pulled towards the outliers, while the median value will remain close to the majority of the data1. References: Difference Between Mean and Median in Statistics (With Example) - BYJU’S

The Owner field in the table page shows the current owner of the table and allows the owner to change it to another user or group. To change the ownership of the table, the owner can click on the Owner field and select the new owner from the drop-down list. This will transfer the ownership of the table to the selected user or group and remove the previous owner from the list of table access control entries1. The other options are incorrect because:

A. Removing the owner’s account from the Owner field will not change the ownership of the table, but will make the table ownerless2.

B. Selecting All Users from the Owner field will not change the ownership of the table, but will grant all users access to the table3.

D. Selecting the Admins group from the Owner field will not change the ownership of the table, but will grant the Admins group access to the table3.

E. Removing all access from the Owner field will not change the ownership of the table, but will revoke all access to the table4. References:

1: Change table ownership

2: Ownerless tables

3: Table access control

4: Revoke access to a table

In Databricks, aviewis a saved SQL query definition that references existing tables or other views. Once created, a view remains persisted in themetastore(such as Unity Catalog or Hive Metastore) until it is explicitly dropped.

Key points:

Views donot store datathemselves but reference data from underlying tables.

Logging out or being inactive doesnotdelete or alter views.

Unless a user or admin explicitlydropsthe view or the underlying data/table is deleted, the view continues to function as expected.

Therefore, after logging back in—even days later—a user can still run SELECT * FROM stakeholders.eur_customers, and it will return the same data (provided the underlying table hasn’t changed).

Partner Connect is a feature that allows you to easily connect your Databricks workspace to Fivetran and other ingestion partners using an automated workflow. You can select a SQL warehouse or a cluster as the destination for your data replication, and the connection details are sent to Fivetran. You can then choose from over 200 data sources that Fivetran supports and start ingesting data into Delta Lake. References: Connect to Fivetran using Partner Connect, Use Databricks with Fivetran

In this scenario, the data engineering team has configured a Structured Streaming pipeline that updates the gold-level tables every 10 minutes. To ensure that the dashboard reflects the most recent data, it is essential to set the dashboard's refresh schedule to an interval of 10 minutes or less. This synchronization ensures that stakeholders view the latest information shortly after it becomes available in the gold-level tables. Options B, C, and D do not directly address the requirement of aligning the dashboard refresh frequency with the data update interval.

The command INSERT INTO stakeholders.suppliers TABLE stakeholders.new_suppliers is not a valid syntax for inserting data into a table in Databricks SQL. According to the documentation12, the correct syntax for inserting data into a table is either:

INSERT { OVERWRITE | INTO } [ TABLE ] table_name [ PARTITION clause ] [ ( column_name [, ...] ) | BY NAME ] query

INSERT INTO [ TABLE ] table_name REPLACE WHERE predicate query

The command in the question is missing the OVERWRITE or INTO keyword, and the query part that specifies the source of the data to be inserted. The TABLE keyword is optional and can be omitted. The PARTITION clause and the column list are also optional and depend on the table schema and the data source. Therefore, the command in the question will fail with a syntax error.