Power Query This option bears a great resemblance to what you might find in Power BI. The same engine that runs Power BI likely runs the Power Query plug‐in. The feature provides an interface for viewing the data from a selected dataset. You can then run through some transformation ideas and see how the data…
FIGUER 3.63 Azure Data Factory Author dataset A dataset is used as parameters for the source and sink configuration in a pipeline that contains a Copy Data activity. Pipeline The Azure Data Factory pipeline user interface is almost identical to the one in Azure Synapse Analytics. Pipelines are groups of activities that can ingest and…
When you drag a Copy activity into the pipeline editor canvas, you will see the properties of that activity. Then, when you click the Source tab, you will see something similar to Figure 3.57. Notice that the value shown in the Source Dataset drop‐down box is the one you created in Exercise 3.9. FIGUER 3.57…
If you will be writing code or queries to manage your data analytics solution running on Azure Synapse Analytics, you should consider storing them in a repository. Source repositories like Azure DevOps and GitHub provide features like protection of losing the code, storage of change history and branching. After spending hours, days, and sometimes precious…
In Exercise 3.3 you enabled managed virtual networking during the provisioning of the Azure Synapse Analytics workspace. This feature enables you to configure outbound workspace connectivity with products, applications, and other services that exist outside of the managed virtual network. When you click the Managed Private Endpoints link, you will see some existing private endpoints….
This feature enables you to grant individuals access to either the Azure Synapse Analytics workspace or a single workspace item. When you initially access the Access Control page, you will see something similar to Figure 3.39. The Access Control page lists the user accounts grouped by Synapse RBAC roles, the type of account, the role,…
An Apache Spark pool is the compute node that will execute the queries you write to pull data from, for example, Parquet files. You can provision a Spark pool from numerous locations. One such place is from the Manage page in Azure Synapse Analytics Studio. After clicking the Manage hub option, select Apache Spark pools….
The Big Data pipeline has been touched on numerous times; if you need a refresher, refer to Figure 2.30. Data ingestion is the first phase of that pipeline (refer to Figure 3.1). The pipeline itself is a series of purposefully sequenced activities. The activities can and often do span all the Big Data stages, from…
Before reading any further, ask yourself what an analytical store is. If you struggle for the answer, refer to Table 3.2, which provides the analytical datastores available in Azure, as well as the data model that works optimally with those products. Table 3.1 provides a list of different ingestion types mapped to the most suitable…
Azure Databricks generates a metastore by default with the provisioning of a cluster. After creating a cluster, you can query the metastore. Doing so produces a visualization of any table. Figure 3.22 illustrates the retrieval from the metastore using the show tables in <databaseName> command. Executing the command lists all the tables in the targeted…