Secure Access to Storage: Azure Databricks and Azure Data Lake Storage Gen2 Patterns 

In a previous article we covered six access control patterns, the advantages and disadvantages of each, and the scenarios in which they would be most appropriate. This article aims to complete the security discussion by providing an overview of network security between these two services, and how to connect securely to ADLS from ADB using Read more

1 Comment

Just-in-time Azure Databricks access tokens and instance pools for Azure Data Factory pipelines using workspace automation 

Using AAD tokens it is now possible to generate an Azure Databricks personal access token programmatically, and provision an instance pool using the Instance Pools API. The token can be generated and utilised at run-time to provide “just-in-time” access to the Databricks workspace. Using the same AAD token, an instance pool can also be provisioned Read more

Enterprise-wide orchestration using multiple data factories 

Building simple data engineering pipelines with a single Azure Data Factory (ADF) is easy, and multiple activities can be orchestrated within a single pipeline. More sophisticated data engineering patterns require flexibility and reusability through Pipeline Orchestration. Read more

1 Comment