Using Connectors to simplify data ingestion at scale in Dynamics 365 for Customer Insights

Microsoft Dynamics 365 for Customer Insights (DCI) SaaS offering provides big data analytics and insights for sales, marketing and service functions. To extract the most out of DCI requires bringing in data from different sources. One of the most important data sources is Microsoft Dynamics 365 (online).  In this article, I want to walk you through the process of managing data sources and integration with DCI.  The process of extracting insights is a typical walk through steps of scoping insights requirements, modeling data, implementing data map, data preparation, transformation and flow, and finally, data ingestion to DCI.

 

Sample DCI application: Retail bank customer 360

The journey of DCI analytics starts with defining the insights and a use case that fit the insights. Take the example of customer 360 view with DCI for a retail banking customer service center.

 

Customer 360 view in sample DCI application

 

Bank client, Mike Price, just enters one of the bank’s branches. The bank has his demographic profile in DCI. DCI shares the client’s phone number and shows the visuals and pictures of Mike to identify him when he is sitting in front of me. It pulls up rich set of various KPIs calculated inside customer insights by its powerful KPI processing engine. The bank want to see and understand who this customer is. Based on all the raw data, DCI constantly calculate those measures. While it spews out insights, DCI could also share trivial measures, like Mike’s checking account balance, savings account balance, KPIs of KPIs, and so on.  DCI provides widgets that highlights Mike’s communication preferences, what channels Mike is using and how Mike is interacting with the bank. DCI shows the financial instrument, and last but not the least, it shows end-to-end customer journey with all interactions and transactional data that you might want to look at.

 

Building Retail 360 experience requires multiple steps on data management side. It starts with identifying the data sources and corresponding connectors to bring data in from those sources, modeling the data, transforming the data, and then activating the connector to start the complete end-to-end flow.

 

Using data source connectors to bring data into DCI

The key to DCI success is in its ability to bring in data from multiple sources, and continuously extract insights from data. Once the scope of the use case is defined (e.g., retail bank customer 360 view defined above), the most important task is to bring the data in from multiple sources and model it in DCI.  DCI excels in integrating data from multiple different kinds of data sources characterized by different attributes.

Connector’s core functionality

A data connector mechanism facilitates data source connection and data pull to DCI.  Connector performs following functions, including

  • Understand the data source schema, attributes and details (not always possible)
  • Perform the required data transformation
  • Land data in DCI in the format ingestible by DCI
  • Provision the necessary resources to scale data extraction from the data source

 

Connectors absorb the data integration complexity

Data source’s attributes compound connector’s complexity. Data sources differ based on multiple factors, including, but not limited to the following:-

  • Client’s internal v/s external data source. E.g., Microsoft Dynamics 365 (online) is an internal data source, as against, data from social media like Twitter or Facebook
  • On premises v/s on cloud data source. E.g., bringing data from a CRM instance on premises, as against, data from a cloud marketing product
  • Transactional v/s behavioral data. E.g., for Amazon, their customer’s invoice or purchase order data is transactional in nature, as against, customer’s purchase preferences and seasonal buying patterns

 

Many other factors contribute to the complexity of connectors for DCI, e.g., data access interface, data model – including the structure or lack thereof, of schema, operations support, bulk export capabilities, governance requirements, interaction model, speed and volume of data change in the source, etc.

 

Identify the right data sources and corresponding ingestion mechanism

To build a retail bank customer 360 with demographics data from all other sources starts with identifying data sources that meet the use case needs. It could be data from CRM systems, social media, web logs, mobile application logs, or any other database.

Data sources for demographic data 

 

There are three different ways to bring data into DCI. These are:-

  • Using an out-of-the-box connector
    • Microsoft Dynamics 365 (online) connector
    • Azure Storage Blob connector
  • Stream the data via DCI end point
    • The documentation for point ingestion API is available at Data APIs
  • Custom built connector that performs the connector’s core functions and leveraging Azure blob connector and DCI end-point

 

Ingestion mechanism choice depends upon multiple factors, most important being, and availability of custom connector for your data source or not.

 

DCI data model support inside connectors

DCI data model models data after schema.org concepts.  Key DCI data building blocks are profiles, interactions, and links.

 

  1. Profile – Profile is a description of an entity/person/organization in a particular context. This is the core concept of a DCI data model. Customers look to profile an entity using DCI. DCI gives the ability to model, enrich and interact with profiles in real time. For retail bank customer 360 use case we discuss above, these could be entities like clients, branches, ATMs, etc.
  2. Interaction – An interaction is an event of relevance to a customer’s business. The goal is to allow customers to capture interactions from across the channels (Web, Mobile, Social) or LOB systems (CRM, ERP etc.).
  3. Links – Links correlate the interaction information with profile and are used to enrich the profile data.
  4. KPI – KPI is key insights about business performance calculated against a profile or interaction.

 

Depending upon the type of connector, DCI may provide an out of the box data model. DCI already does that for Dynamics 365 (online) connector. If not, users need to create their own data model or enhance an existing one when bringing in data from a new source.

 

Out of the box connectors in DCI

Today, there are two out of the box connectors available in DCI December public preview version, Microsoft Dynamics 365 (online) connector and Azure Blob connector. While Dynamics 365 (online) connector provides very structured connectivity to a clear single data source, Azure Blob connector provides free form data ingestion mechanism to scale amount of data and variety of data ingested in DCI.

 

Key differences between Dynamics 365 (online) connector and Azure Blob connector are captured in the table below.

 

Connector characteristics Dynamics 365 (online) connector Azure Blob connector
Data sources to bring data from Dynamics 365 data Any data source
Data modeling Connector creates out of the box data model for Dynamics 365 data. Connector maps Dynamics 365 model to DCI model Users model data in DCI by pushing CSV objects to blob and mapping those to DCI model
Data preparation Connector prepares and transforms Dynamics 365 data Users define their own data preparation, cleansing and quality mechanism
Data sync policy Complete sync of Dynamics 365 data Users create and deploy custom sync mechanism
Data sync frequency Sync as and when data changes on Dynamics 365 Users determine sync frequency
Data sync speed Near real time updates of new / modified data from Dynamics 365 to DCI Users push data to DCI at desired frequency
Data pipeline and resources Pipeline from Dynamics 365 to DCI created and provisioned Users to create data source to Azure Blob data and control resources and pipeline
Data movement errors handling Out of the box error handling management Users provide error handling and recovery mechanism
Data source metadata changes handling Automatic DCI model updates Users manage DCI model changes to reflect source metadata changes

 

Building your own connector

Users can build their own connectors for DCI to emulate some of the custom connector functionality. DCI models data after schema.org concepts.  DCI users can build the data model corresponding to the source to comply with DCI data model.  Next, they can combine the Azure Blob connector and point ingestion mechanism to create the connector with custom orchestration, controls and policies. In particular, users will need to implement the following key functionality to create a custom connector.

 

  • Define and build DCI data model for data source
  • Define and implement logic for data extraction and transformation from source, and load via DCI Azure blob connector or point ingestion API
    • Blob connector typically used for bulk data load into DCI
    • Point ingestion typically used for near real time low volumes of data ingestion
  • Provision and manage resources for above operations

 

Additionally users need to define and implement other aspects of connector like data movement orchestration, sync policy, frequency of data updates, speed of data copy, resources auto-scaling, error handling, etc. for a seamless connector experience.

 

Activate the connector

Once a data source connector is added to DCI, activate the connector to let the data flow in to begin the process of insights creation. Exact technical details and steps of how to add connector data source and deploy it can be found in DCI Step-by-step configuration guide.

 

 

– Dev Vidhani