diff --git a/bs.py b/bs.py index cdb77da..5396fb6 100644 --- a/bs.py +++ b/bs.py @@ -1,4 +1,5 @@ import sys +import re import json from markdownify import markdownify as md @@ -15,7 +16,9 @@ def convert_html_to_markdown(input_file, output_file): html_content = item.get("content", "") # Convert HTML content to Markdown - markdown_content = md(html_content) + markdown_content = md(html_content, strip=['a']) + # Remove unwanted image links (e.g., ![](...) ) + markdown_content = re.sub(r'!\[.*?\]\(.*?\)', '', markdown_content) # Prepend other attributes markdown_output.append(f"# {title}\n\n") diff --git a/data/DCR_mappings_ALL.md b/data/DCR_mappings_ALL.md new file mode 100644 index 0000000..4d3228a --- /dev/null +++ b/data/DCR_mappings_ALL.md @@ -0,0 +1,89 @@ +The `/dcr` API serves as a canonical interface for initiating data change requests. The JSON payload sent to this endpoint contains a standardized representation of the desired changes. The MDM HUB is then responsible for validating this request and mapping its attributes to the distinct models of the target systems. + +*** + +### **`/dcr` API to Target System Attribute Mapping** + +The following sections break down the mapping from the attributes in the `/dcr` request payload to their corresponding fields in OneKey and Veeva. + +--- + +#### **1. Request-Level Attributes** + +These attributes are defined at the top level of each object within the `DCRRequests` array in the JSON payload. + +| HUB /dcr Attribute | OneKey Mapping | Veeva OpenData (VOD) Mapping | Notes / Description | +| :--- | :--- | :--- | :--- | +| **`extDCRRequestId`** | Used to populate the `DCRID` in the Reltio DCR tracking entity. OneKey's `validation.clientRequestId` is typically a HUB-generated internal ID, but `extDCRRequestId` is the primary key for client-side tracking. | **`dcr_key`** (in all CSV files: `change_request.csv`, `change_request_hco.csv`, etc.) | **This is the primary external identifier for the entire DCR.** It is crucial for clients like PforceRx to track the request's status and is used as the main correlation ID across all systems and files. | +| **`extDCRComment`** | **`validation.requestComment`** | **`description`** (in `change_request.csv`) | A free-text field for the requester to provide context or justification for the DCR. For OneKey, it has a special function: due to API limitations, requests to **remove** an attribute are specified in this comment field (e.g., "Please remove attributes: [Address: ...]"). | +| **`country`** | **`isoCod2`** | **`primary_country__v`** (in `change_request_hcp.csv` and `change_request_hco.csv`) and **`country__v`** (in `change_request_address.csv`) | The mandatory two-letter ISO country code. This is a critical routing attribute that determines which validator instance to use and, for Veeva, which S3/SFTP directory the DCR files are placed in. | +| **`action`** (within HCP/HCO object) | This determines the logic. `update` and `insert` map to a `submitVR` call. An `update` requires a valid `individual.individualEid` or `workplace.workplaceEid`. `delete` is handled by updating the entity's end date in Reltio. | **`change_request_type`** (in `change_request.csv`). Mapped to **`ADD_REQUEST`** for an `insert` action, and **`CHANGE_REQUEST`** for an `update` action. | This defines the fundamental operation being performed on the entity. | +| **`refId`** (within HCP/HCO object) | Used to query Reltio to find the OneKey crosswalk (`individual.individualEid` or `workplace.workplaceEid`) which is mandatory for update operations. | Used to query Reltio to find the Veeva crosswalk (`vid__v`), which is mandatory for update operations. The Reltio URI from the `refId` is also used to populate the **`entity_key`** field in the VOD CSVs. | This object contains the necessary identifiers (`CrosswalkTargetObjectId`, `EntityURITargetObjectId`, etc.) to locate the target entity in Reltio for an update or delete operation. | + +--- + +#### **2. Healthcare Professional (HCP) Attributes** + +These attributes are provided within the `HCP` object of a DCR request. + +| HUB /dcr Attribute | OneKey Mapping | Veeva OpenData (VOD) Mapping | Notes / Description | +| :--- | :--- | :--- | :--- | +| **`firstName`** | `individual.firstName` | `first_name__v` | HCP's first name. | +| **`lastName`** | `individual.lastName` | `last_name__v` | HCP's last name. Mandatory for creating a new HCP in OneKey. | +| **`middleName`** | `individual.middleName` | `middle_name__v` | HCP's middle name. | +| **`prefix`** | `individual.prefixNameCode` | `prefix__v` | Name prefix (e.g., Mr., Dr.). Requires a lookup from the canonical code (`HCPPrefix`) to the target system's specific code. | +| **`title`** | `individual.titleCode` | `professional_title__v` | Professional title. Requires a lookup from the canonical code (`HCPTitle` or `HCPProfessionalTitle`) to the target system's specific code. | +| **`gender`** | `individual.genderCode` | `gender__v` | HCP's gender. Requires a lookup to map the canonical value (e.g., M, F) to the target system's code. | +| **`subTypeCode`** | `individual.typeCode` | `hcp_type__v` | The professional subtype of the HCP (e.g., Physician, Nurse). Requires a lookup from the canonical code (`HCPSubTypeCode` or `HCPType`). | +| **`specialties`** (List) | `individual.speciality1`, `individual.speciality2`, `individual.speciality3` | `specialty_1__v` through `specialty_10__v` | The list of specialties is **flattened**. OneKey accepts up to 3 ranked specialties. Veeva accepts up to 10. A lookup is required to map the canonical `HCPSpecialty` code to the target system's value. | +| **`emails`** (List) | `individual.email` | `email_1__v`, `email_2__v` | The list of emails is flattened. OneKey typically takes the highest-ranked email. Veeva takes the top two. | +| **`phones`** (List) | `individual.mobilePhone` | `phone_1__v` to `phone_3__v` (for office), `fax_1__v` to `fax_2__v` (for fax) | The list is filtered by type and ranked. OneKey maps the highest-ranked to `mobilePhone`. Veeva separates numbers into distinct `phone` and `fax` columns based on the Reltio phone type. | + +--- + +#### **3. Healthcare Organization (HCO) Attributes** + +These attributes are provided within the `HCO` object of a DCR request. + +| HUB /dcr Attribute | OneKey Mapping | Veeva OpenData (VOD) Mapping | Notes / Description | +| :--- | :--- | :--- | :--- | +| **`name`** | `workplace.usualName` / `workplace.officialName` | `corporate_name__v` | The primary, official name of the organization. | +| **`otherNames`** (List) | `workplace.usualName2` | `alternate_name_1__v` | The list of alternative names is flattened. Both systems typically take the first or highest-ranked alternative name. | +| **`subTypeCode`** | `workplace.typeCode` | `major_class_of_trade__v` | The HCO's subtype, often representing facility type. Requires a lookup from the canonical code (`COTFacilityType`). | +| **`typeCode`** | Not Mapped | `hco_type__v` | Maps to the HCO Type. Requires a lookup (`HCOType`). The OneKey mapping document indicates this is not used for their system. | +| **`websiteURL`** | `workplace.website` | `URL_1__v` | The official website of the organization. | +| **`specialties`** (List) | `workplace.speciality1` to `speciality3` | `specialty_1__v` to `specialty_10__v` | Similar to HCPs, the list of specialties is flattened and ranked. A lookup from the canonical `COTSpecialty` code is required. | +| **`emails`** (List) | `workplace.email` | `email_1__v`, `email_2__v` | List of emails is flattened, with the highest-ranked ones being used. | +| **`phones`** (List) | `workplace.telephone`, `workplace.fax` | `phone_1__v` to `phone_3__v`, `fax_1__v` to `fax_2__v` | Similar to HCPs, the list is filtered by type (`TEL.OFFICE` vs. `TEL.FAX`) and ranked before mapping. | + +--- + +#### **4. Nested Object Mapping: Addresses** + +Address information is provided as a list of objects within the `HCP` or `HCO` DCR payload. + +| HUB /dcr `addresses` Attribute | OneKey Mapping | Veeva OpenData (VOD) Mapping | Notes / Description | +| :--- | :--- | :--- | :--- | +| **(Address Object)** | Mapped to a single `address` complex object in the JSON request. Only the primary address is sent. | Each address object is mapped to a **separate row** in the **`change_request_address.csv`** file. | OneKey's API takes a single address per DCR, while Veeva's file-based approach can handle multiple address changes in one DCR. | +| **`refId`** | Used to match and update an existing address. | **`address_key`** | This is the `PfizerAddressID`, a unique identifier for the address record in Reltio. | +| **`addressLine1`** | `address.longLabel` or `address.addressLine1` | `address_line_1__v` | The first line of the street address. | +| **`addressLine2`** | `address.longLabel2` or `address.addressLine2` | `address_line_2__v` | The second line of the street address. | +| **`city`** | `address.city` | `locality__v` | The city name. | +| **`stateProvince`** | `address.countyCode` | `administrative_area__v` | The state or province. Requires a lookup from the canonical value to the target system's code. | +| **`zip`** | `address.longPostalCode` or `address.Zip5` | `postal_code__v` | The postal or ZIP code. | +| **`addressType`** | Not explicitly mapped in `submitVR` request, but used in logic. | `address_type__v` | The type of address (e.g., Office, Home). Requires a lookup (`AddressType`). | + +--- + +#### **5. Nested Object Mapping: Affiliations** + +Affiliations are provided as a list of objects (`contactAffiliations` for HCP, `otherHCOAffiliations` for HCO) in the DCR payload. + +| HUB /dcr `affiliations` Attribute | OneKey Mapping | Veeva OpenData (VOD) Mapping | Notes / Description | +| :--- | :--- | :--- | :--- | +| **(Affiliation Object)** | The `refId` of the target HCO is used to find its `workplace.workplaceEid`, which is then sent as part of the HCP or parent HCO record. | Each affiliation object is mapped to a **separate row** in the **`change_request_parenthco.csv`** file. | The mapping for affiliations is fundamentally different. OneKey handles it as an attribute of the child entity, while Veeva treats it as a distinct relationship record. | +| **Relation `refId`** | The `workplace.workplaceEid` of the target (parent) HCO is populated in the request. | **`parenthco_key`** (populated with the Reltio `relationUri`) | Veeva requires a unique key for the relationship itself. | +| **Start/Child Object** | The main body of the request contains the child's details (e.g., the HCP). | **`child_entity_key`** | This field is populated with the `entity_key` of the child entity (e.g., the HCP). | +| **End/Parent Object** | The `workplace.workplaceEid` of the parent HCO is sent. | **`parent_entity_key`** | This field is populated with the `entity_key` of the parent entity (e.g., the HCO). | +| **`type`** | `activity.role` (requires lookup, e.g., `TIH.W.*`) | **`relationship_type__v`** (requires lookup from `RelationType`) | The type or role of the affiliation (e.g., Employed, Manages). | +| **`primary`** | Not explicitly mapped. | **`is_primary_relationship__v`** | A boolean flag (`Y`/`N`) indicating if this is the primary affiliation. | \ No newline at end of file diff --git a/data/DCR_mappings_OK_VOD.md b/data/DCR_mappings_OK_VOD.md new file mode 100644 index 0000000..e00c5b3 --- /dev/null +++ b/data/DCR_mappings_OK_VOD.md @@ -0,0 +1,156 @@ +*** + +### **Attribute Mapping to IQVIA OneKey** + +The following mappings are used when creating or updating entities in OneKey via the `submitVR` operation. The data is compiled from the direct mapping table and the comparator rules for Data Steward-suggested changes. + +#### **1. Common Validation Attributes (HCP & HCO)** + +These attributes are part of the `validation` block in the OneKey request and are mandatory for all DCRs. + +| OneKey Attribute | Value/Source | Description | +| :--- | :--- | :--- | +| `validation.clientRequestId` | HUB\_GENERATED\_ID | A unique identifier generated by the MDM HUB to trace the request. | +| `validation.process` | "Q" | A static value indicating the processing type. | +| `validation.requestDate` | Current Timestamp | The date the DCR was created. | +| `validation.callDate` | Current Timestamp | The date the API call was made. | +| `validation.requestProcess` | "I" | A static value indicating an "Insert" or "Update" process. | +| `validation.requestComment` | `extDCRComment` | A free-text comment provided in the initial DCR, often used for context or to specify removals. | +| `isoCod2` | `Country` from DCR | The two-letter ISO country code, which is mandatory for all requests. | + +--- + +#### **2. Healthcare Organization (HCO) Attribute Mapping** + +| Reltio / HUB Attribute | OneKey Attribute | Mandatory | Description & Notes | +| :--- | :--- | :--- | :--- | +| **Entity Type** | `entityType` | **Yes** | Static value set to **`WORKPLACE`** to identify the entity as an HCO. | +| **OneKey ID** | `workplace.workplaceEid` | **Yes (for updates)** | The unique OneKey identifier for the HCO. It is sourced from the `ONEKEY` crosswalk on the Reltio entity. This is mandatory for any update operation. | +| **Name** | `workplace.usualName` / `workplace.officialName` | Optional | The primary name of the HCO. Both `usualName` and `officialName` are typically populated with the same value from the Reltio `Name` attribute. | +| **Other Names** | `workplace.usualName2` | Optional | Sourced from the `OtherNames.Name` nested attribute in Reltio. | +| **Type Code** | `workplace.typeCode` | Optional | Maps the HCO's `subTypeCode` from the HUB model. This requires a lookup from a canonical code (e.g., `COTFacilityType`, `TET.W.*`) to the OneKey specific code. | +| **Website** | `workplace.website` | Optional | The official website URL of the HCO, sourced from `WebsiteURL`. | +| **Parent HCO Affiliation** | `workplace.parentWorkplaceEid` | Optional | The OneKey ID of the parent HCO. This is sourced by looking up the `otherHCOAffiliations` relation in Reltio and finding the OneKey crosswalk of the referenced parent entity. | +| **Addresses** | `address.*` fields | **Yes** | A complex object representing the HCO's primary address. It is mandatory for creating a new HCO.
- `address.country`
- `address.city`
- `address.addressLine1` (from `addressLine1`)
- `address.addressLine2` (from `addressLine2`)
- `address.Zip5` (from `zip`)
- `address.countyCode` (from `stateProvince`, requires lookup) | +| **Specialties** | `workplace.speciality1` / `2` / `3` | Optional | Reltio's list of `Specialities` is flattened into three separate fields in the OneKey request, ranked by priority. | +| **Phone (Telephone)** | `workplace.telephone` | Optional | The primary telephone number. Sourced from the `Phone` list in Reltio where the type is **not** FAX. The number with the highest rank is chosen. | +| **Phone (Fax)** | `workplace.fax` | Optional | The primary fax number. Sourced from the `Phone` list where the type **is** FAX. The number with the highest rank is chosen. | +| **Email** | `workplace.email` | Optional | The primary email address of the HCO. The email with the highest rank from Reltio's `Email` list is used. | + +--- + +#### **3. Healthcare Professional (HCP) Attribute Mapping** + +| Reltio / HUB Attribute | OneKey Attribute | Mandatory | Description & Notes | +| :--- | :--- | :--- | :--- | +| **Entity Type** | `entityType` | **Yes** | Static value set to **`ACTIVITY`** to identify the entity as an HCP. | +| **OneKey ID** | `individual.individualEid` | **Yes (for updates)** | The unique OneKey identifier for the HCP, sourced from the `ONEKEY` crosswalk on the Reltio entity. Mandatory for updates. | +| **Last Name** | `individual.lastName` | **Yes** | The HCP's last name. This is a mandatory field for creating an HCP. | +| **First Name** | `individual.firstName` | Optional | The HCP's first name. | +| **Middle Name** | `individual.middleName` | Optional | The HCP's middle name. | +| **Type Code** | `individual.typeCode` | Optional | The HCP's subtype (e.g., Physician, Nurse). This is sourced from `subTypeCode` in the HUB model and requires a lookup (`HCPSubTypeCode`, `TYP..*`). | +| **Title** | `individual.titleCode` | Optional | The HCP's professional title (e.g., Dr.). Requires lookup (`HCPTitle`, `TIT.*`). | +| **Prefix** | `individual.prefixNameCode` | Optional | The HCP's name prefix (e.g., Mr., Mrs.). Requires lookup (`HCPPrefix`, `APP.*`). | +| **Gender** | `individual.genderCode` | Optional | The HCP's gender. Requires a lookup to map the Reltio value to the OneKey code. | +| **Year of Birth** | `individual.birthYear` | Optional | The four-digit year of birth of the HCP. | +| **Day of Birth** | `individual.birthDay` | Optional | The day of birth of the HCP. | +| **Language** | `individual.languageEid` | Optional | The preferred language of the HCP. | +| **Website** | `individual.website` | Optional | The personal or professional website of the HCP. | +| **Identifiers** | `individual.externalId1` / `externalId2` | Optional | Mapped from Reltio's `Identifier` values. | +| **Affiliation** | `workplace.workplaceEid` | Optional | The OneKey ID of the HCO to which the HCP is affiliated. This is sourced from the `contactAffiliations` relation in Reltio by finding the OneKey crosswalk of the referenced HCO. | +| **Addresses** | `address.*` fields | **Yes** | A complex object representing the HCP's primary address (often a workplace address). See HCO section for field details. | +| **Specialties** | `individual.speciality1` / `2` / `3` | Optional | Reltio's list of `Specialities` is flattened into three separate fields, ranked by priority. A lookup is required to map the canonical code (`HCPSpecialty`, `SP.W.*`) to the OneKey code. | +| **Phone (Mobile)** | `individual.mobilePhone` | Optional | The primary mobile phone number, sourced from the `Phone` list in Reltio. | +| **Email** | `individual.email` | Optional | The primary email address of the HCP. | + +*** + +### **Attribute Mapping to Veeva OpenData (VOD)** + +DCRs for Veeva are processed asynchronously. The HUB maps the DCR data into a series of CSV file lines, which are then zipped and sent to Veeva via S3/SFTP. The mapping connects Reltio/HUB concepts to specific columns in different Veeva CSV files. + +#### **1. `change_request.csv` - Manifest File** + +This file contains metadata for each DCR. A single line is created for each DCR. + +| Veeva Field Name | Source / Logic | Required | Description | +| :--- | :--- | :--- | :--- | +| `dcr_key` | Mongo Generated DCR ID | **Yes** | A unique ID generated by the HUB for the DCR. This key is used to link all related records across the different CSV files. | +| `description` | `extDCRComment` | **Yes** | Free-text comments from the requester explaining the purpose of the DCR. | +| `created_by` | `createdBy` from DCR | **Yes** | Identifies the user or system that initiated the DCR. | +| `change_request_type` | Inferred | **Yes** | Set to **`ADD_REQUEST`** if a new profile is being created (no Veeva crosswalk exists) or **`CHANGE_REQUEST`** if an existing profile is being updated. | +| `entity_type` | Main DCR object type | **Yes** | Set to **`HCP`** or **`HCO`** depending on the primary subject of the DCR. | + +#### **2. `change_request_hco.csv` - HCO Details** + +This file contains the specific attribute changes for an HCO. + +| Veeva Field Name | Reltio / HUB Attribute | Required (Add) | Description & Notes | +| :--- | :--- | :--- | :--- | +| `dcr_key` | Mongo Generated DCR ID | **Yes** | Links this record back to the manifest file. | +| `entity_key` | `refId.entityURI` or Reltio Crosswalk | **Yes** | The unique identifier for the HCO in the source system (Reltio). It is a concatenation of the source name and value (e.g., `Reltio:rvu44dm`). | +| `vid__v` | VEEVA Crosswalk Value | No (Yes for Change) | The unique Veeva ID for the HCO. This is required for `CHANGE_REQUEST` types to identify which Veeva profile to update. | +| `corporate_name__v` | `Name` | No | The official name of the HCO. | +| `alternate_name_1__v` | `OtherNames.Name` (first element) | Yes | An alternative name for the HCO. | +| `major_class_of_trade__v` | `HCO.subTypeCode` | No | Maps to Reltio's `FacilityType`. This requires a lookup from the `COTFacilityType` canonical code to the Veeva source code. | +| `hco_type__v` | `typecode` | No | The type of the HCO. Requires lookup from `HCOType`. | +| `hco_status__v` | `StatusDetail` | No | The operational status of the HCO. Requires lookup from `HCOStatus`. | +| `specialty_1__v` to `specialty_10__v` | `specialties` | No | Reltio's list of HCO specialties is flattened into 10 separate columns, ranked by priority. | +| `email_1__v`, `email_2__v` | `emails` | No | The top two ranked email addresses from the Reltio `Email` list. | +| `phone_1__v`, `phone_2__v`, `phone_3__v` | `phones` | No | Top-ranked phone numbers where type is `TEL.OFFICE`. | +| `fax_1__v`, `fax_2__v` | `phones` | No | Top-ranked fax numbers where type is `TEL.FAX`. | +| `primary_country__v` | `DCRRequest.country` | No | The two-letter ISO country code. | + +#### **3. `change_request_hcp.csv` - HCP Details** + +This file contains the specific attribute changes for an HCP. + +| Veeva Field Name | Reltio / HUB Attribute | Required (Add) | Description & Notes | +| :--- | :--- | :--- | :--- | +| `dcr_key` | Mongo Generated DCR ID | **Yes** | Links this record back to the manifest file. | +| `entity_key` | `refId.entityURI` or Reltio Crosswalk | **Yes** | The unique identifier for the HCP in Reltio. | +| `vid__v` | VEEVA Crosswalk Value | No (Yes for Change) | The unique Veeva ID for the HCP. Required for `CHANGE_REQUEST` types. | +| `first_name__v` | `firstName` | **Yes** | The HCP's first name. | +| `last_name__v` | `lastName` | **Yes** | The HCP's last name. | +| `middle_name__v` | `middleName` | No | The HCP's middle name. | +| `prefix__v` | `prefix` | No | Name prefix. Requires lookup (`HCPPrefix`). | +| `professional_title__v` | `title` | No | Professional title. Requires lookup (`HCPProfessionalTitle`). | +| `hcp_type__v` | `subTypeCode` | **Yes** | HCP subtype. Requires lookup (`HCPType`). | +| `hcp_status__v` | `StatusDetail` | No | Operational status of the HCP. Requires lookup (`HCPStatus`). | +| `gender__v` | `gender` | No | HCP's gender. Requires lookup (`HCPGender`). | +| `specialty_1__v` to `specialty_10__v` | `specialties` | **Yes (at least 1)** | Reltio's list of HCP specialties is flattened into 10 columns. At least one specialty is required for an `ADD_REQUEST`. | +| `medical_degree_1__v`, `medical_degree_2__v` | Not specified | No | HCP's medical degrees. Requires lookup (`HCPMedicalDegree`). | +| `primary_country__v` | `DCRRequest.country` | **Yes** | The two-letter ISO country code. | + +#### **4. `change_request_address.csv` - Address Details** + +This file contains address information, linked to either an HCP or an HCO. + +| Veeva Field Name | Reltio / HUB Attribute | Required (Add) | Description & Notes | +| :--- | :--- | :--- | :--- | +| `dcr_key` | Mongo Generated DCR ID | **Yes** | Links this record back to the manifest file. | +| `entity_key` | `refId.entityURI` of HCP/HCO | **Yes** | The Reltio identifier of the parent HCP or HCO to which this address belongs. | +| `address_key` | `address.refId` (`PfizerAddressID`) | **Yes** | A unique identifier for the address itself within Reltio. | +| `address_line_1__v` | `addressLine1` | **Yes** | The first line of the street address. | +| `address_line_2__v` | `addressLine2` | No | The second line of the street address. | +| `locality__v` | `city` | **Yes** | The city. | +| `administrative_area__v` | `stateProvince` | **Yes** | The state or province. Requires lookup (`AddressAdminArea`). | +| `postal_code__v` | `zip` | **Yes** | The postal code. | +| `country__v` | `country` | **Yes** | The two-letter ISO country code. | +| `address_type__v` | `addressType` | **Yes** | The type of address (e.g., Office, Home). Requires lookup (`AddressType`). | +| `address_status__v` | Static "A" | No | The status of the address, typically set to "A" for Active. | +| `vid__v` | VEEVA `SourceAddressID` | No (Yes for Change) | The unique Veeva ID for an existing address being updated. | + +#### **5. `change_request_parenthco.csv` - Affiliation/Relation Details** + +This file is used to define or update relationships between entities (HCP-HCO or HCO-HCO). + +| Veeva Field Name | Reltio / HUB Attribute | Required (Add) | Description & Notes | +| :--- | :--- | :--- | :--- | +| `dcr_key` | Mongo Generated DCR ID | **Yes** | Links this record back to the manifest file. | +| `parenthco_key` | Relation URI (`relationUri`) | **Yes** | The unique identifier for the relationship object in Reltio. | +| `child_entity_key` | Start Object's `entity_key` | **Yes** | The `entity_key` of the child entity in the relationship (e.g., the HCP). | +| `parent_entity_key` | End Object's `entity_key` | **Yes** | The `entity_key` of the parent entity in the relationship (e.g., the HCO). | +| `relationship_type__v` | Affiliation `type` | **Yes** | The type of relationship. This requires a lookup from the `RelationType` canonical values. | +| `is_primary_relationship__v` | Affiliation `primary` flag | No | A boolean flag indicating if this is the primary affiliation. | +| `vid__v` | VEEVA Relation ID | No (Yes for Change) | The unique Veeva ID for an existing relationship being updated. | \ No newline at end of file diff --git a/data/DCR_process_overview.md b/data/DCR_process_overview.md new file mode 100644 index 0000000..6c1edc0 --- /dev/null +++ b/data/DCR_process_overview.md @@ -0,0 +1,191 @@ +### **1. Overall DCR Process Overview** + +* **Purpose of DCR Process:** + The Data Change Request (DCR) process is designed to improve and validate the quality of existing data in source systems. It provides a formal mechanism for proposing, validating, and applying data changes. + +* **General DCR Process Flow:** + 1. A source system creates a proposal for a data change, known as a DCR or Validation Request (VR). + 2. The MDM HUB routes this request to the appropriate validation channel. + 3. Validation is performed either by internal Data Stewards (DS) within Reltio or by external, third-party validator services like OneKey or Veeva OpenData. + 4. A response is sent back, which includes metadata about the DCR's status (e.g., accepted, rejected) and the actual data profile update (payload) resulting from the processed DCR. + +* **High-Level Solution Architecture:** + The architecture involves source systems initiating DCRs through the **MDM HUB**. The HUB acts as a central router, directing requests to either **Reltio** for internal data steward review or to **Third-Party Validators** (OneKey, Veeva). The HUB is also responsible for receiving responses and facilitating the update of data in Reltio, often through ETL processes that handle payload delivery (e.g., via S3). + +*** + +### **2. System-Specific DCR Implementations** + +Here are the details for each system involved in the DCR process. + +--- +#### **OneKey (OK)** + +* **Role in DCR Process:** Functions as a third-party validator for DCRs. It receives validation requests, processes them, and returns the results. +* **Key Components:** DCR Service 2, OK DCR Service, OneKey Adapter, Publisher, Hub Store (Mongo DB), Manager (Reltio Adapter). +* **Actors Involved:** PforceRX, Data Stewards (in Reltio), Reltio, HUB, OneKey. +* **Core Process Details:** + * **PforceRX-initiated:** DCRs are created via the HUB's API. The HUB integrates with OneKey's API to submit requests (`/vr/submit`) and periodically checks for status updates (`/vr/trace`). + * **Reltio-initiated:** Data Stewards can suggest changes in Reltio and use the "Send to Third Party Validation" feature, which triggers a flow to submit a validation request to OneKey. Singleton entities created in Reltio can also trigger an automatic validation request to OneKey. +* **Integration Methods:** + * **API:** Real-time integration with OneKey via REST APIs (`/vr/submit`, `/vr/trace`). + * **File Transfer:** Data profile updates (payload) are delivered back to Reltio via CSV files on an S3 bucket, which are then processed by an ETL job. +* **DCR Types Handled:** Create, update, delete operations for HCP/HCO profiles; validation of newly created singleton entities; changes suggested by Data Stewards. + +--- +#### **Veeva OpenData (VOD)** + +* **Role in DCR Process:** Functions as a third-party validator, primarily handling DCRs initiated by Data Stewards from within Reltio. +* **Key Components:** DCR Service 2, Veeva DCR Service, Veeva Adapter, GMTF (Global Master Template & Foundation) jobs. +* **Actors Involved:** Data Stewards (in Reltio), HUB, Veeva OpenData. +* **Core Process Details:** + 1. Data Stewards in Reltio create DCRs using the "Suggest / Send to 3rd Party Validation" functionality. + 2. The HUB stores these requests in a Mongo collection (`DCRRegistryVeeva`). + 3. A scheduled job gathers these DCRs, packages them into ZIP files containing multiple CSVs, and places them on an S3 bucket. + 4. Files are synchronized from S3 to Veeva's SFTP server in batches (typically every 24 hours). + 5. Veeva processes the files and returns response files to an inbound S3 directory, which the HUB traces to update DCR statuses. +* **Integration Methods:** + * **File Transfer:** Asynchronous, batch-based communication via ZIP/CSV files exchanged through S3 and SFTP. +* **DCR Types Handled:** Primarily handles changes suggested by Data Stewards for existing profiles that need external validation from Veeva. + +--- +#### **IQVIA Highlander (HL) - *Decommissioned April 2025*** + +* **Role in DCR Process:** Acted as a wrapper to translate DCRs from a Veeva format into a format that could be loaded into Reltio for Data Steward review. +* **Key Components:** DCR Service (first version), IQVIA DCR Wrapper. +* **Actors Involved:** Veeva (on behalf of PforceRX), Reltio, HUB, IQVIA wrapper, Data Stewards. +* **Core Process Details:** + 1. Veeva uploaded DCR requests as CSV files to an FTP location. + 2. The HUB translated the Veeva CSV format into the IQVIA wrapper's CSV format. + 3. The IQVIA wrapper processed this file and created DCRs directly in Reltio. + 4. Data Stewards would then review, approve, or reject these DCRs within Reltio. +* **Integration Methods:** + * **File Transfer:** Communication was entirely file-based via S3 and SFTP. +* **DCR Types Handled:** Aggregated 21 specific use cases into six generic types: `NEW_HCP_GENERIC`, `UPDATE_HCP_GENERIC`, `DELETE_HCP_GENERIC`, `NEW_HCO_GENERIC`, `UPDATE_HCO_GENERIC`, `DELETE_HCO_GENERIC`. + +*** + +### **3. Key DCR Operations and Workflows** + +--- +#### **Create DCR** + +* **Description:** This is the main entry point for clients like PforceRx to create DCRs. The process validates the request, routes it to the correct target system (Reltio, OneKey, or Veeva), and creates the DCR. +* **Triggers:** An API call to `POST /dcr`. +* **Detailed Steps:** + 1. The DCR service receives and validates the request (e.g., checks for duplicate IDs, existence of referenced objects for updates). + 2. It uses a decision table to determine the target system based on attributes like country, source, and operation type. + 3. It calls the appropriate internal method to create the DCR in the target system (Reltio, OneKey, or Veeva). + 4. A corresponding DCR tracking entity is created in Reltio, and the state is saved in the Mongo DCR Registry. + 5. For Reltio-targeted DCRs, a workflow is initiated for Data Steward review. + 6. Pre-close logic may be applied to auto-accept or auto-reject the DCR based on the country. +* **Decision Logic/Rules:** A configurable decision table routes DCRs based on `userName`, `sourceName`, `country`, `operationType`, `affectedAttributes`, and `affectedObjects`. + +--- +#### **Submit Validation Request** + +* **Description:** This process submits validation requests for newly created "singleton" entities in Reltio to the OneKey service. +* **Triggers:** Reltio events (e.g., `HCP_CREATED`, `HCO_CREATED`) are aggregated in a time window (e.g., 4 hours). +* **Detailed Steps:** + 1. After an event aggregation window closes, the process performs several checks (e.g., entity is active, no existing OneKey crosswalk, no potential matches found via Reltio's `getMatches` API). + 2. If all checks pass, the entity data is mapped to a OneKey `submitVR` request. + 3. The request is sent to OneKey via `POST /vr/submit`. + 4. A DCR entity is created in Reltio to track the status, and the request is logged in Mongo. + +--- +#### **Trace Validation Request** + +* **Description:** This scheduled process checks the status of pending validation requests that have been sent to an external validator like OneKey or Veeva. +* **Triggers:** A timed scheduler (cron job) that runs every N hours. +* **Detailed Steps (OneKey Example):** + 1. The process queries the Mongo DCR cache for requests with a `SENT` status. + 2. For each request, it calls the OneKey `POST /vr/trace` API. + 3. It evaluates the `processStatus` and `responseStatus` from the OneKey response. + 4. If the request is resolved (`VAS_FOUND`, `VAS_NOT_FOUND`, etc.), the DCR status in Reltio and Mongo is updated to `ACCEPTED` or `REJECTED`. + 5. If the response indicates a match was found but an OK crosswalk doesn't yet exist in Reltio, a new workflow is triggered for Data Steward manual review (`DS_ACTION_REQUIRED`). + +--- +#### **Data Steward Response** + +* **Description:** This process handles the final outcome of a DCR that was reviewed internally by a Data Steward in Reltio. +* **Triggers:** Reltio change request events (`CHANGE_REQUEST_CHANGED`, `CHANGE_REQUEST_REMOVED`) that **do not** have the `ThirdPartyValidation` flag. +* **Detailed Steps:** + 1. The process consumes the event from Reltio. + 2. It checks the `state` of the change request. + 3. If the state is `APPLIED` or `REJECTED`, the corresponding DCR entity in Reltio and the record in Mongo are updated to a final status of `ACCEPTED` or `REJECTED`. + +--- +#### **Data Steward OK Validation Request** + +* **Description:** This process handles DCRs created by a Data Steward in Reltio using the "Suggest" and "Send to Third Party Validation" features, routing them to an external validator like OneKey. +* **Triggers:** Reltio change request events that **do** have the `ThirdPartyValidation` flag set to `true`. +* **Detailed Steps:** + 1. The HUB retrieves the "preview" state of the entity from Reltio to see the suggested changes. + 2. It compares the current entity with the preview to calculate the delta. + 3. It maps these changes to a OneKey `submitVR` request. Attribute removals are sent as a comment due to API limitations. + 4. The request is sent to OneKey. + 5. Upon successful submission, the original change request in Reltio is programmatically rejected (since the validation is now happening externally), and a new DCR entity is created for tracking the OneKey validation. + +*** + +### **4. Data Comparison and Mapping Details** + +* **OneKey Comparator:** + When a Data Steward suggests changes, the HUB compares the current Reltio entity with the "preview" state to send to OneKey. + * **Simple Attributes** (e.g., `FirstName`): Values are compared for equality. The suggested value is taken if different. + * **Complex Attributes** (e.g., `Addresses`, `Specialties`): Nested attributes are matched using their Reltio URI. New nested objects are added, and changes to existing ones are applied. + * **Mandatory Fields:** For HCP, `LastName` and `Country` are mandatory. For HCO, `Country` and `Addresses` are mandatory. + * **Attribute Removal:** Due to API limitations, removing an attribute is not done directly but by generating a comment in the request, e.g., "Please remove attributes: [Address: ...]". + +* **Veeva Mapping:** + The process of mapping Reltio canonical codes to Veeva's source-specific codes is multi-layered. + 1. **Veeva Defaults:** The system first checks custom CSV mapping files stored in configuration (`mdm-veeva-dcr-service/defaults`). These files define direct mappings for a specific country and canonical code (e.g., `IN;SP.PD;PD`). + 2. **RDM Lookups:** If no default is found, it queries RDM (via a Mongo `LookupValues` collection) for the canonical code and looks for a `sourceMapping` where the source is "VOD". + 3. **Veeva Fallback:** If no mapping is found, it consults fallback CSV files (`mdm-veeva-dcr-service/fallback`) for certain attributes (e.g., `hco-specialty.csv`). A regular expression is often used to extract the correct code. If all else fails, a question mark (`?`) is used as the default fallback. + +*** + +### **5. Status Management and Error Handling** + +* **DCR Statuses:** + The system uses a combination of statuses to track the DCR lifecycle. + +| RequestStatus | DCRStatus | Internal Cache Status | Description | +| :--- | :--- | :--- | :--- | +| REQUEST\_ACCEPTED | CREATED | SENT\_TO\_OK | DCR sent to OneKey, pending DS review. | +| REQUEST\_ACCEPTED | CREATED | SENT\_TO\_VEEVA | DCR sent to Veeva, pending DS review. | +| REQUEST\_ACCEPTED | CREATED | DS\_ACTION\_REQUIRED | DCR pending internal DS validation in Reltio. | +| REQUEST\_ACCEPTED | ACCEPTED | ACCEPTED | DS accepted the DCR; changes were applied. | +| REQUEST\_ACCEPTED | ACCEPTED | PRE\_ACCEPTED | Pre-close logic automatically accepted the DCR. | +| REQUEST\_REJECTED | REJECTED | REJECTED | DS rejected the DCR. | +| REQUEST\_REJECTED | REJECTED | PRE\_REJECTED | Pre-close logic automatically rejected the DCR. | +| REQUEST\_FAILED | - | FAILED | DCR failed due to a validation or system error. | + +* **Error Codes:** + +| Error Code | Description | HTTP Code | +| :--- | :--- | :--- | +| `DUPLICATE_REQUEST` | The `extDCRRequestId` has already been registered. | 403 | +| `NO_CHANGES_DETECTED` | The request contained no changes compared to the existing entity. | 400 | +| `VALIDATION_ERROR` | A referenced object (HCP/HCO) or attribute could not be found. | 404 / 400 | + +* **DCR State Changes:** + A DCR begins in an `OPEN` state. It is then sent to a target system, moving to states like `SENT_TO_OK`, `SENT_TO_VEEVA`, or `DS_ACTION_REQUIRED`. Pre-close logic can immediately move it to `PRE_ACCEPTED` or `PRE_REJECTED`. Following data steward review (either internal or external), the DCR reaches a terminal state of `ACCEPTED` or `REJECTED`. If an error occurs, it moves to `FAILED`. + +*** + +### **6. Technical Artifacts and Infrastructure** + +* **Mongo Collections:** + * **`DCRRegistryONEKEY` / `DCRRegistryVeeva`:** System-specific collections to store and track the state of DCRs sent to OneKey and Veeva, respectively. They hold the mapped request data and trace the response. + * **`DCRRequest` / `DCRRegistry`:** General-purpose collections for tracking DCRs, their metadata, and overall status within the HUB. + * **`DCRRegistryVeeva`:** Specifically stores Veeva-bound DCRs, including the raw CSV line content, before they are submitted in a batch. + +* **File Specifications:** + * **Veeva Integration:** Uses `ZIP` files containing multiple `CSV` files (`change_request.csv`, `change_request_hcp.csv`, `change_request_address.csv`, etc.). Response files follow a similar pattern and are named with the convention `_DCR_Response_.zip`. + * **Highlander Integration:** Used `CSV` files for requests. + +* **Event Models:** + The system uses internal Kafka events to communicate DCR status changes between components. + * **`OneKeyDCREvent`:** Published by the trace process after receiving a response from OneKey. It contains the DCR ID and the `OneKeyChangeRequest` details (`vrStatus`, `vrStatusDetail`, comments, validated IDs). + * **`VeevaDCREvent`:** Published by the Veeva trace process. It contains the DCR ID and `VeevaChangeRequestDetails` (`vrStatus`, `vrStatusDetail`, comments, new Veeva IDs). \ No newline at end of file diff --git a/data/HUB_DCR_output_2.md b/data/HUB_DCR_output_2.md new file mode 100644 index 0000000..560e11a --- /dev/null +++ b/data/HUB_DCR_output_2.md @@ -0,0 +1,2227 @@ +# DCR flows + +**Page ID:** 415205424 +**Page Link:** /display/GMDM/DCR+flows + +**Overview** +------------ + +DCR (Data Change Request) process helps to improve existing data in source systems. Proposal for change is being created by source systems a as DCR object (sometimes also called VR - Validation Request) which is usually being routed by MDM HUB to DS (Data Stewards) either in Reltio or in Third party validators (OneKey, Veeva OpenData). Response is provided twofold: + +* response for specific DCR - metadata +* profile data update as a direct effect of a DCR processing - payload + +**General DCR process flow** +---------------------------- + + + +**High level solution architecture for DCR flow** +------------------------------------------------- + + + +Source: Lucid + +**Solution for OneKey (OK)** +---------------------------- + +**** + +**Solution for Veeva OpenData (VOD)** +------------------------------------- + +**** + +### **Architecture highlights** + +* **Actors involved**: PforceRX, Reltio, HUB, OneKey +* **Key components**: DCR Service 2 (second version) for AMER, EMEA, APAC, US tenants +* **Process details**: + + DCRs are created directly by PforceRx using DCR's HUB API + + PforceRx checks for DCR status updates every 24h → finds out which DCRs has been updated (since last check 24h ago) and the pulls details from each one with /dcr/\_status + + Integration with OneKey is realized by APIs - DCRs are created with /vr/submit and their status is verified every 8h with /vr/trace + + Data profile updates (payload) are being delivered via CSV and S3 and ETLed (VOD batch) to Reltio with Deloitte's help + + DCRRegistry & DCRRegistryVeeva collections are used in Mongo for tracking purposes + +### **Architecture highlights** + +* **Actors involved**: Data Stewards in Reltio, HUB, Veeva OpenData (VOD) +* **Key components**: DCR Service 2 (second version) for AMER, EMEA, APAC, US tenants +* **Process details**: + + DCRs are created by Data Stewards (DSRs) in Reltio via Suggest / Send to 3rd Party Validation - input for DSRs is being provided by reports from PforceRx + + Communication with Veeva via S3<>SFTP and synchronization GMTF jobs. DCRs are sent and received in batches every 24h + + DCRs metadata is being exchanged via multiple CSV files ZIPed + + Data profile updates (payload) are being delivered via CSV and S3 and ETLed (VOD batch) to Reltio with Deloitte's help + + DCRRegistry & DCRRegistryONEKEY collections are used in Mongofor tracking purposes + +**Solution for IQVIA Highlander (HL)** +-------------------------------------- + +**** + +**Solution for OneKey on GBLUS - sources ICEU, Engage, GRV** +------------------------------------------------------------ + +### **Architecture highlights** + +* **Actors involved**: Veeva on behalf of PforceRX, Reltio, HUB, IQVIA wrapper +* **Key components**: DCR Service (first version) for GBLUS tenant +* **Process details**: + + DCRs are created by sending CSV requests by Veeva - based on information acquired from PforceRx + + Integration HUB <> Veeva → via files and S3<>SFTP. HUB confirms DCR creation by returning file reports back to Veeva + + Integration HUB <> IQVIA wrapper → via files and S3 + + HUB is responsible for translation of Veeva DCR CSV format to IQVIA CSV wrapper which then creates DCR in Reltio + + Data Stewards approve or reject the DCRs in Reltio which updates data profiles accordingly. + + PforceRx receives update about changes in Reltio + + DCRRequest collection is used in Mongo for tracking purposes + +### **Architecture highlights (draft)** + +* **Actors involved**: HUB, IQVIA wrapper +* **Key components**: DCR Service (first version) for GBLUS tenant +* **Process details**: + + POST events from sources are captured - some of them are translated to direct DCRs, some of them are gathered and then pushed via flat files to be transformed into DCRs to OneKey +--- + +# DCR generation process (China DCR) + +**Page ID:** 164470008 +**Page Link:** /pages/viewpage.action?pageId=164470008 + +The gateway supports following DCR types: + +* NewHCP – created when new HCP is registered in Reltio and requires external validation +* NewHCOL1 – created when HCO Level 1 not found in Reltio +* NewHCOL2 – created when HCO Level 2 not found in Reltio +* MultiAffil – created when a profile has multiple affiliations + +DCR generation processes are handled in two steps: + +1. During HCP modification – if initial activation criteria are met, then a DCR request is generated and published to KAFKA *-gw-dcr-requests* topic. +2. In the next step, the internal Camel route *DCRServiceRoute* reads requests generated from the topic and processes as follows: + 1. checks if the time specified by delayPrcInSeconds elapsed since request generation – it makes sure that Reltio batch match process has finished and newly inserted profiles merge with the existing ones. + 2. checks if an entity, that caused DCR generation, still exists; + 3. checks full activation criteria (table below) on the latest state of the target entity, if criteria are not met then the request is closed + 4. creates DCR in Reltio + 5. updates external info + 6. creates PfizerDataChangeRequest entity in Reltio for tracking and exporting purposes. +3. Created DCRs are exported by the Informatica ETL process managed by IQIVIA +4. DCR applying process (reject/approve actions) are executed through MDM HUB DCR response API executed by the external app manged by MDE team. + +The table below presents DCR activation criteria handled by system. + +| | | | | | +| --- | --- | --- | --- | --- | +| **Table 9.** DCR activation criteria | | | | | +| **Rule** | **NewHCP** | **MultiAffiliation** | **NewHCOL2** | **NewHCOL1** | +| Country in | CN | CN | CN | CN | +| Source in | GRV | GRV, MDE, FACE, EVR, CN3RDPARTY | GRV, FACE, CN3RDPARTY | GRV, FACE, CN3RDPARTY | +| ValidationStatus in | pending, partial-validated *or, if merged:* OV: notvalidated, GRV nonOV: pending/partial-validated | validated, pending | validated, pending | validated, pending | +| SpeakerStatus in | enabled, null | enabled, null | enabled, null | enabled, null | +| Workplaces count | | >1 | | | +| Hospital found | true | true | false | true | +| Department found | true | true | | false | +| Similar DCR created in the past | false | false | false | false | + +### Update: December 2021 + +* NewHCP DCR is now created if *ValidationStatus* is *pending* or *partial-validated* +* NewHCP DCR is also created if OV ValidationStatus is *notvalidated*, but most-recently updated GRV crosswalk provides non-ov *ValidationStatus as pending* or *partial-validated* - in case HCP gets merged into another entity upon creation/modification: +* DCR request processing history is now available in Kibana via Transaction Log - dashboard API Calls, transaction type "*CreateDCRRoute*" +* DCR response processing history (DCR approve/reject flow) is now available in Kibana via Transaction Log - dashboard API Calls, transaction type "*DCRResponse*" + + +--- + +# HL DCR [Decommissioned April 2025] + +**Page ID:** 164470085 +**Page Link:** /pages/viewpage.action?pageId=164470085 + +Contacts +-------- + +| Vendor | Contact | +| --- | --- | +| PforceRX | DL-PForceRx-SUPPORT@pfizer.com | +| IQVIA (DCR Wrapper) | Pfizer-MDM-Support@iqvia.com | + +As a part of Highlander project, the DCR processing flow was created which realizes following scenarios: + +1. Update HCP account details i.e. specialty, address, name (different sources of elements), +2. Add new HCP account with primary affiliation to an existing organization, +3. Add new HCP account with a new business account, +4. Update HCP and add affiliation to a new HCO, +5. Update HCP account details and remove existing details i.e. birth date, national id, …, +6. Update HCP account and add new non primary affiliation to an existing organization, +7. Update HCP account and add new primary affiliation to an existing organization, +8. Update HCP account inactivate primary affiliation. Person account has more than 1 affiliation, +9. Update HCP account inactivate non primary affiliation. Person account has more than 1 affiliation, +10. Inactivate HCP account, +11. Update HCP and add a private address, +12. Update HCP and update existing private address, +13. Update HCP and inactivate a private address, +14. Update HCO details i.e. address, name (different sources of elements), +15. Add new HCO account, +16. Update HCO and remove details, +17. Inactivate HCO account, +18. Update HCO address, +19. Update HCO and add new address, +20. Update HCO and inactivate address, +21. Update HCP's existing affiliation. + +Above cases has been aggregated into six generic types in internal HUB model: + +1. NEW\_HCP\_GENERIC - represents cases when the new HCP object is created with or without affiliation to HCO, +2. UPDATE\_HCP\_GENERIC - aggregates cases when the existing HCP object is changed, +3. DELETE\_HCP\_GENERIC - represents the case when HCP is deactivating, +4. NEW\_HCO\_GENERIC - aggregates scenarios when new HCO object is created with or without affiliations to parent HCO, +5. UPDATE\_HCO\_GENERIC - represents cases when existing HCO object is changing, +6. DELETE\_HCO\_GENERIC - represents the case when HCO is deactivating. + +General Process Overview +------------------------ + +**Process steps:** + +1. Veeva uploads DCR request file to FTP location, +2. PforceRx Channel component downloads the DCR request file, +3. PforceRx Channel validates and maps each DCR requests to internal model, +4. PforceRx Channel sends the request to DCR Service, +5. DCR Service process the request: validating, enriching and mapping to Iqvia DCR Wrapper, +6. PforceRx Channel prepares the report file containing technical status of DCR processing - at this time, report will contain only requests which don't pass the validation, +7. Scheduled process in DCR Service, prepares the Wrapper requests file and uploads this to S3 location. +8. DCR Wrapper processes the file: creating DCRs in Reltio or rejecting the request due to errors. After that the response file is published to s3 location, +9. DCR Service downloads the response and updates DCRs status, +10. Scheduled process in PforceRx Channel gets DCR requests and prepares next technical report - at this time the report has technical status which comes from DCR Wrappper, +11. DCRs that was created by DCR Wrapper are reviewed by Data Stewards. DCR can be accepted or rejected, +12. After accepting or rejecting DCR, Reltio publishes the message about this event, +13. DCR Service consumes the message and updates DCR status, +14. PforceRx Channel gets DCR data to prepare a response file. The response file contains the final status of DCRs processing in Reltio. + +Veeva DCR request file specification +------------------------------------ + +The specification is available at following location: + +https://pfizer-my.sharepoint.com/:x:/r/personal/chinj2\_pfizer\_com/Documents/Mig%20In-Prog/Highlander/PMO/09%20Integration/LATAM%20Reltio%20DCR/DCR\_Reltio\_T144\_Field\_Mapping\_Reltio.xlsx + +DCR Wrapper request file specification +-------------------------------------- + +The specification is available at following link: + +https://pfizer.sharepoint.com/:x:/r/sites/HLDCR/Shared%20Documents/ReltioCloudMDM\_LATAM\_Highlander\_DCR\_DID\_PFIZER\_\_DEVMapping\_v2.1.xlsx +--- + +# OK DCR flows (GBLUS) + +**Page ID:** 164469877 +**Page Link:** /pages/viewpage.action?pageId=164469877 + +Description +=========== + +The process is responsible for creating DCRs in Reltio and starting Change Requests Workflow for singleton entities created in Reltio. During this process, the communication to IQVIA OneKey VR API is established.  SubmitVR operation is executed to create a new Validation Request. The TraceVR operation is executed to check the status of the VR in OneKey. All DCRs are saved in the dedicated collection in HUB Mongo DB, required to gather metadata and trace the changes for each DCR request. Some changes can be suggested by the DS using "Suggest" operation in Reltio and "Send to Third Party Validation" button, the process "Data Steward OK Validation Request" is processing these changes and sends them to the OneKey service. + +The process is divided into 4 sections: + +1. Submit Validation Request +2. Trace Validation Request +3. Data Steward Response +4. Data Steward OK Validation Request + +The below diagram presents an overview of the entire process. Detailed descriptions are available in the separated subpages. + +Flow diagram +============ + + + +Model diagram +============= + + + +Steps +===== + +* SubmitVR + + The process of submitting VR is triggered by the Reltio events. The process aggregates events in a time window and once the window is closed the processing is started. + + During SubmitVR process checks are executed, getMatches operation in Relto is invoked to verify potential matches for the singleton entities. + + Once all checks are correct new submitVR request is created in OneKey and DCR is saved in Reltio and in Mongo Cache. +* TraceVR + + The process of tracing VR is triggered each  hours on Mongo DCR cache collection. + + For each DCR the traceVR operation is executed in OneKey to verify the current status for the specific validation request. + + Once the checks are correct the DCR is updated in Reltio and in Mongo Cache. +* Data Steward Response + + The process is responsible for gathering changes on Change Requests objects from Reltio, the process is only accepting events without the ThirdPartyValidation flag + + Based on the received change invoked by the Data Steward DCR is updated in Reltio and in Mongo Cache +* Data Steward OK Validation Request + + The process is responsible for processing changes on Change Requests objects from Reltio, the process is only accepting events with the ThirdPartyValidation flag. This event is generated after DS clicks the "Send to Third Party Validation" button in Reltio. + + The DS is "Suggesting" changes on the specified profile, these changes are next sent to HUB with the DCR event. The changes are not visible in Retlio, it is just a container that keeps the changes. + + HUB is retrieving the "Preview" state from Reltio and calculating the changes that will send to OneKey WebService using submitVR operation + + After successful submitVR response HUB is closing/rejecting the existing DCR in Reltio. The \_reject operation has to be invoked on the current DCR in Reltio because the changes should no be applied to the profile. Changes are now validating in the OneKey system, and appropriate steps will be taken in the next phase (export changed data to Reltio or reject suggestion). + +Triggers +======== + +Described in the separated sub-pages for each process. + +Dependent components +==================== + +Described in the separated sub-pages for each process. +--- + +# Data Steward OK Validation Request + +**Page ID:** 172306908 +**Page Link:** /display/GMDM/Data+Steward+OK+Validation+Request + +Description +=========== + +The process the DS suggested changes based on the Change Request events received from Reltio(publishing) that are marked with the `ThirdPartyValidation` flag. The "suggested" changes are retrieved using the "preview" method and send to IQVIA OneKey or Veeva OpenData for validation. After successful `submitVR` response HUB is closing/rejecting the existing DCR in Reltio and additionally creates a new DCR object with relation to the entity in Reltio for tracking and status purposes.  + +**Because of the ONEKEY interface limitation, removal of attributes is send to IQVIA as a comment.** + +Flow diagram +============ + + + +Steps +===== + +* Event publisher publishes full enriched events to `$env-internal-[onekeyvr|thirdparty]-ds-requests-in`: DCR\_CHANGED("CHANGE\_REQUEST\_CHANGED") and DCR\_CREATED("CHANGE\_REQUEST\_CREATED") +* Only events with ExternalInfo and ThirdPartyValidation flag set to true and the Change Requests status equal to AWAITING\_REVIEW are accepted in this process, otherwise, the event is rejected and processing ends. +* HUB DCR Cache is verified if any ReltioDCR requests exist and are not in a FAILED status, then processing goes to the next step. +* DCR request that contains targetChangeRequest is enriched with the current Entity data using HUB Cache +* Veeva specific: The entity is checked, If no VOD crosswalk exists, then "golden profile" parameters should be used with below logic +* The entity is checked, If active [ONEKEY|VOD] crosswalk exists the following steps are executed: + + The suggested state of the entity is retrieved from Reltio using the `getEntityWithChangeRequests` operation (parameters - entityUri and the changeRequestId from the DCR event). + + Current Entity and Preview Entity are compared using the following rules: (full attributes that are part of comparing process are described **here**) + - Simple attributes (like FirstName/LastName): + * Values are compared using the equals method.  + + if differences are found the suggested value is taken. + + If no differences are found + - for mandatory, the current value is taken + - for optional, the none value is taken (null) + - Complex attributes (like Specialties/Addresses): + * Whole nested attributes are matched using Reltio "uri" attributes key. + * If there is a new Specialty/Address, the new suggested nested attribute is taken + + Veeva specific: If there is a new Specialty/Addresses/Phone/Email/Medical degree\*/HCP Focus area\*, the new suggested nested attribute is taken. Since Veeva uses flat structure for these attributes, we need to calculate specialty attribute number (like specialty\_5\_\_v) to use when sending request. Attribute number = count of existing attributes +1. + * If there is no new Specialty/Address and there is a change in the existing attribute, the suggested nested change is taken. If there are multiple suggested changes, the one with the highest Rank is taken. + * If there are no changes + + for mandatory, the current nested attribute that is connected with the ONEKEY crosswalk is taken. + + for optional, the none nested attribute is taken (no need to send) + - Contact Affiliations / OtherHCOtoHCOAffiliation: + * If there are no changes, return current list + * If there is new Contact Affiliation with ONEKEY crosswalk, add it to current list + - Additional checks: + * If there are changes associated with the other source (different than the [ONEKEY|VOD]), then these changes are ignored and the VR is saved in Reltio with comment listing what attributes were ignored e.g.: "Attributes: [YoB: 1956], [Email: engagetest123@test.com] ignored due to update on non-[onekey|VOD] attribute." + * If attribute associated with [ONEKY|VOD] source is removed, a comment specifying what should be removed on [ONEKY|VOD] side is generated and sent to [ONEKY|VOD], e.g.: "Please remove attributes: [Address: 10648 Savannah Plantation Ct, 32832, Orlando, United States]." + + `DCRRequest` object is created in Mongo for the flow state recording and generation of the new unique DCR ID for validation requests and data tracing. + - | DCR cache attributes | Values for IQVIA | Values for OK | Values for Veeva (R1) | + | --- | --- | --- | --- | + | type | OK\_VR | PFORCERX\_DCR | RELTIO\_SUGGEST | + | status | DCRRequestStatusDetails (DCRRequestStatus.NEW, currentDate) | | | + | createdBy | onekey-dcr-service | User which creates DCR via Suggest button in Reltio | User which creates DCR via Suggest button in Reltio | + | date | now | | | + | SendTo3PartyValidation | true (flag that indicates the DCR objects created by this process) | | | + + Calculated changes are mapped to the OneKey `submitVR` Request and it's submitted using API REST method POST /vr/submit. + - Veeva specific:  submitting DCR request to Veeva requires creation of ZIPed CSV files with agreed structure and placed on S3 bucket + - If the submission is successful then: + - `DCRRequest.status` is updated to `SENT`with [OK|VOD] request and response details + * **DCR**entity is created in Reltio and the relation between the processed entity and the DCR entity + + Reltio source name (crosswalk.type): *DCR* + + Reltio relation type: HCPtoDCR or HCOtoDCR (depending on the object type) + + DCR entity attributes: + + | DCR entity attributes | Mapping for OneKey | Mapping for Veeva | + | --- | --- | --- | + | DCRID | OK VR Reqeust Id (cegedimRequestEid) | ID assigned by MDM HUB | + | EntityURI | the processed entity URI | | + | VRStatus | "OPEN" | | + | VRStatusDetail | "SENT" | | + | Comments | optionally comments | | + | SentDate | current time | | + | SendTo3PartyValidation | true | | + - Otherwise (FAILED) + - `DCRRequest.status` is updated to `FAILED`with OK request and exception response details + - **DCR**entity is created in Reltio and the relation between the processed entity and the DCR entity + * Reltio source name (crosswalk.type): `DCR` + * Reltio relation type: `HCPtoDCR`or `HCOtoDCR`(depending on the object type) + * DCR entity attributes: + * | DCR entity attributes | Mapping | + | --- | --- | + | DCRID | OK VR Reqeust Id (cegedimRequestEid) | + | EntityURI | the processed entity URI | + | VRStatus | "CLOSED" | + | VRStatusDetail | "FAILED" | + | Comments | ``` ONEKEY service failed [exception details] ``` | + | SentDate | current time | + | SendTo3PartyValidation | true | + + The current DCR object in Reltio is closed using the \_reject operation - POST - `/changeRequests//_reject` +* Otherwise, If ONEKEY crosswalk does not exist, or the ONEKEY crosswalk is soft-deleted, or entity is EndDated: the following steps are executed: + + DCRRequest object is created in Mongo for the flow state recording and generation of the new unique DCR ID for validation requests and data tracing. + - | DCR cache attributes | values | + | --- | --- | + | type | DCRType.OK\_VR | + | status | DCRRequestStatusDetails (DCRRequestStatus.NEW, currentDate) | + | created by | onekey-dcr-service | + | date | now | + | SendTo3PartyValidation | *true (flag that indicates the DCR objects created by this process)* | + + *DCRRequest.status* is updated to *FAILED*and comment "No OK crosswalk available" + + ***DCR***entity is created in Reltio and the relation between the processed entity and the DCR entity + - Reltio source name (crosswalk.type): *DCR* + - Reltio relation type: *HCPtoDCR*or *HCOtoDCR*(depending on the object type) + - DCR entity attributes: + - | DCR entity attributes | Mapping | + | --- | --- | + | DCRID | OK VR Reqeust Id (cegedimRequestEid) | + | EntityURI | the processed entity URI | + | VRStatus | "*CLOSED"* | + | VRStatusDetail | "*REJECTED*" | + | Comments | ``` No ONEKEY crosswalk available ``` | + | CreatedBy | *MDM HUB* | + | SentDate | current time | + | SendTo3PartyValidation | true | + + The current DCR object in Reltio is closed using the \_reject operation - POST - /changeRequests//\_reject +* END + +ONEKEY Comparator (suggested changes) +------------------------------------- + +**HCP** + +| Reltio Attribute | ONEKEY attribute | mandatory type | attribute type | +| --- | --- | --- | --- | +| ``` FirstName ``` | individual.firstName | optional | simple value | +| LastName | individual.lastName | mandatory | simple value | +| Country | ``` isoCod2 ``` | mandatory | simple value | +| Gender | individual.genderCode | optional | simple lookup | +| Prefix | individual.prefixNameCode | optional | simple lookup | +| Title | individual.titleCode | optional | simple lookup | +| MiddleName | individual.middleName | optional | simple value | +| YoB | individual.birthYear | optional | simple value | +| Dob | individual.birthDay | optional | simple value | +| TypeCode | individual.typeCode | optional | simple lookup | +| PreferredLanguage | individual.languageEid | optional | simple value | +| ``` WebsiteURL ``` | individual.website | optional | simple value | +| Identifier value 1 | individial.externalId1 | optional | simple value | +| Identifier value 2 | individial.externalId2 | optional | simple value | +| ``` Addresses[] ``` | address.country address.city address.addressLine1 address.addressLine2 address.Zip5 | mandatory | complex (nested) | +| ``` Specialities[] ``` | individual.speciality1 / 2 / 3 | optional | complex (nested) | +| ``` Phone[] ``` | individual.phone | optional | complex (nested) | +| ``` Email[] ``` | individual.email | optional | complex (nested) | +| ``` Contact Affiliations[] ``` | workplace.usualName workplace.officialName workplace.workplaceEid | optional | Contact Affiliation | +| ONEKEY crosswalk | ``` individual.individualEid ``` | mandatory | ID | + +**HCO** + +| Reltio Attribute | ONEKEY attribute | mandatory type | attribute type | +| --- | --- | --- | --- | +| ``` Name ``` | workplace.usualName workplace.officialName | optional | simple value | +| Country | ``` isoCod2 ``` | mandatory | simple value | +| ``` OtherNames.Name ``` | workplace.usualName2 | optional | complex (nested) | +| TypeCode | workplace.typeCode | optional | simple lookup | +| ``` WebisteWebsiteURL ``` | workplace.website | optional | complex (nested) | +| ``` Addresses[] ``` | address.country address.city address.addressLine1 address.addressLine2 address.Zip5 | mandatory | complex (nested) | +| ``` Specialities[] ``` | workplace.speciality1 / 2 / 3 | optional | complex (nested) | +| ``` Phone[] (!FAX) ``` | workplace.telephone | optional | complex (nested) | +| ``` Phone[] (FAX) ``` | workplace.fax | optional | complex (nested) | +| ``` Email[] ``` | workplace.email | optional | complex (nested) | +| ONEKEY crosswalk | ``` workplace.workplaceEid ``` | mandatory | ID | + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Events incoming | mdm-onekey-dcr-service:ChangeRequestStream | process publisher full change request events in the stream that contain ThirdPartyValidation flag | realtime: events stream processing | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| OK DCR Service | Main component with flow implementation | +| Veeva DCR Service | Main component with flow implementation | +| Publisher | Events publisher generates incoming events | +| Hub Store | DCR and Entities Cache | +--- + +# Data Steward Response + +**Page ID:** 164469841 +**Page Link:** /display/GMDM/Data+Steward+Response + +Description +=========== + +The process updates the DCR's based on the Change Request events received from Reltio(publishing). Based on the Data Steward decision the *state* attribute contains relevant information to update DCR status. + +Flow diagram +============ + + + +Steps +===== + +* Event publisher publishes simple events to `$env-internal-[onekeyvr|veeva]-change-requests-in`: DCR\_CHANGED("CHANGE\_REQUEST\_CHANGED") and DCR\_REMOVED("CHANGE\_REQUEST\_REMOVED") +* Only the events without the ThirdPartyValidation flag are accepted, otherwise, the event is Rejected and the process is ended. +* Events are processed in the Stream and based on the *targetChangeRequest.state* attribute decision is made + + If the state is APPLIED or REJECTS, DCR is retrieved from the cache based on the *changeRequestURI* + - If DCR exists in Cache The status in Reltio is updated + + | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus | *CLOSED* | + | VRStatusDetail | state: APPLIED → ACCEPTED state: REJECTED → REJECTED | + - Otherwise, the events are rejected and the transaction is ended + + Otherwise, the events are rejected and the transition is ended. + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Events incoming | mdm-onekey-dcr-service:OneKeyResponseStream mdm-veeva-dcr-service:veevaResponseStream | process publisher full change request events in stream | realtime: events stream processing | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| OK DCR Service | Main component with flow implementation | +| Veeva DCR Service | Main component with flow implementation | +| Publisher | Events publisher generates incoming events | +| Hub Store | DCR and Entities Cache | +--- + +# Submit Validation Request + +**Page ID:** 164469875 +**Page Link:** /display/GMDM/Submit+Validation+Request + +Description +=========== + +The process of submitting new validation requests to the OneKey service based on the Reltio change events aggregated in time windows. During this process, new DCRs are created in Reltio. + +Flow diagram +============ + + + +Steps +===== + +* Event publisher publishes simple events to $env-internal-onekeyvr-in including HCP\_\*, HCO\_\*, ENTITY\_MATCHES\_CHANGED +* Events are aggregated in a time window (recommended the window length 4 hours) and the last event is returned to the process after the window is closed. +* Simple events are enriched with the Entity data using HUB Cache +* Then, the following checks are executed + + check if at least one crosswalk create date is equal or above for a given source name and cut off date specified in configuration - section submitVR/crosswalkDecisionTables + + check if entity attribute values match specified in configuration + + check if there is no valid DCR created for the entity + + check if the entity is active + + check if the OK crosswalk doesn't exist after the full entity retrieval from the HUB cache + + match category is not 99 + + GetMatches operation from Reltio returns 0 potential matches +* If any check is negative then the process is aborted. +* DCRRequest object is created in Mongo for the flow state recording and generation of the new unique DCR ID for validation request and data tracing. +* The entity is mapped to OK VR Request and it's submitted using API REST method POST /vr/submit. +* If the submission is successful then: + + *DCRRequest.status* is updated to *SENT* with OK request and response details + + ***DCR*** entity is created in Reltio and the relation between the processed entity and the DCR entity + - Reltio source name (crosswalk.type): *DCR* + - Reltio relation type: *HCPtoDCR*or *HCOtoDCR*(depending on the object type) + - DCR entity attributes: + + | DCR entity attributes | Mapping | + | --- | --- | + | DCRID | OK VR Reqeust Id (cegedimRequestEid) | + | EntityURI | the processed entity URI | + | VRStatus" | "*OPEN"* | + | VRStatusDetail | "*SENT*" | + | CreatedBy | *MDM HUB* | + | SentDate | current time | +* Otherwise *FAILED*status is recorded in *DCRRequest*with an OK error response. + + *DCRRequest.status* is updated to *FAILED* with OK request and exception response details + + ***DCR*** entity is created in Reltio and the relation between the processed entity and the DCR entity + - Reltio source name (crosswalk.type): *DCR* + - Reltio relation type: *HCPtoDCR* or *HCOtoDCR* (depending on the object type) + - DCR entity attributes: + + | DCR entity attributes | Mapping | + | --- | --- | + | DCRID | OK VR Reqeust Id (cegedimRequestEid) | + | EntityURI | the processed entity URI | + | VRStatus | "*CLOSED"* | + | VRStatusDetail | "*FAILED*" | + | Comments | ONEKEY service failed [exception details] | + | CreatedBy | *MDM HUB* | + | SentDate | current time | + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Events incoming | mdm-onekey-dcr-service:OneKeyStream | process publisher simple events in stream | events stream processing with 4h time window events aggregation | +| **OUT** API request | one-key-client:OneKeyIntegrationService.submitValidation | submit VR request to OneKey | invokes API request for each accepted event | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| OK DCR Service | Main component with flow implementation | +| Publisher | Events publisher generates incoming events | +| Manager | Reltio Adapter for getMatches and created operations | +| OneKey Adapter | Submits Validation Request | +| Hub Store | DCR and Entities Cache | + +Mappings +======== + +Reltio → OK mapping file: onkey\_mappings.xlsx + +OK mandatory / required fields: VR - Business Fields Requirements(Pfizer).xlsx + +OneKey Documentation +==================== + + +--- + +# Trace Validation Request + +**Page ID:** 164469983 +**Page Link:** /display/GMDM/Trace+Validation+Request + +Description +=========== + +The process of tracing the VR changes based on the OneKey VR changes. During this process HUB, DCR Cache is triggered every hour for SENT DCR's and check VR status using OneKey web service. After verification DCR is updated in Reltio or a new Workflow is started in Reltio for the Data Steward manual validation. + +Flow diagram +============ + + + +Steps +===== + +* Every N  hours OK *VR requests* with status *SENT* are queried in *DCRRequests* store*.* +* For each open requests, its status is checked it OK using REST API method /vr/trace +* The first check is the *VR.rsp.status*attribute, checking if the status is *SUCCESS* +* Next, if the process status (*VR.rsp.results.**processStatus***) is REQUEST\_PENDING\_OKE | REQUEST\_PENDING\_JMS | REQUEST\_PROCESSED **or** OK data export date (*VR.rsp.results.*t*race6CegedimOkcExportDate*) is earlier than 24 hours then the processing of the request is postponed to the next check + + *exportDate* or ***processStatus***are optional and can be null. + + The process goes to the next step only if ***processStatus*** is  REQUEST\_RESPONDED | RESPONSE\_SENT + + The process is blocked to next check only if  t*race6CegedimOkcExportDate*is not null and is earlier than 24h +* If the ***processStatus***is validated and *VR.rsp.results**.**responseStatus***is VAS\_NOT\_FOUND | VAS\_INCOHERENT\_REQUEST | VAS\_DUPLICATE\_PROCESS then DCR is being closed with status *REJECTED* + + | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus" | "*CLOSED"* | + | VRStatusDetail | "*REJECTED*" | + | ReceivedDate | current time | + | Comments | *OK.responseComment* | +* Before these 2 next steps, the current Entity status is retrieved from HUB Cache. This is required to check if the entity was merged with OK entity. + + if ***responseStatus*** is*VAS\_FOUND | VAS\_FOUND\_BUT\_INVALID and* OK crosswalk exists in Reltio entity which value equals to OK validated id (i*ndividualEidValidated or workplaceEidValidated) then* DCR is closed with status *ACCEPTED*. + + | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus" | "*CLOSED"* | + | VRStatusDetail | "*ACCEPTED*" | + | ReceivedDate | current time | + | Comments | *OK.responseComment* | + + if ***responseStatus*** is*VAS\_FOUND | VAS\_FOUND\_BUT\_INVALID but*OK crosswalk doesn't exist in Reltio then Relio DCR Request is created and workflow task is triggered for Data Steward review. DCR status entity is updated with *DS\_ACTION\_REQUIRED* status. + + | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus" | "*OPEN"* | + | VRStatusDetail | "DS\_ACTION\_REQUIRED " | + | ReceivedDate | current time | + | Comments | *OK.responseComment* | + + GET /changeRequests operation is invoked to get a new change request ID and start a new workflow + + POST /workflow/\_initiate operation is invoked to init new Workflow in Reltio + + | Workflow attributes | Mapping | + | --- | --- | + | changeRequest.uri | ChangeRequest Reltio URI | + | changeRequest.changes | Entity URI | + | comment | i*ndividualEidValidated or workplaceEidValidated* | + + POST /entities?changeRequestId= - operation is invoked to update change request Entity container with DCR Status to Closed, this change is only visible in Reltio once DS accepts the DCR.  + + | Body attributes | Mapping | + | --- | --- | + | attributes | ``` "DCRRequests": [ { "value": { "VRStatus": [ { "value": "CLOSED" } ] }, "refEntity": { "crosswalks": [ { "type": "configuration/sources/DCR", "value": "$requestId", "dataProvider": false, "contributorProvider": true }, { "type": "configuration/sources/DCR", "value": "$requestId_REF", "dataProvider": true, "contributorProvider": false } ] }, "refRelation": { "crosswalks": [ { "type": "configuration/sources/DCR", "value": "$requestId_REF" } ] } } ] ``` | + | crosswalks | ``` "crosswalks": [ { "type": "configuration/sources/", "value": "", "dataProvider": false, "contributorProvider": true, "deleteDate": "" }, { "type": "configuration/sources/DCR", "value": "$requestId_CR", "dataProvider": true, "contributorProvider": false, "deleteDate": "" } ] ``` | + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Timer (cron) | mdm-onekey-dcr-service:TraceVRService | query mongo to get all SENT DCR's related to OK\_VR process | every hour | +| **OUT** API request | one-key-client:OneKeyIntegrationService.traceValidation | trace VR request to OneKey | invokes API request for each DCR | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| OK DCR Service | Main component with flow implementation | +| Manager | Reltio Adapter for GET /changeRequests and POST /workflow/\_initiate operations | +| OneKey Adapter | TraceValidation Request | +| Hub Store | DCR and Entities Cache | +--- + +# PforceRx DCR flows + +**Page ID:** 209949183 +**Page Link:** /display/GMDM/PforceRx+DCR+flows + +Description +=========== + +MDM HUB exposes Rest API to create and check the status of DCR. The process is responsible for creating DCRs in Reltio and starting Change Requests Workflow DCRs created in Reltio or creating the DCRs (submitVR operation) in ONEKEY. DCR requests can be routed to an external MDM HUB instance handling the requested country. The action is transparent to the caller. During this process, the communication to IQVIA OneKey VR API / Reltio API is established. The routing decision depends on the market, operation type, or changed profile attributes. + +Reltio API:  createEntity (with ChangeReqest) operation is executed to create a completely new entity in the new Change Request in Reltio. attributesUpdate (with ChageRequest) operation is executed after calculation of the specific changes on complex or simple attributes on existing entity - this also creates a new Change Request.  Start Workflow operation is requested at the end, this starts the Wrofklow for the DCR in Reltio so the change requests are started in the Reltio Inbox for Data Steward review. + +IQVIA API: SubmitVR operation is executed to create a new Validation Request. The TraceVR operation is executed to check the status of the VR in OneKey. + +All DCRs are saved in the dedicated collection in HUB Mongo DB, required to gather metadata and trace the changes for each DCR request. The DCR statuses are updated by consuming events generated by Reltio or periodic query action of open DCRs in OneKey + +The Data Steward can decide to route a DCR to IQVIA as well - some changes can be suggested by the DS using the "Suggest" operation in Reltio and "Send to Third Party Validation" button, the process "Data Steward OK Validation Request" is processing these changes and sends them to the OneKey service. + +The below diagram presents an overview of the entire process. Detailed descriptions are available in the separated subpages. + +**API doc URL**: https://api-emea-nprod-gbl-mdm-hub.pfizer.com/api-dcr-spec-emea-dev/swagger-ui/index.html + +Flow diagram +============ + +### DCR Service High-Level Architecture + + + +### DCR HUB Logical Architecture + + + +Model diagram +============= + + + +Flows: +====== + +* Create DCR + + The client call API Post /dcr method and pass the request in JSON format to MDM HUB DCR service + + The request is validated against the following rules: + - mandatory fields are set + - reference object HCP,HCO are available in Reltio + - referenced attributes like specialties, addresses are in the changed object + + The service evaluates the target system based on country, operation type (create, update), changed attributes. The process is controlled by the decision table stored in the config. + + The DCR is created in the target system through the API + + The result is stored in the registry. DCR information entity is created in Reltio for tracking. + + The status with created DCR object ids are returned in response to the Client +* Get DCR status + + The client calls GET /dcr/\_status method + + The DCR service queries DCR registry in Mongo and returns the status to the Client. + + There are processes updating dcr status in the registry: + - DCR change events are generated by Reltio when DCR is accepted or rejected by DS. Events are processed by the service. +* Reltio: process DCR Change Events + + DCR change events are generated by Reltio when DCR is accepted or rejected by DS. Events are processed by the service. +* OneKey: process DCR Change Events + + DCR change events are generated by the OneKey service when DCR is accepted or rejected by DS. Events are processed by the service. +* OneKey: generate DCR Change Events (traceVR) + + Every x configured hours the OneKey status method is queried to get status for open validation requests. +* Reltio: create DCR method - direct + + direct API method that creates DCR in Reltio (contains mapping and logic description) +* OneKey: create DCR method (submitVR) - direct + + direct API method that creates DCR in OneKey - executes the submitVR operation (contains mapping and logic description) + +Triggers +======== + +Described in the separated sub-pages for each process. + +Dependent components +==================== + +Described in the separated sub-pages for each process. +--- + +# Create DCR + +**Page ID:** 209949185 +**Page Link:** /display/GMDM/Create+DCR + +Description +=========== + +The process creates change requests received from PforceRx Client and sends the DCR to the specified target service - Reltio, OneKey or Veeva OpenData (VOD). DCR is created in the system and then processed by the data stewards. The status is asynchronously updated by the HUB processes, Client represents the DCR using a unique *extDCRRequestId* value. Using this value Client can check the status of the DCR (Get DCR status). + +Flow diagram +============ + + + +*Source: Lucid* + + + +*Source: Lucid* + +DCR Service component perspective +================================= + + + +Steps +===== + +1. Clients execute the API POST /dcr request +2. Kong receives requests and handles authentication +3. If the authentication succeeds the request is forwarded to the dcr-service-2 component, +4. DCR Service checks permissions to call this operation and the correctness of the request, then the flow is started and the following steps are executed: + 1. Parse and validate the dcr request. The validation logic checks the following: + 1. Check if the list of `DCRRequests` contains unique `extDCRRequestId`. + 1. Requests that are duplicate will be rejected with the error message - *"Found duplicated request(s)"* + 2. For each `DCRRequest` in the input list execute the following checks: + 1. Users can define the following number of entities in the Request: + 1. at least one entity has to be defined, otherwise, the request will be rejected with an error message - *"No entities found in the request"* + 2. single HCP + 3. singe HCO + 4. singe HCP with single HCO + 5. two HCOs + 2. Check if the main reference objects exist in Reltio for **update** and **delete** action + 1. `HCP.refId` or `HCO.refId`, user have to specify one of: + 1. ***`CrosswalkTargetObjectId`*** - then the entity is retrieved from Reltio using get entity by crosswalk operation + 2. ***`EntityURITargetObjectId`*** - then the entity is retrieved from Reltio using get entity by uri operation + 3. ***`PfizerCustomerIdTargetObjectId`*** - then the entity is retrieved from Reltio using search operation by the `PfizerGlobalCustomerID` + 3. Attributes validation: + 1. Simple attributes - like firstName/lastName e.t.c + 1. for ***update*** action on the main object: + 1. if the input parameter is defined with an empty value - **""** - this will result in the removal of the target attribute + 2. if the input parameter is defined with a non-empty value - this will result in the update of the target attribute + 2. Nested attributes - like Specialties/Addresses e.t.c + 1. for each attribute, the user has to define the **refId** to uniquely identify the attribute + 1. For action "***update***" - if the **refId** is not found in the target object request will be rejected with a detailed error message + 2. For action "***insert***" - the refId is not required - new reference attribute will be added to the target object + 4. Changes validation: + 1. If the validation detected 0 changes (during comparison of applying changes and the target entity) -  the request is rejected with an error message - *"No changes detected"* + 2. Evaluate dcr service (based on the decision table config) + 1. The following decision table is defined to choose the target service + 1. LIST OF the following combination of attributes: + + | attribute | description | + | --- | --- | + | ``` userName ``` | the user name that executes the request | + | ``` sourceName ``` | the source name of the Main object | + | ``` country ``` | the county defined in the request | + | ``` operationType ``` | the operation type for the Main object ``` { insert, update, delete } ``` | + | ``` affectedAttributes ``` | the list of attributes that the user is changing | + | ``` affectedObjects ``` | ``` { HCP, HCO, HCP_HCO } ``` | + | | RESULT →  TargetType {Reltio, OneKey, Veeva} | + 2. Each attribute in the configuration is optional. + 3. The decision table is making the validation based on the input request and the main object- the main object is HCP, if the HCP is empty then the decision table is checking HCO. + 4. The result of the decision table is the `TargetType`, the routing to the Reltio MDM system, OneKey or Veeva service. + 3. Execute target service (reltio/onekey/veeva) + 1. Reltio: create DCR method - direct + 2. OneKey: create DCR method (submitVR) - direct + 3. Veeva: create DCR method (storeVR) + 4. Create DCR in Reltio and save DCR in DCR Registry + * If the submission is successful then: + + ***DCR***entity is created in Reltio and the relation between the processed entity and the DCR entity + - Reltio source name (crosswalk.type): *DCR* + - Reltio relation type: *HCPtoDCR*or *HCOtoDCR*(depending on the object type) + * for "**create**" and "**delete**" operation the Relation have to be created between objects + * if this is just the "**insert**" operation the Relation will be created after the acceptance of the Change Request in Reltio - Reltio: process DCR Change Events + - DCR entity attributes once sent to *OneKey* + + | DCR entity attributes | Mapping | + | --- | --- | + | DCRID | ***extDCRRequestId*** | + | EntityURI | the processed entity URI | + | VRStatus | "*OPEN"* | + | VRStatusDetail | "*SENT\_TO\_OK"* | + | CreatedBy | *MDM HUB* | + | SentDate | current time | + | CreateDate | current time | + | CloseDate | if REJECTED | ACCEPTED -> current time | + | dcrType | evaluate based on config: ``` dcrTypeRules: - type: CR0 size: 1 action: insert entity: com.pfizer.mdm.api.dcr2.HCP ``` | + - DCR entity attributes once sent to *Veeva* + + | *DCR entity attributes* | *Mapping* | + | --- | --- | + | *DCRID* | ***extDCRRequestId*** | + | *EntityURI* | *the processed entity URI* | + | *VRStatus* | *"OPEN"* | + | *VRStatusDetail* | *"SENT\_TO\_VEEVA"* | + | *CreatedBy* | *MDM HUB* | + | *SentDate* | *current time* | + | *CreateDate* | *current time* | + | *CloseDate* | *if REJECTED | ACCEPTED -> current time* | + | *dcrType* | *evaluate based on config:* ``` dcrTypeRules: - type: CR0 size: 1 action: insert entity: com.pfizer.mdm.api.dcr2.HCP ``` | + - DCR entity attributes once sent to *Reltio* → action is passed to DS and workflow is started. + + | DCR entity attributes | Mapping | + | --- | --- | + | DCRID | ***extDCRRequestId*** | + | EntityURI | the processed entity URI | + | VRStatus | "*OPEN"* | + | VRStatusDetail | "DS\_ACTION\_REQUIRED " | + | CreatedBy | *MDM HUB* | + | SentDate | current time | + | CreateDate | current time | + | CloseDate | if REJECTED | ACCEPTED -> current time | + | dcrType | evaluate based on config: ``` dcrTypeRules: - type: CR0 size: 1 action: insert entity: com.pfizer.mdm.api.dcr2.HCP ``` | + + *Mongo Update: `DCRRequest.status`* is updated to *SENT*with OneKey or Veeva request and response details or DS\_ACTION\_REQURIED with all Reltio details + * Otherwise *FAILED*status is recorded in `DCRRequest`with a detailed error message. + + *Mongo Update:  `DCRRequest.status`* is updated to *FAILED* with all required attributes, request, and exception response details + 5. Initialize Workflow in Reltio (only requests that `TargetType`is Reltio) + 1. POST /workflow/\_initiate operation is invoked to init new Workflow in Reltio + + | Workflow attributes | Mapping | + | --- | --- | + | changeRequest.uri | ChangeRequest Reltio URI | + | changeRequest.changes | Entity URI | + 6. Then Auto close logic is invoked to evaluate whether DCR request meets conditions to be auto accepted or auto rejected. Logic is based on decision table `PreCloseConfig`. If `DCRRequest.country` is contained in `PreCloseConfig.acceptCountries` or `PreCloseConfig.rejectCountries` then DCR is accepted or rejected respectively. + 7. return DCRResponse to Client - During the flow, DCRRespone may be returned to Client with the specific ***errorCode*** or ***requestStatus***. The description for all response codes is presented on this page: Get DCR status + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| REST call | DCR Service: POST /dcr | create DCRs in the Reltio, OneKey or Veeva system | API synchronous requests - realtime | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service | Main component with flow implementation | +| OK DCR Service | OneKey Adapter - API operations | +| Veeva DCR Service | Veeva Adapter - API operations and S3/SFTP communication | +| Manager | Reltio Adapter - API operations | +| Hub Store | DCR and Entities Cache | +--- + +# DCR state change + +**Page ID:** 218438617 +**Page Link:** /display/GMDM/DCR+state+change + +Description +=========== + +The following diagram represents the DCR state changes. DCR object stat is saved in HUB and in Reltio DCR entity object. The state of the DCR is changed based on the Reltio/IQVIA/Veeva Data Steward action. + +Flow diagram +============ + + + +Steps +===== + +1. DCR is created (OPEN)  - Create DCR + 1. DCR is sent to Reltio, OneKey or Veeva + 1. When sent to Reltio + 1. Pre Close logic is invoked to auto accept (PRE\_ACCEPT) or auto reject (PRE\_REJECT) DCR + 2. Reltio Data Steward process the DCR - Reltio: process DCR Change Events + 2. OneKey Data Steward process the DCR - OneKey: process DCR Change Events + 3. Veeva Data Steward process the DCR - Veeva: process DCR Change Events + +Data Steward DCR status change perspective +========================================== + + + +Transaction Log +=============== + +There are the following main assumptions regarding the transaction log in DCR service: + +* **Main transaction** + + The user sends to the DCR service list of the DCR Requests and receives the list of the DCR Responses + - Transaction service generates the transaction ID for the input request - this is used as the correlation ID for each separated DCR Request in the list + - Transaction service save: + * METADATA + + main transaction ID + + userName + + extDCRRequestIds (list of all) + * BODY + + the DCR Requests list and the DCR Response List +* **State change transaction** + + DCR object state may change depending on the DS decision, for each state change (represented as a green box in the above diagram) the transaction is saved with the following attributes: + - Transaction METADATA + * main transaction ID + * extDCRRequestId + * dcrRequestId + * Reltio: + + VRStatus + + VRStatusDetail + * HUB: + + DCRRequestStatusDetails + * optionally: + + errorMessage + + ``` + errorCode + ``` + - Transaction BODY: + * Input Event + +**Log appenders:** + +* Kafka Transaction appender - saves whole events(metadata+body) to Kafka - data presented in the Kibana Dashboard +* Simple Transaction logger - saves the transactions details to the file in the following format: + + {ID}    {extDCRRequestId}   {dcrRequestId}   {VRStatus}   {VRStatusDetail}   {DCRRequestStatusDetails}   {errorCode}   {errorMessage} + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| REST call | DCR Service: POST /dcr | create DCRs in the Reltio system or in OneKey | API synchronous requests - realtime | +| **IN** Events incoming | dcr-service-2:DCRReltioResponseStream | process publisher full change request events in the stream | realtime: events stream processing | +| **IN** Events incoming | dcr-service-2:DCROneKeyResponseStream | process publisher full change request events in the stream | realtime: events stream processing | +| **IN** Events incoming | dcr-service-2:DCRVeevaResponseStream | process publisher full change request events in the stream | realtime: events stream processing | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service | Main component with flow implementation | +| OK DCR Service | OneKey Adapter  - API operations | +| Veeva DCR Service | Veeva Adapter  - API operations | +| Manager | Reltio Adapter  - API operations | +| Hub Store | DCR and Entities Cache | +--- + +# Get DCR status + +**Page ID:** 209949187 +**Page Link:** /display/GMDM/Get+DCR+status + +Description +=========== + +The client creates DCRs in Reltio, OneKey or Veeva OpenData using the Create DCR operation. The status is then asynchronously updated in the DCR Registry. The operation retrieves the current status of the DCRs that the updated date is between '`updateFrom`' and '`updateTo`' input parameters. PforceRx first asks what DCRs have been changed since last time they checked (usually 24h) and then iterate for each DCR they get detailed info. + +Flow diagram +============ + +, + + + +Source: Lucid + +##### Dependent flows: + +1. The DCRRegistry is enriched by the DCR events that are generated by Reltio - the flow description is here - Reltio: process DCR Change Events +2. The DCRRegistry is enriched by the DCR events generated in OneKey DCR service component - after submitVR operation is invoked to ONEKEY, each DCR is traced asynchronously in this process - OneKey: process DCR Change Events +3. The DCRRegistry is enriched by the DCR events generated in Veeva OpenData DCR service component - after submitVR operation is invoked to VEEVA, each DCR is traced asynchronously in this process - Veeva: process DCR Change Events + +Steps +===== + +##### Status + +There are the following request statuses that users may receive during Create DCR operation or during checking the updated status using GET `/dcr/_status` operation described below: + +| RequestStatus | DCRStatus | Internal Cache status | Description | +| --- | --- | --- | --- | +| REQUEST\_ACCEPTED | CREATED | SENT\_TO\_OK | DCR was sent to the ONEKEY system for validation and pending the processing by Data Steward in the system | +| REQUEST\_ACCEPTED | CREATED | SENT\_TO\_VEEVA | DCR was sent to the VEEVA system for validation and pending the processing by Data Steward in the system | +| REQUEST\_ACCEPTED | CREATED | DS\_ACTION\_REQUIRED | DCR is pending Data Steward validation in Reltio, waiting for approval or rejection | +| REQUEST\_ACCEPTED | CREATED | OK\_NOT\_FOUND | Used when ONEKEY profile was not found after X retries | +| REQUEST\_ACCEPTED | CREATED | VEEVA\_NOT\_FOUND | Used when VEEVA profile was not found after X retries | +| REQUEST\_ACCEPTED | CREATED | WAITING\_FOR\_ETL\_DATA\_LOAD | Used when waiting for actual data profile load from 3rd Party to appear in Reltio | +| REQUEST\_ACCEPTED | ACCEPTED | ACCEPTED | Data Steward accepted the DCR, changes were applied | +| REQUEST\_ACCEPTED | ACCEPTED | PRE\_ACCEPTED | PreClose logic was invoked and automatically accepted DCR according to decision table in PreCloseConfig | +| REQUEST\_REJECTED | REJECTED | REJECTED | Data Steward rejected the changes presented in the Change Request | +| REQUEST\_REJECTED | REJECTED | PRE\_REJECTED | PreClose logic was invoked and automatically rejected DCR according to decision table in PreCloseConfig | +| REQUEST\_FAILED | - | FAILED | DCR requests failed due to: validation error/ unexpected error e.t.d - details in the errorCode and errorMessage | + +##### Error codes: + +There are the following classes of exception that users may receive during Create DCR operation: + +| Class | errorCode | Description | HTTP code | +| --- | --- | --- | --- | +| 1 | DUPLICATE\_REQUEST | request rejected - extDCRRequestId  is registered - this is a duplicate request | 403 | +| 2 | NO\_CHANGES\_DETECTED | entities are the same (request is the same) - no changes | 400 | +| 3 | VALIDATION\_ERROR | ref object does not exist (not able to find HCP/HCO target object | 404 | +| 3 | VALIDATION\_ERROR | ref attribute does not exist - not able to find nested attribute in the target object | 400 | +| 3 | VALIDATION\_ERROR | wrong number of HCP/HCO entities in the input request | 400 | + +1. Clients execute the API GET`/dcr/_status` request +2. Kong receives requests and handles authentication +3. If the authentication succeeds the request is forwarded to the dcr-service-2 component, +4. DCR Service checks permissions to call this operation and the correctness of the request, then the flow is started and the following steps are executed + 1. Query on mongo is executed to get all DCRs matching input parameters: + 1. updateFrom (date-time) - DCR last update from - *DCRRequestDetails.status.changeDate* + 2. updateTo (date-time) - DCR last update to - *DCRRequestDetails.status.changeDate* + 3. limit (int) the maximum number of results returned through API - the recommended value is 25. The max value for a single request is 50. + 4. offset(int) - result offset - the parameter used to query through results that exceeded the limit. + 2. Resulted values are aggregated and returned to the Client. + 3. The client receives the List body. + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| REST call | DCR Service: GET`/dcr/_status` | get status of created DCRs. Limit the results using query parameters like dates and offset | API synchronous requests - realtime | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service | Main component with flow implementation | +| Hub Store | DCR and Entities Cache | +--- + +# OneKey: create DCR method (submitVR) - direct + +**Page ID:** 209949294 +**Page Link:** /display/GMDM/OneKey%3A+create+DCR+method+%28submitVR%29+-+direct + +Description +=========== + +Rest API method exposed in the OK DCR Service component responsible for submitting the VR to OneKey + +Flow diagram +============ + + + +Steps +===== + +1. Receive the API request +2. Validate - check if the onekey crosswalk exists once there is an update on the profile, otherwise reject the request +3. The DCR is mapped to OK VR Request and it's submitted using API REST method POST /vr/submit. (mapping described below) + 1. If the submission is successful then: + * *DCRRequest*i updated to *SENT\_TO\_OK*with OK request and response details. DCRRegistryONEKEY collection in saved for tracing purposes. The process that reads and check ONEKEY VRs is described here: OneKey: generate DCR Change Events (traceVR) + 2. Otherwise *FAILED*status is recorded and the response is returned with an OK error response + +Mapping +======= + +VR - Business Fields Requirements\_UK.xlsx - file that contains VR UK requirements and mapping to IQVIA model +------------------------------------------------------------------------------------------------------------- + +| | | | | | | | | +| --- | --- | --- | --- | --- | --- | --- | --- | +| **HUB** | | | | | **ONEKEY** | | | +| | ***attributes*** | ***attributes*** | ***codes*** | | ***mandatory*** | ***attributes*** | ***values*** | +| **HCO** | | | | | | | | +| | | | | | Y | entityType | WORKPLACE | +| | | | | | Y | validation.clientRequestId | HUB\_GENERATED\_ID | +| | | | | | Y | validation.process | Q | +| | | | | | Y | validation.requestDate | 1970-01-01T00:00Z | +| | | | | | Y | validation.callDate | 1970-01-01T00:00Z | +| **attributes** | | | | | Y | validation.requestProcess | I | +| | extDCRComment | | | | | validation.requestComment | | +| | | | | | | | | +| | country | | | | Y | isoCod2 | | +| | | | | | | | | +| | | | | | | | | +| | reference Entity | crosswalk | ONEKEY | | | workplace.workplaceEid | | +| | | | | | | | | +| | name | | | | | workplace.usualName | | +| | | | | | | workplace.officialName | | +| | otherHCOAffiliations | parentUsualName | | | | workplace.parentUsualName | | +| | subTypeCode | | COTFacilityType (TET.W.\*) | | | workplace.typeCode | | +| | ~~typeCode~~ | no value in PFORCERX | ~~HCOSubType~~ ~~(LEX.W.\*)~~ | | | ~~workplace.activityLocationCode~~ | | +| | addresses | | | | | | | +| | | sourceAddressId | | | | N/A | | +| | | addressType | | | | N/A | | +| | | addressLine1 | | | | address.longLabel | | +| | | addressLine2 | | | | address.longLabel2 | | +| | | addressLine3 | | | | N/A | | +| | | stateProvince | AddressState (DPT.W.\*) | | | address.countyCode | | +| | | city | | | Y | address.city | | +| | | zip | | | | address.longPostalCode | | +| | | country | | | Y | address.country | | +| | | rank | | | | get address with rank=1 | | +| | emails | | | | | | | +| | | type | | | | N/A | | +| | | email | | | | workplace.email | | +| | | rank | | | | get email with rank=1 | | +| | otherHCOAffiliations | | | | | | | +| | | type | | | | N/A | | +| | | rank | | | | get affiliation with rank=1 | | +| | reference Entity | otherHCOAffiliations reference entity onekeyID | ONEKEY | | | workplace.parentWorkplaceEid | | +| | phones | | | | | | | +| | | type | contains FAX | | | | | +| | | number | | | | workplace.telephone | | +| | | rank | | | | get phone with rank=1 | | +| | | | | | | | | +| | | type | not contains FAX | | | | | +| | | number | | | | workplace.fax | | +| | | rank | | | | get phone with rank=1 | | +| HCP | | | | | | | | +| | | | | | Y | entityType | ACTIVITY | +| | | | | | Y | validation.clientRequestId | HUB\_GENERATED\_ID | +| | | | | | Y | validation.process | Q | +| | | | | | Y | validation.requestDate | 1970-01-01T00:00Z | +| | | | | | Y | validation.callDate | 1970-01-01T00:00Z | +| **attributes** | | | | | Y | validation.requestProcess | I | +| | extDCRComment | | | | | validation.requestComment | | +| | | | | | | | | +| | country | | | | Y | isoCod2 | | +| | | | | | | | | +| | | | | | | | | +| | reference Entity | crosswalk | ONEKEY | | | individual.individualEid | | +| | | | | | | | | +| | firstName | | | | | individual.firstName | | +| | lastName | | | | Y | individual.lastName | | +| | middleName | | | | | individual.middleName | | +| | typeCode | | | | | N/A | | +| | subTypeCode | | HCPSubTypeCode (TYP..\*) | | | individual.typeCode | | +| | title | | HCPTitle (TIT.\*) | | | individual.titleCode | | +| | prefix | | HCPPrefix (APP.\*) | | | individual.prefixNameCode | | +| | suffix | | | | | N/A | | +| | gender | | Gender (.\*) | | | individual.genderCode | | +| | specialties | | | | | | | +| | | typeCode | HCPSpecialty (SP.W.\*) | | | individual.speciality1 | | +| | | type | | | | N/A | | +| | | rank | | | | get speciality with rank=1 | | +| | | typeCode | HCPSpecialty (SP.W.\*) | | | individual.speciality2 | | +| | | type | | | | N/A | | +| | | rank | | | | get speciality with rank=2 | | +| | | typeCode | HCPSpecialty (SP.W.\*) | | | individual.speciality3 | | +| | | type | | | | N/A | | +| | | rank | | | | get speciality with rank=3 | | +| | addresses | | | | | | | +| | | sourceAddressId | | | | N/A | | +| | | addressType | | | | N/A | | +| | | addressLine1 | | | | address.longLabel | | +| | | addressLine2 | | | | address.longLabel2 | | +| | | addressLine3 | | | | N/A | | +| | | stateProvince | AddressState (DPT.W.\*) | | | address.countyCode | | +| | | city | | | Y | address.city | | +| | | zip | | | | address.longPostalCode | | +| | | country | | | Y | address.country | | +| | | rank | | | | get address with rank=1 | | +| | identifiers | | | | | | | +| | | type | | | | N/A | | +| | | id | | | | N/A | | +| | phones | | | | | | | +| | | type | | | | N/A | | +| | | number | | | | individual.mobilePhone | | +| | | rank | | | | get phone with rank=1 | | +| | emails | | | | | | | +| | | type | | | | N/A | | +| | | email | | | | individual.email | | +| | | rank | | | | get phone with rank=1 | | +| | | | | | | | | +| --- | --- | --- | --- | --- | --- | --- | --- | +| | ~~contactAffiliations~~ | no value in PFORCERX | | | | | | +| | | ~~type~~ | ~~RoleType~~ ~~(TIH.W.\*)~~ | | | ~~activity.role~~ | | +| | | ~~primary~~ | | | | ~~N/A~~ | | +| | | ~~rank~~ | | | | ~~get affiliation with rank=1~~ | | +| | contactAffiliations reference Entity | crosswalks | ONEKEY | | | workplace.workplaceEid | | +| HCP & HCO | | | | | | | | +| | | | | | Y | entityType | ACTIVITY | +| | For HCP full mapping check the HCP section above | | | | Y | validation.clientRequestId | HUB\_GENERATED\_ID | +| | For HCO full mapping check the HCO section above | | | | Y | validation.process | Q | +| | | | | | Y | validation.requestDate | 1970-01-01T00:00Z | +| | | | | | Y | validation.callDate | 1970-01-01T00:00Z | +| **attributes** | | | | | Y | validation.requestProcess | I | +| | extDCRComment | | | | | validation.requestComment | | +| | | | | | | | | +| | country | | | | Y | isoCod2 | | +| | addresses | | | | | | | +| | | If the HCO address exists map to ONEKEY address | | | | address (mapping HCO) | | +| | | else | | | | | | +| | | If the HCP address exists map to ONEKEY address | | | | address (mapping HCP) | | +| | ~~contactAffiliations~~ | no value in PFORCERX | | | | | | +| | | ~~type~~ | ~~RoleType~~ ``` (TIH.W.*) ``` | | | ~~activity.role~~ | | +| | | ~~primary~~ | | | | ~~N/A~~ | | +| | | ~~rank~~ | | | | ~~get affiliation with rank=1~~ | | + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| REST call | DCR Service: POST /dcr | create DCRs in the ONEKEY | API synchronous requests - realtime | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service 2 | Main component with flow implementation | +| Hub Store | DCR and Entities Cache | +--- + +# OneKey: generate DCR Change Events (traceVR) + +**Page ID:** 209950500 +**Page Link:** /pages/viewpage.action?pageId=209950500 + +Description +=========== + +This process is triggered after the DCR was routed to Onekey based on the decision table configuration. The process of tracing the VR changes is based on the OneKey VR changes. During this process HUB, DCR Cache is triggered every hour for SENT DCR's and check VR status using OneKey web service. After verification, the DCR Change event is generated. The DCR event is processed in the OneKey: process DCR Change Events and the DCR is updated in Reltio with Accepted or Rejected status. + +Flow diagram +============ + + + +Steps +===== + +* Every N  hours OK *VR requests* with status *SENT*are queried in *DCRRegistryONEKEY* store*.* +* For each open requests, its status is checked it OK using REST API method /vr/trace +* The first check is the *VR.rsp.status*attribute, checking if the status is *SUCCESS* +* Next, if the process status (*VR.rsp.results.**processStatus***) is REQUEST\_PENDING\_OKE | REQUEST\_PENDING\_JMS | REQUEST\_PROCESSED **or** OK data export date (*VR.rsp.results.*t*race6CegedimOkcExportDate*) is earlier than 24 hours then the processing of the request is postponed to the next check + + *exportDate* or ***processStatus***are optional and can be null. + + The process goes to the next step only if ***processStatus***is  REQUEST\_RESPONDED | RESPONSE\_SENT + + The process is blocked to next check only if  t*race6CegedimOkcExportDate*is not null and is earlier than 24h +* If the ***processStatus***is validated and *VR.rsp.results**.**responseStatus***is VAS\_NOT\_FOUND | VAS\_INCOHERENT\_REQUEST | VAS\_DUPLICATE\_PROCESS then OneKeyDCREvent is being generated with status *REJECTED* + + | OneKeyChangeRequest attributes | Mapping | + | --- | --- | + | vrStatus | "*CLOSED"* | + | vrStatusDetail | "*REJECTED*" | + | traceResponseReceivedDate | current time | + | oneKeyComment | *OK.responseComment* | +* Next. + + if ***responseStatus***is*VAS\_FOUND | VAS\_FOUND\_BUT\_INVALID* then OneKeyDCREvent is being generated with status *ACCEPTED*. ( now the new ONEKEY profile will be loaded to Reltio using ETL data load. The OneKey: process DCR Change Events is processing this events ad checks the Reltio if the ONEKEY is created and PfizerCustomerGlobalId is assigned, this process will wait until ONEKEY is in Reltio so the client received the ACCEPTED DCR only after this condition is met) + + | DCR entity attributes | Mapping | + | --- | --- | + | vrStatus | "*CLOSED"* | + | vrStatusDetail | "*ACCEPTED*" | + | traceResponseReceivedDate | current time | + | oneKeyComment | *OK.responseComment* *\n* *ONEKEY ID = individualEidValidated or workplaceEidValidated* | + + events are published to the $env-internal-onekey-dcr-change-events-in topic + +#### Event Model + +``` +data class OneKeyDCREvent(val eventType: String? = null, + val eventTime: Long? = null, + val eventPublishingTime: Long? = null, + val countryCode: String? = null, + val dcrId: String? = null, + val targetChangeRequest: OneKeyChangeRequest, +) +``` + +``` + + +``` + +``` +data class OneKeyChangeRequest( + val vrStatus : String? = null, + val vrStatusDetail : String? = null, + val oneKeyComment : String? = null, + val individualEidValidated : String? = null, + val workplaceEidValidated : String? = null, + val vrTraceRequest : String? = null, + val vrTraceResponse : String? = null, +) +``` + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Timer (cron) | dcr-service:TraceVRService | query mongo to get all SENT DCR's related to the PFORCERX process | every hour | +| **OUT** Events | dcr-service:TraceVRService | generate the OneKeyDCREvent | every hour | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service | Main component with flow implementation | +| Hub Store | DCR and Entities Cache | +--- + +# OneKey: process DCR Change Events + +**Page ID:** 209949303 +**Page Link:** /display/GMDM/OneKey%3A+process+DCR+Change+Events + +Description +=========== + +The process updates the DCR's based on the Change Request events received from `[ONEKEY|VOD]` (after trace VR method result). Based on the [IQVIA|VEEVA] Data Steward decision the `state`attribute contains relevant information to update DCR status. During this process also the comments created by IQVIA DS are retrieved and the relationship (optional step) between the DCR object and the newly created entity is created. DCR status is accepted only after the `[ONEKEY|VOD]` profile is created in Reltio, only then the Client will receive the ACCEPTED status. The process is checking Reltio with delay and retries if the ETL load is still in progress waiting for `[ONEKEY|VOD]` profile. + +Flow diagram +============ + +**OneKey variant** + + + +**Veeva variant:** + +Steps +===== + +* OneKey: generate DCR Change Events (traceVR) publishes simple events to $env-internal-onekey-dcr-change-events-in: DCR\_CHANGED + + `Veeva specific`: Veeva: generate DCR Change Events (traceVR) publishes simple events to `$env-internal-veeva-dcr-change-events-in`: `DCR_CHANGED` +* Events are aggregated in a time window (recommended the window length 24 hours) and the last event is returned to the process after the window is closed. +* Events are processed in the Stream and based on the `OneKeyDCREvent.OneKeyChangeRequest``.vrStatus | VeevaDCREvent.VeevaChangeRequestDetails.vrStatus` attribute decision is made +* DCR is retrieved from the cache based on the `_id` of the DCR +* If the event state is **ACCEPTED** + + Get Reltio entity `PfizerCustomerID` by `[ONEKEY|VOD]` crosswalk + + If such crosswalk entity exists in Reltio: + - **PfizerGlobalCustomerId** is saved in Registry and will be returned to the Client + - During the process, the optional check is triggered - create the relation between the DCR object and newly created entities + * if DCRRegistry contain an empty list of `entityUris`, or some of the newly created entity is not present in the list, the Relation between this object and the DCR has to be created + + **DCR**entity is updated in Reltio and the relation between the processed entity and the DCR entity + - Reltio source name (crosswalk. type): DCR + - Reltio relation type: `HCPtoDCR`or `HCOtoDCR`(depending on the object type) + + Newly created entities uris should be retrieved by the `individualEidValidated` or `workplaceEidValidated` (it may be both) attributes from the events that represent the HCP or HCO crosswalks. + - The status in Reltio and in Mongo is updated + + | DCR entity attributes | Mapping for OneKey | Mapping for Veeva | + | --- | --- | --- | + | VRStatus | CLOSED | | + | VRStatusDetail | state: ACCEPTED | | + | Comments | ONEKEY comments ({`VR.rsp.responseComments`}) ONEKEY ID = `i``ndividualEidValidated` or `workplaceEidValidated` | VEEVA comments = `VR.rsp.responseComments` VEEVA ID = `entityUris` | + | PfizerGlobalCustomerId | This is required in ACCEPTED status | | + + If the `[ONEKEY|VOD]` does not exist in Reltio + - Regenerate the Event with a new timestamp to the input topic so this will be processed in the next hours + - Update the Reltio DCR status + * | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus | OPEN | + | VRStatusDetail | ACCEPTED | + - update the Mongo status to the `OK_NOT_FOUND | VEEVA_NOT_FOUND` and increase the "`retryCounter`" attribute +* If the event state is **REJECTED** + + If a Reltio DS has already seen this request, REJECT the DCR and end the flow (if the initial target type is Reltio) + + The status in Reltio and in Mongo is updated + + | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus | CLOSED | + | VRStatusDetail | state: REJECTED | + | Comments | `[ONEKEY|VOD]` comments ({`VR.rsp.responseComments`}) | + + If this is based on the routing table and it was never sent to the Reltio DS, then create the DCR workflow and send this to the Reltio DS. Add the information comment that this was Rejected by the OneKey, so now Reltio DS has to decide if this should be REJECTED or APPLIED in Reltio. Add the comment that this is not possible to execute the sendTo3PartyValidation button in this case. Steps: + - Check if the initial target type is `[ONEKEY|VOD]` + - Use the DCR Request that was initially received from PforceRx and is a Domain Model request (after validation) + - Send the DCR to Reltio the service returns the following response: + * ACCEPTED (change request accepted by Reltio) + + update the status to `DS_ACTION_REQUIERED`and in the comment add the following: "This DCR was REJECTED by the `[ONEKEY|VOD]` Data Steward with the following comment: <`[ONEKEY|VOD]` reject comment>. Please review this DCR in Reltio and APPLY or REJECT. It is not possible to execute the sendTo3PartyValidation button in this case" + + initialize new Workflow in Reltio with the comment. + + save data in the DCR entity status in Reltio and update Mongo DCR Registry with workflow ID and other attributes that were used in this Flow. + * REJECTED  (failure or error response from Reltio) + + CLOSE the DCR with the information that DCR was REJECTED by the `[ONEKEY|VOD]` and Reltio also REJECTED the DCR. Add the error message from both systems in the comment. + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Events incoming | dcr-service-2:DCROneKeyResponseStream dcr-service-2:DCRVeevaResponseStream (`$env-internal-veeva-dcr-change-events-in)` | process publisher full change request events in the stream | realtime: events stream processing | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service 2 | Main component with flow implementation | +| Manager | Reltio Adapter  - API operations | +| Publisher | Events publisher generates incoming events | +| Hub Store | DCR and Entities Cache | +--- + +# Reltio: create DCR method - direct + +**Page ID:** 209949292 +**Page Link:** /display/GMDM/Reltio%3A+create+DCR+method+-+direct + +Description +=========== + +Rest API method exposed in the Manager component responsible for submitting the Change Request to Reltio + +Flow diagram +============ + + + +Steps +===== + +1. Receive the DCR request generated by DCR Service 2 component +2. Depending on the Action execute the method in the Manager component: + 1. insert - Execute standard Create/Update HCP/HCO/MCO operation with additional changeRequest.id parameter + 2. update - Execute Update Attributes operation with additional changeRequest.id parameter + 1. the combination of IGNORE\_ATTRIBUTE & INSERT\_ATTRIBUTE once updating existing parameter in Reltio + 2. the INSERT\_ATTRIBUTE once adding new attribute to Reltio + 3. delete - Execute Update Attribute operation with additional changeRequest.id parameter + 1. the UPDATE\_END\_DATE on the entity to inactivate this profile +3. Based on the Reltio response the DCR Response is returned: + 1. REQUEST\_ACCEPTED - Reltio processed the request successfully + 2. REQUEST\_FAILED - Reltio returned the exception, Client will receive the detailed description in the errorMessage + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| REST call | DCR Service: POST /dcr2 | Create change Requests in Reltio | API synchronous requests - realtime | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service | Main component with flow implementation | +| Hub Store | DCR and Entities Cache | +--- + +# Reltio: process DCR Change Events + +**Page ID:** 209949300 +**Page Link:** /display/GMDM/Reltio%3A+process+DCR+Change+Events + +Description +=========== + +The process updates the DCR's based on the Change Request events received from Reltio(publishing). Based on the Data Steward decision the `state`attribute contains relevant information to update DCR status. During this process also the comments created by DS are retrieved and the relationship (optional step) between the DCR object and the newly created entity is created. + +Flow diagram +============ + + + +Steps +===== + +* Event publisher publishes simple events to `$env-internal-reltio-dcr-change-events-in`: DCR\_CHANGED("CHANGE\_REQUEST\_CHANGED") and DCR\_REMOVED("CHANGE\_REQUEST\_REMOVED") +* When the events do not contain the ThirdPartyValidation flag it means that DS APPLIED or REJECTED the DCR, the following logic is applied + + Events are processed in the Stream and based on the `targetChangeRequest.state` attribute decision is made + - If the state is APPLIED or REJECTS, DCR is retrieved from the cache based on the `changeRequestURI` + * If DCR exists in Cache The status in Reltio is updated + + | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus | CLOSED | + | VRStatusDetail | state: APPLIED → ACCEPTED state: REJECTED → REJECTED | + * Otherwise, the events are rejected and the transaction is ended + - The `PfizerCustomerGlobalId` is retrieved for newly created entities in Reltio based on the main entity URI. + - During the process, the optional check is triggered - create the relation between the DCR object and newly created entities + * if `DCRRegistry` contain an empty list of `entityUris`, or some of the newly created entity is not present in the list, the Relation between this object and the DCR has to be created + + **DCR**entity is updated in Reltio and the relation between the processed entity and the DCR entity + - Reltio source name (crosswalk. type): DCR + - Reltio relation type: `HCPtoDCR`or `HCOtoDCR`(depending on the object type) + - The comments added by the DataSteward during the processing of the Change request is retrieved using the following operation: + * ***GET*** /tasks?objectURI=entities/ + * The processInstanceComments is retrieved from the response and added to DCRRegistry.changeRequestComment + +* Otherwise, when the events contain the ThirdPartyValidation flag it means that DS decided to send the DCR to IQVIA or VEEVA for the validation, the following logic is applied: + + If the current targetType is ONEKEY | VEEVA + - REJECT the DCR and add the comment on the DCR in Retlio that "DCR was already processed by `[ONEKEY|VEEVA]` Data Stewards, REJECT because it is not allowed to send this DCR one more time to `[IQVIA|VEEVA]`" + + If the current targetType is Reltio, it means that we can send this DCR to `[IQVIA|VEEVA]` for validation  + - Use the DCR Request that was initially received from PforceRx and is a Domain Model request (after validation) + - Execute the POST /dcr method in `[ONEKEY|VEEVA]` DCR Service, the service returns the following response: + * ACCEPTED - update the status to `[SENT_TO_OK|SENT_TO_VEEVA]` + * REJECTED - it means that some unexpected exception occurred in `[ONEKEY|VEEVA]`, or request was rejected by `[ONEKEY|VEEVA]`, or the ONEKEY crosswalk does not exist in Reltio, and `[ONEKEY|VEEVA]`service rejected this request + + `Veeva specific`: When VOD crosswalk does not exist in Reltio, current version of profile is being sent to Veeva for validation independent from initial changes which where incorporated within DCR + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Events incoming | dcr-service-2:DCRReltioResponseStream | process publisher full change request events in the stream | realtime: events stream processing | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service DCR Service 2 | Main component with flow implementation | +| Manager | Reltio Adapter  - API operations | +| Publisher | Events publisher generates incoming events | +| Hub Store | DCR and Entities Cache | +--- + +# Reltio: Profiles created by DCR + +**Page ID:** 510266969 +**Page Link:** /display/GMDM/Reltio%3A+Profiles+created+by+DCR + +| DCR type | Approval/Reject | Record visibility in MDM | Crosswalk Type | Crosswalk Value | Source | +| --- | --- | --- | --- | --- | --- | +| DCR create for HCP/HCO | Approved by OneKey/VOD | HCP/HCO created in MDM | ONEKEY|VOD | onekey id | ONEKEY|VOD | +| Approved by DSR | HCP/HCO created in MDM | System source name from DCR (KOL\_OneView, PforceRx, etc) | DCR ID | System source name from DCR (KOL\_OneView, PforceRx, etc) | +| DCR edit for HCP/HCO | Approved by OneKey/VOD | HCP/HCO requested attribute updated in MDM | ONEKEY|VOD | | ONEKEY|VOD | +| Approved by DSR | HCP/HCO requested attribute updated in MDM | Reltio | entity uri | Reltio | +| DCR edit for HCPaddress/HCO address | Approved by OneKey/VOD | New address created in MDM, existing address marked as inactive | ONEKEY|VOD | | ONEKEY|VOD | +| Approved by DSR | New address created in MDM, existing address marked as inactive | Reltio | entity uri | Reltio | +--- + +# Veeva DCR flows + +**Page ID:** 379332475 +**Page Link:** /display/GMDM/Veeva+DCR+flows + +Description +=========== + +The process is responsible for creating DCRs which are stored (Store VR) to be further transferred and processed by Veeva. Changes can be suggested by the DS using "Suggest" operation in Reltio and "Send to Third Party Validation" button. All DCRs are saved in the dedicated collection in HUB Mongo DB, required to gather metadata and trace the changes for each DCR request. During this process, the communication to Veeva Opendata is established via S3/SFTP communication. SubmitVR operation is executed to create a new ZIP files with DCR requests spread across multiple CSV files. The TraceVR operation is executed to check if Veeva responded to initial DCR Requests via ZIP file placed Inbound S3 dir. + +The process is divided into 3 sections: + +1. Create DCR request - Veeva +2. Submit DCR Request - Veeva +3. Trace Validation Request - Veeva + +The below diagram presents an overview of the entire process. Detailed descriptions are available in the separated subpages. + +Business process diagram for R1 phase +===================================== + + + +Flow diagram +============ + + + +Steps +===== + +* CreateVR + + Process of saving DCR requests in Mongo Cache after being triggered by DCR Service 2. + + DCR request information is translated to Veeva's model and stored in dedicated collection for Veeva DCRs. +* SubmitVR + + The process of submitting VR stored in Mongo Cache to Veeva's SFTP via S3 bucket. The process aggregates events stored in Mongo Cache since last submit. + + New ZIP is created with CSV files containing DCR request for Veeva. ZIP is placed in outbound dir in S3 bucket which is further synchronized to Veeva's SFTP. + + Each DCR is updated with ZIP file name which was used to transfer request to Veeva. +* TraceVR + + The process of tracing VR is triggered each hours by Spring Scheduler. + + Inbound S3 bucket is searched for ZIP files with CSVs containing DCR responses from Veeva. There are multiple dirs in S3 buckets, each for specific group of countries (currently CN and APAC). + + Parts of DCR responses are spread across multiple files. Combined information is being processed. + + Finally information about DCR is updated in Mongo Cache and events are produced to dedicated topic for DCR Service 2 for further processing. + +Triggers +======== + +DCR service 2 is being triggered via `/dcr` API calls which are triggered by Data Stewards actions (R1 phase) → "Suggests 3rd party validation" which pushes DCR from Reltio to HUB. + +Dependent components +==================== + +Described in the separated sub-pages for each process. + +Design document for HUB development +=================================== + +1. Design → VeevaOpenData-implementation.docx +2. Reltio HUB-VOD mapping → VeevaOpenDataAPACDataDictionary.xlsx +3. VOD model description (v4) → Veeva\_OpenData\_APAC\_Data\_Dictionary v4.xlsx +--- + +# Create DCR request - Veeva + +**Page ID:** 386814533 +**Page Link:** /display/GMDM/Create+DCR+request+-+Veeva + +Description +=========== + +The process of creating new DCR requests to the Veeva OpenData. During this process, new DCRs are created in DCRregistryVeeva mongo collection. + +Flow diagram +============ + + + +Steps +===== + +* Service is called by Rest API +* Input request is validated. If request is invalid - return response with status REJECTED +* Transform input request to Veeva DCR model + + translate lookup codes to Veeva source codes + + fill the Veeva DCR model with input request values +* Save DCR request to DCRRegistryVeeva mongo collection with status NEW + +Mappings +======== + +DCR domain model→ VOD mapping file: VeevaOpenDataAPACDataDictionary-mmor-mapping.xlsx + +Veeva integration guide +======================= + + +--- + +# Submit DCR Request - Veeva + +**Page ID:** 379333348 +**Page Link:** /display/GMDM/Submit+DCR+Request+-+Veeva + +Description +=========== + +The process of submitting new validation requests to the Veeva OpenData service via VeevaAdapter (communication with S3/SFTP) based on DCRRegistryVeeva mongo collection . During this process, new DCRs are created in VOD system. + +Flow diagram +============ + + + +Steps +===== + +### Veeva DCR service flow: + +* Every N hours Veeva DCR requests with status NEW are queried in DCRRegistryVeeva store. +* DCR are group by country +* For each country: + + - merge Veeva DCR requests - create one zip file for each country + - upload zip file to S3 location + - update DCR status to SENT if upload status is successful + + | DCR entity attributes | Mapping | + | --- | --- | + | DCRID | Veeva VR Request Id | + | VRStatus | "OPEN" | + | VRStatusDetail | "SENT" | + | CreatedBy | MDM HUB | + | SentDate | current time | + +### SFTP integration service flow: + +* Every N  hours grab all zip files from S3 locations +* Upload files to corresponding SFTP server + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **Spring scheduler** | mdm-veeva-dcr-service:VeevaDCRRequestSender | prepare ZIP files for VOD system | Called every specified interval | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| Veeva adapter | Upload DCR request to s3 location | +--- + +# Trace Validation Request - Veeva + +**Page ID:** 379333358 +**Page Link:** /display/GMDM/Trace+Validation+Request+-+Veeva + +Description +=========== + +The process of tracing the VR changes based on the Veeva VR changes. During this process HUB, DCRRegistryVeeva Cache is triggered every hour for SENT DCR's and check VR status using Veeva Adapter (s3/SFTP integration). After verification DCR event is sent to DCR Service 2  Veeva response stream. + +Flow diagram +============ + +** + +Steps +===== + +* Every N get all Veeva DCR responses using Veeva Adapter +* For each response: + + check if status is terminal - (CHANGE\_ACCEPTED, CHANGE\_PARTIAL, CHANGE\_REJECTED, CHANGE\_CANCELLED) + - if not - go to next response + + query DCRregistryVeeva mongo collection for DCR with given key and SENT status + + get Veeva ID (vid\_\_v) from response file + + generate Veeva DCR change event + + update DCR status in DCRRegistryVeeva mongo collection + + - resolution is CHANGE\_ACCEPTED, CHANGE\_PARTIAL + + | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus | "CLOSED" | + | VRStatusDetail | "ACCEPTED" | + | ResponseTime | veeva response completed date | + | Comments | veeva response resolution notes | + - resolution is CHANGE\_REJECTED, CHANGE\_CANCELLED + + | DCR entity attributes | Mapping | + | --- | --- | + | VRStatus | "CLOSED" | + | VRStatusDetail | "REJECTED" | + | ResponseTime | veeva response completed date | + | Comments | veeva response resolution notes | + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Spring scheduler | mdm-veeva-dcr-service:VeevaDCRRequestTrace | start trace validation request process | every hour | +| **OUT** Kafka topic | mdm-dcr-service-2:VeevaResponseStream | update DCR status in Reltio, create relations | invokes Kafka producer for each veeva DCR response | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service 2 | Process response event | +--- + +# Veeva: create DCR method (storeVR) + +**Page ID:** 379332642 +**Page Link:** /pages/viewpage.action?pageId=379332642 + +Description +=========== + +Rest API method exposed in the Veeva DCR Service component responsible for creating new DCR requests specific to Veeva OpenData (VOD) and storing them in dedicated collection for further submit. Since VOD enables communication only via S3/SFTP, it's required to use dedicated mechanism to actually trigger CSV/ZIP file creation and file placement in outbound directory. This will periodic call to Submit VR method will be scheduled once a day (with cron) which will in the end call VeevaAdapter with method createChangeRequest. + +Flow diagram +============ + + + +Steps +===== + +1. Receive the API request +2. Validate initial request + 1. check if the Veeva crosswalk exists once there is an update on the profile + 2. otherwise it's required to prepare DCR to create new Veeva profile + 3. If there is any formal attribute missing or incorrect: skip request +3. Then the DCR is mapped to Veeva Request by invoking mapper between HUB DCR → VEEVA model + 1. For mapping purpose below mapping table should be used + 2. If there is not proper LOV mapping between HUB and Veeva, default fallback should be set to question mark → ? +4. Once proper request has been created,it should be stored as a `VeevaVRDetails` entry in dedicated `DCRRegistryVeeva` collection to be ready for actually send via Submit VR job and for future tracing purposes +5. Prepare return response for initial API request with below logic + 1. Generate sample request after successful mongo insert →  `generateResponse(dcrRequest, RequestStatus.REQUEST_ACCEPTED, null, null)` + 2. Generate error when validation or exception →  `generateResponse(dcrRequest, RequestStatus.REQUEST_FAILED, getErrorDetails(), null);` + +Mapping HUB DCR → Veeva model +============================= + +* Below table does not contain all new attributes which are new in Reltio. Only the most important ones were mentioned there. +* File STTM Stats\_SG\_HK\_v3.xlsx contains full mapping requirements from Veeva OpenData to Reltio data model. It does contain full data mapping which should be covered in target DCR process for VOD. + +| | | | | | | | | | | | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| **Reltio** | | **HUB** | | **VEEVA** | | | | | | | +| **Attribute Path** | **Details** | **DCR Request path** | **Details** | **File Name** | **Field Name** | **Required for Add Request?** | **Required for Change Request?** | **Description** | **Reference (RDM/LOV)** | NOTE | +| **HCO** | | | | | | | | | | | +| N/A | | Mongo Generated ID for this DCR | Kafka KEY | once mapping from HUB Domain DCRRequest take this from DCRRequestD.dcrRequestId: String, // HUB DCR request id - Mongo ID - required in ONEKEY service | change\_request | dcr\_key | Y | Y | Customer's internal identifier for this request | | | +| Change Requests comments | | extDCRComment | | change\_request | description | Y | Y | Requester free-text comments explaining the DCR | | | +| targetChangeRequest.createdBy | | createdBy | | change\_request | created\_by | Y | Y | For requestor identification | | | +| N/A | | if new objects - ADD, if veeva ID CHANGE | | change\_request | change\_request\_type | Y | Y | ADD\_REQUEST or CHANGE\_REQUEST | | | +| N/A | depends on suggested changes (check use-cases) | main entity object type HCP or HCO | | change\_request | entity\_type | Y | N | HCP or HCO | EntityType | | +| N/A | | Mongo Generated ID for this DCR | Kafka KEY | | change\_request\_hco | dcr\_key | Y | Y | Customer's internal identifier for this request | | | +| Reltio Uri and Reltio Type | when insert new profile | entities.HCO.updateCrosswalk.type (Reltio) entities.HCO.updateCrosswalk.value (Reltio id) and refId.entityURI | concatenate Reltio:rvu44dm | change\_request\_hco | entity\_key | Y | Y | Customer's internal HCO identifier | | | +| Crosswalks - VEEVA crosswalk | when update on VEEVA | entities.HCO.updateCrosswalk.type (VEEVA) entities.HCO.updateCrosswalk.value (VEEVA ID) | | change\_request\_hco | vid\_\_v | Y | N | Veeva ID of existing HCO to update; if blank, the request will be interpreted as an add request | | | +| configuration/entityTypes/HCO/attributes/OtherNames/attributes/Name | first element | TODO - add new attribute | | change\_request\_hco | alternate\_name\_1\_\_v | Y | N | | | | +| ?? | | ?? | | change\_request\_hco | business\_type\_\_v | Y | N | | HCOBusinessType | TO BE CONFIRMED | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/FacilityType | | HCO.subTypeCode | | change\_request\_hcp | major\_class\_of\_trade\_\_v | N | N | | COTFacilityType | In PforceRx - Account Type, more info: MR-9512 - Getting issue details... STATUS | +| configuration/entityTypes/HCO/attributes/Name | | name | | change\_request\_hco | corporate\_name\_\_v | N | Y | | | | +| configuration/entityTypes/HCO/attributes/TotalLicenseBeds | | TODO - add new attribute | | change\_request\_hco | count\_beds\_\_v | N | Y | | | | +| configuration/entityTypes/HCO/attributes/Email/attributes/Email | email with rank 1 | emails | | change\_request\_hco | email\_1\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/Email/attributes/Email | email with rank 2 | | change\_request\_hco | email\_2\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/Phone/attributes/Number | phone type TEL.FAX with best rank | phones | | change\_request\_hco | fax\_1\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/Phone/attributes/Number | phone type TEL.FAX with worst rank | | change\_request\_hco | fax\_2\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/StatusDetail | | TODO - add new attribute | | change\_request\_hco | hco\_status\_\_v | N | N | | HCOStatus | | +| configuration/entityTypes/HCO/attributes/TypeCode | | typecode | | change\_request\_hco | hco\_type\_\_v | N | N | | HCOType | | +| configuration/entityTypes/HCO/attributes/Phone/attributes/Number | phone type TEL.OFFICE with best rank | phones | | change\_request\_hco | phone\_1\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/Phone/attributes/Number | phone type TEL.OFFICE with worst rank | | change\_request\_hco | phone\_2\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/Phone/attributes/Number | phone type TEL.OFFICE with worst rank | | change\_request\_hco | phone\_3\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/Country | | DCRRequest.country | | change\_request\_hco | primary\_country\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | elements from COT | specialties | | change\_request\_hco | specialty\_1\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_10\_\_v | N | N | | Speciality | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_2\_\_v | N | N | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_3\_\_v | N | N | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_4\_\_v | N | N | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_5\_\_v | N | N | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_6\_\_v | N | N | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_7\_\_v | N | N | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_8\_\_v | N | N | | | +| configuration/entityTypes/HCO/attributes/ClassofTradeN/attributes/Specialty | | change\_request\_hco | specialty\_9\_\_v | N | N | | | +| configuration/entityTypes/HCO/attributes/Website/attributes/WebsiteURL | first element | websiteURL | | change\_request\_hco | URL\_1\_\_v | N | N | | | | +| configuration/entityTypes/HCO/attributes/Website/attributes/WebsiteURL | N/A | N/A | | change\_request\_hco | URL\_2\_\_v | N | N | | | | +| **HCP** | | | | | | | | | | | +| N/A | | Mongo Generated ID for this DCR | Kafka KEY | | change\_request\_hcp | dcr\_key | Y | Y | Customer's internal identifier for this request | | | +| Reltio Uri and Reltio Type | when insert new profile | entities.HCO.updateCrosswalk.type (Reltio) entities.HCO.updateCrosswalk.value (Reltio id) and refId.entityURI | concatenate Reltio:rvu44dm | change\_request\_hcp | entity\_key | Y | Y | Customer's internal HCP identifier | | | +| configuration/entityTypes/HCP/attributes/Country | | DCRRequest.country | | change\_request\_hcp | primary\_country\_\_v | Y | Y | | | | +| Crosswalks - VEEVA crosswalk | when update on VEEVA | entities.HCO.updateCrosswalk.type (VEEVA) entities.HCO.updateCrosswalk.value (VEEVA ID) | | change\_request\_hcp | vid\_\_v | N | Y | | | | +| configuration/entityTypes/HCP/attributes/FirstName | | firstName | | change\_request\_hcp | first\_name\_\_v | Y | N | | | | +| configuration/entityTypes/HCP/attributes/Middle | | middleName | | change\_request\_hcp | middle\_name\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/LastName | | lastName | | change\_request\_hcp | last\_name\_\_v | Y | N | | | | +| configuration/entityTypes/HCP/attributes/Nickname | | TODO - add new attribute | | change\_request\_hcp | nickname\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Prefix | | prefix | | change\_request\_hcp | prefix\_\_v | N | N | | HCPPrefix | | +| configuration/entityTypes/HCP/attributes/SuffixName | | suffix | | change\_request\_hcp | suffix\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Title | | title | | change\_request\_hcp | professional\_title\_\_v | N | N | | HCPProfessionalTitle | | +| configuration/entityTypes/HCP/attributes/SubTypeCode | | subTypeCode | | change\_request\_hcp | hcp\_type\_\_v | Y | N | | HCPType | | +| configuration/entityTypes/HCP/attributes/StatusDetail | | TODO - add new attribute | | change\_request\_hcp | hcp\_status\_\_v | N | N | | HCPStatus | | +| configuration/entityTypes/HCP/attributes/AlternateName/attributes/FirstName | | TODO - add new attribute | | change\_request\_hcp | alternate\_first\_name\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/AlternateName/attributes/LastName | | TODO - add new attribute | | change\_request\_hcp | alternate\_last\_name\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/AlternateName/attributes/MiddleName | | TODO - add new attribute | | change\_request\_hcp | alternate\_middle\_name\_\_v | N | N | | | | +| ?? | | TODO - add new attribute | | change\_request\_hcp | family\_full\_name\_\_v | N | N | | | TO BE CONFRIMED | +| configuration/entityTypes/HCP/attributes/DoB | | birthYear | | change\_request\_hcp | birth\_year\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Credential/attributes/Credential | by rank 1 | TODO - add new attribute | | change\_request\_hcp | credentials\_1\_\_v | N | N | | | TO BE CONFIRMED | +| configuration/entityTypes/HCP/attributes/Credential/attributes/Credential | 2 | TODO - add new attribute | | change\_request\_hcp | credentials\_2\_\_v | N | N | | | In reltio there is attribute but not used | +| configuration/entityTypes/HCP/attributes/Credential/attributes/Credential | 3 | TODO - add new attribute | | change\_request\_hcp | credentials\_3\_\_v | N | N | | | "uri": "configuration/entityTypes/HCP/attributes/Credential/attributes/Credential", | +| configuration/entityTypes/HCP/attributes/Credential/attributes/Credential | 4 | TODO - add new attribute | | change\_request\_hcp | credentials\_4\_\_v | N | N | | | "lookupCode": "rdm/lookupTypes/Credential", | +| configuration/entityTypes/HCP/attributes/Credential/attributes/Credential | 5 | TODO - add new attribute | | change\_request\_hcp | credentials\_5\_\_v | N | N | | HCPCredentials | "skipInDataAccess": false | +| ?? | | TODO - add new attribute | | change\_request\_hcp | fellow\_\_v | N | N | | BooleanReference | TO BE CONFRIMED | +| configuration/entityTypes/HCP/attributes/Gender | | gender | | change\_request\_hcp | gender\_\_v | N | N | | HCPGender | | +| ?? Education ?? | | TODO - add new attribute | | change\_request\_hcp | education\_level\_\_v | N | N | | HCPEducationLevel | TO BE CONFRIMED | +| configuration/entityTypes/HCP/attributes/Education/attributes/SchoolName | | TODO - add new attribute | | change\_request\_hcp | grad\_school\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Education/attributes/YearOfGraduation | | TODO - add new attribute | | change\_request\_hcp | grad\_year\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_10\_\_v | N | N | | | TO BE CONFRIMED | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_1\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_2\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_3\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_4\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_5\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_6\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_7\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_8\_\_v | N | N | | | | +| ?? | | | | change\_request\_hcp | hcp\_focus\_area\_9\_\_v | N | N | | HCPFocusArea | | +| ?? | | | | change\_request\_hcp | medical\_degree\_1\_\_v | N | N | | | TO BE CONFRIMED | +| ?? | | | | change\_request\_hcp | medical\_degree\_2\_\_v | N | N | | HCPMedicalDegree | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | by rank from 1 to 100 | specialties | | change\_request\_hcp | specialty\_1\_\_v | Y | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_10\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_2\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_3\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_4\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_5\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_6\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_7\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_8\_\_v | N | N | | | | +| configuration/entityTypes/HCP/attributes/Specialities/attributes/Specialty | specialties | | change\_request\_hcp | specialty\_9\_\_v | N | N | | Specialty | | +| configuration/entityTypes/HCP/attributes/WebsiteURL | | TODO - add new attribute | | change\_request\_hcp | URL\_1\_\_v | N | N | | | | +| **ADDRESS** | | | | | | | | | | | +| | | Mongo Generated ID for this DCR | Kafka KEY | | change\_request\_address | dcr\_key | Y | Y | Customer's internal identifier for this request | | | +| Reltio Uri and Reltio Type | when insert new profile | entities.HCP OR HCO.updateCrosswalk.type (Reltio) entities.HCP OR HCO.updateCrosswalk.value (Reltio id) and refId.entityURI | concatenate Reltio:rvu44dm | change\_request\_address | entity\_key | Y | Y | Customer's internal HCO/HCP identifier | | | +| attributes/Addresses/attributes/PfizerAddressID | | address.refId | | change\_request\_address | address\_key | Y | Y | Customer's internal address identifier | | | +| attributes/Addresses/attributes/AddressLine1 | | addressLine1 | | change\_request\_address | address\_line\_1\_\_v | Y | N | | | | +| attributes/Addresses/attributes/AddressLine2 | | addressLine2 | | change\_request\_address | address\_line\_2\_\_v | N | N | | | | +| attributes/Addresses/attributes/AddressLine3 | | addressLine3 | | change\_request\_address | address\_line\_3\_\_v | N | N | | | | +| N/A | | N/A | A | change\_request\_address | address\_status\_\_v | N | N | | AddressStatus | | +| attributes/Addresses/attributes/AddressType | | addressType | | change\_request\_address | address\_type\_\_v | Y | N | | AddressType | | +| attributes/Addresses/attributes/StateProvince | | stateProvince | | change\_request\_address | administrative\_area\_\_v | Y | N | | AddressAdminArea | | +| attributes/Addresses/attributes/Country | | country | | change\_request\_address | country\_\_v | Y | N | | | | +| attributes/Addresses/attributes/City | | city | | change\_request\_address | locality\_\_v | Y | Y | | | | +| attributes/Addresses/attributes/Zip5 | | zip | | change\_request\_address | postal\_code\_\_v | Y | N | | | | +| attributes/Addresses/attributes/Source/attributes/SourceName attributes/Addresses/attributes/Source/attributes/SourceAddressID | when VEEVA map VEEVA ID to | sourceAddressId | | change\_request\_address | vid\_\_v | N | Y | | | | +| map from relationTypes/OtherHCOtoHCOAffiliations or relationTypes/ContactAffiliations | | This will be HCP.ContactAffiliation or HCO.OtherHcoToHCO affiliation | | | | | | | | | +| | | Mongo Generated ID for this DCR | Kafka KEY | | change\_request\_parenthco | dcr\_key | Y | Y | Customer's internal identifier for this request | | | +| | | HCO.otherHCOAffiliations.relationUri or HCP.contactAffiliations.relationUri | (from Domain model) information about Reltio Relation ID | change\_request\_parenthco | parenthco\_key | Y | Y | Customer's internal identifier for this relationship | RELATION ID | | +| | | KEY entity\_key from HCP or HCO (start object) | | change\_request\_parenthco | child\_entity\_key | Y | Y | Child Identifier in the HCO/HCP file | START OBJECT ID | | +| endObject entity uri mapped to refId.EntityURITargetObjectId | | KEY entity\_key from HCP or HCO (end object, by affiliation) | | change\_request\_parenthco | parent\_entity\_key | Y | Y | Parent identifier in the HCO file | END OBJECT ID | | +| | changes in Domain model mapping | map Reltion.Source.SourceName - VEEVA map Relation.Source.SourceValue - VEEVA ID | add to Domain model map if relation is from VEEVA ID | change\_request\_parenthco | vid\_\_v | N | Y | | | | +| | | start object entity type | | change\_request\_parenthco | entity\_type\_\_v | Y | N | | | | +| attributes/RelationType/attributes/PrimaryAffiliation | | if is primary TODO - add new attribute to otherHcoToHCO | | change\_request\_parenthco | is\_primary\_relationship\_\_v | N | N | | BooleanReference | | +| | | HCO\_HCO or HCP\_HCO | | change\_request\_parenthco | hierarchy\_type\_\_v | | | | RelationHierarchyType | | +| attributes/RelationType/attributes/RelationshipDescription | | type from affiliation based on ContactAffliation or OtherHCOToHCO affiliation | I think it will be 14-Emploted for HCP\_HCO and 4-Manages for HCO\_HCO but maybe we can map from affiliation.type | change\_request\_parenthco | relationship\_type\_\_v | Y | N | | RelationType | | + +Mongo collection +================ + +All DCRs initiated by the `dcr-service-2` API and to be sent to Veeva will be stored in Mongo in new collection `DCRRegistryVeeva`. The idea is to gather all DCRs requested by the client through the day and schedule ‘`SubmitVR`’ process that will communicate with Veeva adapter. + +Typical use case: + +* Client requests 3 DCRs during the day +* `SubmitVR` contains the schedule that gathers all DCRs with **NEW** status created during the day and using VeevaAdapter to push requests to S3/SFTP. + +In this store we are going to keep both types of DCRs: + +``` +initiated by PforceRX - PFORCERX_DCR("PforceRxDCR") +initiated by Reltio SubmitVR - SENDTO3PART_DCR("ReltioSuggestedAndSendTo3PartyDCR"); +``` + +Store class idea: + +* \_id – this is the same ID that was assigned to DCR in dcr-service-2 + +**VeevaVRDetails** + +``` +@Document("DCRRegistryVEEVA") +@JsonIgnoreProperties(ignoreUnknown = true) +@JsonInclude(JsonInclude.Include.NON_NULL) +data class VeevaVRDetails( +    @JsonProperty("_id") +    @Id +    val id: String? = null, +    val type: DCRType, +    val status: DCRRequestStatusDetails, +    val createdBy: String? = null, +    val createTime: ZonedDateTime? = null, +    val endTime: ZonedDateTime? = null, +    val veevaRequestTime: ZonedDateTime? = null, +    val veevaResponseTime: ZonedDateTime? = null, +    val veevaRequestFileName: String? = null +    val veevaResponseFileName: String? = null    val veevaResponseFileTime: ZonedDateTime? = null +    val country: String? = null, +    val source: String? = null, +    val extDCRComment: String? = null, // external DCR Comment (client comment) +    val trackingDetails: List = mutableListOf(), + +    RAW FILE LINES mapped from DCRRequestD to Veeva model +    val veevaRequest: +            val change_request_csv: String, +            val change_request_hcp_csv: String +            val change_request_hco_csv: List +            val change_request_address_csv: List +            val change_request_parenthco_csv: List + +    RAW FILE LINES mapped from Veeva Response model +    val veevaResponse: +            val change_request_response_csv: String, +            val change_request_response_hcp_csv: String +            val change_request_response_hco_csv: List +            val change_request_response_address_csv: List +            val change_request_response_parenthco_csv: List +) +``` + +Mapping Reltio canonical codes → Veeva source codes +=================================================== + +There are a couple of steps performed to find out a mapping for canonical code from Reltio to source code understood by VOD. Below steps are performed (in this order) once a code is found. + +Veeva Defaults +-------------- + +Configuration is stored in `mdm-config-registry > config-hub/stage_apac/mdm-veeva-dcr-service/defaults` + +The purpose of these logic is to select one of possible multiple source codes on VOD end for a single code on Pfizer side (1:N). The other scenario is when there is no actual source code for a canonical code on VOD end (1:0), however this is usually covered by fallback code logic. + +There are a couple of files, each containing source codes for a specific attribute. The ones related to `HCO.Specialty` and `HCP.Specialty` have logic which selects proper code. + +* Usually there are constructed as a three column CSV: `Country, Canonical Code, Source Code` +* For specific `Country` we're looking for `Canonical code` and then we're sending `Source code` as it is (no trim required) + + Examples: `IN;SP.PD;PD` → `PD` source code will be sent to VOD + +RDM lookups with RegExp +----------------------- + +The main logic which is used to find out proper source code for canonical code. We're using codes configured in RDM, however mongo collection LookupValues are used. For specific canonical code (code) we looking for sourceMappings with source = VOD. Often country is embedded within source code so we're applying regexpConfig (more in Veeva Fallback section) to extract specific source code for particular country. + +Veeva Fallback +-------------- + +Configuration is stored in `mdm-config-registry > config-hub/stage_apac/mdm-veeva-dcr-service/fallback` + +* Available for a couple of attributes: + + `hco-cot-facility-type.csv` + - COTFacilityType + + `hco-specialty.csv` + - COTSpecialty + + `hco-type-code.csv` + - HCOType + + `hcp-specialty.csv` + - HCPSpecialty + + `hcp-title.csv` + - HCPTitle + + `hcp-type-code.csv` + - HCPSubTypeCode +* Usually files are constructed as a one column CSV, however the logic for extracting source code may be different +* Source code is extracted using RegExp for each parameter. Check `application.yml` for this mdm-veeva-dcr-server component - `mdm-inboud-services > mdm-veeva-dcr-service/src/main/resources/application.yml` to find out proper line and extract code sent to VOD. + + Example value for `hco-specialty-type.csv: IN_?` + + Regexp value for HCP.specialty: `regexpConfig > HCPSpecialty: ^COUNTRY_(.+)$` + + Source code sent to VOD for India country: "`?`" (only question mark without country prefix) + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| REST call | mdm-veeva-dcr-service: POST /dcr → veevaDCRService.createChangeRequest(request) | Creates DCR and stores it in collection without actual send to Veeva. | API synchronous requests - realtime | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service 2 | Main component with flow implementation | +| Hub Store | DCR and Entities Cache | +--- + +# Veeva: create DCR method (submitVR) + +**Page ID:** 386796763 +**Page Link:** /pages/viewpage.action?pageId=386796763 + +Description +=========== + +Gather all stored DCR entities in `DCRRegistryVeeva` collection (status = NEW) and sends them via S3/SFTP to Veeva OpenData (VOD). This method triggers CSV/ZIP file creation and file placement in outbound directory. This method is triggered from cron which invokes VeevaDCRRequestSender.sendDCRs() from the Veeva DCR Service + +Flow diagram +============ + + + +Steps +===== + +1. Receive the API request via scheduled trigger, usually every 24h (`senderConfiguration.schedulerConfig.fixedDelay`) at specific time of day (`senderConfiguration.schedulerConfig.initDelay)` +2. All DCR entities (`VeevaVRDetails`) with status NEW are being retrieved from `DCRRegistryVeeva` collection +3. Then `VeevaCreateChangeRequest` object is created which aggregates all CSV content which should be placed in actual CSV files. + 1. Each object contains only DCRs specific for `country` + 2. Each `country` has its own S3/SFTP directory structure as well as dedicated SFTP server instance +4. Once CSV files are created with header and content, they are packed into single ZIP file +5. Finally ZIP file is placed in outbound S3 directory +6. If file was placed + 1. successfuly - then `VeevaChangeRequestACK` status = `SUCCESS` + 2. otherwise - then `VeevaChangeRequestACK` status = `FAILURE` and process ends +7. Finally, status of `VeevaVRDetails` entity in `DCRRegistryVeeva` collection is updated and set to `SENT_TO_VEEVA` + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| Timer (cron) | mdm-veeva-dcr-service: VeevaDCRRequestSender.sendDCRs() | Takes all unsent entities (status = NEW) from Veeva collection and actually puts file on S3/SFTP directory via `veevaAdapter.createDCRs` | Usually every 24h (`senderConfiguration.schedulerConfig.fixedDelay`) at specific time of day (`senderConfiguration.schedulerConfig.initDelay)` | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service 2 | Main component with flow implementation | +| Hub Store | DCR and Entities Cache | +--- + +# Veeva: generate DCR Change Events (traceVR) + +**Page ID:** 379329922 +**Page Link:** /pages/viewpage.action?pageId=379329922 + +Description +=========== + +The process is responsible for gathering DCR responses from Veeva OpenData (VOD). Responses are provided via CSV/ZIP files placed on S3/SFTP server in inbound directory which are specific for each country. During this process files should be retrieved, mapped from VOD to HUB DCR model and published to Kafka topic to be properly processed by DCR Service 2, Veeva: process DCR Change Events. + +Flow diagram +============ + + + +Source: Lucid + +Steps +===== + +1. Method is trigger via cron, usually every 24h (`traceConfiguration.schedulerConfig.fixedDelay`) at specific time of day (`traceConfiguration.schedulerConfig.initDelay`) +2. For each country, each inbound directory in scanned for ZIP files +3. Each ZIP files (`_DCR_Response_.zip`) should be unpacked and processed. A bunch of CSV files should be extracted. Specifically: + 1. `change_request_response.csv` → it's a manifest file with general information in specific columns + 1. `dcr_key` → ID of DCR which was established during DCR request creation + 2. `entity_key` → ID of entity in Reltio, the same one we provided during DCR request creation + 3. `entity_type` → type of entity (HCO, HCP) which is being modified via this DCR + 4. `resolution` → has information whether DCR was accepted or rejected. Full list of values is below. + 1. | **`resolution` value** | **Description** | + | --- | --- | + | CHANGE\_PENDING | This change is still processing and hasn't been resolved | + | CHANGE\_ACCEPTED | This change has been accepted without modification | + | CHANGE\_PARTIAL | This change has been accepted with additional changes made by the steward, or some parts of the change request have been rejected | + | CHANGE\_REJECTED | This change has been rejected in its entirety | + | CHANGE\_CANCELLED | This change has been cancelled | + 5. `change_request_type` + 1. | `change_request_type` value | Description | + | --- | --- | + | ADD\_REQUEST | whether DCR caused to create new profile in VOD with new `vid__v`  (Veeva id) | + | CHANGE\_REQUEST | just update of existing profile in VOD with existing and already known `vid__v` (Veeva id) | + 2. `change_request_hcp_response.csv` - contains information about DCR related to HCP + 3. `change_request_hco_response.csv` - contains information about DCR related to HCO + 4. `change_request_address_response.csv` - contains information about DCR related to addresses which are related to specific HCP or HCO + 5. `change_request_parenthco_response.csv` - contains information about DCR which correspond to relations between HCP and HCO, and HCO and HCO + 6. File with log: `_DCR_Request_Job_Log.csv` can be skipped. It does not contain any useful information to be processed automatically +4. For all DCR responses from VOD, we need to get corresponding DCR entity (`VeevaVRDetails)`from collection `DCRRegistryVeeva` should be selected. +5. In general, specific response files are not that important (VOD profiles updates will be ingested to HUB via ETL channel) however when new profiles are created (`change_request_response.csv.change_request_type = ADD_REQUEST`) we need to extract theirs Veeva ID. + 1. We need to deep dive into `change_request_hcp_response.csv` or `change_request_hco_response.csv` to find `vid__v` (Veeva ID) for specific `dcr_key` + 2. This new Veeva ID should be stored in `VeevaDCREvent.vrDetails.veevaHCPIds` + 3. It should be further used as a crosswalk value in Reltio: + + 1. entities.HCO.updateCrosswalk.type (VEEVA) + 2. entities.HCO.updateCrosswalk.value (VEEVA ID) +6. Once data has been properly mapped from Veeva to HUB DCR model, new `VeevaDCREvent` entity should be created and published to dedicated Kafka topic `$env-internal-veeva-dcr-change-events-in` + 1. Please be advised, when the status of resolution is not final (CHANGE\_ACCEPTED, CHANGE\_REJECTED, CHANGE\_CANCELLED, CHANGE\_PARTIAL) we should not sent event to DCR-service-2 +7. Then for each successfully processed DCR entity (`VeevaVRDetails`) in Mongo  DCRRegistryVeeva collection should be updated + 1. | Veeva CSV: resolution | **Mongo: DCRRegistryVeeva** Entity: VeevaVRDetails.status: DCRRequestStatusDetails | Topic: $env-internal-veeva-dcr-change-events-in Event: VeevaDCREvent.vrDetails.vrStatus | Topic: $env-internal-veeva-dcr-change-events-in Event: VeevaDCREvent.vrDetails.vrStatusDetail | + | --- | --- | --- | --- | + | CHANGE\_PENDING | *status should not be updated at all (stays as SENT)* | *do not send events to DCR-service-2* | *do not send events to DCR-service-2* | + | CHANGE\_ACCEPTED | ACCEPTED | CLOSED | ACCEPTED | + | CHANGE\_PARTIAL | ACCEPTED | CLOSED | ACCEPTED *resolutionNotes / veevaComment should contain more information what was rejected by VEEVA DS* | + | CHANGE\_REJECTED | REJECTED | CLOSED | REJECTED | + | CHANGE\_CANCELLED | REJECTED | CLOSED | REJECTED | +8. Once files are processed, ZIP file should be moved from inbound to archive directory + +Event VeevaDCREvent Model +------------------------- + +``` +data class VeevaDCREvent (val eventType: String? = null, +                          val eventTime: Long? = null, +                          val eventPublishingTime: Long? = null, +                          val countryCode: String? = null, +                          val dcrId: String? = null, +                          val vrDetails: VeevaChangeRequestDetails) + +data class VeevaChangeRequestDetails ( +    val vrStatus: String? = null, - HUB CODEs +    val vrStatusDetail: String? = null, - HUB CODEs +    val veevaComment: String? = null, +    val veevaHCPIds: List? = null, +    val veevaHCOIds: List? = null) +``` + +Triggers +======== + +| Trigger action | Component | Action | Default time | +| --- | --- | --- | --- | +| **IN** Timer (cron) | mdm-veeva-dcr-service: VeevaDCRRequestTrace.traceDCRs() | get DCR responses from S3/SFTP directory, extract CSV files from ZIP file and publish events to kafka topic | every hour usually every 6h (`traceConfiguration.schedulerConfig.fixedDelay`) at specific time of day (`traceConfiguration.schedulerConfig.initDelay`) | +| **OUT** Events on Kafka Topic | mdm-veeva-dcr-service: VeevaDCRRequestTrace.traceDCRs() $env-internal-veeva-dcr-change-events-in | VeevaDCREvent event published to topic to be consumed by DCR Service 2 | every hour usually every 6h (`traceConfiguration.schedulerConfig.fixedDelay`) at specific time of day (`traceConfiguration.schedulerConfig.initDelay`) | + +Dependent components +==================== + +| Component | Usage | +| --- | --- | +| DCR Service 2 | Main component with flow implementation | +| Hub Store | DCR and Entities Cache | +--- +