Overview: The Silver Layer, also referred to as the ‘Staging’ SQL database, is where data undergoes further refinement and transformation. This layer focuses on structuring the data into well-defined snapshots, adhering to a common data model. The goal is to create standardized and consistent data sets that are ready for analytical purposes and easy integration … Read more
Overview: The Bronze Layer is the initial raw data storage tier in the data warehouse, representing the true start of the data lake. It is designed to store data in its raw, untransformed state, yet in an organised and accessible manner. This layer ensures that the data is preserved in its original form, allowing for … Read more
Overview: The Import Layer serves as the initial landing zone for all incoming data into the data warehouse. This layer is designed to handle and store data in a variety of formats and from different sources, ensuring a flexible and robust intake process. It acts as a staging area where data can be validated, archived, … Read more
Purpose The Common View for Supplier Details, excluding dates or measures, acts as a unified and reliable source for data on business suppliers. It is accessible to different departments and systems across the organisation. This view supports integration with fact tables for reporting and analysis and provides a limited set of dimension attributes. Use Case … Read more
Alterations Busopp.Details_Plus (&202407) Remove CONVERT(NVARCHAR(32),HashBytes(‘MD5’, CONCAT_WS(‘|’,src.COMPANY,src.OPPORTUNITY_NO)),2) as busopp_item_id, as this is now generated in get.myView Created dvw.customer_item_202406 as part of removing Company reference from customer (& Supplier) Customer.Details Removed reference to COMPANY (and the associated check of customer.invoice) Customer (and Supplier) COMPANY defaults to 9900 Check for Alterations to Customer.InvoiceCutomer.OrderlineProject.CustomerInvoiceProject.CustomerOrderlineProject.ActivityDetails get.myView COMPANY now included as … Read more
Step 1: Download SQL Files from Azure Storage Step 2: Flatten Directory Structure and Compare Files Using Python Notebook Step 3: Analyse Differences Using Visual Studio Code
Released immediately due to data error discovered in [QFACT_BMT_ACTUAL_HOURS_BY_IAS$] PRD 20240731 Alterations Employee.Hours Change UPLIFT to CAST(GROSS AS DECIMAL(10, 3)) – CAST(NETT AS DECIMAL(10, 3)) AS UPLIFT, As UPLIFT was being delivered with discrepancies between rounding of GROSS and NETT i.e. 21.8104 – 21.81 = -4.0E-4 This was causing an error in both CAST and … Read more
Purpose: Transfer SharePoint List data through azure blob storage and into the DataWarehouse Requires: Prerequisites: Process Steps: 4. Data Pipeline 5. Create pipeline parameters. Create two pipeline parameters. 5. Activities Tab 6. Look up Activity Click on the Lookup activity and go to settings, from there if not already set up create a new connection … Read more
Purpose: Transfer SharePoint list the Database environment for DataWarehouse Requires: Process Steps: 4. DataFlow Gen2 5. Power Query Window 6. Get Data 7. More… 8. SharepPoint Online List (NOT Sharepoint List) 9. Enter URL 10. Set Implementation to [2.0] 11. [Next] 12. Choose your list 13. [Create] 14 . Rename Dataflow 15. Select [Cog] on … Read more
1. Create a blank query in Power Query Editor like so: 2. Keep “Include in report refresh” unselected. 3. Create a Card visual to show “LastPublishedDate”. 4. Before publishing the report, you need to refresh the data of the “LastPublishedDate” table, not refresh the report.