What constitutes ‘sensitive personal data’?
In its most basic definition, sensitive data is a specific set of “special categories”: Article 9 of the GDPR prohibits special category data from being processed unless you can rely on an exemption.
Common Data Model
Standardised DataMart Model
Core Views and their Standard Structures Each DataMart will be built on six primary views: 1. Details (or Core) View Purpose: Schema: Column Name Data Type Description company VARCHAR Primary entity identifier object_id VARCHAR Unique object identifier (project_id) seq_json JSON Structured metadata reference for linking name VARCHAR Name of the project/entity description TEXT Brief descriptive … Read more
Service Level Agreement (SLA) for DataMart availability
This draft framework provides a robust SLA for DataMart availability, balancing business needs, technical feasibility, and stakeholder communication. 1. Availability Target Set a clear uptime percentage that aligns with the criticality of the DataMart to your business processes. 2. Operational Hours Define the expected operational hours of the DataMart. Specify if it needs to be … Read more
Model-View-Controller
MVC, which stands for Model-View-Controller, is a design pattern used in software engineering, particularly in web and application development. It divides an application into three interconnected components, which separates internal data logic from the user interface, allowing for more manageable and scalable code. 1. Model: 2. View: 3. Controller: Workflow Example: Benefits of MVC:
Medallion.Platinum
Overview: The Platinum Layer is the most advanced tier in the data warehouse, dedicated to performing complex business transformations and machine learning (ML) exercises. This layer leverages advanced analytics to generate insights that support budget, forecasting and predictive analytics. The resulting transformed data and predictive models are then returned to the Gold Layer for delivery. … Read more
Medallion.Gold
Overview: The Gold Layer serves as the delivery centre of the data warehouse, providing secure and efficient access to the finalised data. This layer is responsible for delivering data to end-users and applications, ensuring that it is ready for analysis, reporting, and other business intelligence activities. Key Characteristics: Workflow Summary: Importance: RAID Risks: Issues: Dependencies: … Read more
Medallion.Silver
Overview: The Silver Layer, also referred to as the ‘Staging’ SQL database, is where data undergoes further refinement and transformation. This layer focuses on structuring the data into well-defined snapshots, adhering to a common data model. The goal is to create standardized and consistent data sets that are ready for analytical purposes and easy integration … Read more
Medallion.Bronze
Overview: The Bronze Layer is the initial raw data storage tier in the data warehouse, representing the true start of the data lake. It is designed to store data in its raw, untransformed state, yet in an organised and accessible manner. This layer ensures that the data is preserved in its original form, allowing for … Read more
Import Layer of Data Warehouse
Overview: The Import Layer serves as the initial landing zone for all incoming data into the data warehouse. This layer is designed to handle and store data in a variety of formats and from different sources, ensuring a flexible and robust intake process. It acts as a staging area where data can be validated, archived, … Read more
Components of the MVC Design Pattern
1. Model The Model component in the MVC (Model-View-Controller) design pattern represents the data and business logic of an application. It is responsible for managing the application’s data, processing business rules, and responding to requests for information from other components, such as the View and the Controller. 2. View Displays the data from the Model … Read more