Project Plan or other project management document. Data Discovery Stage: Objective: Identify data sources, understand data requirements, and define the scope of the DataMart project. Key Activities: Deliverables: Design Stage: Objective: Design the architecture, schema, and data models for the DataMart. Key Activities: Deliverables: MVP (Minimum Viable Product) Stage: Objective: Develop and deploy a minimum … Read more

Organisation/Team responsible for management of Service Desk requests/incidents. Escalation Routes All reports WILL provide a method of raising a Service Desk request for support.  Once raised the following escalation routes will be followed. Issues Reporting (IT Service Desk) Stage Activity Comments Call Received Greet & Verify IdentityDocument Caller Details   Problem Identification Listen & Clarify … Read more

Identification of controlled data, its classification, and associated processes. Data Identification Inventory of Data Assets Data Cataloguing: Create a comprehensive inventory of all data assets across your organisation, including databases, data warehouses, data marts, and data lakes. Data Source Mapping: Identify and map all data sources, including internal systems and third-party sources. All Data Sources … Read more

Design methodology, development platform, hardware requirements, database. All Software Quality Procedures Continuous Integration and Continuous Deployment (CI/CD) Continuous Integration and Continuous Deployment (CI/CD) ensures that changes and updates to data pipelines are tested, integrated, and deployed to production, facilitating consistent and reliable data processing and delivery. In dynamic data environments where sources, formats, and requirements … Read more

Reporting of progress to customer/project manager[SW1] [JK2] . Data Engineering SHALL use a Jira(R) style ticket management tool to plan, track and release tickets and tasks for the purpose demonstrating the entire development lifecycle.  Allowing the wider team to move work forward, stay aligned and communicate in context. Data Engineering has chosen to use an Open Source … Read more

Assurance System Test, Regression Test, Unit Test activities, UA Tests, Peer Reviews (with estimated dates), code review methodology (Pull Requests/Pair Programming/other) A new DataMart feature is more than several lines of code. It is usually a multilayer, complex pipeline system, incorporating separate functional components and third-party data integrations. Therefore, efficient development testing should go far … Read more

SQP, requirements catalogue, package details, method of rolling out. DataMart Lifecycle (Development Plan) Output Purpose Owner Location Data Source Inventory Identify relevant data sources within the organisation, including databases, applications, and external sources. Data Architect/Engineer (with Data Specialist) myBMT(Data Objects) Data profiling reports Profile data sources to understand data quality, structure, and relationships. Data Architect/Engineer … Read more

Frequency, location and style of meetings, who should attend. To include standups/sprint meetings, code review methods and end of project review. Incorporating an Agile approach forms an integral aspect of the Data Engineering development process. By dividing the development cycle into smaller segments, iterations, and sprints, Agile methodologies enable engineers and data specialists to collaborate … Read more

People involved and their roles, chains of command and reporting methods, distribution list of SQP. Data Steering Committee Data Stewards Data Specialists Quality Behaviours of Data Specialists Data Engineering Data Mesh Principle The Data Mesh decentralises data ownership by transferring the responsibility from the central data team to the business units that create and consume … Read more

Scope covered by the project and any documents received. Technical Perspective The Data Warehouse Development Quality Plan describes the creation, organisation, movement, change and storage of data from the perspective of the technical and physical implementation. The scope of the Data Warehouse Development Quality Plan encompasses the following key areas: Business Perspective The business perspective … Read more