Overview The myOSH_To_Bronze notebook is designed to handle multiple MyOSH API endpoints dynamically, meaning it can process different endpoints such as records or users without hardcoding logic. For most endpoints (like users), the process is straightforward: it performs a regular API call and writes the response directly to JSON in the source container and to … Read more
Overview The myosh_Archive notebook is designed to maintain a complete and accurate archive of records from the MyOSH API. It ensures that no data is lost by identifying gaps in the current API response and backfilling missing records, then merging everything into a single, consolidated archive stored in the Azure Data Lake container. Why Do … Read more
Overview The cdm__Archive_upsert notebook is designed to help keep accurate historical records of data. It implements a technique called Slowly Changing Dimension Type 2 (SCD2), which means instead of overwriting old information, it creates a timeline of changes so you can see what was true at any point in time. This method ensures that previous … Read more
When working with Microsoft Fabric, one common challenge is keeping the SQL endpoint in sync with the Lakehouse endpoint after a pipeline run. Fabric provides two endpoints for every Lakehouse: Why does the SQL endpoint sometimes lag? The SQL endpoint doesn’t automatically refresh its metadata the moment new data lands in the Lakehouse. Instead, a … Read more
SharePoint list → cleaned, flat, ISO-dated table A reusable Power Query (M) snippet for Dataflows Gen2 that: When to use: Landing SharePoint list data into a Bronze/Silver layer (Fabric/Synapse/SQL) where flat, predictable schemas, text-formatted dates, and stable column names are preferred. Full code (drop-in) Step-by-step Customisation points Validation checklist Troubleshooting Notes
Purpose To provide a clear, sustainable leadership structure for the Data & Analytics function by separating accountability for: while maintaining shared ownership of standards and governance. This model is intended to: Context The Data & Analytics function currently spans: These responsibilities place competing demands on a single leadership role: To reduce strain and increase effectiveness, … Read more
Role Purpose The Head of Reporting & Analytics Delivery provides leadership for the organisation’s reporting, analytical outputs, and stakeholder-facing data products. The role exists to ensure that: The role focuses on: Scope and Impact The role: The Head of Reporting & Analytics Delivery operates with: Key Responsibilities (SFIA-8 Aligned) Strategy and Direction (SFIA: Information Strategy, … Read more
Weekly Release Note – Week Ending 03 November 2025 New Views project_meta_codes_pmn A new DataMart view based on data sourced from the SharePoint List PMN, processed via a GEN2 Dataflow to the Bronze layer.🔗 View Documentation This view enables portfolio and project management insights by transforming Project Manager Narratives (PMNs) into a structured, analysable format. … Read more
Category: Utility Functions → String & EncodingAuthor: Data EngineeringLanguage: Power Query (M)Purpose: Converts a text string into a URL-safe format by encoding non-alphanumeric characters as percent-encoded hexadecimal values (%XX).Compatibility: Power BI, Excel Power Query, and Fabric Dataflows (Gen2) 📝 Description UrlEncode transforms text into a format suitable for inclusion in URLs or API query strings.All … Read more
Purpose This view supports financial control and transparency by standardising balance-level data for consistent reporting and reconciliation. It helps finance and business leaders understand balance movements, inter-company positions, and the relationship between projects, programmes, and cost centres. Use Case Suggested dashboard insights: Source Integration with Dimension / Reference Tables Joins used (per SQL): (none in … Read more