High A Profile — Series Analysis How do you know when you’ve done a good job? It sounds like a simple question, yet the answer often reveals something deeper about how we are naturally wired. For some people, a sense of achievement emerges when a manager expresses appreciation, when colleagues offer praise, or when an … Read more
Overview The cdm_Archive_to_STG notebook is a critical component in the cdm_today and cdm_Archive pipeline. Its primary role is to create a staging table that represents a point-in-time snapshot of source data, which is then used by the cdm__Archive_upsert notebook to accurately update the main dimension table. This staging layer acts as a buffer between raw … Read more
Overview The myOSH_To_Bronze notebook is designed to handle multiple MyOSH API endpoints dynamically, meaning it can process different endpoints such as records or users without hardcoding logic. For most endpoints (like users), the process is straightforward: it performs a regular API call and writes the response directly to JSON in the source container and to … Read more
Overview The myosh_Archive notebook is designed to maintain a complete and accurate archive of records from the MyOSH API. It ensures that no data is lost by identifying gaps in the current API response and backfilling missing records, then merging everything into a single, consolidated archive stored in the Azure Data Lake container. Why Do … Read more
Overview The cdm__Archive_upsert notebook is designed to help keep accurate historical records of data. It implements a technique called Slowly Changing Dimension Type 2 (SCD2), which means instead of overwriting old information, it creates a timeline of changes so you can see what was true at any point in time. This method ensures that previous … Read more
When working with Microsoft Fabric, one common challenge is keeping the SQL endpoint in sync with the Lakehouse endpoint after a pipeline run. Fabric provides two endpoints for every Lakehouse: Why does the SQL endpoint sometimes lag? The SQL endpoint doesn’t automatically refresh its metadata the moment new data lands in the Lakehouse. Instead, a … Read more
Purpose:To run the stored procedure get.myArchive from Power BI for a series of previous Sundays (for example, the last six Sundays), and optionally combine or parameterise the results. 🧭 Background The get.myArchive procedure is designed to retrieve snapshot data for a specified date.By default, Power BI passes a single date parameter such as: However, when … Read more
A calm, consistent, and clever approach to data engineering. 🔹 1. Look in the Box First Before inventing a workaround, check what already exists.If Microsoft or IFS built it, use it — it’s likely more robust, secure, and supported. 🔹 2. Easy Landings Data should arrive safely and predictably.Keep import containers simple, standardised, and transformation-free.Bronze … Read more
To ensure efficiency, compliance, and trust in the Data Warehouse, we commit to the principle of non-proliferation of information. This means that data will be held and processed in a way that avoids unnecessary duplication, ensures clear lineage, and retains only what is required for business and regulatory purposes. Standards This approach ensures the Data … Read more
1. Collaboration & Communication 2. Reliability & Delivery 3. Technical & Professional Expertise 4. Innovation & Improvement 5. Customer/Stakeholder Focus 6. Leadership & Influence (even without formal authority) 7. Personal Attributes