I had been following the hints and clues on Twitter, where some Power BI team members were changing their job titles to Synapse. I was curious and hopeful about what they were working on, but I also didn’t want Power BI to lose momentum. When I saw the leadership changes, I felt reassured that whatever they did would be amazing.
Fast forward, at Build, Microsoft announced the long-awaited Microsoft Fabric. What is it? It’s a SaaS Data Fabric platform with a common data lake called OneLake. It’s a one-stop shop for all the analytics needs of every enterprise. With fabric, there is no need to stitch together services from multiple vendors and all data is saved as delta parquet files. No more vendor lock-in or proprietary data formats!
The data Fabric concept is not new; as per datanami, Noel Yuhanna has been at the forefront of Data Fabric. In his words, “A data fabric is essentially an abstraction layer that links a disparate collection of data tools that address key pain points in big data projects. A data fabric solution should deliver capabilities in the areas of data access, discovery, transformation, integration, security, governance, lineage, and orchestration. It should also provide self-service capabilities, as well as some graph capabilities to identify connected data.
By providing a way to bring these data management capabilities to bear on data spanning all these silos, a data fabric can help alleviate core data management challenges holding companies back from higher-level data use cases, including advanced analytics and AI in the cloud.”
Microsoft Fabric takes this concept to the next level by integrating it with Power BI, Microsoft Office and Fluent UI frameworks, Integration with Office and Power BI makes the time and steps to data insights very flexible. Fluent UI provides a consistent and beautiful user experience across all artefacts. I think Microsoft Fabric is the ultimate solution for anyone who wants to leverage the power of data in the cloud. Whether you prefer a centralized or distributed approach, Microsoft Fabric can handle it all. You don’t have to change your existing skills, and you can use the tool or language you are comfortable with. There is plenty of information on MS Docs, and I am sure many people will be talking about it. I want to take my take on Microsoft Fabric from an ex-Microsoft SQL Server BI developer.
You see, I have a long history with Microsoft SQL Server and BI. I used to work with SSIS, SSRS and SSAS to create amazing data solutions. I loved how I could integrate, transform and analyze data using these tools. I felt like I had everything I needed in one place: SQL Server. Ever since, no matter what tool I worked with, I still somehow try to connect the dots back to MSBI. This might be because it helps me to understand the topic better or not to get intimidated by changing world.
Fabric sounds game changer, but as an ex-MS SQL BI developer, I find it familiar. In my older days, I accumulated data in SQL Server databases. I made ETL or ELT using SQL-stored procedures and my first love SSIS. All my data integration tasks were mainly done using SSIS but sometimes with a few bits of C# and hardcore SQL Procedures. I used SSAS to create dimensional models; all calculations were traditionally created by MDX and then DAX.
I created beautiful SSRS reports with cool features like data-driven parameters, conditional formatting, look-up sets. I kept my end users on top of their KPIs using SSRS subscriptions. Not only end users, hardcore IT guys knew when the server was hitting limits when jobs were failing, even some occasional happy birthday emails using SQL Server Task Manager and sometimes with a bit of help from the Windows Server task manager. Whatever I did, I kind of moved, transformed and analysed all the data on my SQL Server. There were occasional intruders like Oracle, My Sql etc but I was faithful to SQL Server.
But times have changed, and so has the data landscape. Now we have more data sources, more data types, and more data challenges. We need to be able to handle big data, streaming data, unstructured data and more. We need to be able to scale, secure and optimize our data pipelines. We need to be able to deliver insights faster and easier.
That’s where Microsoft Fabric comes in. It’s a platform that lets me connect all these different data sources and destinations. If I have an existing data platform, nicely formatted and all; I used to use linked servers, now I would use Shortcuts on Fabric. Back then, used to access data files with a mapped network drive on Server. Now I would probably mount my storage on Fabric. Microsoft Fabric lets me connect all kinds of data sources and destinations. Whether I have a well-structured data platform or a bunch of files on a server, I can easily access them with Shortcuts and Mounts on Fabric.
For Data Integration, I used a combination of SSIS and SQL Server functionalities like stored procedures and functions. Now on Fabric, I can use Data Pipelines and Data Flows Gen-II to create visual and intuitive workflows for my data transformations. Or I can use Notebooks to write code in any language I want. I think I will be mostly using Pipelines and Data flows Gen-II. I am a more visual person.
I haven’t played enough, but it looks like, I can create multiple data flows and call those data flows from Data Factory pipelines. That would be really useful for scenarios like For Each Loop.
Next comes methodologies or frameworks of data storage. The proven approach then was ETL – Extract Raw data from operational data sources, Transform and store in a Transactional layer, then Load into a Data warehouse; in a presentable format for further data analysis. I think the old ETL is now Medallion Lakehouse architecture. For larger data warehouses we used to create Databases schemas or data marts for each department or domain, and now that’s domain-based data mesh architecture.
Microsoft Fabric supports any data storage framework or architecture I need. I can use Workspaces and Domains to organize my data into different stages and domains. I can follow the Medallion Lakehouse approach or create custom data marts for each department or domain.
Next data analytics, in the past, we had three options for creating an analytics layer: SQL Views, SQL DW and SSAS Model. The choice depended on many factors, such as the urgency, the use case, the budget and the developer’s skills. But now, with Fabric, which gives us more flexibility and power. We can use gold-level lakehouse tables for fast and reliable data access, or we can build a Warehouse with custom measures and calculations, or we can create a Power BI Dataset for interactive visualization.
In my opinion, the priority would be time to insights and performance. Using the benefits of DeltaLake and one copy would be my highest priority. But again, there are best practices and client scenarios. It’s not always easy to follow a recommended framework.
But that’s not all. Analytics is not just about making nice reports with actionable insights. We are in the age of AI. We want those insights to reach the end users in the most effective way possible. In the past, I used SQL Task Manager and Windows Task Manager to automate things. I used SSRS and Excel for further analysis. I relied on support teams for ad hoc report requests. In the modern era, different export options, and self-service BI is not enough. We need scalable platforms with no propriety locks to use the final output for further analysis. Gold-level data is not an end. We may want to analyse the final dataset created by the self-service end user. We may want to add pixie dust of Open AI on that final data frame created by the citizen developer. That’s where Fabric is perfect. I am no longer limited to the final stage of Load in my ETL framework or Gold layer of Medallion Lakehouse architecture. We are emerging into the world of creating Platinum and diamond layers.
Microsoft Fabric, there are so many elements I don’t know yet. There are these whole data science bits, I don’t understand yet. But as an ex-MS SQL BI developer, I find it familiar. It reminds me of how I used to work with SQL Server and BI tools. It’s like a modern version of them. It’s like a new way of doing old things. It’s time to bring back Data Platforms and analytics platforms together. Together in Cloud 🙂
What do you think? Are you excited about Microsoft Fabric? Do you have any questions or comments? Let me know in the comments section below.
Until next time, Prathy 🙂
