Mirror Monitoring Wins!

Last week, Team D(ata) Rex successfully ingested real telemetry data into Microsoft Fabric. This week, we decided to move forward with Azure Monitor Mirroring, eliminating the extra step of exporting data through Azure Blob Storage, and shifted our focus toward visualization!

Our sponsor shared several existing Azure dashboards with us, giving us a great starting point for our next challenge: recreating them in Fabric! We started exploring the queries behind these dashboards and figuring out how to turn our mirrored telemetry data into similar visualizations.

Of course, it wasn’t quite as simple as copying and pasting! We had to adjust queries to account for differences between Azure and Fabric, but each successful query brought us one step closer to our goal.

What’s Next?

We’re excited to get our Power BI licenses so we can start building interactive dashboards and really bring our telemetry data to life!

Stay tuned for more updates from Team D(ata) Rex 🙂

Last week, Team D(ata) Rex officially selected Microsoft Fabric as our data warehouse platform and began exploring ways to bring Vispero’s telemetry into it. This week, we took a major step forward by successfully ingesting real Application Insights data and running our first SQL queries in Fabric!

Our team tested two approaches to bringing telemetry into Fabric: Azure Monitor Mirroring and Diagnostic Settings. Both were successful! Mirroring allowed us to access telemetry directly in Fabric, while Diagnostic Settings let us export data to Azure Blob Storage and query it through a Fabric Lakehouse.

Getting everything working wasn’t without challenges. We ran into workspace access restrictions and configuration issues, but with help from our liaison, Jake, we were able to resolve them and move forward.

Our next step is to compare both ingestion methods based on performance, cost, and data completeness to determine which best fits Vispero’s needs. We’re also beginning to explore how additional data sources, such as Salesforce, could be integrated into our solution.

With our first successful queries behind us, we’re excited to move beyond setup and start turning Vispero’s telemetry into meaningful insights!

Plot Twist: We Went with Microsoft Fabric!

After weeks of researching data warehouses, our team had narrowed our options to Google BigQuery and Amazon Redshift. Just when we thought we were approaching a final decision, our project took an unexpected turn: Microsoft Fabric!

Following discussions with Vispero, we decided to move forward with Fabric because it integrates directly with their existing Azure environment. Rather than introducing another cloud provider, Fabric offers a fully managed platform for data ingestion, storage, and analytics, all while keeping infrastructure management relatively simple.

With our warehouse selected, we shifted from research to hands-on implementation. Our team gained access to Vispero’s Azure environment and began exploring Application Insights to better understand the telemetry we’re working with. We also investigated two ingestion approaches: Azure diagnostic settings, which export data through Blob Storage, and Azure Monitor mirroring, which could offer a more direct connection to Fabric.

We encountered a few challenges with Azure permissions, but our next focus is establishing a working ingestion pipeline and preparing for Power BI integration.

This week showed us that engineering decisions don’t always follow a straight path. With our warehouse selected, we’re excited to move from research to implementation!

From PDR to Prototyping: Where We Stand

Tuesday’s Preliminary Design Review was a pivotal moment for our project. It marked the first concrete step toward turning Vispero’s fragmented telemetry into a unified, decision‑ready data platform.

Our proposed direction is therefore three‑fold. First, we keep Application Insights in its monitoring role and layer a cloud‑agnostic warehouse that can ingest both Azure and AWS sources. Second, we build a transformation pipeline that normalizes the raw streams, enforces a 730‑day retention policy, and prepares the data for rapid analytics. Third, we connect the warehouse to Power BI (and other visualization tools) so product managers, analysts, and engineers can explore the data without writing code.

During the review we walked the audience through the customer needs: cost‑effective operation, sub‑10‑second dashboard loads, role‑based access, and LLM‑augmented analytics into concrete technical measures. Compute pricing for the warehouse, not storage, will drive our long‑term cost, and that connector compatibility is the real make‑or‑break factor when pairing a warehouse with downstream tools.

We narrowed the shortlist to two likely candidates: Google BigQuery, whose scan‑metered model offers generous free‑tier transfers, and Amazon Redshift, whose time‑metered compute with auto‑pause aligns well with bursty, role‑based query patterns. Both platforms price storage similar per month, so the decision hinges on how their compute billing matches Vispero’s usage profile.

What remains is the sponsor’s final stack selection. Once Vispero confirms the warehouse, we will spin up a prototype with a synthetic‑telemetry dataset, validate ingestion pathways, benchmark query latency, and confirm Power BI integration.

https://www.canva.com/design/DAHWhVJa1sU/PAVwpxvxLzfxvZRCdStGzA/edit

This week our team completed a shift from scouting individual analytics tools to exploring a unified data warehouse topped with a flexible visualization layer. Microsoft Application Insights will remain the primary source for product telemetry, while the new stack will centralize that data for deeper, SQL‑driven exploration.

We evaluated Metabase, Tableau, Power BI, and Google Analytics on cost, usability, integration ease, and overall fit. Metabase emerged as the front‑runner. At the same time, refreshed cost models for the leading warehouse candidates incorporating revised query volume to provide a more accurate cost estimate.

The Trello backlog has been updated to reflect these decisions.

We will prepare slides for the September 29 presentation, and present the updated comparisons to Vispero. Their feedback will lock in the prototype environment and ingestion pipeline.

As we close Week 4, the clarity of our research is propelling us toward a decisive technical path. By distilling our data warehousing options into actionable deliverables and aligning closely with our engineering liaison, we’re ready to leap from evaluation on a higher level to more realistic cost estimates.

Our front-runners of research in data warehouses were Snowflake, Google BigQuery, DataBricks, Azure Synapse Analytics, and Databricks.

We also met with our company liaisons in person and presented our research and refined our specifications!

Last week the team met with the company liaison and our coach, clearing up what the project needs for managing their data and how we will keep weekly accountability as the year progresses. We also created our team name, Data (D) Rex and our logo. We set up a shared tracker so everyone could report their research as we started hunting for the best data‑warehouse solution for Vispero’s needs. In the coming weeks we’ll dive deeper into cost, scalability, and security for each option and run proof‑of‑concept tests and demos.