Overcoming Data Ingestion and Normalization Challenges for FinOps

Best practice

Overcoming Data Ingestion and Normalization Challenges for FinOps

The following article was written by Andrew Sweeney, Co-founder and CEO of ReadyWorks.

---

The previous few years have taught us all to expect the unexpected, and cloud costs are no exception. The promise of ‘pay for what you use’ proved appealing and companies migrated to the cloud expecting cost reductions. For many, that promise wasn’t realized.

Today more are adopting a hybrid cloud environment. Gartner predicts that by 2026, 75% of organizations will adopt a digital transformation model predicated on the cloud as the fundamental underlying platform. Companies want to take advantage of scalability using a mix of public cloud providers’ resources to support customers and employees but must also factor in operational performance needs requiring workloads to be placed on-premises at the edge cloud.  

With resources spread this way it can be hard to see the forest for the trees, but that’s exactly what FinOps teams must do to maximize the value of a hybrid cloud infrastructure. A collaborative approach is key, and FinOps must align with executives’ strategic, operational, and financial goals while working closely with ITAM professionals.

The challenge of data ingestion and normalization

FinOps teams need access to huge amounts of data to identify where workloads should be placed, based on usage, performance, cost, and other business needs, including security and regulatory requirements. Each of their many internal customers wants to view that data in a different way – a business team may want to optimize performance, or understand their portion of cloud spending, a CFO may want a more detailed overview and breakdown, while a Chief Compliance Officer will need to prove to regulators that the business remains in-compliance. That means tweaking and presenting data differently for each stakeholder.

The amount of data FinOps has access to isn’t the problem. There’s a wealth of it out there including cloud service provider bills and usage details, performance data, records from on-premises CMDBs and other business-specific and ITAM data. But it’s held in tools on-premises and from multiple service providers and these tools don’t interact. Information is stored in different formats and is often incomplete. The challenge is to access, ingest, and normalize that data in a timely manner to make informed decisions.

This is a familiar story for ITAM professionals. They have long been tasked with aggregating, normalizing, and validating data to manage company assets. It takes time to filter out the noise and manipulate data into a reasonable and accurate format. In the dynamic cloud environment, the time this takes means FinOps will always be making decisions based on outdated information, which means they will never be able to fully optimize cost and performance in the cloud.

Checklist for Data Ingestion and Normalization success

To make the most informed decisions at the right time, FinOps teams should:

• Work with the teams below to identify the data required to build financial insights:
  • Operations: to understand KPIs or workload performance requirements.
  • Finance:  to identify critical financial metrics.
  • Compliance: to understand regulatory requirements for data placement.
  • ITAM: to identify appropriate data sources as well as cost synergies that can be made, for example through software license optimization.
• Connect to relevant data sources and automate data aggregation to access a constant stream of cost, performance, and usage data, that, combined with business requirements, allow them to make accurate informed decisions.
• Find ways to automate data normalization and validation.
• Be able to view data from any angle and manipulate it to meet the needs of all stakeholders.
• Automate change, using tools to move workloads where the company can gain the most benefits.

Some companies are now choosing to develop their own solutions, but that’s going to take development time, money, and effort. As new digital tools and capabilities are introduced in the company it’s going to mean more of the same to develop new connections.  But by using a tool now highlighted as transformational in six Gartner hype cycles – a tool called a digital platform conductor (DPC) – there’s no need to tie up in-house resources.

A DPC unlocks the data tied up in siloed platforms, spreadsheets, and the minds of specialists.  It connects to data sources and aggregates the information they hold to provide valuable insights into critical aspects of a business. It complements large-scale platforms already in-place, leveraging their capabilities, and orchestrating workflows across them. It also cleans up source data, correcting formats, gaps, and errors, using bi-directional connectors.

FinOps teams can then use this data to answer questions such as:

• How will a move to the cloud impact software licenses and costs?
• Am I risking regulatory compliance by making change?
• How much is ‘X’ business unit spend in the cloud?  
• Where should I place workloads to optimize performance?
• What is our projected cloud spending overall and by business unit?
• Can I cut cloud spending while maintaining performance?

Reporting is simplified, as FinOps can provide report links that allow stakeholders to see the data they need in real-time from their preferred perspective. FinOps can use this data to optimize workload placement across the dynamic hybrid environment to meet all business needs.

Conclusion

The ever-evolving landscape of cloud computing has brought both promises and challenges to organizations seeking to optimize their digital infrastructure. As hybrid cloud becomes the norm, FinOps and ITAM teams are at the forefront of this transformation, tasked with deciphering vast volumes of data to make crucial decisions. The challenge lies mainly in the fragmentation and complexity of the data sources. To navigate this data maze effectively, collaboration is key, bridging the gap between finance, operations, compliance, and ITAM.

The journey towards success involves embracing automation, streamlining access to pertinent data, and enabling versatility in data presentation. It’s about empowering FinOps teams to act on real-time insights and optimize cloud costs and performance dynamically. While some organizations contemplate developing proprietary solutions, others look to the Digital Platform Conductor (DPC) to liberate data from silos, unlock valuable business insights, and orchestrate workflows.

With a DPC among their toolkit – FinOps & ITAM teams can confidently answer critical questions, anticipate the impact of cloud migration on licenses and costs, ensure regulatory compliance, and optimize workload placement for peak performance. Reporting becomes effortless, providing stakeholders with real-time access to the data they need, allowing FinOps to fine-tune workload placement across the dynamic hybrid environment, meeting the diverse needs of the business. The future of FinOps lies in the hands of those who embrace innovation and automation, enabling organizations to thrive in an ever-expanding cloud-driven world.

Can’t find what you’re looking for?