Creating Business Advantage

Big Data Platforms: How to Control Costs and Manage Resources Effectively

Written by Alana Hoponoa | Dec 12, 2024 9:51:18 PM

Cost Optimisation of a Big Data Platform for a Large Enterprise Client

A key client was embarking on an exciting journey to consolidate two existing data platforms into a single unified platform —one hosted in Azure and the other in AWS. During the initial phase of this project, we quickly realised that managing multiple data platforms simultaneously would increase costs for some time before they started to decline post-consolidation. This realisation prompted us to initiate a proactive cost optimisation effort from the very start.

The Challenge

The primary challenge was to identify and eliminate unused resources and services across the AWS accounts. These unused assets were not only inflating costs but also cluttering the infrastructure. The client needed a comprehensive strategy to clean up these resources and optimise spend during the transition period.

Our Solution

Our approach focused on streamlining their data storage and compute resources to achieve significant cost savings without disrupting services. Key actions included:

  • Data Management: Changing the storage options for some data and removing others altogether.
  • Resource Cleanup: Decommissioning some environments, removing unused instances and upgrading end-of-life instances.

Project Outcomes

The cost optimisation initiative yielded great results. By the project phase's conclusion, we successfully saved over $15,000 NZD per month. This significant cost reduction was accomplished without compromising IT services, demonstrating the effectiveness of our strategy.

Lessons Learned

This project reinforced several key aspects of our approach:

  • Effective Communication is Key: Regular follow-ups and clear communication are crucial to confirm whether resources can be deleted or moved.
  • Cost Optimisation is Continuous: Cost optimisation is an ongoing effort. Continuous monitoring and optimisation are essential as cloud environments and resource utilisation evolve over time.

The client's journey highlights the importance of proactive cost management and resource optimisation in maintaining efficient and cost-effective operations during major platform transitions. Through continuously refining operational strategies, even greater savings and efficiencies can be achieved, following a clearly defined and executable roadmap.

Reach out now to discuss your cost optimisation challenge with us!