Airlines are large and complex companies. Like all big businesses, they collect a variety of data on their customers and processes, which they use to inform their business decisions. However, data is getting bigger, and one of New Zealand’s airlines saw that the volume of data and the type of analyses they envisaged for the future were beyond the capacity of their current legacy platforms. Since they have a cloud-first strategy as an organisation, they knew their solution would be to move their data and analyses to the cloud.
Paul Mumby, a principal architect at OSS Group, was one of the first on the scene to bring the airline’s vision to production.
“One of the early things we identified is that the company wanted to remain fluid,” says Mumby. “They wanted to use a platform that would allow them to keep their options open so that if they changed their minds at a future date, the costs of moving to a new platform would be minimal.”
Implementing AWS
With ease and flexibility in mind, the first version of the new platform was built on Amazon Web Services (AWS), a globally recognized cloud platform that is scalable and cost-effective. Mumby designed the platform and another consultancy manually built and hand-crafted it on AWS with a full Cloudera Hadoop stack.
“While it worked well,” said Mumby, “it wasn’t capable of being altered quickly and easily without a lot of complexity and testing.”
Instead of a manual build, Mumby advocated for a DevOps approach. This way, everything could be automated and nothing would need to be hand-crafted.
“In the DevOps approach, you design the code that builds it for you. When you need to make a change, you change the code instead of manually changing the platform.”
The airline asked Mumby and the OSS Group team to take over the building of the platform. They built the second version in AWS, again with a Cloudera Hadoop stack. Instead of stopping there, Mumby’s team then automated the process they used to build it, using a combination of Ansible, an open-source tool, and CloudFormation, a free AWS tool.
“Ansible can be used to build template machines based on code,” says Mumby.
In essence, Ansible asks Cloudera’s software repository what it needs to stand up the specified stack. The required stack of infrastructure is then built on the AWS platform. Cloudera Director, an automation product that leverages the AWS automation APIs, proceeds to further build out all of their software onto those instances.
“We have scripts that are pushed out to act as CI/CD servers. We’re using Bitbucket, a Git platform, as a revision control solution for the code and GoCD to trigger builds of the infrastructure.”
Automation reduces rebuild time
Building the second version took about the same number of hours as building the first version.
“The original hand-crafted build took five months with a team of five people. It took our team, which had an average of eight people, three months to build the automation.”
However, with the automation capacity of the new version, building an entirely new stack from scratch takes only two hours.
This speed is critical to the agility of a platform. For example, by the time the OSS Group team had finished the build, Cloudera released a new version of their solution stack, leaping from version 5.15 to version 6. Without automation, upgrading to the new stack would have required the team to start over again. With automation, Mumby was able to make the switch to version 6 in less than three weeks.
“We made the appropriate changes to the automation scripts to roll out the version 6 upgrades. We deployed it, tested it, fixed a few problems, fixed the scripts, redeployed it, and now we’ve got a working version 6 cluster.”
As a result of this speed, Mumby suspects that the airline is the first organisation in New Zealand to run version 6 in production.
AWS spot instances reduce cost
While the airline is interested in enhancing agility, they are also interested in reducing cost. Mumby and the OSS Group team were able to leverage some aspects of how AWS works to minimise cost. Specifically, they changed the way they purchased instances.
“When we are testing, we sometimes deploy a whole new cluster three times a day.”
Typically, new clusters are large and expensive, with infrastructure costing thousands of dollars per week.
“Building five different versions to try out what works can cost hundreds of dollars, even for a few hours of testing.”
AWS caters for this testing phase by offering spot instances. They offer unused virtual servers at a discount when used for short-term projects.
“We only pay for what we need to because we’re doing it on AWS.”
Spot instances can then be scaled up to reserved instances and on-demand instances according to the company’s requirements and level of commitment to AWS. By using a considered combination of spot instances, reserved instances and On-Demand instances, OSS Group were able to cut costs by 80% while building the second version.
“It also means that when we need to test something, we can scale back down again.”
A bright future with AWS
From the very beginning of the first build, the airline insisted on maintaining flexibility with their data platform strategy.
“Because of their requirements, we had to use a platform that was compatible with tools that were not proprietary.”
Instead of insisting on using their specific tools, AWS provided extensive flexibility, from scalable pricing to numerous APIs. Their compatibility with open-source tools made the automation solutions chosen by OSS Group easy to implement.
“This means that the effort required to move to another platform solution would be relatively low.”
This flexibility gave the airline the peace of mind it needed to adopt AWS as their platform of choice.
It is perhaps ironic that the very qualities that make AWS easy to move away from have resulted in the development of a loyal customer.
“I don’t anticipate the airline moving away from AWS in the foreseeable future,” Mumby concludes.
No Comments Yet
Let us know what you think