The project that our team took part in and completed shows how to create a dedicated infrastructure that impacts cost optimization by adapting computing power to the actual demand in a given time.
How our Azure skills redefined virtual farm scaling
The implementation of an innovative solution for automatic scaling of virtual machine farms based on a task list and processing time is an extremely complex process. What is more, if this is done in the Microsoft Azure environment, and production of video films is the target area, the whole process takes on a completely new meaning, requiring an analytical approach and extensive knowledge.
Our client needed the support of experienced architects to meet these challenges, so we provided the necessary expertise for the task. On the example of the mechanism developed by our team, we will show that despite significant challenges and resulting limitations, the implementation can be carried out efficiently, quickly, and, most importantly, failure-free.
A hard nut to crack
The virtual farm scaling project was carried out for a Polish company responsible for production of personalized video films and automation of sales processes and marketing activities impacting individual communication with a customer. The company’s solution operates on a large number of individual data that ultimately were to be found in the Cloud.
Therefore, we search for experts specializing in the Azure environment. We were recommended by Microsoft which actively participated in the project of solution migration to the Cloud. As a team of enthusiasts creating innovative solutions in the Cloud technology for many years, we have decided to face this extraordinary project. The idea was to create the possibility of automatic scaling of entire virtual farms of machines depending on the number of tasks waiting in queue and their average anticipated processing time.
Since the addition of a new machine is relatively long, and the scaling had to be fast, a mechanism was needed that did not work on simple removal and creation of new machines, but that additionally had a certain pool of stopped machines, which, if needed, can be run in a much shorter time than necessary to create a new one. It is worth adding that at the time of starting work on the project, Microsoft Azure did not have a ready-made solution to achieve the implementation assumptions.
Creativity as a key to success
To implement the project, we had to choose the right course of action from two possible options. The first and at the same time obvious one was the use of the Azure VM Scale Set. Unfortunately, it had significant disadvantages that disqualified it at the start:
- scaling machines are always removed and created from scratch, and their biggest disadvantage is the inability to stop them;
- no control over the order of machine removal.
The second option was to manage individual machines through REST API Azure Resource Manager. Therefore, it was obvious to us to choose this method. However, it was only a direction for us, not a solution to the problem we were facing. As time has shown, we made a good decision as on its basis we created our own solution that recognized machines that are currently active, stopped, in the process of creation, etc.
As a result, on this basis, we developed a dedicated application that can create a new machine, launch it from the pool of stopped ones, stop an active one and even remove it. Thus, we gained full control over individual machines, that Scale Set failed to provide.
Application stamina
Has the application developed by us worked? Yes, despite the fact that its implementation was much more complicated than using Scale Set (if possible). However, it was not the development of the software that gave us the greatest satisfaction. We are most proud of its trouble-free operation immediately after implementation and during further use.
Thanks to the changes introduced by us, the solution uses Cloud’s capabilities much more effectively than it was initially assumed. Our researches and developed implementations allowed the company to deliver materials much faster and more conveniently and, what is important, significantly reduce expenses.
Time machine
Would we decide to change anything with DeLorean DMC-12 returning us to the beginning of work on the project? No, because in the case of control over individual virtual machines, direct use of Resource Manager API is the best solution that does not limit operations that may be done on machines in any way. What is more, nothing should be changed in a well-functioning implementation that works great …, but it would be interesting to test the results of Service Fabric’s performance in terms of costs and ease of implementation.
Billennium LAB. We provide support at every stage of cloud management in the organization. We have implemented the first cloud project in public sector on a large scale. Billennium with a Microsoft partner was responsible for the production implementation of the gate for the Ministry to accept files in the SCF format.
Learn more about our Cloud and Software Development Competence Centers.
