Reduce operational costs

How We Reduced a Million Dollar Cloud Bill by 75 percent

When we built an initial version of a probabilistic device identifier that enabled enterprise customers to add layers of security to their web applications, we deployed it with several enterprise customers. The project closeout was touted as a success since the service met all the functional requirements.

However, once we started gathering operational metrics for the service, we realized that we needed to meet some non-functional requirements, such as cost and response time. The service level objectives required us to reduce latency by 40 percent and cost by 50 percent. Thus, we initiated an architecture review of the product. The architectural review resulted in a significant change in the system architecture of the product.

First, we considered the networking configuration of the product. It turns out the serverless component we used in VPC integrated configuration was expensive, and replacing with a simpler and cheaper load balancer enabled us to make a significant system-level change without changing much of the code in the application.

Second, VPC services communicated with serverless components via the public internet. This communication was a security risk and costly due to the need to maintain and route the traffic through NAT gateways in the VPC. Thus, we replaced public internet communication with private communication over VPC endpoints.

Third, the data science team gave us a probabilistic component of the service as a container image. Initially, we deployed the container image in a container service and the rest of the business logic serverless components. Running this complex compute environment incurred a redundant compute cost and added latency due to extra network hop between serverless and container components. Thus, we boldly decided to rewrite the probabilistic code as part of serverless component business logic and eliminate container service.

Together, these changes enabled us to develop an architectural north star for the product. We forecasted that the proposed changes would reduce costs by 60 percent. This data allowed us to secure the necessary team resources for this project. As we incrementally implemented each milestone, we could show near-immediate cost reduction. The final results were even better than forecasted. At the end of the project, we reduced service cost by 75 percent and latency by 50 percent. Thus, the service exceeded the service level objectives and met the non-functional requirements of the product.