CostOptimization

The AWS bill shrank by 70%, but so did my DevOps dignity

Fourteen thousand dollars a month.

That number wasn’t on a spreadsheet for a new car or a down payment on a house. It was the line item from our AWS bill simply labeled “API Gateway.” It stared back at me from the screen, judging my life choices. For $14,000, you could hire a small team of actual gateway guards, probably with very cool sunglasses and earpieces. All we had was a service. An expensive, silent, and, as we would soon learn, incredibly competent service.

This is the story of how we, in a fit of brilliant financial engineering, decided to fire our expensive digital bouncer and replace it with what we thought was a cheaper, more efficient swinging saloon door. It’s a tale of triumph, disaster, sleepless nights, and the hard-won lesson that sometimes, the most expensive feature is the one you don’t realize you’re using until it’s gone.

Our grand cost-saving gamble

Every DevOps engineer has had this moment. You’re scrolling through the AWS Cost Explorer, and you spot it: a service that’s drinking your budget like it’s happy hour. For us, API Gateway was that service. Its pricing model felt… punitive. You pay per request, and you pay for the data that flows through it. It’s like paying a toll for every car and then an extra fee based on the weight of the passengers.

Then, we saw the alternative: the Application Load Balancer (ALB). The ALB was the cool, laid-back cousin. Its pricing was simple, based on capacity units. It was like paying a flat fee for an all-you-can-eat buffet instead of by the gram.

The math was so seductive it felt like we were cheating the system.

We celebrated. We patted ourselves on the back. We had slain the beast of cloud waste! The migration was smooth. Response times even improved slightly, as an ALB adds less overhead than the feature-rich API Gateway. Our architecture was simpler. We had won.

Or so we thought.

When reality crashes the party

API Gateway isn’t just a toll booth. It’s a fortress gate with a built-in, military-grade security detail you get for free. It inspects IDs, pats down suspicious characters, and keeps a strict count of who comes and goes. When we swapped it for an ALB, we essentially replaced that fortress gate with a flimsy screen door. And then we were surprised when the bugs got in.

The trouble didn’t start with a bang. It started with a whisper.

Week 1: The first unwanted visitor. A script kiddie, probably bored in their parents’ basement, decided to poke our new, undefended endpoint with a classic SQL injection probe. It was the digital equivalent of someone jiggling the handle on your front door. With API Gateway, its built-in Web Application Firewall (WAF) would have spotted the malicious pattern and drop-kicked the request into the void before it ever reached our application. With the ALB, the request sailed right through. Our application code was robust enough to reject it, but the fact that it got to knock on the application’s door at all was a bad sign. We dismissed it. “An amateur,” we scoffed.

Week 3: The client who drank too much. A bug in a third-party client integration caused it to get stuck in a loop. A very, very fast loop. It began hammering one of our endpoints with 10,000 requests per second. The servers, bless their hearts, tried to keep up. Alarms started screaming. The on-call engineer, who was attempting to have a peaceful dinner, saw his phone light up like a Christmas tree. API Gateway’s built-in rate limiting would have calmly told the misbehaving client, “You’ve had enough, buddy. Come back later.” It would have throttled the requests automatically. We had to scramble, manually blocking the IP while the system groaned under the strain.

Week 6: The full-blown apocalypse. This was it. The big one. A proper Distributed Denial-of-Service (DDoS) attack. It wasn’t sophisticated, just a tidal wave of garbage traffic from thousands of hijacked machines. Our ALB, true to its name, diligently balanced this flood of garbage across all our servers, ensuring that every single one of them was completely overwhelmed in a perfectly distributed fashion. The site went down. Hard.

The next 72 hours were a blur of caffeine, panicked Slack messages, and emergency calls with our new best friends at Cloudflare. We had saved $9,800 on our monthly bill, but we were now paying for it with our sanity and our customers’ trust.

Waking up to the real tradeoff

The hidden cost wasn’t just the emergency WAF setup or the premium support plan. It was the engineering hours. The all-nighters. The “I’m so sorry, it won’t happen again” emails to customers.

We had made a classic mistake. We looked at API Gateway’s price tag and saw a cost. We should have seen an itemized receipt for services rendered:

Built-in WAF: Free bouncer that stops common attacks.
Rate Limiting: A Free bartender who cuts off rowdy clients.
Request Validation: Free doorman that checks for malformed IDs (JSON).
Authorization: Free security guard that validates credentials (JWTs/OAuth).

An ALB does none of this. It just balances the load. That’s its only job. Expecting it to handle security is like expecting the mailman to guard your house. It’s not what he’s paid for.

Building a smarter hybrid fortress

After the fires were out and we’d all had a chance to sleep, we didn’t just switch back. That would have been too easy (and an admission of complete defeat). We came back wiser, like battle-hardened veterans of the cloud wars. We realized that not all traffic is created equal.

We landed on a hybrid solution, the architectural equivalent of having both a friendly greeter and a heavily-armed guard.

1. Let the ALB handle the easy stuff. For high-volume, low-risk traffic like serving images or tracking pixels, the ALB is perfect. It’s cheap, fast, and the security risk is minimal. Here’s how we route the “dumb” traffic in serverless.yml:

# serverless.yml
functions:
  handleMedia:
    handler: handler.alb  # A simple handler for media
    events:
      - alb:
          listenerArn: arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/my-main-alb/a1b2c3d4e5f6a7b8
          priority: 100
          conditions:
            path: /media/*  # Any request for images, videos, etc.

2. Protect the crown jewels with API Gateway. For anything critical, user authentication, data processing, and payment endpoints, we put it back behind the fortress of API Gateway. The extra cost is a small price to pay for peace of mind.

# serverless.yml
functions:
  handleSecureData:
    handler: handler.auth  # A handler with business logic
    events:
      - httpApi:
          path: /secure/v2/{proxy+}
          method: any

3. Add the security we should have had all along. This was non-negotiable. We layered on security like a paranoid onion.

AWS WAF on the ALB: We now pay for a WAF to protect our “dumb” endpoints. It’s an extra cost, but cheaper than an outage.
Cloudflare: Sits in front of everything, providing world-class DDoS protection.
Custom Rate Limiting: For nuanced cases, we built our own rate limiter. A Lambda function checks a Redis cache to track request counts from specific IPs or API keys.

Here’s a conceptual Python snippet for what that Lambda might look like:

# A simplified Lambda function for rate limiting with Redis
import redis
import os

# Connect to Redis
redis_client = redis.Redis(
    host=os.environ['REDIS_HOST'], 
    port=6379, 
    db=0
)

RATE_LIMIT_THRESHOLD = 100  # requests
TIME_WINDOW_SECONDS = 60    # per minute

def lambda_handler(event, context):
    # Get an identifier, e.g., from the source IP or an API key
    client_key = event['requestContext']['identity']['sourceIp']
    
    # Increment the count for this key
    current_count = redis_client.incr(client_key)
    
    # If it's a new key, set an expiration time
    if current_count == 1:
        redis_client.expire(client_key, TIME_WINDOW_SECONDS)
    
    # Check if the limit is exceeded
    if current_count > RATE_LIMIT_THRESHOLD:
        # Return a "429 Too Many Requests" error
        return {
            "statusCode": 429,
            "body": "Rate limit exceeded. Please try again later."
        }
    
    # Otherwise, allow the request to proceed to the application
    return {
        "statusCode": 200, 
        "body": "Request allowed"
    }

The final scorecard

So, after all that drama, was it worth it? Let’s look at the final numbers.

Verdict: For us, yes, it was ultimately worth it. Our traffic profile justified the complexity. But we traded money for time and complexity. We saved cash on the AWS bill, but the initial investment in engineering hours was massive. My DevOps dignity took a hit, but it has since recovered, scarred but wiser.

Your personal sanity check

Before you follow in our footsteps, do yourself a favor and perform an audit. Don’t let a shiny, low price tag lure you onto the rocks.

1. Calculate your break-even point. First, figure out what API Gateway is costing you.

# Get your usage data from API Gateway 
aws apigateway get-usage \ --usage-plan-id your_plan_id_here \ 
   --start-date 2024-01-01 
   --end-date 2024-01-31 \ 
   --query 'items'

Then, estimate your ALB costs based on request volume.

# Get request count to estimate ALB costs
aws cloudwatch get-metric-statistics \
    --namespace AWS/ApplicationELB \
    --metric-name RequestCount \
    --dimensions Name=LoadBalancer,Value=app/my-main-alb/a1b2c3d4e5f6a7b8 \
    --start-time 2024-01-01T00:00:00Z \
    --end-time 2024-01-31T23:59:59Z \
    --period 86400 \
    --statistics Sum

2. Test your defenses (or lack thereof). Don’t wait for a real attack. Simulate one. Tools like Burp Suite or sqlmap can show you just how exposed you are.

# A simple sqlmap test against a vulnerable parameter 
sqlmap -u "https://api.yoursite.com/endpoint?id=1" --batch

If you switch to an ALB without a WAF, you might be horrified by what you find.

In the end, choosing between API Gateway and an ALB isn’t just a technical decision. It’s a business decision about risk, and one that goes far beyond the monthly bill. It’s about calculating the true Total Cost of Ownership (TCO), where the “O” includes your engineers’ time, your customers’ trust, and your sleep schedule. Are you paying with dollars on an invoice, or are you paying with 3 AM incident calls and the slow erosion of your team’s morale?

Peace of mind has a price tag. For us, we discovered its value was around $7,700 a month. We just had to get DDoS’d to learn how to read the receipt. Don’t make the same mistake. Look at your architecture not just for what it costs, but for what it protects. Because the cheapest option on paper can quickly become the most expensive one in reality.

Lower costs with Valkey on Amazon ElastiCache

Amazon ElastiCache is a fully managed, in-memory caching service that helps you boost your application performance by retrieving information from fast, managed, in-memory caches, instead of relying solely on slower disk-based databases. Until now, you’ve had a couple of main choices for your caching engine: Memcached and Redis. Memcached is the simple, no-frills option, while Redis is the powerful, feature-rich one. Many companies, including mine, skip Memcached entirely due to its limitations. Now, there’s a new kid on the block: Valkey. And it’s not here to replace either of them but to give us more options. So, what’s the big deal?

What’s the deal with Valkey and why should we care?

Valkey is essentially a fork of Redis, meaning it branched off from the Redis codebase. It’s open-source, under the BSD 3-Clause license, and developed by a community of developers. Think of it like this: Redis was a popular open-source project, but its licensing changed slightly. So, a group of folks decided to take the core idea and continue developing it with a more open and community-focused approach. That’s Valkey in a nutshell. Importantly, Valkey uses the same protocol as Redis. This means you can use the same Redis clients to interact with Valkey, making it easy to switch or try out.

Now, you might be thinking, “Another caching engine? Why bother?”. Well, the interesting part about Valkey is that it claims to be just as powerful as Redis, but potentially more cost-effective. This is achieved by focusing on performance and resource usage. While Valkey has similarities with Redis, its community is putting in effort to improve some internal aspects. The end goal is to offer performance comparable to Redis but with better resource utilization. This can lead to significant cost savings in the long term. Also, being open source means no hefty licensing fees, unlike some commercial versions of Redis. This makes Valkey a compelling option, especially for applications that rely heavily on caching.

Valkey vs. Redis? As powerful as Redis but with a better price tag

This is where things get interesting. Valkey is designed to be compatible with the Redis protocol. This is crucial because it means migrating from Redis to Valkey should be relatively straightforward. You can keep using your existing Redis client libraries, which is a huge plus.

Now, when it comes to speed, early benchmarks suggest that Valkey can go toe-to-toe with Redis, and sometimes even surpass it, depending on the workload. This could be due to some clever optimizations under the hood in how Valkey handles memory or manages data structures.

But the real kicker is the potential for cost savings. How does Valkey achieve this? It boils down to efficiency. It seems that Valkey might be able to do more with less. For example, it could potentially store more data in the same instance size compared to Redis, meaning you pay less for the same amount of cached data. Or, it might use less CPU power for the same workload, allowing you to choose smaller, cheaper instances.

Why choose Valkey on ElastiCache? The key benefits

Let’s break down the main advantages of using Valkey:

Cost reduction: This is probably the biggest draw. Valkey’s efficiency, combined with its open-source nature, can lead to a smaller AWS bill. Imagine needing fewer or smaller instances to handle the same caching load. That’s money back in your pocket.
Scalable performance: Valkey is built to scale horizontally, just like Redis. You can add more nodes to your cluster to handle increased demand, ensuring your application remains snappy even under heavy load. It supports replication and high availability, so your data is safe and your application keeps running smoothly.
Flexibility and control: Because Valkey is open source, you have more transparency and control over the software you’re using. You can peek under the hood, understand how it works, and even contribute to its development if you’re so inclined.
Active community: Valkey is driven by a passionate community. This means continuous development, quick bug fixes, and a wealth of shared knowledge. It’s like having a global team of experts working to make the software better.

So, when should you pick Valkey over Redis?

Valkey seems particularly well-suited for a few scenarios:

Cost-sensitive applications: If you’re looking to optimize your infrastructure costs without sacrificing performance, Valkey is worth considering.
High-Throughput workloads: Applications that do a lot of reading and writing to the cache can benefit from Valkey’s efficiency.
Open source preference: Companies that prefer using open-source software for philosophical or practical reasons will find Valkey appealing.

Of course, it’s important to remember that Valkey is relatively new. While it’s showing great promise, it’s always a good idea to keep an eye on its development and adoption within the industry. Redis remains a solid, battle-tested option, so the choice ultimately depends on your specific needs and priorities.

The bottom line

Adding Valkey to ElastiCache is like getting a new, potentially more efficient tool in your toolbox. It doesn’t replace Redis, but it gives you another option, one that could save you money while delivering excellent performance. So, why not give Valkey a try on ElastiCache and see if it’s the right fit for your application? You might be pleasantly surprised. Remember, the best way to know is to test it yourself and see those cost savings firsthand.

February 4, 2025 by Fernando SRE Cloud stuff DevOps stuff

Choosing the Right AWS Reserved Instance Regional or Zonal

Let’s talk buffets. You’ve got your “all-access” pass. The one that lets you roam freely, sampling a bit of everything the dining hall offers. That’s your “regional” pass. Then you’ve got the “specialist” pass, unlimited servings, but only at that one table with the perfectly cooked prime rib. This, my friends, is the heart of the matter when we’re talking about Regional and Zonal Reserved Instances (RIs) in the world of Amazon Web Services (AWS). Let’s break it down.

Think of Reserved Instances (RIs) as pre-paid meal tickets for your cloud computing needs. You commit to using a certain amount of computing power for a year or three, and in return, Amazon gives you a hefty discount compared to paying by the hour (on-demand pricing). It’s like saying, “Hey Amazon, I’m gonna need a lot of computing power. Can you give me a better price if I promise to use it?”

Now, within this world of RIs, you have two main flavors: Regional and Zonal.

Regional RIs the flexible diners

These are your “roam around the buffet” passes. They’re not tied to a specific table (Availability Zone or AZ, in AWS lingo).

AZ flexibility: You can use your computing power in any AZ within a specific region. If one table is full, no problem, just move to another. If your application can work in any part of the region, it’s all good.
Instance size flexibility: This is like saying you can use your meal ticket for a large plate, a medium one, or even just a small snack, as long as it’s from the same food group (instance family). A t3.large reservation, for instance, can be used as a t3.medium or even a t3.xlarge, it uses a normalization factor to do it.
Automatic discount: The discount applies automatically to any instance in the region that matches the attributes of your RI. You don’t have to do any special configurations.

But there’s a catch (isn’t there always?). Regional RIs don’t guarantee you a seat at any specific table. If it’s a popular buffet (a busy AZ), and you need a seat there, you might be out of luck.

Zonal RIs the reserved table crowd

These are for those who know exactly what they want and where they want it.

Capacity reservation in a specific AZ: You’re reserving a specific table at the buffet. You’re guaranteed to have a seat (computing power) in that particular AZ.
No size flexibility: You need to choose exactly your plate size. Your reservation only applies to the exact instance type and size you picked. If you reserved a table for roast beef, you can’t use it for the pasta, sadly.
Discount locked to your AZ: Your discount only works at your reserved table, in the specific AZ you’ve chosen.

So, when do you pick one over the other?

Go Regional when:

Your app is flexible: It can run happily in any AZ within a region. You care more about the discount than about being tied to a specific location. You like flexibility.
You want maximum savings: You want to squeeze every penny of savings by taking advantage of instance size flexibility.
You like things simple: Easier management, no need to juggle reservations across different AZs.
Use cases: Think web applications with load balancing, development, and testing environments, or batch processing jobs. They don’t care too much where they are located, just that they have the power to do what they have to do.

Go Zonal when:

You need guaranteed capacity: You absolutely, positively need computing power in a specific AZ. For example, maybe your app needs to be close to your users in a certain area of the world.
Your app is picky about location: Some apps need to be in a specific AZ for latency, compliance, or architectural reasons. Maybe you have a database that needs to be super close to your application server.
You know your needs: You have a good handle on your future computing needs in that specific AZ.
Use cases: Think primary databases that need to be close to the application layer, mission-critical applications that demand high availability in a single AZ.

A real example to chew on

Imagine you’re running a popular online game. Your player base is spread across a whole region. You use Regional RIs for your game servers because they’re load-balanced and can handle players connecting from anywhere in the region. You take advantage of the Regional flexibility.

But your game’s main database? That needs to be rock-solid and always available in a specific AZ for the lowest latency. For that, you’d use a Zonal RI, reserving capacity to ensure it’s always there when your players need it.

The Bottom Line

Choosing between Regional and Zonal RIs is about understanding your application’s needs and your priorities. It’s like choosing between a flexible buffet pass or a reserved table. Both can be great, it just depends on what you’re hungry for. If you want flexibility and maximum savings, go Regional. If you need guaranteed capacity in a specific location, go Zonal.

So, there you have it. Hopefully, this makes the world of AWS Reserved Instances a bit clearer, and perhaps a bit more appetizing. Now, if you’ll excuse me, all this talk of food has made me hungry. I’m off to find a buffet… I mean, to optimize some cloud instances. 🙂

December 29, 2024 by Fernando SRE Cloud stuff

Scaling for Success. Cost-Effective Cloud Architectures on AWS

One of the most exciting aspects of cloud computing is the promise of scalability, the ability to expand or contract resources to meet demand. But how do you design an architecture that can handle unexpected traffic spikes without breaking the bank during quieter periods? This question often comes up in AWS Solution Architect interviews, and for good reason. It’s a core challenge that many businesses face when moving to the cloud. Let’s explore some AWS services and strategies that can help you achieve both scalability and cost efficiency.

Building a Dynamic and Cost-Aware AWS Architecture

Imagine your application is like a bustling restaurant. During peak hours, you need a full staff and all tables ready. But during off-peak times, you don’t want to be paying for idle resources. Here’s how we can translate this concept into a scalable AWS architecture:

Auto Scaling Groups (ASGs): Think of ASGs as your restaurant’s staffing manager. They automatically adjust the number of EC2 instances (your servers) based on predefined rules. If your website traffic suddenly spikes, ASGs will spin up additional instances to handle the load. When traffic dies down, they’ll scale back, saving you money. You can even combine ASGs with Spot Instances for even greater cost savings.
Amazon EC2 Spot Instances: These are like the temporary staff you might hire during a particularly busy event. Spot Instances let you take advantage of unused EC2 capacity at a much lower cost. If your demand is unpredictable, Spot Instances can be a great way to save money while ensuring you have enough resources to handle peak loads.
Amazon Lambda: Lambda is your kitchen staff that only gets paid when they’re cooking, and they’re really good at their job, they can whip up a dish in under 15 minutes! It’s a serverless compute service that runs your code in response to events (like a new file being uploaded or a database change). You only pay for the compute time you actually use, making it ideal for sporadic or unpredictable workloads.
AWS Fargate: Fargate is like having a catering service handle your entire kitchen operation. It’s a serverless compute engine for containers, meaning you don’t have to worry about managing the underlying servers. Fargate automatically scales your containerized applications based on demand, and you only pay for the resources your containers consume.

How the Pieces Fit Together

Now, let’s see how these services can work together in harmony:

Core Application on EC2 with Auto Scaling: Your main application might run on EC2 instances within an Auto Scaling Group. You can configure this group to monitor the CPU utilization of your servers and automatically launch new instances if the average CPU usage reaches a threshold, such as 75% (this is known as a Target Tracking Scaling Policy). This ensures you always have enough servers running to handle the current load, even during unexpected traffic spikes.
Spot Instances for Cost Optimization: To save costs, you could configure your Auto Scaling Group to use Spot Instances whenever possible. This allows you to take advantage of lower prices while still scaling up when needed. Importantly, you’ll also want to set up a recovery policy within your Auto Scaling Group. This policy ensures that if Spot Instances are not available (due to high demand or price fluctuations), your Auto Scaling Group will automatically launch On-Demand Instances instead. This way, you can reliably meet your application’s resource needs even when Spot Instances are unavailable.
Lambda for Event-Driven Tasks: Lambda functions excel at handling event-driven tasks that don’t require a constantly running server. For example, when a new image is uploaded to your S3 bucket, you can trigger a Lambda function to automatically resize it or convert it to a different format. Similarly, Lambda can be used to send notifications to users when certain events occur in your application, such as a new order being placed or a payment being processed. Since Lambda functions are only active when triggered, they can significantly reduce your costs compared to running dedicated EC2 instances for these tasks.
Fargate for Containerized Microservices: If your application is built using microservices, you can run them in containers on Fargate. This eliminates the need to manage servers and allows you to scale each microservice independently. By decoupling your microservices and using Amazon Simple Queue Service (SQS) queues for communication, you can ensure that even under heavy load, all requests will be handled and none will be lost. For applications where the order of operations is critical, such as financial transactions or order processing, you can use FIFO (First-In-First-Out) SQS queues to maintain the exact order of messages.

Monitoring and Optimization: Imagine having a restaurant manager who constantly monitors how busy the restaurant is, how much food is being wasted, and how satisfied the customers are. This is what Amazon CloudWatch does for your AWS environment. It provides detailed metrics and alarms, allowing you to fine-tune your scaling policies and optimize your resource usage. With CloudWatch, you can visualize the health and performance of your entire AWS infrastructure at a glance through intuitive dashboards and graphs. These visualizations make it easy to identify trends, spot potential issues, and make informed decisions about resource allocation and optimization.

The Outcome, A Satisfied Customer and a Healthy Bottom Line

By combining these AWS services and strategies, you can build a cloud architecture that is both scalable and cost-effective. This means your application can gracefully handle unexpected traffic spikes, ensuring a smooth user experience even during peak demand. At the same time, you won’t be paying for idle resources during quieter periods, keeping your cloud costs under control.

Final Analysis

Designing for scalability and cost efficiency is a fundamental aspect of cloud architecture. By leveraging AWS services like Auto Scaling, EC2 Spot Instances, Lambda, and Fargate, you can create a dynamic and responsive environment that adapts to your application’s needs. Remember, the key is to understand your workload patterns and choose the right tools for the job. With careful planning and the right AWS services, you can build a cloud architecture that is both powerful and cost-effective, setting your business up for success in the cloud and in the restaurant. 😉

July 24, 2024 by Fernando SRE Cloud stuff