How Can I Drop Repeated Long-Running Requests with NGINX and AWS ELB?

by Huron St. Clair   Last Updated June 15, 2019 16:00 PM - source

In our environment, we have:

  1. AWS Application Load Balancer shunting HTTPS to EC2 Instances
  2. EC2 Instances running NGINX that shunt requests to either PHP-FPM or Passenger (Ruby Server)
  3. The PHP and Rails applications at the end of the chain

We have some optimization problems with a few processes, and they can take a second or two to resolve. If someone gets clicky, they can cause the following process:

  1. Click a link. Start a request that makes it to the application server (PHP-FPM or Passenger) and starts processing.
  2. Click the same link. Start a second request that makes it to the application server. ELB and NGINX hang up the first request at this point (HTTP status 499), but the abort signal is ignored by PHP-FPM and Passenger. So a thread on the App Server is still processing the first request when the second one comes in. The second one also starts processing.
  3. Repeat Step 2 until all servers are busy responding to long-running processes.

We've been able to mitigate this somewhat in two ways: 1. Stopping some of the requests that were caused by buttons (ie- shoehorning in JS to disable the button that was causing the request.) 2. Implementing NGINX Rate Limiting.

The problem we get is that at scale it doesn't take much to bog down the system.

In plain English, all I want to do is:

If the same requester asks for the same thing three times in a short period, stop passing that through to the application for a time.

It appears that NGINX rate limiting doesn't allow this (unless I'm missing something.)

The Web Application Firewall rules for AWS are for "The maximum number of requests from a single IP address that are allowed in a five-minute period." With a minimum of 2000.

I'm looking more for "3 times in 10 seconds" not "2000 times in 5 minutes".

Perhaps it's something we'll need to include at the application layer.

Mostly I'm fishing for strategies. We can't be the only ones to have this problem where long-running application processes chew up resources even though they've been cancelled.

Is there a silver bullet method for dropping these requests?

Is there a way to actually-cancel the application after NGINX sends the abort signal?

Is there a way in NGINX or ELB/WAF to deny identical requests?

Thanks team.

Related Questions

All requests go into ELB surge queue

Updated August 30, 2017 16:00 PM

AWS S3, CloudFront, web font CORS error

Updated August 05, 2017 12:00 PM