I've been trying to think through a problem in utilizing Terraform (though any infra language would do, like CloudFormation, ARM or whatever Google uses as this is not a cloud applicable problem to my knowledge). I've often done canaries in software engineering, and its pretty much a staple besides red/black (or blue/green, pick your word poison). To build a scenario, let's say that we already have our networking layer in in place, and it will not be changed. I'll use AWS to draw this out into manageable chunks.
Let's say our VPC has the following attributes
Pretty standard. Okay we have our baseline, let's say our application will be something leveraging AWS specific services.
How would you canary this? Let's say we want to test out ALB instead of ELB, or remove Redis entirely and replace it with Elasticache. We evidently don't want to roll this straight to production, it'd be suicide.
Would I consider something like a second Route53 coupled with Traffic Flow (AWS' version of traffic shaping to my knowledge) and a set of the desired infrastructure to gradually give load to see if when or how it breaks? Please let me know your thoughts, has been something tickling the back of my brain for months.