Feeling safe with AWS Security Groups
One database, from the worst setup I can think of to least privilege, one step at a time. At every step: what is the impact if something does go wrong?
It's really fun being the security guy; you get to look at configurations and think, "Wow, why are we doing that? This looks incredibly vulnerable." That's why I am talking about Security Groups today. I am focusing purely on Security Groups in this article.
Before you start reading this, I want to say there are strict ways to configure security groups, and they come with a lot of operational overhead that large companies struggle to track because so much is happening across multiple accounts. I get it; I have been there and experienced it.
The other side of the coin is where people just want something working, and they bypass security. Would I do that? No. But people do.
There can also be a happy middle ground where some people are okay with some risk if something gets compromised, just to avoid the complete operational overhead. But that's a risk you have to determine, not me. You signed up to the cloud; you make your own decisions and do what's best for you.
So instead of telling you the one right way, I'm going to take one database from the worst setup I can think of to least privilege, step by step. At every step, I'll ask the same question: "What is the impact if something does go wrong?" Where you stop is up to you.
The groundwork
What is a Virtual Private Cloud?
A Virtual Private Cloud (VPC) is a private network inside your AWS region. You select a CIDR block for your address range (basically an IP address range allocated for that VPC). Nothing gets in or out unless you allow it. Some of your AWS resources will live there, and I'll cover how they communicate with each other within that VPC in this article.
The AWS Public Zone
The AWS Public Zone holds services managed by AWS. You can reach them over HTTPS on the public internet, but IAM guards them. Not every managed resource lives here; for example, RDS runs on instances inside your VPC, so you reach it over your network rather than a public endpoint. A DynamoDB table, by contrast, has a public endpoint.
The credentials making the call need permission to use the service. If you are logged into a role that can send SQS messages, you can send one; if you try to put an item into DynamoDB without permissions, it will be denied.
If an attacker steals credentials, they can call whatever the permissions allow. If you have a DynamoDB table with data and your role can read that table, they have access to all your data. That's why least privilege is important. A role that doesn't need access to a service shouldn't have it. Reduce your blast radius before it's too late.
Subnets: public vs. private
They vary in access and functionality. A subnet stays private unless you explicitly make it public; to do that, attach an Internet Gateway (IGW) to the VPC it's in and add a route to the IGW in the route table. An IGW is a two-way gateway that allows inbound and outbound traffic. A route that sends 0.0.0.0/0 to the IGW catches any traffic headed outside the VPC that no other route handles. Other routes, like local, provide a path between subnets in the VPC.
Route table Destination Target
public-rt 10.0.0.0/16 local ← within the VPC
0.0.0.0/0 igw-0abc123 ← makes the subnet public
A path is not explicit permission. When traffic leaves the subnet to communicate with another subnet, it might not actually have access to anything.
Private subnets that require outbound traffic will use a NAT Gateway. It only allows outbound traffic from the private subnet. It's important to understand that it only allows inbound traffic back from what is sent outbound. You cannot initiate inbound traffic on a NAT Gateway.
Route table Destination Target
private-rt 10.0.0.0/16 local
0.0.0.0/0 nat-0def456 ← outbound only, via the NAT
The route table associated with the private subnet has a 0.0.0.0/0 route that points to the NAT Gateway in a public subnet, which then sends traffic out via the IGW. Remember, a subnet is only public if its VPC has an IGW attached and its route table sends traffic to it.
Simply put, a VPC can have an IGW attached, and a route table provides a path for inbound and outbound traffic. A NAT Gateway sits in a public subnet; traffic from a private subnet routes out via the NAT Gateway and on through the IGW.
A private subnet with no 0.0.0.0/0 route at all is often called an isolated subnet. It can reach the rest of your VPC, and nothing else.
Route table Destination Target
isolated-rt 10.0.0.0/16 local ← no route out at all
Security groups
So now we understand how the traffic enters a VPC and flows between subnets within that VPC. When you have resources, how do you control what traffic reaches those resources? Remember, just because traffic can reach a subnet doesn't mean it has permission to do anything with the resources inside the VPC and subnets.
A security group is a stateful firewall, which means it tracks the state of inbound and outbound traffic. If security group rules allow outbound traffic, it will allow the inbound response. It applies on the inverse as well: if the security group rules allow inbound traffic, it can send an outbound response back to that same call.
Security groups will attach to your resources inside the VPC and allow traffic based on inbound and outbound rules. They don't have deny rules; they're explicit allow-only. Anything that doesn't match the rules for that resource will get dropped.
The demonstration: resources that progressively get more strict
The setup: a Postgres RDS instance on port 5432, and an application (a Lambda or ECS service; it changes based on the example) that needs to query it. We'll start where no one should start and tighten it step by step. For this example, we'll focus on traffic inside the VPC and subnets, not what is sending it there. That's for a different article.
I am not a huge fan of demonstrations that show you one way and one way only. I think it's more beneficial to understand why something is bad and why others are acceptable.
For reference:
Name Resource Action
orders-api Lambda/ECS Calls orders-db on TCP 5432
orders-db RDS (Postgres) Listens on TCP 5432
Using a public subnet
First, please don't put your database in a public subnet. You can have an RDS instance in a public subnet that no traffic can reach if the security group has no inbound rules. Meaning that no traffic can reach it at all. It's essentially useless.
rds-sg Inbound: none ← nothing can reach it, including your app
Outbound: All traffic to 0.0.0.0/0 ← AWS default
Even if you did scope it down to something you think is safe, it's going to get hammered with automated scanners. This is the worst thing you could possibly do to your database. It still needs credentials to access it, but this is basically inviting anyone to breach your database. It's one misconfigured security group away from being exposed.
rds-sg Inbound: TCP 5432 from 203.0.113.10/32 ← one IP, but still on the internet
Outbound: All traffic to 0.0.0.0/0 ← AWS default
RDS usually doesn't need outbound rules at all. It only answers requests, and because security groups are stateful, the replies go back automatically. The same goes for any resources whose only job is to respond to requests from a network perspective.
Outbound is easy to overlook because AWS gives every new security group an "allow all traffic to 0.0.0.0/0" rule by default. If an attacker gets remote code execution on one of your servers, nothing stops it from sending your data out, mining Bitcoin on your account, or deleting everything if access allows.
rds-sg Outbound: none ← a database only answers; replies go back anyway
I know I always say your infrastructure, not mine. But please listen on this one.
Moving to a private subnet
So you decided, after getting attacked on a public subnet, that it's a horrendous idea and moved your database into a private subnet with an inbound rule that allows TCP on all ports, with your VPC CIDR block as the source range. (RDS requires subnets across at least 2 AZs fyi, but for simplicity's sake let's keep going.)
rds-sg Inbound: TCP 0-65535 from 10.0.0.0/16 ← every port, the whole VPC
Outbound: none
Is it better? Yes. This is what I normally hear: "at least it's better than…" when, in reality, when designing your architecture and system, you should be asking, "What is the impact if something does go wrong?"
The blast radius is still big. This means that everything inside the VPC can access your RDS instance. Even resources in another subnet. It's not open to the public, and anything sending traffic in would need an explicit path.
After you ask yourself the number one question, "What is the impact if something does go wrong?", you notice the blast radius is big, and your database instance is reachable on all ports. So you scope inbound to your private subnet's CIDR block and allow TCP only on port 5432.
rds-sg Inbound: TCP 5432 from 10.0.1.0/24 ← one port, one subnet
Outbound: none
The blast radius is smaller; is it better? Yes. But it's the same problem as above with a smaller blast radius. Anything in the subnet can reach it, even resources that have no business accessing the database.
Multi-tier subnetting
So you moved from a public subnet to a private subnet and scoped the security group to that subnet's CIDR block. Your blast radius is now everything in that subnet.
Now you can isolate the database tier in a separate subnet and keep your application in a different private subnet. An isolated subnet has no route to the internet in either direction: no IGW, no NAT. Nothing on the internet can reach it, and nothing in it can reach the internet. Even if the database were compromised, it would have no way to send your data out.
Subnet CIDR Route out Contains
app 10.0.1.0/24 0.0.0.0/0 → NAT orders-api
database 10.0.2.0/24 none (isolated) orders-db
Is this better? Yes, but the real impact will come later; it's more of a building block.
Guess what I am going to ask next? "What is the impact if something does go wrong?" It's not much different to the example above: if your inbound security group on the RDS is scoped to the application subnet, everything in that subnet can still reach the RDS instance on port 5432.
rds-sg Inbound: TCP 5432 from 10.0.1.0/24 ← the app subnet: same rule as before
Outbound: none
Allowlisting individual IPs
Now you want to reduce the blast radius and start thinking about least-privilege access, which means you are in the right frame of mind. You are now thinking about individual resources and how your system actually works, instead of it working because of broad permissions and not knowing why.
So you start scoping inbound down to IP addresses. Then you realise this isn't scalable and adds operational overhead, and some resources don't even have static IPs. Lambdas and ECS tasks, for example, don't have them, and they are a big part of compute in applications running on AWS.
rds-sg Inbound: TCP 5432 from 10.0.1.23/32 ← orders-api task 1
TCP 5432 from 10.0.1.47/32 ← orders-api task 2
TCP 5432 from 10.0.1.89/32 ← orders-api task 3
Outbound: none
Security-wise, it's not the worst, but I wouldn't recommend it. When a resource's IP is released, it goes back into the subnet pool, meaning a new, unrelated resource could inherit access to your database when the IP is reallocated. In ECS Fargate, every task gets a fresh IP when it starts, so addresses are constantly being released and reused. A task with no reason to touch your database can pick up an IP your security group rules still trust, and it gains access.
Before deploy 10.0.1.23 orders-api task ✅ allowed, and should be
After deploy 10.0.1.23 (released)
Days later 10.0.1.23 log-shipper task ✅ allowed, and shouldn't be
"What is the impact if something does go wrong?" Probably less than before, but the operational overhead isn't worth the strain. It's a nightmare to maintain, and it is a sure way to create deployment bottlenecks; a simple feature becomes a fountain of useless meetings.
This will still be necessary if an external resource has a static IP. So it's not Gospel that you can't use a static IP.
Self-referencing security groups
If you and your team cannot handle the operational overhead and risk of static IPs for the security group rules. I don't blame you. The next step is to look into self-referencing security groups.
Self-referencing security groups solve the problem with IP rules: create a group called rds-sg-access with inbound rules that allow TCP connections on port 5432 from anything in the rds-sg-access group, and then an outbound rule that allows the same.
rds-sg-access Inbound: TCP 5432 from rds-sg-access ← itself
Outbound: TCP 5432 to rds-sg-access ← itself
Attached to: orders-db, orders-api
Attach this to the RDS instance and any other resources that need access. Anything in the group can now reach anything else in it on 5432. Your Lambda can reach the database, and you don't need any IP rules.
To remove access, remove the group from the resource and communication stops. Each resource can carry up to 5 security groups. For most resources, that's a soft limit you can raise to 16, but Lambda and ECS cap it at 5 regardless. Every shared group like this uses one of those slots.
Is it better? Yes. Is it less operational overhead? Yes.
But "What is the impact if something does go wrong?"
If one resource in the group is compromised, it can reach every member on port 5432, not just the database that the lambda needs. Because the group allows TCP on 5432, every member is exposed on that port to every other member. For some resources, this doesn't matter. A Lambda doesn't listen on 5432, so being reachable on it changes nothing. But anything that does listen on 5432, like a second database, is exposed to every member.
Attached to: orders-db, orders-api, reports-api, billing-db
orders-api → orders-db ✅ needed
orders-api → billing-db ✅ not needed, but allowed
reports-api → orders-db ✅ not needed, but allowed
orders-db → billing-db ✅ nobody asked for this
The blast radius is significantly smaller than before. Only resources explicitly attached to the group are affected; resources in the same VPC or the application subnets without the group can't reach the database in the isolated subnet.
If you have groups with wider rules, the problem grows with the number of resources, and it becomes harder to understand what has access to what.
Obviously, no breach is good, but knowing that the blast radius is smaller is a massive improvement over everything being affected. For a business or some people, this level of security would be acceptable. I don't blame them: it's scalable, and it's explicit. Group membership tells you exactly which resources should be accessible.
(Self-referencing groups do have a place: clusters whose members genuinely need to talk to each other. The problem is unrelated resources sharing one.)
One-way security group references
Each resource gets its own group, and each rule names the other group in one direction only. In the section above, the lambda and the database shared the same rds-sg-access group. The group is now split into two.
orders-api-sg Inbound: none
Outbound: TCP 5432 to orders-db-sg
orders-db-sg Inbound: TCP 5432 from orders-api-sg
Outbound: none
Only the resources you've granted can open connections to the database, and the database accepts connections only from them. It doesn't start connections of its own.
Is it better? Yes. Is it less operational overhead? No. Every relationship now needs a rule on each side.
But "What is the impact if something does go wrong?"
A compromised resource can only reach what you explicitly granted. The blast radius is tiny, but the impact could still be huge: a compromised Lambda can reach the database. Security group rules limit how far a breach can go; the database's own permissions limit what it can do once it gets there.
The compromised Lambda can only reach the database, and that's it. Add a second database, and it gets its own group.
orders-api-sg Inbound: none
Outbound: TCP 5432 to orders-db-sg
orders-db-sg Inbound: TCP 5432 from orders-api-sg ✅ orders-api → orders-db
Outbound: none
billing-db-sg Inbound: none ❌ orders-api → billing-db
Outbound: none
The Lambda can't reach it until you add a rule on both sides.
orders-api-sg Outbound: TCP 5432 to orders-db-sg
TCP 5432 to billing-db-sg ← new
billing-db-sg Inbound: TCP 5432 from orders-api-sg ← new
This is least privilege at the network layer. Each rule describes one relationship, so your security groups describe your architecture. If you can't explain why a rule exists, then you don't need it.
The catch
Every relationship now needs two rules: one on each side. The caller needs permission to send, and the target needs permission to accept. The more resources you add, the more rules you write, review, and keep in sync. It's still stateful, so you're not writing rules for replies, but you are writing both ends of every connection.
There are limits too. A security group can have 60 inbound and 60 outbound rules by default. With one-way references, every caller adds an inbound rule to the target, so a database or endpoint that everything talks to hits 60 first. You can raise it to 200 without touching anything else, but quota increases apply to your whole account in that region. The upside: each resource carries only its own group, so the 5-group cap that hurt self-referencing groups doesn't apply.
orders-db-sg Inbound: TCP 5432 from orders-api-sg
TCP 5432 from reports-api-sg
TCP 5432 from api-3-sg
...
TCP 5432 from api-60-sg ← default limit: 60 inbound rules
TCP 5432 from api-61-sg ❌ fails until the quota is raised
Outbound: none
orders-api-sg Inbound: none
Outbound: TCP 5432 to orders-db-sg ← 1 of 60 outbound rules
TCP 5432 to billing-db-sg ← 2 of 60
...
TCP 5432 to db-61-sg ❌ fails until the quota is raised
orders-api Security groups: orders-api-sg ← always 1 of 5 slots used
Operationally, this is a lot to track by hand, and hitting limits with large serverless apps is a real possibility. That might not be a trade-off some places are willing to make, and I think that's fair.
This is why I built Anvil Cloud
Anvil takes care of all of this for you. Everything is explicitly stated in your infrastructure-as-code framework.
Every resource gets its own security group, and it starts empty: no inbound, no outbound, not even AWS's default. Nothing can talk to anything until you grant it:
import { App } from '@anvil-cloud/sdk';
import * as anvil from '@anvil-cloud/sdk';
export default new App({
defaults: {
tags: { team: 'orders', costCentre: 'engineering' },
},
providers: {
aws: { region: 'ap-southeast-2' },
},
compliance: {
frameworks: ['soc2', 'cis'],
schedule: 'daily',
scanOnDeploy: true,
},
run(ctx) {
const vpc = new anvil.aws.Vpc('core', {
nat: { natType: 'gateway' },
isolatedSubnets: true,
});
const app = { vpcId: vpc.vpcId, privateSubnetIds: vpc.privateSubnetIds };
const data = { vpcId: vpc.vpcId, privateSubnetIds: vpc.isolatedSubnetIds };
const db = new anvil.aws.Rds('orders-db', {
engine: 'postgres',
vpc: data, // isolated subnets: no route out
});
const fn = new anvil.aws.Lambda('orders-api', {
runtime: 'nodejs24.x',
handler: 'src/index.handler',
vpc: app, // private subnets: outbound via NAT
});
// Outbound 5432 on orders-api's group, inbound 5432 on orders-db's group.
db.grantConnect(fn);
},
});
It's really that simple.
As your application grows, Anvil handles it all for you. Remove a grant, and its rules go with it. Remove a resource, and its security group goes too.
After everything we've just been through, every level, every trade-off, every "it depends on your tolerance for risk", this almost feels anticlimactic. A handful of lines, and you have least privilege at the network layer without writing a single rule yourself.
You don't need Anvil to do any of this, but I promise you it makes it a lot easier. Everything in this article is plain AWS, and now you know how to build it by hand. I know most people would rather spend time on their app that makes them money than keep security group rules in order.
So before you start building without Anvil Cloud, ask yourself one last time: "What is the impact if something does go wrong?"
Get started with Anvil Cloud. It's just one anvil init away.
As always, it's your infrastructure, not mine.
Damien.
Reference docs for this topic: VPC.