Everything you want to know about AWS Direct Connect but were afraid to ask

AWS Direct Connect is one of those AWS services that everybody knows about but not too many people use. I’ve recently been involved in the set up of a redundant AWS Direct Connect link. To assist others considering doing the same, I’m sharing what I’ve learned.

MTUs?

This is big.

Within an AWS availability zone, and between availability zones in the same region, EC2 instances use jumbo frames. However, jumbo frames are not supported on AWS Direct Connect links, so you will be limited to a maximum MTU of 1500. You may wish to consider the implications of this before you consider using AWS Direct Connect.

Otherwise…

What is AWS Direct Connect?

Its a dedicated link between a 3rd party network and AWS. That means data flows over a dedicated isolated connection, which means you get dedicated, consistent bandwidth, unlike a VPN, which flows over the public Internet.

How is it provisioned?

You have 2 choices. AWS partners with co-location data centre providers across their various regions. This involves AWS dropping wholesale connectivity directly into the Meet Me Rooms in these 3rd party data centres. If your equipment is located in one of these data centres, your AWS Direct Connect connection is then simply patched from your cabinet into the Meet Me Room. This is called a Cross Connect.

If you are not using one of AWS’s co-location data centre partners, you can still make a Direct Connect link from your corporate network to AWS. This involves linking your corporate network to one of the data centres where AWS has a presence in their Meet Me Room, from where you can make on onward connection to AWS. The Direct Connect documentation lists telecoms providers in each region who can provide this service, and the data centres to which they can make connections.
https://aws.amazon.com/directconnect/partners/

What speeds are available?

By default, you can get either a 10GB or 1GB connection, but you can also consult directly with the AWS partners to get lower speed connections.

What do you pay?

You pay per hour for the amount of time your connection is “up” (connected at both ends). What you pay per hour depends on the speed of your connection. If you provision a connection but it isn’t “up”, you don’t pay, unless you leave that unconnected connection in place for > 90 days (after which you start paying the standard rate).

You also pay per GB of data transferred from AWS to your location. You don’t pay for data transferred from your location to AWS.

What if I need more than 10GB?

You can aggregate multiple 10GB connections together.

How stable are the connections?

Whereas connecting to AWS with a VPN provides for 2 BGP routes from your location to AWS, a Direct Connect link is a single point of failure. It is thought (presumed?) that AWS provide for a certain level of redundancy once the connection leaves the Meet Me Room in the data centre, but there are no guarantees about this and AWS do not offer an SLA for connectivity.

What hardware do I need?

You will need L3 network hardware. It will need to be able to do BGP routing and support encrypted BGP passphrases. It will need to have sufficient port speed to connect to the Direct Connect uplinks you have provided. If this is a virgin install in a co-location data centre, there are switches available that can do both L3 and L2, handle BGP and provide redundancy for 2 Direct Connect connections. This negates the need to purchase both routers and switches. You should be able to get this kit for < €20,000. Providers will almost certainly try to sell you more expensive kit. If you’re using Direct Connect, they presume money is no object for you.

What are the steps required to set up a connection?

Decide if you need a single connection or if your going to need a pair of redundant connections.

Decide what speed connection you need. Don’t guess this. Estimate it based on current network traffic in your infrastructure.

Design you IP topology

If you are going to use one of the co-location data centres, contact them. Otherwise, contact one of the Telecoms Provider partners. They will provide pricing/guidance in terms of connecting your equipment or location to the relevant Meet Me Room.

Procure the termination hardware on your side of the connection.

Once you have provisioned your connection and hardware, starting building your configuration on the AWS side of the connection.

What do I need on the in terms of configuring the VPC I am connecting to?

Typically, you will be connecting resources in a VPC to your co-location data centre of on-premises infrastructure. There are a number of hops between a VPC and a Direct Connect connection.

Working out from the VPC, the first thing you need is a Virtual Private Gateway (AWS denotes these as VGW, rather than VPG). This is logically a point of ingress/egress to your VPC. You will be asked to chose a BGP identifier when creating this. If you use BGP already, supply what you need. Otherwise, let AWS generate one for you.

When you have created this, you next create a Route Table that contains a route for the CIDR of your co-location data centre or on-premises infrastructure that points to the VGW you created earlier.

Next, create a subnet(s) (or use an existing one) and attach the Route Table to that subnet. Anything resources that need to use the Direct Connect connection need to be deployed in that subnet(s). Its probably worth deploying an EC2 instance in that subnet for testing.

This is all you need to do in the VPC configuration (you can apply NACLs, security etc later. Leave everything open for now for testing.)

How do I set up the Direct Connect configuration on the AWS side?

Once you’ve configured your VPC, you now need to configure your Direct Connect service (you don’t need to do these in any particular order. You can start with Direct Connect if you like).

Create the connections (dxcon) you require in the AWS Direct Connect console. You’ll be asked for a location to connect to and chose a speed of either 10GB or 1GB (if you want a lower speed, you’ll need to talk to your Telco or data centre before you can proceed).

The connection will be provisioned fairly quickly, and show itself in a “provisioning” state. After a few hours, it will be in a “down” state. At this point, you can select actions and download what is called a Letter of Authority (LOA) for the connection. This will specify what ports in the Meet Me Room your connection should be patched in to. You need to forward this to your co-location data centre or Telco for them to action.

Note: it is not infrequent to find the ports you have been allocated are already in use by someone else. In this case, delete the connection and start again. If you can, check with the data centre verbally that the ports are free before you submit the LOA to them. Repeat all of above if you have multiple connections. Redundancy is dealt with later in the process.

To be able to use your connection, you now need to attach a Virtual Interface (dxvif) to it. You have options here, and as is always the case, options make things a bit more complicated.

You can connect a Virtual Interface to either a VGW (Virtual Private Gateway) or a Direct Connect Gateway (not the same thing as a Direct Connect connection).

If you connect to a VGW, you will only ever be able to connect to the VPC to which that VGW provides access.

If you connect to a Direct Connect Gateway, you can associate multiple VGWs with that Gateway, allowing you access to multiple VPCs *across all AWS regions*. If you want to use this option, you need to create a Direct Connect Gateway before you create a Virtual Interface.

I can’t see any reason other than corporate governance and security why you would not want to use a Direct Connect Gateway, so I’d suggest using that option if in doubt.

So now proceed and create your Virtual Interface. If you only want to attach it to the VGW you created earlier, that option is there for you. Otherwise, attach it to the Direct Connect Gateway you created.

Once you have your Virtual Interface, go back to the Connections panel and associate that with one of your connections. You will need a dedicated Virtual Interface for each connection (you can also attach multiple Virtual Interfaces to the same connection, but that isn’t relevant here).

The final step here only occurs if you are using a Direct Connect Gateway. If you are, you need to associate the VGW you created in your VPC with the Direct Connect Gateway. It should be presented as option for you in the list of available VGWs. Start typing its identifier into the search field if not. The UI can be a bit flaky here.

That should be everything. Redundancy is the next piece.

How do I configure redundancy on the AWS side?

If you want redundant connectivity, you really need to use a Direct Connect Gateway rather than linking your connection directly to a VGW. I *think* this is a requirement for redundancy. If not, its still my recommendation.

If you have done that, you should now have 2 Virtual Interfaces and 1 VGW associated with your Direct Connect Gateway. Think of the Direct Connect Gateway as a router. The 2 Virtual Interfaces are on the external side of the router, linking in to 2 Direct Connect connections. The VGW is on the AWS side of the router, linking back to the VPC.

That should be all that is required. Traffic will flow out of the VPC through the VGW into the Direct Connect Gateway, which is BGP enabled and links into the 2 Virtual Interfaces, which are also BGP enabled. If one connection goes down, BGP routes the traffic on to the other connection. This is transparent to the VPC.

What about redundancy on the other side of the connection?

This is matter for your network administrator or service provider. Typically, the 2 connections will terminate in a logical stack of redundant routers/switches which are BGP enabled and can transfer traffic flow between the external connections.

How do I know its working?

You won’t see the state of your connections and Virtual Interfaces switch to “available” until L2 connectivity is established and the necessary BGP authentication handshake has occurred. At that point, you should be able to send ICMP requests from your termination hardware to the EC2 instance you created in your VPC earlier.

Good luck!