Monday, June 25, 2012

Poor Man's Static IP for EC2 a.k.a. Elastic Network Interface

Amamzon's Elastic Network Interface (EIN) allows you to "reserve" an IP address. This is immensely useful in VPC because an EIN can function as a pseudo static IP for elastic instances. Granted, you have to use two IPs for a single instance. But EIN lets you assign a fixed private IP address to an elastic instance without having to go through the trouble of setting up dynamic DNS update. Unfortunately, Amazon's documentation is missing key information on configurting secondary IP with EIN. Even if you have attached an EIN to an elastic instance, you cannot access the instance using the private IP associated with the EIN. What gives? The missing piece is IP interface and routing configuration. Below is a step-to-step guide to configure the EIN interface. This guide assumes that you have followed the official AWS guide to the point where you have configured an EIN and have brought up an elastic instance that is attached with that EIN. Further, it assumes that the primary interface is assigned an IP 10.3.1.190 and the secondary interface, which is the EIN, is assigned an IP 10.3.1.191. At the end of the exercise, we will be able to ssh to the secondary IP address in addition to the primary one.

First, check IP address binding to each network interface.

$ sudo ip address

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:26:69:f0:87:46 brd ff:ff:ff:ff:ff:ff
inet 10.3.1.190/24 brd 10.3.1.255 scope global eth0
inet6 fe80::26:69ff:fef0:8746/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 02:26:69:dc:cc:62 brd ff:ff:ff:ff:ff:ff


We see from the output the current network interface assignment is of the following:

eth0: 10.3.1.190
eth1: none

Therefore,  the first order of business is to assign the EIN IP address to the interface eth1:

$ sudo ip address add 10.3.1.191/24 brd + dev eth1

Next, bring up the interface:

$ sudo ip link set dev eth1 up

Verify that eth1 is indeed up:

$ sudo ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:26:69:f0:87:46 brd ff:ff:ff:ff:ff:ff
inet 10.3.1.190/24 brd 10.3.1.255 scope global eth0
inet6 fe80::26:69ff:fef0:8746/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 02:26:69:dc:cc:62 brd ff:ff:ff:ff:ff:ff
inet 10.3.1.191/24 brd 10.3.1.255 scope global eth1
inet6 fe80::26:69ff:fedc:cc62/64 scope link
valid_lft forever preferred_lft forever


Next, find out the default gateway:

$ ip route show
default via 10.3.1.1 dev eth0
10.3.1.0/24 dev eth0 proto kernel scope link src 10.3.1.190
10.3.1.0/24 dev eth1 proto kernel scope link src 10.3.1.191


The default gateway is 10.3.1.1 in the output. It is bound to the virtual gateway associated with the VPC.  Since it is currently only bound to eth0, any traffic from eth1 that is destined to IP addresses outside the 10.3.1.0/24 IP block will be dropped! We need to reconfigure IP routing on the elastic instance to allow IP packets leaving eth1 to be routed through the default gateway. Here is how you do it.

First, add a new routing table called "awsein":

$ sudo echo 2 awsein >> /etc/iproute2/rt_tables

It will add a table called "awsein" to rt_tables as entry 2:
$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#1 inr.ruhep
2 awsein


Now adds a default route in the new table to use the same default gateway as the one used by eth0:

$ sudo ip route add default via 10.3.1.1 dev eth1 table awsein
$ sudo ip route flush cache

Confirm that the new route is indeed added:

$ ip route show table awsein
default via 10.3.1.1 dev eth1 metric 1000


Next, we need to create a new routing rule to trigger the default route on eth1 by its source IP. To do this, we first check existing routes:
$ ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup default


Note the number 32766 for the rule "main". We will now add a new rule to "awsein" with a priority smaller than the one for "main".

$ sudo ip rule add from 10.3.1.191 lookup awsein prio 1000
Finally, verify the new rule configuration:

$ ip rule
0:      from all lookup local

1000:   from 10.3.1.191 lookup awsein
32766:  from all lookup main
32767:  from all lookup default

Now you can ssh into the instance using the EIN IP 10.3.1.191! Happy hacking.


11 comments:

  1. Hi,

    I know this is quite an old post, but i need to do exactly what you are outlining above. After following your guide, all works exactly as it should, however, as soon as i restart the Ubuntu server, the configuration gets lost.

    Once running these 3 commands again, it comes right.

    $ sudo ip route add default via 10.3.1.1 dev eth1 table awsein

    $ sudo ip route flush cache

    $ sudo ip rule add from 10.3.1.191 lookup awsein prio 1000

    Is there a way to have these settings become persistent?

    Thanks
    Byron

    ReplyDelete
    Replies
    1. Figured my issue out. Thought i would just leave the answer here in case anyone else was having the problem as well:

      You need to add these 2 lines to "/etc/network/interfaces" under the eth1 adaptor.

      up ip route add default via 10.3.1.1 dev eth1 table awsein
      up ip rule add from 10.3.1.191 lookup awsein prio 1000

      Delete
    2. Hi Bryon,

      This is the content of my /etc/network/interfaces file. Even after entering last two lines I have to reconfigure. Is there something wrong in my interfaces file



      # This file describes the network interfaces available on your system
      # and how to activate them. For more information, see interfaces(5).

      # The loopback network interface
      auto lo
      iface lo inet loopback

      # The primary network interface
      auto eth0
      iface eth0 inet dhcp


      #Secondary network interface
      #auto eth1
      #iface eth1 inet dhcp
      up ip route add default via 10.0.0.1 dev eth1 table awsein
      up ip rule add from 10.0.0.178 lookup awsein prio 1000

      Delete
    3. This is my interfaces file. If I use DHCP or static configuration for eth1, then I am not able to ssh using primary ip. The basic issue it that I have to reconfigure after reboot


      # This file describes the network interfaces available on your system
      # and how to activate them. For more information, see interfaces(5).

      # The loopback network interface
      auto lo
      iface lo inet loopback

      # The primary network interface
      auto eth0
      iface eth0 inet dhcp


      #Secondary network interface
      #auto eth1
      #iface eth1 inet dhcp
      auto eth1
      iface eth1 inet static
      address 10.0.0.178
      netmask 255.255.255.0
      network 10.0.0.0
      broadcast 10.0.0.255
      gateway 10.0.0.1
      up ip route add default via 10.0.0.1 dev eth1 table awsein
      up ip rule add from 10.0.0.178 lookup awsein prio 1000


      Is there something I am missing in my interfaces file

      Thanks
      Lavesh

      Delete
    4. Hi Lavesh,

      After looking at your network settings, all looks fine except for the fact that you have specified your gateway under the second interface. Try removing your gateway line from the second interface, you only need to specify a gateway once.
      So your second interface should look like this

      auto eth1
      iface eth1 inet static
      address 10.0.0.178
      netmask 255.255.255.0
      network 10.0.0.0
      broadcast 10.0.0.255
      up ip route add default via 10.0.0.1 dev eth1 table awsein
      up ip rule add from 10.0.0.178 lookup awsein prio 1000

      If that doesn't work, try set your primary, eth0, adapter to a static IP as well and post that config here.

      Thanks
      Byron

      Delete
    5. Thanks Bryon that worked.
      A single line could have screwed everything.

      Thanks a lot
      Lavesh

      Delete
  2. thanks a lot. I was searching this solution for a long time

    Seriously Thanks a ton

    ReplyDelete
  3. Hi, what if I need eth0, eth1 and eth2? When I run

    ip route add default via 10.3.1.1 dev eth2 table awsein

    I get an error saying "file already exist"

    Any help is appreciated.

    ReplyDelete
  4. why you used 1000 as priority and if i want to add more virtual ips will i use the same number for each ,

    need help

    ReplyDelete
  5. Can you please change "EIN" in this article to "ENI": Elastic Network Interface. The article is useful but doesn't appear in an internet search for "ENI", which is the correct term.

    ReplyDelete