{"id":537,"date":"2010-10-27T13:35:31","date_gmt":"2010-10-27T12:35:31","guid":{"rendered":"http:\/\/craig.dubculture.co.nz\/blog\/?p=537"},"modified":"2010-10-27T13:37:55","modified_gmt":"2010-10-27T12:37:55","slug":"clustering-an-amazon-elastic-ip-address","status":"publish","type":"post","link":"http:\/\/craig.dubculture.co.nz\/blog\/2010\/10\/27\/clustering-an-amazon-elastic-ip-address\/","title":{"rendered":"Clustering an Amazon Elastic IP address"},"content":{"rendered":"<div style=\"float: right; margin-left: 20px\"><a href=\"http:\/\/www.flickr.com\/photos\/richard-g\/303117497\/\" title=\"Balls of red elastic bands by Richard-G, on Flickr\"><img decoding=\"async\" loading=\"lazy\" src=\"http:\/\/farm1.static.flickr.com\/117\/303117497_035101af14_m.jpg\" width=\"240\" height=\"180\" alt=\"Balls of red elastic bands\" \/><\/a><\/div>\n<p>If you have a problem that <a href=\"craig.dubculture.co.nz\/blog\/2010\/10\/25\/migrating-your-servers-to-amazon-ec2-load-balancing\/\">Amazon's Elastic Load Balancing<\/a> can't solve, you might want to do the old fashioned \"two machine IP failover\" cluster.<\/p>\n<p>Amazon instances only have one internal, and one external, IP address at a time. Consider this:<\/p>\n<ul>\n<li>Instance 1: 256.256.256.4 [Elastic IP]<\/li>\n<li>Instance 2: 257.257.257.8<\/li>\n<\/ul>\n<p>If you claim the elastic IP on instance 2, then a new IP will be allocated to instance 1:<\/p>\n<ul>\n<li>Instance 1: \u00bf?<\/li>\n<li>Instance 2: 256.256.256.4 [Elastic IP]<\/li>\n<\/ul>\n<p>You won't know what it is unless you query the web services, or look at the console, for instance 1. Be sure you are aware of the implications of this before proceeding.<\/p>\n<p>I found <a href=\"http:\/\/developer.amazonwebservices.com\/connect\/thread.jspa?messageID=112816\">a forum post from Alex Polvi<\/a> which, with some tidying, does the job nicely. When the slave node realises that its master mate has gone offline, it will claim the IP address; when the master returns, you can have the master claim it back, or you can have the slave just become the new master.<\/p>\n<h2>Claiming the shared\/elastic IP<\/h2>\n<p>Your script needs a command that the master machine can call to claim the elastic IP address. \u00a0Alex's example uses <a href=\"http:\/\/timkay.com\/aws\/\">Tim Kay's 'aws' script<\/a>, which doesn't require Java like the official Amazon ec2-utils.<\/p>\n<p>You need <em>\/root\/.awssecret<\/em> to contain the Access Key ID on the first line and the Secret Access Key on the second line:<\/p>\n<pre>AK47QWERTY7890ASDFG0H\r\n01mM4Rkl4RmArkLArmaRK14rM4rkL4MarKLar<\/pre>\n<p>You can now test this:<\/p>\n<pre>$ export AWS_PARAMS=\"--region=eu-west-1\"\r\n$ export ELASTIC_IP=256.256.256.4\r\n$ export MY_ID=$(curl -s http:\/\/169.254.169.254\/latest\/meta-data\/instance-id)\r\n$ aws $AWS_PARAMS associate-address \"$ELASTIC_IP\" -i \"$MY_ID\"<\/pre>\n<p>The MY_ID command uses the <a href=\"http:\/\/docs.amazonwebservices.com\/AWSEC2\/latest\/DeveloperGuide\/index.html?AESDG-chapter-instancedata.html\">instance data<\/a> service to get the instance ID for the machine you're running on, so you can use this script, unedited, on both machines.<\/p>\n<p>This should claim the IP 256.256.256.4 for the instance on which the script is run.<\/p>\n<p>In order for Heartbeat to be able to use this script, we need a simple init script.  When run with 'start' it should claim the IP, and when run with 'stop' it should relinquish it.  You will need to edit the parameters at the top (or better yet, put them in \/etc\/default\/elastic-ip and source that in your file). \u00a0Remember to ensure this script is executable.<\/p>\n<h3>\/etc\/init.d\/elastic-ip<\/h3>\n<pre>#!\/bin\/bash\r\nDESC=\"elastic-ip remapper\"\r\nMY_ID=$(curl -s http:\/\/169.254.169.254\/latest\/meta-data\/instance-id)\r\nELASTIC_IP=\"256.256.256.4\"\r\nAWS_PARAMS=\"--region=eu-west-1\"\r\n\r\nif ! [ -f ~\/.awssecret ] &amp;&amp; ! [ -f \/root\/.awssecret ]; then\r\n    echo \"$DESC: cannot find ~\/.awssecret or \/root\/.awssecret\"\r\n    exit 1\r\nfi\r\n\r\ncase $1 in\r\n    start)\r\n        aws $AWS_PARAMS associate-address \"$ELASTIC_IP\" -i \"$MY_ID\" &gt; \/dev\/null\r\n        [ $? -eq 0 ] &amp;&amp; echo $DESC: IP $ELASTIC_IP associated with $MY_ID || echo $DESC: Could not map IP $ELASTIC_IP to $MY_ID\r\n        ;;\r\n    stop)\r\n        aws $AWS_PARMAS disassociate-address \"$ELASTIC_IP\" &gt; \/dev\/null\r\n        [ $? -eq 0 ] &amp;&amp; echo $DESC: IP $ELASTIC_IP disowned || echo $DESC: Could not disown $ELASTIC_IP\r\n        ;;\r\n    status)\r\n        aws $AWS_PARAMS describe-addresses | grep \"$ELASTIC_IP\" | grep \"$MY_ID\" &gt; \/dev\/null\r\n        # grep will return true if this ip is mapped to this instance\r\n        [ $? -eq 0 ] &amp;&amp; echo $DESC: I have $ELASTIC_IP || echo $DESC: I do not have $ELASTIC_IP\r\n        ;;\r\nesac<\/pre>\n<h2>Heartbeat<\/h2>\n<p>Each server needs the heartbeat package installed:<\/p>\n<pre>$ apt-get install heartbeat<\/pre>\n<p>Allow heartbeat traffic between your instances:<\/p>\n<pre>$ ec2-authorize <em>$group<\/em> -P udp -p 694 -u <em>$YOURUSERID<\/em> -o <em>$group<\/em> # heartbeat<\/pre>\n<p>Heartbeat is configured by three files, all in <em>\/etc\/ha.d<\/em>, and in our case, all identical on both servers:<\/p>\n<h3>authkeys<\/h3>\n<pre>auth 1\r\n1 sha1 foobarbaz<\/pre>\n<p><a href=\"http:\/\/www.linux-ha.org\/wiki\/Authkeys\">The authkeys page<\/a> on the heartbeat wiki offers a script to help generate a key.<\/p>\n<h3>ha.cf<\/h3>\n<pre># Log to syslog as facility \"daemon\"\r\nlogfacility daemon \r\n\r\n# List of cluster members by short hostname (uname -n)\r\nnode server1 server2\r\n\r\n# Send one heartbeat each second\r\nkeepalive 1 \r\n\r\n# Declare nodes dead after 10 seconds\r\ndeadtime 10 \r\n\r\n# internal IP of the peer\r\nucast eth0 10.256.256.4\r\nucast eth0 10.257.257.8\r\n\r\n# Fail back, so we're normally running on the primary server\r\nauto_failback on<\/pre>\n<p>All pretty self-explanatory: set your own 'node' and 'ucast' entries with your hostnames and internal IP addresses. Even when the external IPs are bouncing around, the internal IPs should stay the same. auto_failback is optional, as mentioned above. <a href=\"http:\/\/www.linux-ha.org\/wiki\/Ha.cf\">Read the docs<\/a> for more options.<\/p>\n<h3>haresources<\/h3>\n<pre>server1 elastic-ip<\/pre>\n<p>Here, we set up a link between the \u00a0primary server (server1) and the script we want to run (elastic-ip). <a href=\"http:\/\/www.linux-ha.org\/wiki\/Haresources\">The wiki shows you what else you can do<\/a>.<\/p>\n<h2>Putting it all together<\/h2>\n<p>Start heartbeat on both nodes, and server1 should claim the IP address. \u00a0Stop heartbeat on server1 (or if server1 crashes), and server2 will notice after 10 seconds \u00a0and claim the IP address. As soon as server1 is back up, it should claim it back too.  You can run \/etc\/init.d\/elastic-ip status to prove this:<\/p>\n<pre>\r\nserver1:~$ sudo \/etc\/init.d\/elastic-ip status\r\nelastic-ip remapper: I have 256.256.256.4\r\nserver2:~$ sudo \/etc\/init.d\/elastic-ip status\r\nelastic-ip remapper: I do not have 256.256.256.4\r\n<\/pre>\n<p>Whatever happens, your elastic IP will always point to a good instance!<\/p>\n<h3>Postscript: what Heartbeat will not do<\/h3>\n<p>Heartbeat will notice if a server goes away, and claim the IP. However, it will not notice if a service stops running but the machine stays alive. Your good work may all be for nothing!<\/p>\n<p>To solve this, I suggest <a href=\"http:\/\/mmonit.com\/monit\/\">monit<\/a>, or if you're a ruby fan, <a href=\"http:\/\/asemanfar.com\/Why-We-Wrote-Bluepill\">bluepill<\/a>. These will monitor a service, and restart it if it is not responding.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you have a problem that Amazon's Elastic Load Balancing can't solve, you might want to do the old fashioned \"two machine IP failover\" cluster. Amazon instances only have one internal, and one external, IP address at a time. Consider this: Instance 1: 256.256.256.4 [Elastic IP] Instance 2: 257.257.257.8 If you claim the elastic IP [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[83,66,68],"tags":[],"_links":{"self":[{"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/posts\/537"}],"collection":[{"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=537"}],"version-history":[{"count":5,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/posts\/537\/revisions"}],"predecessor-version":[{"id":545,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/posts\/537\/revisions\/545"}],"wp:attachment":[{"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=537"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=537"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/craig.dubculture.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=537"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}