Skip to content

AWS Deployment

Jump to bottom Edit New page
sheetalparade edited this page Jun 28, 2012 · 4 revisions

Prerequisites

  1. Zookeeper running in AWS.
  2. Security Group with Zookeeper port opened.

AMI Creation

To run Dempsy Applications in AWS infrastructure, you will need access to dempsy executable along with application artifacts. If these are accessible from AWS infrastructure, cloudInit could be used to download and install application on any instances created. Other approach is to create an custom AMI with Dempsy and Application installed.

To create an Custom AMI, start instance with default Amazon AMI.

  • SSH into the box.
  • Install dempsy by unzip the compressed file or using the rpm (example directory /rel-code/tools/dempsy).
  • Install application jar files (example directory /rel-code/apps/).
  • Copy or install any other resources/files the application might need.
  • Update the zookeeper connect string in the configuration (dempsy.conf) and test the install by starting application run.sh -a Application -c Cluster -d /rel-code/apps/.
  • If successful update the zookeeper connect string to 'PLACEHOLDER'
  • Clean all the log & tmp directories.
  • Stop the Instance. DO NOT TERMINATE THE INSTANCE.
  • Using AWS EC2 console or EC2 tools, create new AMI.

For more information on Custom AMI creation please refer AWS Documentation.

CloudInit

CloudInit developed by Ubuntu community to initialize the cloud instances uses data associated with an instance for instructions to execute on start-up. This data could be cloudinit commands or unix shell script.

This examples uses script in the user-data to execute commands on start-up. More information on cloudInit can be found at Ubuntu and AWS

Following shell script will be used to initialize the cloud instance.

#!/bin/sh
sudo yum -y update
sed -i 's/PLACEHOLDER/$ZkConnectStr/' $DempsyHomeconf/dempsy.conf
rm $DempsyHomelog/*
$DempsyHomerun.sh -a $ApplicationName -c $ClusterName -d $AppHome start

This script will

  • Update all the software from yum repositories.
  • Update dempsy.conf for zookeeper connect string by replacing string 'PLACEHOLDER' with value from parameter.
  • Clean dempsy log directory.
  • Start dempsy application based on parameters passed in.

This script in cloudformation template is part of user-data. CloudFormation templates are defined in JSON. The script is represents in JSON as value of userdata.

"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[
   "#!/bin/sh","\n",
   "sudo yum -y update", "\n",
   "sed -i 's/PLACEHOLDER/",{"Ref" : "ZkConnectStr"},"/' ", {"Ref" : "DempsyHome"}, "conf/dempsy.conf","\n",
   "rm ", {"Ref" : "DempsyHome"}, "log/*","\n",
   {"Ref" : "DempsyHome"}, "run.sh -a ",{"Ref" : "ApplicationName"}," -c ",{"Ref" : "ClusterName"}," -d ",{"Ref" : "AppHome"}," start","\n" ]]}}
}

CloudFormation

CloudFormation template files defined in JSON format enables provisioning of AWS infrastructure deployments predictably and repeatedly. All the resources defined in one template are group together as a single unit (called stack). Entire stack can be created/initialized together or destroyed together. If one of the resource in the stack fails, entire stack fails and is roll backed.

CloudFormation offers many advantages like

  1. CloudFormation supports wide variety of AWS resources like EC2 instances, EBS, Load balances, IPS, Autoscaling, CloudWatch, Security groups and Virtual Private Cloud to name few.
  2. CloudFormation takes care of order in which resources to allocate.
  3. CloudFormation templates are in JSON format, thus they can be stored and maintained outside AWS infrastructure.
  4. These template can be customized using parameters. Parameters allow templates to select the region or EC2 instance type at runtime or while the stack is being built.

All input parameters/variables are defined under "Parameters"

"Parameters" : {
  "InstanceType" : {
     "Description" : "Type of EC2 instance to launch",
     "Type" : "String",
     "Default" : "t1.micro"
  }
}

Each parameter had name or key, description, Type and optional default value. The sample template has following parameters defined

  • InstanceType - Type of EC2 instance to use. Default - t1.micro
  • KeyName - Key Name to use for the EC2 instance.
  • ApplicationAMI - Custom AMI id.
  • ZkConnectStr - Zookeeper connect string.
  • DempsyHome - Directory dempsy is installed. Default - /rel-code/tools/dempsy/
  • AppHome - Directory application will be installed. Default - /rel-code/apps/
  • ApplicationName - Name of the application to start.
  • ClusterName- Name of the cluster to start.
  • DesiredSize - Desired size of cluster. Default - 1
  • MinSize - Minimum size of cluster. Default - 1
  • MaxSize - Maximum size of cluster. Default - 4

Resources are defined as

"Resources" : {
  "DempsyAutoScalingGroup" : {
     "Type" : "AWS::AutoScaling::AutoScalingGroup",
     "Properties" : {
       "AvailabilityZones" : { "Fn::GetAZs" : ""},
       "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
       "MinSize" : { "Ref" : "MinSize" },
       "MaxSize" : { "Ref" : "MaxSize" },
       "DesiredCapacity" : { "Ref" : "DesiredSize" }
     }
  }
}

Following AWS resources are defined in the sample template

  • DempsyAutoScalingGroup - Auto scaling resource
  • LaunchConfig - Defines AWS resources like EC2 instances, Load balancers which are part of auto scaling group.
  • ScaleUpPolicy - Defines how to scale up, by adding more resources.
  • ScaleDownPolicy - Defines how to scale down by taking out resources.
  • CPUAlarmHigh - Create an alarm for high CPU usage and calls onto scale up policy.
  • CPUAlarmLow - Create an alarm for low CPU usage and calls onto scale down policy.

Additional documentation for CloudFormation can be found on AWS website

{
   "Description" : "Create simple word count application using Dempsy framework",
   "Parameters" : {
      "InstanceType" : {
         "Description" : "Type of EC2 instance to launch",
         "Type" : "String",
         "Default" : "t1.micro"
      },
      "KeyName" : {
         "Description" : "The EC2 Key Pair to allow SSH access to the instances",
         "Type" : "String"
      },
      "ApplicationAMI" : {
        "Description" : "The AMI with application and dempsy installed.",
        "Type" : "String",
        "Default" : "ami-3efb5957"
      },
      "ZkConnectStr" : {
        "Description" : "The zookeeper connection string (host:port,host:port)",
        "Type" : "String",
        "Default" : "localhost:2181"
      },
      "DempsyHome" : {
        "Description" : "The directory were Dempsy is installed.",
        "Type" : "String",
        "Default" : "/rel-code/tools/dempsy/"
      },
      "AppHome" : {
        "Description" : "The directory were Application is installed.",
        "Type" : "String",
        "Default" : "/rel-code/apps/"
      },
      "ApplicationName" : {
        "Description" : "The name of Application to start.",
        "Type" : "String",
        "Default" : "Demo"
      },
      "ClusterName" : {
        "Description" : "The name of Cluster to start.",
        "Type" : "String"
      },
      "DesiredSize" : {
        "Description" : "Desired Size of cluster.",
        "Type" : "String",
        "Default" : "1"
      },
      "MinSize" : {
        "Description" : "Minimum size of cluster.",
        "Type" : "String",
        "Default" : "1"
      },
      "MaxSize" : {
        "Description" : "Max size cluster is allowed to grow.",
        "Type" : "String",
        "Default" : "4"
      }
   },
   "Resources" : {
      "DempsyAutoScalingGroup" : {
         "Type" : "AWS::AutoScaling::AutoScalingGroup",
         "Properties" : {
           "AvailabilityZones" : { "Fn::GetAZs" : ""},
           "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
           "MinSize" : { "Ref" : "MinSize" },
           "MaxSize" : { "Ref" : "MaxSize" },
           "DesiredCapacity" : { "Ref" : "DesiredSize" }
         }
      },
      "LaunchConfig" : {
         "Type" : "AWS::AutoScaling::LaunchConfiguration",
         "Properties" : {
            "KeyName" : { "Ref" : "KeyName" },
            "ImageId" : { "Ref" : "ApplicationAMI" },
            "SecurityGroups" : [ "Dempsy-Open" ],
            "InstanceType" : { "Ref" : "InstanceType" },
           "UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[
               "#!/bin/sh","\n",
               "sudo yum -y update", "\n",
               "sed -i 's/PLACEHOLDER/",{"Ref" : "ZkConnectStr"},"/' ", {"Ref" : "DempsyHome"}, "conf/dempsy.conf","\n",
               "rm ", {"Ref" : "DempsyHome"}, "log/*","\n",
               {"Ref" : "DempsyHome"}, "run.sh -a ",{"Ref" : "ApplicationName"}," -c ",{"Ref" : "ClusterName"}," -d ",{"Ref" : "AppHome"}," start","\n" ]]}}
         }
      },
      "ScaleUpPolicy" : {
         "Type" : "AWS::AutoScaling::ScalingPolicy",
         "Properties" : {
            "AdjustmentType" : "ChangeInCapacity",
            "AutoScalingGroupName" : { "Ref" : "DempsyAutoScalingGroup" },
            "Cooldown" : "60",
            "ScalingAdjustment" : "1"
         }
      },
     "ScaleDownPolicy" : {
       "Type" : "AWS::AutoScaling::ScalingPolicy",
       "Properties" : {
         "AdjustmentType" : "ChangeInCapacity",
         "AutoScalingGroupName" : { "Ref" : "DempsyAutoScalingGroup" },
         "Cooldown" : "60",
         "ScalingAdjustment" : "-1"
       }
     },
     "CPUAlarmHigh": {
        "Type": "AWS::CloudWatch::Alarm",
        "Properties": {
           "AlarmDescription": "Scale-up if CPU > 30% for 10 minutes",
           "MetricName": "CPUUtilization",
           "Namespace": "AWS/EC2",
           "Statistic": "Average",
           "Period": "300",
           "EvaluationPeriods": "2",
           "Threshold": "30",
           "AlarmActions": [ { "Ref": "ScaleUpPolicy" } ],
           "Dimensions": [
             {
               "Name": "AutoScalingGroupName",
               "Value": { "Ref": "DempsyAutoScalingGroup" }
             }
           ],
           "ComparisonOperator": "GreaterThanThreshold"
         }
     },
     "CPUAlarmLow": {
        "Type": "AWS::CloudWatch::Alarm",
        "Properties": {
           "AlarmDescription": "Scale-down if CPU < 20% for 10 minutes",
           "MetricName": "CPUUtilization",
           "Namespace": "AWS/EC2",
           "Statistic": "Average",
           "Period": "300",
           "EvaluationPeriods": "2",
           "Threshold": "20",
           "AlarmActions": [ { "Ref": "ScaleDownPolicy" } ],
           "Dimensions": [
             {
               "Name": "AutoScalingGroupName",
               "Value": { "Ref": "DempsyAutoScalingGroup" }
             }
           ],
           "ComparisonOperator": "LessThanThreshold"
        }
     }
   }
}
Add a custom footer
Add a custom sidebar

Clone this wiki locally