autoscaling, tutorial

Credential Setup: Toil Autoscaling with AWS

This post will cover credential setup and the initial launching of a toil autoscaling cluster.

Note: To manage autoscaling, toil mimics a mesos cluster by spinning up an AWS EC2 instance with a docker image of a mesos leader node. This sort of virtual cluster can then spin up mesos worker nodes as needed to manage autoscaling according to resource requirements. This tutorial covers only setting up credentials and initially launching this leader node.

This tutorial was run on Ubuntu 16.04 using Toil version 3.12.0.

First, setup an rsa key for AWS. On Ubuntu, generate your key locally with the command:

ssh-keygen -t rsa

This should prompt you to save your key. Please save it in:

/home/"your user name"/.ssh/id_rsa

Now move this to where Ubuntu can see it as an authorized key:

cat /home/"your user name"/.ssh/id_rsa.pub >> /home/"your user name"/.ssh/authorized_keys
eval `ssh-agent -s`
ssh-add

Next, you’ll need to add the key to AWS via the browser. For example, on us-west1, this address would accessible at:

https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#KeyPairs:sort=keyName

You also need to chmod your private key (good practice but also enforced by AWS):

chmod 400 id_rsa

Now click on the “Import Key Pair” button to add your key.

keypair

Next, you need to create an AWS access key. First go to the IAM dashboard, again, for “us-west1”, the example link would be here:

https://console.aws.amazon.com/iam/home?region=us-west-1#/home

The directions (transcribed from: https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html ) are now:

  1. On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.
  2. Expand the Access keys (access key ID and secret access key) section.
  3. Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can’t retrieve this secret access key again.

Now you should have a newly generated “AWS Access Key ID” and “AWS Secret Access Key”. We can now install the AWS CLI and make sure that it has the proper credentials:

pip install awscli --upgrade --user

Now configure your AWS credentials with:

aws configure

Add your “AWS Access Key ID” and “AWS Secret Access Key” from earlier and your region and output format:

" AWS Access Key ID [****************Q65Q]: "
" AWS Secret Access Key [****************G0ys]: "
" Default region name [us-west-1]: "
" Default output format [json]: "

Toil also relies on boto, and you’ll need to create a boto file containing your credentials as well. To do this, run:

nano ~/.boto

And paste in the following (with your actual “AWS Access Key ID” and “AWS Secret Access Key”):

[Credentials]
aws_access_key_id = ****************Q65Q
aws_secret_access_key = ****************G0ys

Now install toil from source to get the latest version:

git clone https://github.com/BD2KGenomics/toil.git
cd toil
virtualenv venv
source venv/bin/activate
make prepare
make develop extras=[aws,mesos,azure,google,encryption,cwl]

Now that toil is installed and you are running a virtualenv, you can now launch a toil leader node with the following command:

TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:3.12.0 toil launch-cluster clustername --leaderNodeType t2.medium --zone us-west-1a --keyPairName id_rsa

To further break down each of these commands:

TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:latest – This is optional. It specifies a mesos docker image that we maintain with the latest version of toil installed on it. If you want to use a different version of toil, please specify the image tag you need from: https://quay.io/repository/ucsc_cgl/toil?tag=latest&tab=tags

toil launch-cluster – Base command in toil to launch a cluster.

clustername – Just choose a name for your cluster.

–leaderNodeType t2.medium – Specify the leader node type. Make a t2.medium (2CPU; 4Gb RAM; $0.0464/Hour). List of available AWS instances: https://aws.amazon.com/ec2/pricing/on-demand/

–zone us-west-1a – Specify the AWS zone you want to launch the instance in. Must have the same prefix as the zone in your awscli credentials (which, in the example of this tutorial is: “us-west-1”).

–keyPairName id_rsa – The name of your key pair, which should be “id_rsa” if you’ve followed this tutorial.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s