This post will cover credential setup and the initial launching of a toil autoscaling cluster.
Note: To manage autoscaling, toil mimics a mesos cluster by spinning up an AWS EC2 instance with a docker image of a mesos leader node. This sort of virtual cluster can then spin up mesos worker nodes as needed to manage autoscaling according to resource requirements. This tutorial covers only setting up credentials and initially launching this leader node.
This tutorial was run on Ubuntu 16.04 using Toil version 3.12.0.
First, setup an rsa key for AWS. On Ubuntu, generate your key locally with the command:
ssh-keygen -t rsa
This should prompt you to save your key. Please save it in:
/home/"your user name"/.ssh/id_rsa
Now move this to where Ubuntu can see it as an authorized key:
cat /home/"your user name"/.ssh/id_rsa.pub >> /home/"your user name"/.ssh/authorized_keys eval `ssh-agent -s` ssh-add
Next, you’ll need to add the key to AWS via the browser. For example, on us-west1, this address would accessible at:
You also need to chmod your private key (good practice but also enforced by AWS):
chmod 400 id_rsa
Now click on the “Import Key Pair” button to add your key.
Next, you need to create an AWS access key. First go to the IAM dashboard, again, for “us-west1”, the example link would be here:
The directions (transcribed from: https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html ) are now:
- On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.
- Expand the Access keys (access key ID and secret access key) section.
- Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can’t retrieve this secret access key again.
Now you should have a newly generated “AWS Access Key ID” and “AWS Secret Access Key”. We can now install the AWS CLI and make sure that it has the proper credentials:
pip install awscli --upgrade --user
Now configure your AWS credentials with:
Add your “AWS Access Key ID” and “AWS Secret Access Key” from earlier and your region and output format:
" AWS Access Key ID [****************Q65Q]: " " AWS Secret Access Key [****************G0ys]: " " Default region name [us-west-1]: " " Default output format [json]: "
Toil also relies on boto, and you’ll need to create a boto file containing your credentials as well. To do this, run:
And paste in the following (with your actual “AWS Access Key ID” and “AWS Secret Access Key”):
[Credentials] aws_access_key_id = ****************Q65Q aws_secret_access_key = ****************G0ys
Now install toil from source to get the latest version:
git clone https://github.com/BD2KGenomics/toil.git cd toil virtualenv venv source venv/bin/activate make prepare make develop extras=[aws,mesos,azure,google,encryption,cwl]
Now that toil is installed and you are running a virtualenv, you can now launch a toil leader node with the following command:
TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:3.12.0 toil launch-cluster clustername --leaderNodeType t2.medium --zone us-west-1a --keyPairName id_rsa
To further break down each of these commands:
TOIL_APPLIANCE_SELF=quay.io/ucsc_cgl/toil:latest – This is optional. It specifies a mesos docker image that we maintain with the latest version of toil installed on it. If you want to use a different version of toil, please specify the image tag you need from: https://quay.io/repository/ucsc_cgl/toil?tag=latest&tab=tags
toil launch-cluster – Base command in toil to launch a cluster.
clustername – Just choose a name for your cluster.
–leaderNodeType t2.medium – Specify the leader node type. Make a t2.medium (2CPU; 4Gb RAM; $0.0464/Hour). List of available AWS instances: https://aws.amazon.com/ec2/pricing/on-demand/
–zone us-west-1a – Specify the AWS zone you want to launch the instance in. Must have the same prefix as the zone in your awscli credentials (which, in the example of this tutorial is: “us-west-1”).
–keyPairName id_rsa – The name of your key pair, which should be “id_rsa” if you’ve followed this tutorial.