Did you know that you can navigate the posts by swiping left and right?
All successful projects grow. In the beginning we usually don’t care about scalability so much. The need of it comes often suddenly. You cannot predict effects of being advertised on popular blog or going viral. Then you need to scale up to serve the traffic.
There are many options to fix this. You can start with Heroku, but you might need more flexibility later and move to bare servers on Hetzner. And Hetzner might be a fit for you for several months. But suddenly you meet server’s limits, like we did recently, and you have to decide whether to buy another metal or go back into the cloud. Buying second server has own pitfalls and might be tricky to maintain. Amazon’s EC2 might be an option, but manual management might be annoying (it was to me).
Here I want to show you, how to manage EC2 instances using fog. I believe that crafting own, custom solution for such thing is good idea, because you can expect to have effects pretty fast.
Each a little larger application can be divided into some logic parts, which might need to scale. In case of our app I’d do following split:
In the beginning we will focus on scaling web and background workers. Problems with scaling databases come later, so we will deal with them in following posts.
For our custom tool we will use Thor. It’s nicer than Rake and have more feature while it doesn’t have odd DSL in favor of ruby classes. Let’s create a project!
I use RVM, so I’ll setup RVM gemset:
I like bundler:
We need to install thor and fog, so put them into
Gemfile and do
Let’s test thor. Create a
Here is content for
You can easily test it in the terminal:
And that’s it. We have simple project ready to grow.
Our first task would be to boot an instance and ensure it’s connectivity. Before that we need to prepare configuration.
Go to Key Pairs section of Management Console (warning: link leads to EU West 1 zone!) and create new key pair. You will be prompted for name for keypair.
Keys are created on per-region basis. My convention is to provide region name followed by user name (
euwest1_sevos in my case).
Download your private key (I will use
euwest1_sevos.pem as key file name)
and place it in
config/keys/ directory in our project.
Remember to change rights to the key file to 600:
Go here and scroll to Access Credentials section. Grab AccessKey ID and Secret Access Key. We will need those to make API calls to AWS.
config/credentials.yml file for our credentials and commons:
Now, we can use these data to manage instances.
I use current (as of writing this article) Ubuntu 12.04 64-bit AMI from EU West datacenter
Amazon uses security groups to protect instances from unwanted traffic.
For sake of simplicity let’s create one security group opening port 22 (SSH)
for world. Go to Security Groups
and crete new group. Name it
mycloud or whatever you’ll use later as
security_group_name, provide description and leave VPC setting unchanged.
After creating security group set it up for accepting SSH connection.
Remember to click Add Rule and apply changes!
up method and do some magic!
Now it’s time to check our code. Type in the console:
After several seconds you should see similar message:
Just copy the SSH command and try to connect to the server. Note that even if server seems to be ready, sshd server might be not up yet. Just wait few seconds and try again.
That’s it for today. Turn off instance manually using AWS Console. Select an instance and from Actions menu select Terminate command.
We prepared a thor task for creating AWS instance from an AMI image. We still need some role-system, since we don’t care about single instance.
In next posts I will write also about provisioning and grouping instances.