Merb and Rails are merging!

23 Dec 2008

No, it's not April 1st, and as far as I know, hell hasn't frozen over either. The Merb and Rails teams have in fact announced that they will be joining forces. The end result will eventually be released as Rails 3.

Rather than repeating all the details here, below are links to the original announcements by various team members:

It is unclear how much of the actual Merb code will make it into Rails 3, but the important thing is that Rails will embrace many of Merb's core principles, such as a lightweight core, performance, modularity (i.e. you'll be able to easily swap ActiveRecord out for DataMapper or some other ORM framework), and a well-defined and stable public API that plugins can be based on.

I've been a big fan of Rails for many years, and it is certainly relatively mature and has a large developer community. At the same time, I've been drawn to Merb and related technologies (such as DataMapper) lately, and I strongly agree with their core principles.

Integrating these two frameworks will be no small feat, so the new combined team definitely has their work cut out for them. But as different as the two frameworks are, they also have a lot in common. By agreeing on this new direction, the team will be able to focus on the important tasks without having to deal with redundant functionality. I am looking forward to the new direction and am excited about trying out the first Rails 3 alpha whenever it is released.

Over the past few months, there's been a fair amount of bickering between the Rails and Merb teams, and I'm all the more impressed with both teams for reaching this decision and deciding to work together. I am convinced that the Ruby Community will be a lot better for it.

Rails in the Cloud: AWS, Heroku, and Morph

13 Nov 2008

Amazon Web Services

Over the course of the past 6 months or so, I've had the opportunity to explore various cloud hosting services, starting with Amazon Web Services. I don't want to go into detail about AWS here, but suffice to say that I like this suite of services a lot, and it was a great fit for an SMS app / messaging server I deployed on this infrastructure. The application consisted of a set of loosely coupled components running on EC2 (some daemon apps and some Merb based web frontends), communicating via the Simple Queue Service (SQS). I leveraged S3 for deployments and backups.

AWS is great, and the ability to bring up new EC2 instances any time is very powerful. Need to test something out real quick? Simply launch a fresh instance, then kill it when you no longer need it. One of your app servers running at capacity? Simply launch another one, or even automate this by monitoring your load. Need to bring up 1000 instances for an hour in order to load test your web app, then shut them right down again? No problem! No need to ever call up an IT person at your local hosting company and get a new server provisioned, or even buy your own server and drive to a colocation facility yourself to set it up.

But the services that AWS provides are still very low level. This can be a good thing because it gives you complete control and flexibility over how to architect your application, but it also means that you'll spend a good amount of time configuring Linux images and writing deployment, monitoring, and backup scripts. For many complex applications, this makes perfect sense, and I would pretty much always pick AWS over VPS or dedicated hosting services these days. But for a large class of straightforward web apps, it would be great if I didn't have to worry about these details (especially since I'm admittedly not that much of an IT guy). Ideally I would simply push my code out into the cloud and delegate all deployment concerns to the hosting service. When taken to the extreme, I would not even want to have to know any details about the deployment infrastructure. All I would care about is that the app runs reliably and scales up as needed (and affordably).

Google App Engine

Google App Engine has a similar philosophy, and if I was a Python fan, this would be a very interesting option. As soon as Google starts supporting Ruby, I will definitely check this out. My guess is that Rails might be difficult to support, because it is very tightly coupled with relational databases, but other Ruby based web frameworks (in particular Merb, perhaps even with a special DataMapper adapter) seem like they could be a great fit for App Engine. DataMapper already has adapters for other non-relational data stores (such as CouchDB), so it seems like it should be fairly straightforward to build an adapter for the Google App Engine Datastore.

But I'm getting off-track from what I was really planning to write about:

Lately, several Rails based services have emerged that take a big step in this direction. I've had the chance to work with Heroku and Morph AppSpace, so I figured I'd share my early impressions.

Heroku

Still in private beta, Heroku is a Rails specific hosting service that prides itself on its ease of use and elastic scalability. You create a new Rails app using their web interface, and then either edit it straight in the browser via a nifty browser based IDE, or check it out via Git and edit it locally. Changes made via the browser take effect immediately, changes made via Git are automatically deployed upon pushing to the remote repository, including applying migration scripts. It can't get much more streamlined than that!

Heroku sits on top of EC2, but this fact is completely hidden from the developer, who never interfaces directly with the underlying EC2 instance. This means that, among other limitations, you don't have shell or FTP access to the server. You do however have access to the Rails console via Heroku's web interface, allowing you to work with your model objects and take care of the odd ad-hoc task. You also have access to your Rake tasks and the Rails code generators. Overall, I didn't find this limitation all that bad.

Heroku uses PostgreSQL (we would have preferred MySQL, but that is unfortunately not an option at this point), but you don't have direct access to the database. The web interface has some rudimentary functionality for viewing your database schema and data, but there is no way to run SQL queries. I have found this a much bigger limitation than the lack of shell access, as it prevents a lot of ad-hoc analytics or quick and dirty data changes that occasionally come in handy (although I suppose it is easy enough to whip up a quick migration for the latter). There is a way to download a yaml based database dump, which you can then import into your local database (regardless of whether this runs PostgreSQL, MySQL, or Sqlite3). It's a bit cumbersome, but at least the option is there, so your data is never held hostage.

Another major limitation is that Heroku does not support background tasks of any sort. This includes both cron jobs as well as tasks offloaded by the Rails app onto a job server such as BackgrounDRb or Workling. This may not be critical for all applications, but it is becoming increasingly common for Rails apps to offload a lot of the processing into asynchronous background tasks, allowing for a more responsive user experience as well as better scalability.

Heroku is free for now, but as far as I am aware they are aiming for a utility based pricing model, where you only pay for the actual bandwidth and CPU utilization you have consumed. If done right, this should be a great model that allows developers to launch their application without committing to any large upfront costs, and scale up the cost linearly with utilization.

Overall Heroku is very impressive, and if you're starting out with a brand-new Rails app, it's well worth considering, at least in the early phases. It does come with some significant limitations compared to traditional hosting options, which may or may not be a big deal for you. Of course these are offset with the elimination of IT related tasks that are no longer necessary in this environment.

But as much as I like Heroku's premise, we were underwhelmed with the performance of our Rails app on Heroku, which was our main reason for exploring alternatives such as Morph. Of course, since Heroku is still in its early stages, I'm sure they will be able to improve this over time.

Morph AppSpace

Morph is similar in principle to Heroku, but in my experience it is generally a bit more flexible.

One of the main differences is that scaling isn't quite as transparent as in Heroku. Morph applications run on one or more "cubes", which they describe as a "virtualized application compute environment". A single cube is free and may very well be sufficient during development, so as with Heroku there is no up-front cost (although free plans don't support custom domains). Additional cubes cost $31 per month (the price goes down after 4 cubes) and also come with increased bandwidth and storage limits. But unlike typical VPS accounts, cubes are charged on a daily basis, so you can ramp up (or down) your cubes any time as needed to adjust to your application's utilization. This does mean that you need to manually allow your app to scale by adding additional cubes, but this is easily accomplished in the Morph Control Panel. Apparently automatic scaling based on application load is planned for a future release.

The next difference is in the range of applications that Morph supports: Rails, Java (including Grails), and PHP (experimental). I have only used Morph for Rails applications and can't speak to any of the other options.

Morph supports both PostgreSQL and MySQL (although the latter costs an extra $0.33 per day, apparently due to licensing issues). One very useful feature is that Morph gives you direct access to the database via a web based admin tool (phpMyAdmin in case of MySQL).

Unlike Heroku, Morph does not offer a web based IDE. It also doesn't create a blank Rails app for you, nor does it offer source code hosting. Instead, it integrates with your existing source control system via a customized Capistrano deployment script. You simply specify the type (supported are Git, Subversion, Mercurial, Bazaar, and Local Directory) and URL of your SCM (GitHub in our case) and their wizard spits out a customized Capistrano script. After that, deployments are a breeze, not unlike Heroku. One difference is that Morph's deployment process involves multiple stages, the first of which consists of uploading your code to S3. As a result, you don't actually see detailed deployment status in the console and need to refer to the deployment logs in the Control Panel instead.

Another critical feature for us is that Morph has at least rudimentary support for cron jobs. The Control Panel allows you to run Rake tasks directly (just like Heroku), but it also allows Rake tasks to be scheduled at arbitrary intervals (the minimum interval is 1 minute). This still doesn't allow us to offload arbitrary processing jobs to a job server, but it does at least enable jobs to be queued up in the database and processed asynchronously by a Rake task. We are using this mechanism to offload email processing to ar_mailer, which works great and leads to a significantly improved user experience.

On the downside, Morph does not provide access to the Rails console. For me this is largely offset by the ability to access the database, but console access would have been nice.

Last not least, at least for our application, Morph resulted in a significant speed boost (at least multiple X, close to an order of magnitude).

Verdict

So which service (if any) do I recommend? I wholeheartedly recommend both services. Which one is a better fit for you will likely depend on the specific features (and limitations) that matter to you. Personally, I'm mostly enamored with Morph at this point, the improved performance and ability to run background jobs being the biggest differentiators for me. It remains to be seen whether either Heroku or Morph remain good options for us as our application grows (the fact that neither support true background tasks or Memcached servers might become a limiting factor at some point), but if nothing else they're an ideal way to get off the ground.

If you're starting out with a new Rails application, you may want to simply try out both services (it really doesn't take that long) to see which one you like better. The easiest way to do this is to start with Heroku, as this creates a new Rails app for you and provides Git hosting. You should be able to then use Morph's Capistrano wizard to generate a custom Capistrano script for you and check this into your Git repository, at which point you'll be able to deploy the same Rails app to both Heroku and Morph. Of course, if you later decide to stick with Morph, you should find a different source code host (I heartily recommend GitHub).

The bottom line is that this new breed of hosting services is extremely compelling. Sure, both Heroku and Morph still have some rough edges and won't be a good match for all applications at this point, but the direction is very promising and I am excited to find out what the future holds.

Amazon EC2 Out Of Beta

23 Oct 2008

Today, Amazon removed the beta label from their EC2 service, along with a bunch of related announcements. This is great news!

Over the past half year, I have become an enthusiastic user of the various AWS services, including EC2, which has been very stable for me thus far. But now Amazon is formalizing this by offering a 99.95% availability guarantee as part of the new EC2 SLA.

I don't personally care about the new Windows support, but I suppose this might make some Microsoft aficionados happy...

Amazon also announced some exciting future features:

Management Console - The management console will simplify the process of configuring and operating your applications in the AWS cloud. You'll be able to get a global picture of your cloud computing environment using a point-and-click web interface.

I've been using Ylastic as a Management Console, and can highly recommend this service. It allows me to monitor and manage our various AWS services from any machine, without having to install any apps locally. But having a Management Console built right into AWS would be neat (assuming it is as solid as Ylastic).

Load Balancing - The load balancing service will allow you to balance incoming requests and traffic across multiple EC2 instances.

This is great! Currently, AWS users have to roll their own load balancing implementation or rely on a more limited, DNS based solution. Most of my EC2 deployments don't involve public websites, so I have not had to tackle this issue. But having a solid load balancing solution built right into AWS will be tremendously useful for me in the future.

Automatic Scaling - The auto-scaling service will allow you to grow and shrink your usage of EC2 capacity on demand based on application requirements.

Another feature that's been on our roadmap, and I'm excited to hear there's going to be an officially supported solution. Hopefully Amazon's implementation will be flexible enough to allow different criteria to determine when to start and stop instances, such as CPU usage or SQS queue status.

Cloud Monitoring - The cloud monitoring service will provide real time, multi-dimensional monitoring of host resources across any number of EC2 instances, with the ability to aggregate operational metrics across instances, Availability Zones, and time slots.

Another awesome feature! So far I've shied away from setting up a tool like Nagios for in-depth monitoring of our EC2 instances. It sounds like Amazon's built-in monitoring solution will meet this need.

Overall I'm very excited about the pace at which AWS has been improving over the past half year or so. Availability Zones and particularly Elastic IPs made a major difference, followed closely by EBS.

Here's a small wishlist of features I'd like to see in EC2:

  1. Instance Aliases: When running more than a handful of instances, each one for a specific service, it becomes very difficult to keep track of which instance ID maps to which service. Ideally instances would have a user-defined alias, but unfortunately the EC2 API does not offer this functionality. Luckily I am currently managing all instances via Ylastic, which supplements this functionality, but it means that I would be lost if I had to manage my existing instances using a different tool. This really should be implemented on the EC2 API level.

  2. Querying User Data: Along the same lines, EC2 instances do support arbitrary user data (which I use to specify the role of the instance upon startup), but unfortunately this can only be queried from within the instance, not externally. Again, Ylastic solves this issue by keeping track of the user data itself, but this should be supported by the EC2 API.

This list used to be a lot longer, but the recent release of EBS, coupled with the announcements above, took care of much of it. Nice work, Amazon!

Rails And JSON Containing Unicode Characters

27 Aug 2008

As I mentioned in a previous blog post, Rails 2.1 natively supports incoming JSON requests. Unfortunately, it still struggles with JSON data containing non-ASCII characters.

According to the JSON spec, JSON fully supports UTF-8 encoded text, so with a few exceptions it generally should not be necessary to escape non-ASCII characters with \u Unicode escape sequences. However, many JSON libraries appear to escape all non-ASCII text in this fashion. This in itself should not be a problem, but ActiveSupport::JSON currently does not properly parse JSON containing \u escapes, resulting in strings with literal \u escape sequences rather than the desired UTF-8 encoded characters. This is especially confusing since ActiveSupport:JSON itself encodes all non-ASCII characters as \u escapes, so one might think that the reverse transformation yields the original data. But this behavior is likely explained by an odd implementation choice for its decoder: Rather than using the json (or json-pure) library, it converts the JSON data to YAML and then uses the YAML library to decode the data into Ruby objects.

Monkey-patching to the rescue! I decided to replace ActiveSupport::JSON::decode with an implementation that uses the json library. The easiest way is to stick the following code into a file named something like activesupport_json_unicode_patch.rb inside the config/initializers/ directory, where Rails will automatically pick it up.

require 'json'

module ActiveSupport
  module JSON
    def self.decode(json)
      ::JSON.parse(json)
    end
  end
end

You can verify the fix by adding a test case (I added a file named activesupport_json_test.rb to the test/unit/ directory):

require File.dirname(__FILE__) + '/../test_helper'

class ActiveSupportJsonTest < Test::Unit::TestCase
  
  def test_json_encoding
    unicode_escaped_json = '{"foo":"G\u00fcnter","bar":"El\u00e8ne"}'
    hash = ActiveSupport::JSON.decode(unicode_escaped_json)
    assert_equal({'foo' => 'Günter', 'bar' => 'Elène'}, hash)
  end
  
end

This test should fail without the patch and pass after adding it.

In addition to fixing the JSON / Unicode problem, this patch should also provide a nice speed boost, as we're replacing the somewhat roundabout YAML based JSON decode method with a native one (particularly if you're using the native json implementation rather than json-pure.)

It's been a while...

27 Aug 2008

I realized that it's been quite a while since my last update. Unfortunately it seems like the amount of interesting stuff I have to write about is inversely proportional to the spare time I have available for writing... ;)

Anyway, I figure I'll try to get back into the habit of publishing smaller posts, but hopefully more regularly. Let's see how it goes...

Ever since I went back into startup life four months ago, I've had the chance to play with a lot of exciting technologies, so there's plenty of stuff to write about. Below are just some quick notes and mini-reviews, but I'll be writing more about the various topics in the future.

  • Merb
    • This is quickly becoming my web framework of choice. It is well-thought-out, highly modular, and ideally suited for both small webservices as well as large scale websites.
    • The default application layout is very similar to Rails, making it easy for Rails developers to get up to speed on Merb. However, it also supports highly compact app layouts that are ideal for smaller webservices.
    • I'm sure I'll write more about Merb later.
  • DataMapper
    • An awesome ORM framework, and one of several supported by Merb. Like Merb it is still pre-1.0 and therefore still evolving a lot, but it is already quite solid and very powerful.
    • More on this later...
  • RSpec
    • For my new projects, I have been using RSpec exclusively, and after a small learning curve it has really grown on me. Granted, I'm probably not taking advantage of all its features (for example I haven't actually written my own matchers yet), but with a bit of discipline (and a heavy dose of mocking and stubbing), specs written in RSpec are probably an order of magnitude clearer and more readable than with Test::Unit.
    • I took the opportunity to get rid of test fixtures as well (and good riddance!)
  • Ruby in general + Rails
    • Right now I am working with a mix of Merb projects, Rails projects, and standalone Ruby apps.
    • I've had the opportunity to build a Ruby based framework for SMS based mobile apps, consisting of a processing pipeline with several independent services that communicate via asynchronous message queues, with some DSLs and meta-programming thrown in for good measure. This has allowed me to become much more familiar with Ruby than I was back when I just played with some Rails apps.
    • There's definitely a right language for every task, but I have to admit that I find it harder and harder to imagine ever going back to Java...
  • Amazon Web Services (EC2, SQS, and S3)
    • Using Amazon AWS as a deployment platform is a whole new experience for me, and I am very impressed with the various services. They were already solid when I started working with them, but in the past few months Amazon managed to launch several major features (such as static IP addresses and persistent storage) that significantly lower the barrier of entry.
    • EC2 is particularly well-suited for the type of loosely coupled architecture we are developing, and I envision eventually being able to dynamically start and stop instances to adjust both to general growth over time, as well as fluctuations throughout the day (as mobile apps tend to exhibit certain usage patterns.)
    • SQS is a convenient way to tie the different components together (albeit with some limitations, such as an 8KB maximum message size), and of course S3 is available for any storage needs.
    • I will definitely blog in more detail about various AWS strategies and recipes.
  • Git
    • We've been using Git (and GitHub) for all new applications. We're definitely not exploiting it to its full potential so far (given that we're only a few developers that mostly work on different products, we haven't had much of a need to leverage Git's distributed nature), but I am definitely growing quite fond of it.
    • It is extremely fast, I love being able to commit code and browse the revision history even without a network connection, topic branches are convenient and powerful, and more.
    • Tool support is definitely still a weak point, but for the most part I've been happy using Git on the command line.
  • Lots of sysadmin / deployment stuff
    • System administration is definitely not my forte, but I get by (at least on Linux; Joyent's Solaris servers still manage to throw me off occasionally...).
    • I have had a chance to get more exposure to deploying Rails apps (including Mongrel, Monit, Passenger, etc.), as well as building a deployment framework for AWS from scratch.
    • Deploying applications to such a cloud based architecture is quite different from a typical Rails or LAMP stack, but I'm quite happy with the initial version of the deployment framework I've built. I'm essentially using S3 to store versioned app packages (which correspond directly to a Git tag) as well as third party gems, and a configuration file (also in S3) for each environment (such as production or staging) defines which version should be deployed on it. Each service regularly polls S3 for configuration changes and updates itself if appropriate. We use a single machine image and configure each instance via user data upon startup, which allows the instance to pull down the appropriate files from S3 and start running the service it is intended for. A small set of Rake tasks manage deploying releases and promoting them from one environment to another one.
    • There are still some challenges ahead, though, such as a proper logging system (I'm planning to migrate our apps to log to a central syslog server.)
  • Various web based services, including Google Apps, GitHub, Lighthouse, Scout, and Ylastic
    • All of these are great utilities.
    • Google Apps is a must-have; any startup should use it at least for email and calendaring, as well as collaborating on documents or spreadsheets. We also use Google Sites as our Intranet / Wiki.
    • I have briefly blogged about Scout before and will likely blog about various Scout plugins in the future.
    • GitHub is an amazing application, not only for hosting our own source code, but also for following the various open source projects we depend on (such as Rails, Merb, and DataMapper.)
    • We haven't had a chance to really dive into Lighthouse yet, so the jury is still out on it. I do believe that its refreshingly simple, tag based approach should work pretty well, though.
    • Last not least, Ylastic has been invaluable and a great way to manage our EC2 instances, images, debug SQS based apps, browse S3, and more. I'll probably write up a more thorough review soon.
  • Honorary mention: Pandora and Airfoil
    • For keeping us supplied with music and allowing us to stream it to our Airport Express. :)

That's it for now. Hopefully it won't be two months before my next post...

Network Monitoring With Scout

24 Jun 2008

I've been meaning to set up a network monitoring tool at work for a while. We have a couple of different applications using various technologies (currently mainly Ruby on Rails and PHP), running on various VPS servers. While we are using Monit to keep an eye on our Rails apps and restart them if necessary, as well as a couple of custom webpages to track vital and growth stats of our apps, we currently don't use any monitoring or (perhaps more importantly) alerting tools beyond that. After one of our PHP / MySQL apps stopped responding (due to the fact that we ran out of disk space, as we later discovered), I figured it was about time to put some more sophisticated network monitoring in place.

The de-facto standard application seems to be Nagios, which is quite powerful and configurable, but has an extremely steep learning curve. It also does not offer a friendly UI for configuring services and relies on static configuration files instead. There is also a newer crop of network monitoring apps that I was hoping might be a bit less daunting to get up to speed on, such as Zenoss (here is a brief overview of open source network monitoring apps). I downloaded a few of these, but ultimately realized that I would not be able to get my head wrapped around any of these apps purely using intuition, and that I would actually have to invest a fair amount of time to master at least the basics. I'm simply not enough of an operations expert... It is clear that all of these apps are extremely powerful, and probably great for larger deployments, but I really just needed a simple tool to check some operating system level vital stats or ping some URLs, for a handful of machines.

That's when I remembered Scout, a hosted network monitoring service that launched fairly recently and that sounded very interesting when I first came across it. Their subscription plans are pretty reasonable (the $29/month plan for 4 servers should suffice for us at this stage), particularly given that we would have had to pay for an additional VPS slice or EC2 instance to host Nagios or some other deployed solution anyway. Best of all, Scout offers a free plan, and even though this only supports a single server, this is a good way to evaluate how well it works for our purposes.

Scout uses an interesting approach at monitoring servers. Rather than using SNMP or an agent that is continuously running on each server, Scout uses a lightweight client (installed via a Ruby gem) that needs to be run periodically (10 minutes being the minimal reporting interval), generally via a cron job. Once the client app is installed on each server to be monitored, the servers don't need to be touched for future configuration changes. Instead, everything is configured on the Scout website, and pulled down by the client the next time it checks in. The entire configuration consists of a number of plugins that can be installed for each client. Out of the box, Scout supports around 20 plugins that range from basic monitoring tasks for server load or disk space to more specific plugins for Ruby on Rails, Mongrel, or MySQL.

Even better, Scout offers a very simple Plugin API for integrating your own plugins. Plugins are written in Ruby and mainly consist of a single method that either returns a bunch of stats as a hash, which is exposed by Scout both in tabular report and graph form, or triggers an alert in case of a problem. Since plugins have the full Ruby stack at their disposal, it is easy to write a plugin that shells out to a Unix command, performs an HTTP request, hits a database, or anything else you can think of.

One minor downside is that (as far as I can tell), there is no way to simply upload a plugin. Instead, Scout relies on a pull mechanism, which means that we would need to expose any proprietary plugins via a publicly accessible URL. This might be an issue if the plugin itself contains sensitive information, although settings (such as passwords or paths) can be decoupled from the code and configured via the web interface. While not ideal, putting the plugin code in a publicly accessible but not automatically discoverable location and only making it available for the duration of the initial download or future updates should minimize this risk and turn it into a minor inconvenience.

Based on my initial impression, Scout looks very promising. The reporting functionality is fairly basic, and particularly the graphs could perhaps use a bit more polish, but everything is very easy to use. Scout is clearly geared towards developers rather than sysadmins, so perhaps that is why it appeals to me. If your monitoring needs are relatively straightforward and you don't need all the functionality that a deployed solution like Nagios offers, Scout is definitely worth a look, at least for relatively small deployments. I am not sure how well it scales beyond 16 servers (both in terms of administration and pricing), so it is possible that a deployed application might make more sense at that point.

Phusion Passenger Now Rack Compatible?

01 Jun 2008

According to this blog post and several mentions on Twitter, Phusion announced today's release of Passenger (aka mod_rails) 2.0 at RailsConf. Apparently, Passenger 2.0 will be Rack compliant and thus support not only Rails, but any Rack compatible web framework, including Merb and Sinatra. Interestingly, Passenger will not even be limited to Ruby any more and extend their support to WSGI, the Python web adapter framework that inspired Rack in the first place. For example, this will allow Passenger 2.0 to run the popular Django web framework. In light of these changes, Passenger will drop the name mod_rails.

I think these are fantastic news! As I mentioned in my previous post on Phusion Passenger, it makes deploying Rails apps trivially easy, and I am planning to use it as the default deployment platform for my Rails apps. I have also been flirting with Merb lately, and knowing that I am going to be able to deploy it just as easily as Rails makes a big difference to me.

Support for Rack is the logical next step for Passenger, so I am not all that surprised about the direction they are going. I am however a bit surprised about the timing. After all, version 1.0 was only released fairly recently and prominently branded as mod_rails. The http://www.modrails.com/documentation.html even states (although I assume that this will be updated within the next few days):

Does it support other Ruby frameworks (Merb, Camping, etc.)?

No.

What?! Why??

Because this is an evil plot created by evil overlords, with the goal of world domination destroying all other Ruby frameworks. …

Actually…

There is the following saying: Jack of all trades, master of none. Our intention is to be masters, not Jacks. The primary goal of version 1.0 was to create an easy-to-use, low-maintenance, stable and fast Ruby on Rails deployment system for Apache. And we've put a lot of effort into reaching that goal. Implementing support for other Ruby frameworks would have deviated us from that goal and would have increased development time significantly.

That said, nothing prevents future versions from supporting other Ruby frameworks, or from becoming a generic Ruby web application deployment platform. Please discuss it with us if you're interested in steering development towards that direction.

My guess is that there was a lot of demand for Rack support from users (even Rails now supports Rack), and after looking into it, they realized it was easier to integrate this than they initially expected. Either way, this is great news, and I look forward to trying out Passenger 2.0 with Merb and DataMapper, or even Sinatra for smaller apps.

Update: 2.0 RC1 has been released, and you can find more details in the release announcement on the Phusion blog. In addition to Rack and WSGI support, 2.0 sounds like a more solid and stable release overall, with a significantly smaller memory footprint, faster startup time, fair load balancing, upload buffering, and some convenient analysis tools. There's also a native Ubuntu package now, in case you want to avoid compiling from source.

Rails 2.1 and Incoming JSON Requests

25 May 2008

Earlier this week, we tried to figure out the cleanest and easiest way to get our Rails app to accept incoming JSON requests. Up until recently, developers were able to use various Rails plugins for this purpose, such as the json_request plugin.

Luckily, it turns out that full support for JSON was added to Rails in April, making it a first class citizen along with XML and regular URL-encoded form fields. This functionality will be officially released in Rails 2.1, but in addition to Edge Rails, it is already included in Rails 2.0.991, which is available from the Ruby on Rails Gem Repository. You can install this pre-release via:

sudo gem update rails --source http://gems.rubyonrails.org

Using this functionality is really simple. Let's say we have created the following scaffolded Rails app with a Book resource, perhaps to manage your library:

rails library
cd library
script/generate scaffold book title:string author:string isbn:string price:decimal
rake db:migrate

As you know, you can now access the books controller in the browser via http://localhost:3000/books, and use the "New Book" link to create a new book via the scaffolded form that Rails provides. But you can also create books via JSON (or XML, for that matter). In fact, we will try XML first, which has been natively supported in Rails for a while:

curl -H "Content-Type:text/xml" -H "Accept:text/xml" \
  -d "<book><title>Posted via XML</title><author>Ex Emel</author><isbn>1234567890</isbn><price>34.99</price></book>" \
  http://localhost:3000/books

Note that I am setting both the Content-Type and Accept header to "text/xml", indicating that the incoming request consists of XML and that we would like to receive an XML-formatted response as well. The response looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<book>
  <author>Ex Emel</author>
  <created-at type="datetime">2008-05-26T05:58:38Z</created-at>
  <id type="integer">2</id>
  <isbn>1234567890</isbn>
  <price type="decimal">34.99</price>
  <title>Posted via XML 2</title>
  <updated-at type="datetime">2008-05-26T05:58:38Z</updated-at>
</book>

Now let's try the same thing in JSON:

curl -H "Content-Type:application/json" -H "Accept:application/json" \
  -d "{\"book\":{\"title\":\"Posted via JSON\", \"author\":\"Jason Bourne\", \"isbn\":1234567890, \"price\":49.95}}" \
  http://localhost:3000/books

As you can verify in the browser, this request was successful and had the desired effect. However, unlike the XML case, there was no response this time. This is because our controller does not know how to render JSON results yet. Looking at the create method in the BooksController, we notice that the responds_to block contains entries for HTML and XML, but not for JSON. Simply copy the XML lines and replace all occurrences of xml with json. The updated method should look like this:

def create
  @book = Book.new(params[:book])
    
  respond_to do |format|
    if @book.save
      flash[:notice] = 'Book was successfully created.'
      format.html { redirect_to(@book) }
      format.xml  { render :xml => @book, :status => :created, :location => @book }
      format.json { render :json => @book, :status => :created, :location => @book }
    else
      format.html { render :action => "new" }
      format.xml  { render :xml => @book.errors, :status => :unprocessable_entity }
      format.json { render :json => @book.errors, :status => :unprocessable_entity }
    end
  end
end

If you run the same curl command again, you should now get the following response:

{"book": {"isbn": 1234567890, "updated_at": "2008-05-26T06:11:33Z",
"title": "Posted via JSON", "price": 49.95, "author": "Jason Bourne",
"id": 4, "created_at": "2008-05-26T06:11:33Z"}}

One important thing to note is that both the incoming and outgoing JSON contain an outermost element called book. This is in fact required for resource based JSON requests to work. The same is true for XML, but since XML requires an enclosing element (unlike JSON, which can be "naked"), it is perhaps less obvious in this case. The outermost JSON element should always have the same name as the resource it corresponds to.

A few notes on testing JSON requests:

We initially banged our heads against the wall trying to figure out how to convince our functional test to pass JSON-formatted parameters in the post method, including various hacks to override different settings on the @request object. Admittedly it had been a while since I had seriously used Rails the last time, but it later dawned on me that we were going about this the wrong way. Functional tests (in Rails, anyways... let's not talk about its confusing and non-standard test terminology) bypass most of the actual HTTP request handling and are not meant to test this aspect of an application. They essentially pick up at the point where the controller has received its (already parsed) parameters in the params hash, regardless of whether these originated from an XML, JSON, or URL-encoded form request.

Since JSON support is implemented by Rails (and thus covered by its own unit tests), it probably does not make sense to focus too much on testing this general functionality in the individual application. But if you do want to test JSON requests, you can use integration tests for this purpose. The post method in integration tests is more low level and simulates the actual HTTP request, along with the parameter parsing.

So in our library example, we might use an integration test case such as the one below to specifically test creating a book via JSON:

def test_create_via_json
  assert_difference('Book.count') do
    post '/books/create',
      '{"book":{"title":"Posted via JSON", "author":"Jason Bourne", "isbn":1234567890, "price":49.95}}',
      {'Content-Type' => 'application/json', 'Accept' => 'application/json'}
  end
end

Native JSON support in Rails is definitely a useful feature. In fact, I was fairly surprised that this wasn't already implemented until recently. But now that it's here, it should come in very handy.

Phusion Passenger (aka mod_rails) on DreamHost

14 May 2008

A couple weeks ago, Phusion Passenger (aka mod_rails) was released. I recently tested this at work, on an EC2 instance, and my initial experience was so smooth that I am already planning to use it to deploy our various Rails applications. The benchmarks I've seen put its performance on approximately the same level as a Mongrel cluster, but its ease of use is an order of magnitude better. All you need to do is install an Apache mod and set up a virtual host config that points to your Rails app's public directory. You don't even need to tell it that the directory you're pointing to represents a Rails app -- mod_rails is smart enough to figure this out by itself (although there are a few Rails specific options you can use to control the base URI or the Rails environment). No more juggling Mongrel PIDs, complicated proxy configs, or anything of that sort. Simply create a tmp/restart.txt file to have Apache reload the Rails app after you deploy a new version. Tom Copeland posted a very simple Capistrano script for mod_rails, which essentially does just that and stubs out the usual Rails Capistrano tasks that are no longer necessary in this setup.

Yesterday, DreamHost announced their support for mod_rails. I had played with Rails on DreamHost several years ago (back when FCGI was still the generally accepted way to run Rails apps), but ultimately gave up on this because of the frustrating experience (performance, stability, and ease of deployment wise). Since then, VPS hosting services (such as SliceHost) have become the prevalent solution for hosting Rails apps. But with DreamHost officially supporting mod_rails, I figured I'd give this a spin to see how well it works in practice.

I am happy to say, it seems to work just as advertised! In order to test Rails on DreamHost, I downloaded the popular Mephisto Blog Application, unzipped it into a directory on my DreamHost account, and configured the database settings (I didn't even bother with MySQL and opted for Sqlite3 for the purpose of this test). I then went into my domain's settings on the DreamHost Web Panel, checked the "Ruby on Rails Passenger (mod_rails)" checkbox, and pointed to my Mephisto directory's public subdirectory as the web directory for my domain (this is important, as the web directory defaults to yourdomain.com, without the /public that mod_rails expects).

A minute or two later, my changes had been applied and I was greeted by the Mephisto blog when I hit my domain in the browser. I configured my blog's settings and entered some dummy articles, and found the performance to be very snappy -- no different from PHP apps that I am hosting at DreamHost (such as this WordPress blog).

I think this is pretty exciting. Sure, there are many other cost-effective options to deploy Rails apps these days (such as the unique and highly promising Heroku or a cheap $20 VPS slice on SliceHost), but for a personal blog or another small, reasonably low-traffic website (such as the 12 or so random Rails apps all of us are concurrently working on and too cheap to spring for VPS hosting, since most of them will never go anywhere), having the option to easily deploy these on a shared hosting account is great.

Now I am hoping that mod_rails will be extended beyond just Rails to support any Rack compliant Ruby web framework, such as Merb or Sinatra.