According to the JSON spec, JSON fully supports UTF-8 encoded text, so with a few exceptions it generally should not be necessary to escape non-ASCII characters with \u Unicode escape sequences. However, many JSON libraries appear to escape all non-ASCII text in this fashion. This in itself should not be a problem, but ActiveSupport::JSON currently does not properly parse JSON containing \u escapes, resulting in strings with literal \u escape sequences rather than the desired UTF-8 encoded characters. This is especially confusing since ActiveSupport:JSON itself encodes all non-ASCII characters as \u escapes, so one might think that the reverse transformation yields the original data. But this behavior is likely explained by an odd implementation choice for its decoder: Rather than using the json (or json-pure) library, it converts the JSON data to YAML and then uses the YAML library to decode the data into Ruby objects.

Monkey-patching to the rescue! I decided to replace ActiveSupport::JSON::decode with an implementation that uses the json library. The easiest way is to stick the following code into a file named something like activesupport_json_unicode_patch.rb inside the config/initializers/ directory, where Rails will automatically pick it up.

require 'json'
 
module ActiveSupport
  module JSON
    def self.decode(json)
      ::JSON.parse(json)
    end
  end
end

You can verify the fix by adding a test case (I added a file named activesupport_json_test.rb to the test/unit/ directory):

require File.dirname(__FILE__) + '/../test_helper'
 
class ActiveSupportJsonTest < Test::Unit::TestCase
 
  def test_json_encoding
    unicode_escaped_json = '{"foo":"G\u00fcnter","bar":"El\u00e8ne"}'
    hash = ActiveSupport::JSON.decode(unicode_escaped_json)
    assert_equal({'foo' => 'Günter', 'bar' => 'Elène'}, hash)
  end
 
end

This test should fail without the patch and pass after adding it.

In addition to fixing the JSON / Unicode problem, this patch should also provide a nice speed boost, as we’re replacing the somewhat roundabout YAML based JSON decode method with a native one (particularly if you’re using the native json implementation rather than json-pure.)

Anyway, I figure I’ll try to get back into the habit of publishing smaller posts, but hopefully more regularly. Let’s see how it goes…

Ever since I went back into startup life four months ago, I’ve had the chance to play with a lot of exciting technologies, so there’s plenty of stuff to write about. Below are just some quick notes and mini-reviews, but I’ll be writing more about the various topics in the future.

• Merb
  • This is quickly becoming my web framework of choice. It is well-thought-out, highly modular, and ideally suited for both small webservices as well as large scale websites.
  • The default application layout is very similar to Rails, making it easy for Rails developers to get up to speed on Merb. However, it also supports highly compact app layouts that are ideal for smaller webservices.
  • I’m sure I’ll write more about Merb later.
• DataMapper
  • An awesome ORM framework, and one of several supported by Merb. Like Merb it is still pre-1.0 and therefore still evolving a lot, but it is already quite solid and very powerful.
  • More on this later…
• RSpec
  • For my new projects, I have been using RSpec exclusively, and after a small learning curve it has really grown on me. Granted, I’m probably not taking advantage of all its features (for example I haven’t actually written my own matchers yet), but with a bit of discipline (and a heavy dose of mocking and stubbing), specs written in RSpec are probably an order of magnitude clearer and more readable than with Test::Unit.
  • I took the opportunity to get rid of test fixtures as well (and good riddance!)
• Ruby in general + Rails
  • Right now I am working with a mix of Merb projects, Rails projects, and standalone Ruby apps.
  • I’ve had the opportunity to build a Ruby based framework for SMS based mobile apps, consisting of a processing pipeline with several independent services that communicate via asynchronous message queues, with some DSLs and meta-programming thrown in for good measure. This has allowed me to become much more familiar with Ruby than I was back when I just played with some Rails apps.
  • There’s definitely a right language for every task, but I have to admit that I find it harder and harder to imagine ever going back to Java…
• Amazon Web Services (EC2, SQS, and S3)
  • Using Amazon AWS as a deployment platform is a whole new experience for me, and I am very impressed with the various services. They were already solid when I started working with them, but in the past few months Amazon managed to launch several major features (such as static IP addresses and persistent storage) that significantly lower the barrier of entry.
  • EC2 is particularly well-suited for the type of loosely coupled architecture we are developing, and I envision eventually being able to dynamically start and stop instances to adjust both to general growth over time, as well as fluctuations throughout the day (as mobile apps tend to exhibit certain usage patterns.)
  • SQS is a convenient way to tie the different components together (albeit with some limitations, such as an 8KB maximum message size), and of course S3 is available for any storage needs.
  • I will definitely blog in more detail about various AWS strategies and recipes.
• Git
  • We’ve been using Git (and GitHub) for all new applications. We’re definitely not exploiting it to its full potential so far (given that we’re only a few developers that mostly work on different products, we haven’t had much of a need to leverage Git’s distributed nature), but I am definitely growing quite fond of it.
  • It is extremely fast, I love being able to commit code and browse the revision history even without a network connection, topic branches are convenient and powerful, and more.
  • Tool support is definitely still a weak point, but for the most part I’ve been happy using Git on the command line.
• Lots of sysadmin / deployment stuff
  • System administration is definitely not my forte, but I get by (at least on Linux; Joyent’s Solaris servers still manage to throw me off occasionally…).
  • I have had a chance to get more exposure to deploying Rails apps (including Mongrel, Monit, Passenger, etc.), as well as building a deployment framework for AWS from scratch.
  • Deploying applications to such a cloud based architecture is quite different from a typical Rails or LAMP stack, but I’m quite happy with the initial version of the deployment framework I’ve built. I’m essentially using S3 to store versioned app packages (which correspond directly to a Git tag) as well as third party gems, and a configuration file (also in S3) for each environment (such as production or staging) defines which version should be deployed on it. Each service regularly polls S3 for configuration changes and updates itself if appropriate. We use a single machine image and configure each instance via user data upon startup, which allows the instance to pull down the appropriate files from S3 and start running the service it is intended for. A small set of Rake tasks manage deploying releases and promoting them from one environment to another one.
  • There are still some challenges ahead, though, such as a proper logging system (I’m planning to migrate our apps to log to a central syslog server.)
• Various web based services, including Google Apps, GitHub, Lighthouse, Scout, and Ylastic
  • All of these are great utilities.
  • Google Apps is a must-have; any startup should use it at least for email and calendaring, as well as collaborating on documents or spreadsheets. We also use Google Sites as our Intranet / Wiki.
  • I have briefly blogged about Scout before and will likely blog about various Scout plugins in the future.
  • GitHub is an amazing application, not only for hosting our own source code, but also for following the various open source projects we depend on (such as Rails, Merb, and DataMapper.)
  • We haven’t had a chance to really dive into Lighthouse yet, so the jury is still out on it. I do believe that its refreshingly simple, tag based approach should work pretty well, though.
  • Last not least, Ylastic has been invaluable and a great way to manage our EC2 instances, images, debug SQS based apps, browse S3, and more. I’ll probably write up a more thorough review soon.
• Honorary mention: Pandora and Airfoil
  • For keeping us supplied with music and allowing us to stream it to our Airport Express.

That’s it for now. Hopefully it won’t be two months before my next post…

The de-facto standard application seems to be Nagios, which is quite powerful and configurable, but has an extremely steep learning curve. It also does not offer a friendly UI for configuring services and relies on static configuration files instead. There is also a newer crop of network monitoring apps that I was hoping might be a bit less daunting to get up to speed on, such as Zenoss (here is a brief overview of open source network monitoring apps). I downloaded a few of these, but ultimately realized that I would not be able to get my head wrapped around any of these apps purely using intuition, and that I would actually have to invest a fair amount of time to master at least the basics. I’m simply not enough of an operations expert… It is clear that all of these apps are extremely powerful, and probably great for larger deployments, but I really just needed a simple tool to check some operating system level vital stats or ping some URLs, for a handful of machines.

That’s when I remembered Scout, a hosted network monitoring service that launched fairly recently and that sounded very interesting when I first came across it. Their subscription plans are pretty reasonable (the $29/month plan for 4 servers should suffice for us at this stage), particularly given that we would have had to pay for an additional VPS slice or EC2 instance to host Nagios or some other deployed solution anyway. Best of all, Scout offers a free plan, and even though this only supports a single server, this is a good way to evaluate how well it works for our purposes.

Scout uses an interesting approach at monitoring servers. Rather than using SNMP or an agent that is continuously running on each server, Scout uses a lightweight client (installed via a Ruby gem) that needs to be run periodically (10 minutes being the minimal reporting interval), generally via a cron job. Once the client app is installed on each server to be monitored, the servers don’t need to be touched for future configuration changes. Instead, everything is configured on the Scout website, and pulled down by the client the next time it checks in. The entire configuration consists of a number of plugins that can be installed for each client. Out of the box, Scout supports around 20 plugins that range from basic monitoring tasks for server load or disk space to more specific plugins for Ruby on Rails, Mongrel, or MySQL.

Even better, Scout offers a very simple Plugin API for integrating your own plugins. Plugins are written in Ruby and mainly consist of a single method that either returns a bunch of stats as a hash, which is exposed by Scout both in tabular report and graph form, or triggers an alert in case of a problem. Since plugins have the full Ruby stack at their disposal, it is easy to write a plugin that shells out to a Unix command, performs an HTTP request, hits a database, or anything else you can think of.

One minor downside is that (as far as I can tell), there is no way to simply upload a plugin. Instead, Scout relies on a pull mechanism, which means that we would need to expose any proprietary plugins via a publicly accessible URL. This might be an issue if the plugin itself contains sensitive information, although settings (such as passwords or paths) can be decoupled from the code and configured via the web interface. While not ideal, putting the plugin code in a publicly accessible but not automatically discoverable location and only making it available for the duration of the initial download or future updates should minimize this risk and turn it into a minor inconvenience.

Based on my initial impression, Scout looks very promising. The reporting functionality is fairly basic, and particularly the graphs could perhaps use a bit more polish, but everything is very easy to use. Scout is clearly geared towards developers rather than sysadmins, so perhaps that is why it appeals to me. If your monitoring needs are relatively straightforward and you don’t need all the functionality that a deployed solution like Nagios offers, Scout is definitely worth a look, at least for relatively small deployments. I am not sure how well it scales beyond 16 servers (both in terms of administration and pricing), so it is possible that a deployed application might make more sense at that point.

I think these are fantastic news! As I mentioned in my previous post on Phusion Passenger, it makes deploying Rails apps trivially easy, and I am planning to use it as the default deployment platform for my Rails apps. I have also been flirting with Merb lately, and knowing that I am going to be able to deploy it just as easily as Rails makes a big difference to me.

Support for Rack is the logical next step for Passenger, so I am not all that surprised about the direction they are going. I am however a bit surprised about the timing. After all, version 1.0 was only released fairly recently and prominently branded as mod_rails. The [http://www.modrails.com/documentation.html](online documentation) even states (although I assume that this will be updated within the next few days):

Does it support other Ruby frameworks (Merb, Camping, etc.)?

No.

What?! Why??

Because this is an evil plot created by evil overlords, with the goal of world domination destroying all other Ruby frameworks. …

Actually…

There is the following saying: Jack of all trades, master of none. Our intention is to be masters, not Jacks. The primary goal of version 1.0 was to create an easy-to-use, low-maintenance, stable and fast Ruby on Rails deployment system for Apache. And we’ve put a lot of effort into reaching that goal. Implementing support for other Ruby frameworks would have deviated us from that goal and would have increased development time significantly.

That said, nothing prevents future versions from supporting other Ruby frameworks, or from becoming a generic Ruby web application deployment platform. Please discuss it with us if you’re interested in steering development towards that direction.

My guess is that there was a lot of demand for Rack support from users (even Rails now supports Rack), and after looking into it, they realized it was easier to integrate this than they initially expected. Either way, this is great news, and I look forward to trying out Passenger 2.0 with Merb and DataMapper, or even Sinatra for smaller apps.

Update: 2.0 RC1 has been released, and you can find more details in the release announcement on the Phusion blog. In addition to Rack and WSGI support, 2.0 sounds like a more solid and stable release overall, with a significantly smaller memory footprint, faster startup time, fair load balancing, upload buffering, and some convenient analysis tools. There’s also a native Ubuntu package now, in case you want to avoid compiling from source.

Luckily, it turns out that full support for JSON was added to Rails in April, making it a first class citizen along with XML and regular URL-encoded form fields. This functionality will be officially released in Rails 2.1, but in addition to Edge Rails, it is already included in Rails 2.0.991, which is available from the Ruby on Rails Gem Repository. You can install this pre-release via:

sudo gem update rails --source http://gems.rubyonrails.org

Using this functionality is really simple. Let’s say we have created the following scaffolded Rails app with a Book resource, perhaps to manage your library:

rails library
cd library
script/generate scaffold book title:string author:string isbn:string price:decimal
rake db:migrate

As you know, you can now access the books controller in the browser via http://localhost:3000/books, and use the “New Book” link to create a new book via the scaffolded form that Rails provides. But you can also create books via JSON (or XML, for that matter). In fact, we will try XML first, which has been natively supported in Rails for a while:

curl -H "Content-Type:text/xml" -H "Accept:text/xml" \
  -d "<book><title>Posted via XML</title><author>Ex Emel</author><isbn>1234567890</isbn><price>34.99</price></book>" \
  http://localhost:3000/books

Note that I am setting both the Content-Type and Accept header to “text/xml”, indicating that the incoming request consists of XML and that we would like to receive an XML-formatted response as well. The response looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<book>
  <author>Ex Emel</author>
  <created-at type="datetime">2008-05-26T05:58:38Z</created-at>
  <id type="integer">2</id>
  <isbn>1234567890</isbn>
  <price type="decimal">34.99</price>
  <title>Posted via XML 2</title>
  <updated-at type="datetime">2008-05-26T05:58:38Z</updated-at>
</book>

Now let’s try the same thing in JSON:

curl -H "Content-Type:application/json" -H "Accept:application/json" \
  -d "{\"book\":{\"title\":\"Posted via JSON\", \"author\":\"Jason Bourne\", \"isbn\":1234567890, \"price\":49.95}}" \
  http://localhost:3000/books

As you can verify in the browser, this request was successful and had the desired effect. However, unlike the XML case, there was no response this time. This is because our controller does not know how to render JSON results yet. Looking at the create method in the BooksController, we notice that the responds_to block contains entries for HTML and XML, but not for JSON. Simply copy the XML lines and replace all occurrences of xml with json. The updated method should look like this:

def create
  @book = Book.new(params[:book])
 
  respond_to do |format|
    if @book.save
      flash[:notice] = 'Book was successfully created.'
      format.html { redirect_to(@book) }
      format.xml  { render :xml => @book, :status => :created, :location => @book }
      format.json { render :json => @book, :status => :created, :location => @book }
    else
      format.html { render :action => "new" }
      format.xml  { render :xml => @book.errors, :status => :unprocessable_entity }
      format.json { render :json => @book.errors, :status => :unprocessable_entity }
    end
  end
end

If you run the same curl command again, you should now get the following response:

{"book": {"isbn": 1234567890, "updated_at": "2008-05-26T06:11:33Z",
"title": "Posted via JSON", "price": 49.95, "author": "Jason Bourne",
"id": 4, "created_at": "2008-05-26T06:11:33Z"}}

One important thing to note is that both the incoming and outgoing JSON contain an outermost element called book. This is in fact required for resource based JSON requests to work. The same is true for XML, but since XML requires an enclosing element (unlike JSON, which can be “naked”), it is perhaps less obvious in this case. The outermost JSON element should always have the same name as the resource it corresponds to.

A few notes on testing JSON requests:

We initially banged our heads against the wall trying to figure out how to convince our functional test to pass JSON-formatted parameters in the post method, including various hacks to override different settings on the @request object. Admittedly it had been a while since I had seriously used Rails the last time, but it later dawned on me that we were going about this the wrong way. Functional tests (in Rails, anyways… let’s not talk about its confusing and non-standard test terminology) bypass most of the actual HTTP request handling and are not meant to test this aspect of an application. They essentially pick up at the point where the controller has received its (already parsed) parameters in the params hash, regardless of whether these originated from an XML, JSON, or URL-encoded form request.

Since JSON support is implemented by Rails (and thus covered by its own unit tests), it probably does not make sense to focus too much on testing this general functionality in the individual application. But if you do want to test JSON requests, you can use integration tests for this purpose. The post method in integration tests is more low level and simulates the actual HTTP request, along with the parameter parsing.

So in our library example, we might use an integration test case such as the one below to specifically test creating a book via JSON:

def test_create_via_json
  assert_difference('Book.count') do
    post '/books/create',
      '{"book":{"title":"Posted via JSON", "author":"Jason Bourne", "isbn":1234567890, "price":49.95}}',
      {'Content-Type' => 'application/json', 'Accept' => 'application/json'}
  end
end

Native JSON support in Rails is definitely a useful feature. In fact, I was fairly surprised that this wasn’t already implemented until recently. But now that it’s here, it should come in very handy.

Yesterday, DreamHost announced their support for mod_rails. I had played with Rails on DreamHost several years ago (back when FCGI was still the generally accepted way to run Rails apps), but ultimately gave up on this because of the frustrating experience (performance, stability, and ease of deployment wise). Since then, VPS hosting services (such as SliceHost) have become the prevalent solution for hosting Rails apps. But with DreamHost officially supporting mod_rails, I figured I’d give this a spin to see how well it works in practice.

I am happy to say, it seems to work just as advertised! In order to test Rails on DreamHost, I downloaded the popular Mephisto Blog Application, unzipped it into a directory on my DreamHost account, and configured the database settings (I didn’t even bother with MySQL and opted for Sqlite3 for the purpose of this test). I then went into my domain’s settings on the DreamHost Web Panel, checked the “Ruby on Rails Passenger (mod_rails)” checkbox, and pointed to my Mephisto directory’s public subdirectory as the web directory for my domain (this is important, as the web directory defaults to yourdomain.com, without the /public that mod_rails expects).

A minute or two later, my changes had been applied and I was greeted by the Mephisto blog when I hit my domain in the browser. I configured my blog’s settings and entered some dummy articles, and found the performance to be very snappy — no different from PHP apps that I am hosting at DreamHost (such as this WordPress blog).

I think this is pretty exciting. Sure, there are many other cost-effective options to deploy Rails apps these days (such as the unique and highly promising Heroku or a cheap $20 VPS slice on SliceHost), but for a personal blog or another small, reasonably low-traffic website (such as the 12 or so random Rails apps all of us are concurrently working on and too cheap to spring for VPS hosting, since most of them will never go anywhere), having the option to easily deploy these on a shared hosting account is great.

Now I am hoping that mod_rails will be extended beyond just Rails to support any Rack compliant Ruby web framework, such as Merb or Sinatra.

Of course, the article’s comments attracted the usual, ignorant TechCrunch trolls. Most took the opportunity to pitch their framework of choice (such as PHP, Java, .NET, or Django), which they claimed would of course magically solve all of Twitter’s scalability issues.

I have to say I am appalled at this level of ignorance. People just don’t seem to realize that Twitter is a complex messaging application and that the front-end is only a relatively small aspect of it. Even if one particular front-end technology happens to be faster than another one (and admittedly Rails, as much as I like it, is not the fastest technology out there), this fact is bound to be negligible compared to the real challenges in scaling the back-end, starting with the database (trust me, like many developers I’ve learned this the hard way ). Even for a typical web application (which Twitter is not), there are many performance improvements than can be implemented at that level (such as leveraging database replicas to separate writes from reads, or utilizing Memcached to cache queries and other data), all of which can be applied equally well to any front-end framework.

I’m not saying that it does not make sense to consider other technologies (there might very well be a breaking point at which it makes sense to evaluate Java or even rewriting parts of the system in C/C++), but in my opinion this should be considered a cost-savings measure when the application reaches a scale at which the cost of hardware far outweighs any savings due to increased developer productivity (think Google), and not a magic bullet for solving fundamental scalability issues (performance != scalability!)

One of the real difficulties in scaling Twitter lies in the fact that all Twitter hits are completely personalized and need to return fresh data, making it difficult to fully leverage caching. Also, since Twitter is a social application and the returned data is generated by each user’s social graph, there is no straightforward way to shard the database by user, as one might be able to do in a typical e-commerce or enterprise application (or pretty much any non-social app…). Without knowing more about Twitter’s internal architecture and their actual profiling results, it would be foolish of me to make any concrete recommendations — particularly silly ones like “Use technology XYZ, it will magically solve all your problems!” Too bad many of the developers out there don’t seem to realize this…

There were probably about 20 other people (mainly Rails developers) at the event, and since the space was pretty small and we didn’t have a dedicated room, it was initially difficult to actually speak with David. But towards the end of the evening the group started getting smaller, which made it easier to participate in the conversation.

In terms of what David had to say, I did not catch too many noteworthy items that are not already known in Ruby on Rails circles. But a few things seem to be key to understanding both the history and the future of Rails:

1. David created Rails to solve his specific problems. It wasn’t meant to directly solve every possible problem out there, and if there are useful features missing from it that simply means that he (or presumably the rest of the core team, now that it has become a larger community project) has not needed that particular functionality, not that it would not be useful.

2. Rails is (as we all know) a very opinionated framework. It makes a lot of reasonable assumptions about various things (such as the names of “id” columns), which enables developers to accomplish a lot with very little code. However, all of these opinions are purely about the internals of the system and therefore relevant for developers, but never affect the end user in any way. David is strongly against including any features in Rails that would result in any default application flows or otherwise affect the actual behavior of the application. That’s why Rails still does not (and probably will never) include a built-in authentication system.

Otherwise, he seems pretty indifferent about a lot of the details about how people are working with Rails. For example, Ruby and Ruby on Rails’ performance was good enough for him several years ago, so this area has never been too much of a concern of his. He also doesn’t feel strongly about using Test::Unit vs. RSpec. In his opinion, Test::Unit is good enough, and actually easier to understand for someone who does not have much experience with testing, whereas RSpec probably has more of a learning curve (and he did not seem to excited about the particular syntax). He felt that the most important thing that BDD and RSpec brought to the table is the “should” keyword, and that changing Rails to allow this in test method names made a big enough difference in readability and helped enforce the convention of using a single assertion for each test case. (I actually was not aware that you can use “should” in Rails test names; I’ll have to look into this.)

I probably missed a bunch of other nuggets of wisdom because the room was a bit noisy and crowded, but I’m sure that all of these have been posted on various blogs and mentioned in various keynote speeches before.

Anyway, it was nice to meet David in person. He seems like a great guy, and I definitely appreciate that he took the time to meet with a small fraction of his user base tonight. It’s refreshing to work with technologies that are owned and driven by real-world people like this, rather than large corporations that have lost touch with their user base.

First off, Google is an amazing company. Especially for a company of this size (and impact), it is highly impressive that they have managed to maintain this kind of work environment, company culture, and integrity. On the most basic level, there are all the perks, from great health benefits to free food (there are about 15 cafeterias on the Mountain View campus alone, many of which offer breakfast, lunch, and dinner), micro-kitchens with free snacks (including fresh fruit or fresh cut carrots and celery), on-campus gyms and a beach volleyball field, and more. Another great thing are the many tech talks (usually there are at least a couple every day), which feature both internal and external speakers. There are also many opportunities for training. In the one year that I have worked at Google, I had the opportunity to take a one-day class on Agile Estimating and Planning, as well as an excellent four-day workshop on Design Patterns and Refactoring, and numerous Google and product specific classes. Then there are the various off-sites and team building events, which probably take up another good week each year. Not many companies send their employees to Disneyland for three days… And of course there’s Google itself, with its various products that have become so ubiquitous and synonymous with the web that life would be difficult to imagine without them. Anybody remember the web before Google Search came along? Let me give you a hint: it sucked! But even newer products like Gmail, Google Reader, or Calendar have caught on quickly and become established as best-of-breed applications, and I definitely felt proud to work at the company that has built all of these applications. And speaking of work, the actual work environment is also nice in many ways: Every engineer gets either two 24″ or one 30″ monitor, as well as a company laptop (either MacBook Pro or Thinkpad). Depending on which project you work on, you might get to work with innovative internal tools and frameworks (such as BigTable), and I have definitely developed an entirely different perspective on scale, which humbles any project I’ve worked on before. Then there’s the community. In general, Googlers are a great bunch, and very smart. There are internal mailing lists on pretty much any subject, and you can pretty much guarantee that you will be able to get a solid answer to whatever question you might have (work or non-work related). In fact, this reminded me quite a bit of Usenet, back when it was still popular and usable, and not totally overrun with idiots. Of course, keeping up with all of this can become quite a time sink as well… Google is highly engineering-driven, and engineers enjoy a lot of trust and power, which is a very different and refreshing experience from working at a more product-driven company.

So if everything is so great at Google, why am I stupid enough to leave? And part of me does indeed feel a bit guilty about not being able to fully appreciate and enjoy working at Google — after all, there are many people out there in the world that have dreary and monotonous minimum-wage jobs, without any benefits or perks to speak of. But in the end, I have realized that I am just much more of a startup person than a big-company person. Perks and everything are great, but this is ultimately not what motivates me. At an early stage startup, every single individual has a tremendous impact on the company (good or bad…), along with a much broader set of responsibilities (everybody has to wear many hats). Then, there’s the pioneering spirit, which is extremely energizing and contagious. These days, it seems like a lot of the true innovations are made at small startups, which have the benefit of being orders of magnitude times more agile and efficient than a large company will ever be. Sure, many ideas don’t go anywhere, but every once in a while, something new comes along that leaves a big footprint (and let’s not forget that even Google started out like this). Last not least, there is of course a significantly bigger upside to working at a startup. Of course the harsh truth is that most startups fail, but at least there is that 1 in a 10 chance of being tremendously successful (and the sense of actually being able to contribute to this chance). As a recent Google employee, I would have never gotten rich there, even if the stock had doubled or tripled in price.

There are a few other Google-specific problems I should mention as well. For one thing, it is unlikely to initially be able to work in an area that one is passionate about. Many of the Google products are exciting, but unfortunately I was unable to be passionate about my particular product area. That is not to say that there weren’t any interesting aspects about it, and I do have a lot of respect for the team I worked with. Overall this is less of a problem later, as it is generally encouraged to switch projects every 1-2 years, but this first year makes a big difference, particularly for experienced engineers that have a good understanding of what kind of things they enjoy working on (or perhaps more importantly, don’t enjoy working on) or what kind of environments are a good match. I feel that the hiring process should be improved to better take this into consideration, although this is admittedly a difficult logistical problem at Google’s scale. Another scale-related problem: Due to the sheer size of the code base and the vast number of Google-specific tools and frameworks, it also takes a very long time to learn how to actually become productive at Google, which can be frustrating at times.

But overall, I feel privileged that I had the chance to work at Google. I’m sure they will still be around for a long time, and 20 years from now I will be able to tell my grandchildren “Oh, Google, yes, I worked there once…”. I will certainly miss the Google campus, which had the vibrant feel of a University campus. I will also miss many things about the Google culture, and hopefully be able to take many of these inspirations with me into my future career.

Which brings me to my new job, which I am enormously excited about starting this Monday. I can’t say too much, as the company is just getting founded and still in stealth mode, but I am going to be a co-founder of a brand-new startup in the Social Networking / Mobile space, two areas I am very interested in. I have worked at small startups, but so far I’ve never had the opportunity to come in on the ground floor like this, so this should be a great adventure. My role is going to be that of Director of Engineering, but I expect to be wearing a lot of hats, ranging from architecture and implementation to being involved in the product direction, taking care of hosting and other IT stuff, and ultimately building a great team and helping define the company culture (although we will initially be a small core team). We will be working with some exciting technologies, including Ruby and a sprinkling of Ruby on Rails, which I am strongly looking forward to as well. Besides using a bit of Ruby at my previous job, I have mostly played with Ruby in my spare time, and I am glad about being able to finally use it again at my day job as well (though we will be pragmatic and use whatever tool makes most sense for the job at hand). I think I am officially done with Java at this point (but never say never… I am pretty sure I will at least use it again as a platform at some point in the future).

Well, this blog post turned out a bit longer than expected. But don’t worry, I am sure I will be busy enough at my startup that I won’t have a chance to bug you with any additional long posts for a while.

Evernote is a next generation note taking service. What sets it apart from similar applications are the numerous ways of entering as well as retrieving notes. First of all there is the Evernote Website. It is simple and straightforward, and seems to be very responsive. You can create as many notebooks as you want. Each notebook acts like a folder that contains individual notes as files, and can be viewed using different perspectives (such as thumbnail view or list view), much like a native file browser application like Finder or Windows Explorer.

Notes can be entered and edited on the website, but there are several alternatives. For example, you can email a note to a special email address that is generated for you when you sign up. You can also use this mechanism to send notes from your mobile phone, using your provider’s SMS-Email gateway. However, I have been running into some technical issues when I attempted this from my mobile phone (I’m getting the following error message: “Your MSG could not be DELIVERED because InvalidPduContent”), so I need to look into this. Curiously, sending a picture via MMS worked just fine. Speaking of pictures: Evernote’s support for images is pretty amazing. It uses advanced text recognition to extract text from images, such that it becomes searchable. I have tested this with a low quality photo of a poster snapped from my mobile phone, and it worked as advertised. Very impressive!

You can also use a nifty bookmarklet that allows you to either submit an entire webpage to Evernote, or just the text that you have selected.

But perhaps most importantly, Evernote has a downloadable client app for both Windows and Mac. I have only tested the Mac client, which was released very recently. It seems to lag slightly behind its Windows counterpart in terms of features, but on the positive side it looks and feels like a proper Mac application rather than a quick port. It relies on a Sync feature to synchronize notes between the server and the client (either manually or in a configurable interval). Otherwise, the client offers pretty much the same functionality as the website. In addition, it comes with a clipping service that shows up in the menu bar (and registers some convenient keyboard shortcuts) and allows you to easily submit any copied text to the client, or even to clip a screenshot.

Evernote also has many ways to browse notes. It has a search box that allows you to find notes using full text search, but also using other criteria. For example, notes can be tagged (much like emails in Gmail), after which you can search for them by tag (or just click on the tag in the navigation bar). Notes can also be located via various attributes, such as creation or modification date, source (website, email, mobile, etc.), or whether or not they contain images or audio. This works from both the website or the native client. Evernote also has a mobile website, which I have not tested. This should be convenient when you need to look up a note while you’re on the road.

Overall, I am pretty impressed with Evernote. I have played with various note taking applications, and this comes very close to perfect in terms of features. I often have ideas that I need to capture, and right now I’m using a personal wiki for this purpose. However, I would prefer to work in a native application when possible, since this is generally more convenient. The combination of native client and website for Evernote is quite powerful, with a well-implemented synchronization mechanism. I just wish there were some wiki-like features, such as easy linking between individual notes. But the most important drawback is the lack of formatting options. The website only supports straight text entry, with no formatting at all. The Mac client supports rudimentary formatting (such as font, color, bold, italic, underline, alignment), but unfortunately this formatting appears to be lost when the note is uploaded to the website. The thumbnail view does show the properly formatted note, but the full view does not. And any edits on the website completely reset any formatting that might have been applied on the client. I don’t require any sophisticated formatting, but at the minimum I would need support bullet lists (which neither the Mac client nor the website supports), headings, and emphasis.

As it is right now, Evernote seems like a somewhat useful scratch pad to collect short notes, web clippings, etc. Particularly the mobile features might come in handy. With a bit of additional work, I think it could be extended to become a more comprehensive solution for organizing information, but right now I am going to continue using my wiki for this purpose.

If you would like to see it in action, there is a short Evernote Screencast.