Rails And JSON Containing Unicode Characters

As I mentioned in a previous blog post, Rails 2.1 natively supports incoming JSON requests. Unfortunately, it still struggles with JSON data containing non-ASCII characters.

According to the JSON spec, JSON fully supports UTF-8 encoded text, so with a few exceptions it generally should not be necessary to escape non-ASCII characters with \u Unicode escape sequences. However, many JSON libraries appear to escape all non-ASCII text in this fashion. This in itself should not be a problem, but ActiveSupport::JSON currently does not properly parse JSON containing \u escapes, resulting in strings with literal \u escape sequences rather than the desired UTF-8 encoded characters. This is especially confusing since ActiveSupport:JSON itself encodes all non-ASCII characters as \u escapes, so one might think that the reverse transformation yields the original data. But this behavior is likely explained by an odd implementation choice for its decoder: Rather than using the json (or json-pure) library, it converts the JSON data to YAML and then uses the YAML library to decode the data into Ruby objects.

Monkey-patching to the rescue! I decided to replace ActiveSupport::JSON::decode with an implementation that uses the json library. The easiest way is to stick the following code into a file named something like activesupport_json_unicode_patch.rb inside the config/initializers/ directory, where Rails will automatically pick it up.

require 'json'
 
module ActiveSupport
  module JSON
    def self.decode(json)
      ::JSON.parse(json)
    end
  end
end

You can verify the fix by adding a test case (I added a file named activesupport_json_test.rb to the test/unit/ directory):

require File.dirname(__FILE__) + '/../test_helper'
 
class ActiveSupportJsonTest < Test::Unit::TestCase
 
  def test_json_encoding
    unicode_escaped_json = '{"foo":"G\u00fcnter","bar":"El\u00e8ne"}'
    hash = ActiveSupport::JSON.decode(unicode_escaped_json)
    assert_equal({'foo' => 'Günter', 'bar' => 'Elène'}, hash)
  end
 
end

This test should fail without the patch and pass after adding it.

In addition to fixing the JSON / Unicode problem, this patch should also provide a nice speed boost, as we’re replacing the somewhat roundabout YAML based JSON decode method with a native one (particularly if you’re using the native json implementation rather than json-pure.)

This entry was posted on Wednesday, August 27th, 2008 at 11:17 pm and is filed under Development. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

9 Responses to “Rails And JSON Containing Unicode Characters”

glenn says:

8/28/2008 at 4:07 am

Nicely done. I get the feeling there is still a lot most of us (myself especially) have to learn about doing l18n properly in Rails 2.1
DigitalHobbit says:

8/28/2008 at 9:23 am

I’m definitely still learning how to do i18n with Ruby / Rails as I go along.

As a recovering Java developer, I have to admit that Java absolutely nailed i18n, pretty much right from the beginning. Strings are always proper Unicode strings, as opposed to Ruby’s (at least pre-1.9) fancy byte arrays. There’s built-in support to decode text encoded in all the common encodings into Java Strings, solid support for resource bundles and locale awareness, and more.

Oh well, I’m still having more fun with Ruby… (and it looks like both the Rails community as well as Ruby itself are committed to improving the current i18n situation)
Darren says:

11/2/2008 at 3:51 am

I had a similar problem with the flickraw library. Internally it used YAML to parse JSON which completely failed on Unicode.

In my case I modified the library to use JSON. Will email the author once I’m happy it’s not created any new problems.
mcm says:

12/23/2008 at 9:54 am

Nice – but the required json gem overrides the to_json methods added by ActiveSupport. So if your are calling to_json (with AS defined options) on your rails objects requiring the gem will break your app. Any solutions or alternatives for that?

See also http://groups.google.com/group/rubyonrails-core/browse_thread/thread/54e5453eaac6687b
mcm says:

12/23/2008 at 10:40 am

I found another solution which seems to work. Instead of using the json gem (see my comment above). I’m using a unicode unescape method found at http://d.hatena.ne.jp/cesar/20070401/p1 and override the decode function like that:

module ActiveSupport module JSON class << self alias orig_decode decode def decode(json) orig_decode(MyUnicode.unescape(json)) end end end end

(Note: Renamed the ‘Unicode’ module from the link above to ‘MyUnicode’ – just to prevent any naming troubles)
DigitalHobbit says:

12/23/2008 at 10:59 am

Good catch! I’m not passing any options to to_json right now, so I’m safe for now. Still, this is unfortunate.

Glad you found a workaround! Hopefully this (or a similar) Unicode fix will make it into Rails soon. It seems like they’ve had a lot of i18n related features lately, so proper Unicode support seems like a very basic requirement…
moelee says:

4/21/2009 at 3:17 pm

Looks like this monkey patch breaks the ability of ActiveResource to parse timestamped attributes (i.e. created_at, updated_at) into Time objects. A fix for this is much needed in the next version of Rails
DigitalHobbit says:

4/21/2009 at 11:13 pm

Thanks for the heads-up. You’re right, this definitely needs to be fixed in Rails. I thought I saw something about improved Unicode / JSON support for Rails in a recent commit, but I might be wrong…
lobo_tuerto says:

6/3/2009 at 7:27 am

Arghh! they fixed the decoding of Unicode chars, but to a certain extent…

It has some issues still, here is the problem illustrated, check this out: http://pastie.org/497986

Rails And JSON Containing Unicode Characters

9 Responses to “Rails And JSON Containing Unicode Characters”

Leave a Reply