Rails Metal: a micro-framework with the power of Rails: \m/

Updates:

  • Clarified the distinction between Rails Metal and Rack middleware after more information from @Josh in the comments. Thanks!
  • Read more about metal from DHH on the Official Rails Blog.
  • Demonstrate Testing Metal end points
  • Update Poller example to match new style
  • Cover Sinatra Integration
  • Correct benchmarks

Josh Peek committed a new feature to Edge Rails today: Rails Metal. After the recent work to replace Rails’ crufty request processing code with Rack and integrate its middleware support, Rails Metal is a logical progression that allows Rails apps to use the power of Rack middleware to create super-fast actions.

For example, here’s a sample “Hello World” Metal:

  class Poller < Rails::Rack::Metal
    def call(env)
      if env["PATH_INFO"] =~ /^\/poller/
        [200, {"Content-Type" => "text/html"}, "Hello, World!"]
      else
        [404, {"Content-Type" => "text/html"}, "Not Found"]
      end
    end
  end

And for comparison, a “Hello World” controller:

    class OldPollerController < ApplicationController
      def poller
        render :text => "Hello World!"
      end
    end

So, let’s fire up ruby script/server and see what this gives us:

  # traditional Controller
  $ curl 127.0.0.1:3000/old_poller/poller
    Hello World!

  # the new Metal
  $ curl 127.0.0.1:3000/poller
    Hello World!

So, the point of all of these other “micro-frameworks” is that they’re faster than Rails, right? Let’s benchmark this new “Hello World” Metal:

  # first, let's benchmark the traditional controller
  $ ab -n 1000 http://127.0.0.1:3000/old_poller/poller
  ... snip ...
  Requests per second:    408.45 [#/sec] (mean)
  Time per request:       2.448 [ms] (mean)

  # now for the Metal middleware
  $ ab -n 1000 http://127.0.0.1:3000/poller
  ... snip ...
  Requests per second:    1154.66 [#/sec] (mean)
  Time per request:       0.866 [ms] (mean)

For this trivial “Hello World” benchmark, Rails Metal is 2.8x faster than a Controller. Awesome. Have a couple actions of your app you need to optimize? Instead of breaking them out into a separate application using a micro-framework, add a Metal inside your existing app. You get the performance benefits of processing requests outside of ActionPack, and it’s all integrated as a part of your Rails app. Easy!

Sinatra Metal

You can now also use Sinatra to create Metal end points:

  Sinatra::Application.default_options.merge!(:run => false, :env => 
  :production)
  Api = Sinatra.application unless defined? Api

  get '/interesting/new/ideas' do
    'Hello Sinatra!'
  end

First person to show the use of a Merb app as a Metal end point wins a prize.

Standalone Execution

Additionaly, Rails Metal are able to be executed in a separate process from your Rails application using rackup:

  rackup -s mongrel app/metal/poller.rb

This runs the Poller Metal separeately from Rails, on it’s own port (rackup defaults to 9292). This is perfect if you have an action that’s taking a very long time (for example a file upload) that you’d like to split out from the normal Rails request processing queue.

Testing Metal

Update: After several people commented asking how to test metal, DHH chimed in and recommend Integration Testing for Metal end points, as they hit the whole stack, and I submitted a patch cleaning up the Integration Testing behavior of Metal. Testing Metal end points now works just like any other Integration test:

      class PollerTest < ActionController::IntegrationTest
        test "poller returns hello world" do
          get "/poller"
          assert_response 200
          assert_response :success
          assert_response :ok
          assert_equal "Hello World!", response.body
        end
      end

Fun With Middleware

So, essentially, Rails Metal is a thin wrapper around Rails’ new Rack middleware support. Rack middleware is pretty powerful stuff: framework-independent components that process requests independently or in concert with other middleware. For example, here’s a simple piece of Rack middleware that runs a regex on responses:

class RegexMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, response = @app.call(env)
    new_response = []
    response.each do |part|
      new_response << part.gsub(/World/, 'Middleware')
    end
    [status, headers, new_response]
  end
end

To use this rack middleware in Rails, add this line to your environment.rb

Rails::Initializer.run do |config|
  ...
  config.middleware.use RegexMiddleware
end 

Restart your server, and check out what happens:

  $ curl 127.0.0.1:3000/poller
    Hello Middleware!

The Rack middleware filtered the output of the Metal we created before. This works with output generated by normal controllers and everything too. The possible uses of this pattern are endless:

  • Single Sign On
  • Request/Response Signing (think OAuth)
  • Asset Compression

rack-contrib is a nice collection of Rack middleware if you’re interested in more examples.

Rails Metal is a simple wrapper around the existing (yet undocumented) Rack middleware support in Edge Rails that attempts to DRY the process of using middleware to create endpoints (like a poller) as opposed to filters (which are better implemented as traditional middleware, like the examples above). For example, Rails Metal might be used:

  • To speed up a ‘poller’ action called by all active users of a popular web-based chat application every 3 seconds (hint: Campfire).
  • To improve the performance of any API endpoint
  • To process file uploads outside of the Rails request queue
  • To authorize delivery of cached content

We don’t need no stinking micro-frameworks

With the additional of Metal and Rack middleware support, Rails effectively includes a micro-framework of its own; one that either tighly integrates with Rails or is executed separately – whichever the need dictates.

This is a great response by the Rails team to all of the buzz surrouding micro-frameworks: a micro-framework with the power of Rails. I’m definitely going to try this approach to squeeze a couple extra requests per second out of a heavily trafficked API call – let me know in the comments if you find a use for it.

Read up a bit more on Rack and then take a look at Josh’s commit Introducing Rails Metal (and the ensuing comments) if you’re interested for more information.

Fire Eagle: Location-Aware Applications Without the Hassle

Tom Coates said it best yesterday morning at ETech: people have been touting ‘location-aware services’ as the next big thing for years. However, they’ve never taken off.

What’s been holding them back: acquiring reliable location data about users is a hard problem for developers to solve.

With yesterday’s release of Fire Eagle, that problem is now a whole lot easier to solve.

Ride the Fire Eagle Danger Day!

So what is Fire Eagle? It’s not Twitter for location, that’s for damn sure. Here’s how the Yahoo! copywriting wizards describe it:

The secure and stylish way to share your location with sites and services online while giving you unprecedented control over your data and privacy. We’re here to make the whole web respond to your location and help you to discover more about the world around you.

At the diagram to the left shows, the Fire Eagle platform acts as an broker for your location data. One or many applications can set your location, and, provided you give them access, any other service can access this data.

This is one giant piece in the puzzle for location-based services. Users set their location in one place, and any number of other services are able to then act on this data however they please.

The other piece in the puzzle: a Fire Eagle updater that requires absolutely NO user interaction. If I’m carrying around my iPhone in my pocket all day, why can’t it tell Fire Eagle where I am?

Of course, Erica Sadun has already whipped up an unofficial iPhone app to ping Fire Eagle called firefindme. Installation isn’t the easiest thing in the world – it assumes some launchd skillz to setup automatic updates. However, I’m sure a user friendly iPhone updater is coming very shortly ;).

Developing Location-Aware Applications, Sites and Services with Fire Eagle

I’ve got a full-on tutorial coming detailing how to make your Rails app talk to Fire Eagle, but in the meantime, check out my Fire Eagle Ruby Gem:

sudo gem install fireeagle

If Ruby’s not your bag, don’t worry – there are libraries for working with Fire Eagle in javascript, php, perl and python.

Proof-of-Concept Twitter Bot

Just like last time, I’ve created a proof-of-concept twitter bot for testing out Fire Eagle: firebot.

First off: you need an invite to talk to Fire Eagle right now. Luckily, firebot is handing out a few. follow firebot on Twitter, and then direct message it with ‘invite’

  • d firebot invite

Once you have an invite, direct message firebot with ‘auth’:

  • d firebot auth

firebot will then reply with a link. You’ll need to visit that link, authenticate with your Yahoo! account, and then authorize firebot with Fire Eagle.

Once that’s done, you can update your location with a direct message to firebot like so:

  • d firebot u Atlanta, GA
  • d firebot u Belize
  • d firebot u 30022
  • d firebot u 123 Anytown USA
  • etc

To look up the location of someone else using firebot:

  • d firebot q jnewland
  • d firebot q cjmartin
  • d firebot q plasticbagUK

Disclaimer

By telling firebot your location, you agree to share your location information with all other users of firebot. All direct messages you send to firebot are stored permanently at Twitter. If at any point you’d like all of your information deleted from firebot, please contact @jnewland.

What’s next?

Get hackin’ on your awesome location-based web app! Extra bonus points if you use the Fire Eagle Rubygem. If you’ve got a great idea for a Fire Eagle app and don’t have an invite, just ask firebot for one!

PS: If you hack up a Fire Eagle javascript sidebar widget that works on pages served as application/xml (preferably using the brilliant wedje technique) AND embraces the draft geo microformat, I’ll buy you a pony. Seriously.

resource_this - DRY Rails Resource Controllers

I’ve always been annoyed at the lack of maintainability that comes with using multiple resource controllers in my Rails apps. Each generated resource controller clocks in at 85 lines, and most of mine only differ from each other by a line or two – an added before_filter or a change in the url that the users is redirected to after the creation of a new Widget. Not very DRY. When coming back to each one of these controllers to add or adjust features, it takes me entirely too much time to sift through the stock 85 lines and find my application-specific behavior.

Enter resource_this

git clone git://github.com/jnewland/resource_this.git

resource_this aims to solve this maintainability problem by making your stock resource controllers look like this:

  class PostsController < ActionController::Base
   resource_this
  end

Behind the scenes, this code is generated:

  class PostsController < ActionController::Base
    before_filter :load_post, :only => [ :show, :edit, :update, :destroy ]
    before_filter :load_posts, :only => [ :index ]
    before_filter :new_post, :only => [ :new ]
    before_filter :create_post, :only => [ :create ]
    before_filter :update_post, :only => [ :update ]
    before_filter :destroy_post, :only => [ :destroy ]

  protected
    def load_post
      @post = Post.find(params[:id])
    end

    def new_post
      @post = Post.new
    end

    def create_post
      @post = Post.new(params[:post])
      @created = @post.save
    end

    def update_post
      @updated = @post.update_attributes(params[:post])
    end

    def destroy_post
      @post = @post.destroy
    end

    def load_posts
      @posts = Post.find(:all)
    end

  public
    def index
      respond_to do |format|
        format.html
        format.xml  { render :xml => @posts }
        format.js
      end
    end

    def show          
      respond_to do |format|
        format.html
        format.xml  { render :xml => @post }
        format.js
      end
    end

    def new          
      respond_to do |format|
        format.html { render :action => :edit }
        format.xml  { render :xml => @post }
        format.js
      end
    end

    def create
      respond_to do |format|
        if @created
          flash[:notice] = 'Post was successfully created.'
          format.html { redirect_to @post }
          format.xml  { render :xml => @post, :status => :created, :location => @post }
          format.js
        else
          format.html { render :action => :new }
          format.xml  { render :xml => @post.errors, :status => :unprocessable_entity }
          format.js
        end
      end
    end 

    def edit
      respond_to do |format|
        format.html
        format.js
      end
    end

    def update
      respond_to do |format|
        if @updated
          flash[:notice] = 'Post was successfully updated.'
          format.html { redirect_to @post }
          format.xml  { head :ok }
          format.js
        else
          format.html { render :action => :edit }
          format.xml  { render :xml => @post.errors, :status => :unprocessable_entity }
          format.js
        end
      end
    end

    def destroy          
      respond_to do |format|
        format.html { redirect_to :action => posts_url }
        format.xml  { head :ok }
        format.js
      end
    end
  end

Nested resources like so:

  class CommentsController < ActionController::Base
    resource_this :nested => [:posts]
  end

This generates a very similar controller to the one above with adjusted redirects and one additional before_filter / loader method pair to grab the parent resource. In this case:

  before_filter :load_post

  def load_post
    @post = Post.find(params[:post_id])
  end

The separation of logic – DB operations in before_filters, rendering in the standard resource controller methods – makes this approach ridiculously easy to customize. Need to load an additional object for the :show action? Slap another before_filter on it. Need to change the path that the :update action redirects to? Override the :update action with your new rendering behavior. And this customized behavior sticks out like a sore thumb – making it infinitely easier to maintain.

Oh, there’s also a generator:

./script/generate resource_this FooKlass [title:string body:text]

This works just like the resource generator, with the addition of the resource_this line to your controller and a functional test. No views are generated, so the test focuses on the XML behavior of this controller.

Contributing

resource_this is hosted on GitHub, so feel free to fork it and send a pull request with your changes.

Fire Eagle, meet Danger Day

UPDATE: These instructions are out of date. See here for instructions that work with the new Fire Eagle!

A couple days ago, a friend of mine sent me an invite for Fire Eagle, Yahoo! Research Berkley’s nifty closed-Alpha location storage and query engine, and I’ve been hooked ever since. For the rest of you without access, here’s a brief overview of what FireEagle does, straight from the FAQ page:

Fire Eagle is a site that keeps track of your current location and helps you share it with other sites and services safely. There are hundreds of potential applications.

Fire Eagle allows you to share your locations with other sites and services safely, through a secure server – you are always in control. You can decide to share your location with any application that can use it, and even choose how much detail to give that application (exact point, neighborhood, city, state, country).

So, I whipped up a quick Fire Eagle Rubygem to make it easier to deal with Fire Eagle’s API. The next logical step? A twitter bot.

Fire Eagle, meet Danger Day

If you’re lucky enough to have an invite to Fire Eagle, here’s how you can use it on Twitter:

  1. Follow Danger Day on Twitter
  2. Sign in to Fire Eagle
  3. Authorize Danger Day with your FireEagle account
  4. Get a mobile token to confirm your authentication with Danger Day
  5. Send a direct message to Danger Day with your token.

Once that’s done, you can update your location with a direct message to Danger Day like so:

  • u Atlanta, GA
  • u Belize
  • u 30022
  • u 123 Anytown USA
  • etc

If you’d like to do this via your mobile phone, make sure your mobile is setup with Twitter, then send the following text massage to 40404, Twitter’s short code:

  • d dangerday u Atlanta, GA

To look up the location of someone else using Danger Day:

  • q jnewland
  • q cjmartin
  • q plasticbagUK

or from your mobile:

  • d dangerday q jnewland

What’s next?

I’m getting married in a week, so I leave the creation of cooler Fire Eagle apps as an exercise to the reader. Extra bonus points if you use the Fire Eagle Rubygem. If you’ve got a great idea for a Fire Eagle app and don’t have an invite, get in touch with me – I might be able to make that happen.

PS: If you hack up a Fire Eagle javascript sidebar widget that works on pages served as application/xml (preferably using the brilliant wedje technique) AND embraces the draft geo microformat, I’ll buy you a pony. Seriously. Here’s my location in XML – go to town.

RailsConf '07 Roundup

David Heinemeier Hansson, by James Duncan Davidson. Creative Commons BY-NC-ND

I spent the past several days in Portland, OR, for RailsConf, the yearly gathering of the vibrant Ruby on Rails community. O’Reilly Media and Ruby Central put on an incredible conference. My only disappointment was that I couldn’t attend all of the presentations. Luckily, most of the presentation slides are online (some with accompanying code!!):

But by far, the most valuable part of the whole event was the time I spent in the hallways and around Portland with other Rails developers. In the two years I’ve been working with Rails, I’ve networked and collaborated with dozens if not hundreds of Rails developers online. It was great to finally be able to associate faces and voices with their respective names, blogs, and chat handles.

I also wrangled Erik Kastner and Charles Brian Quinn into the Capazon project while in Portland – look for some updates on that front in the near future.

A special thanks to the following folks for making my RailsConf an especially great time:

Joyent Slingshot Demo Notes

Eric Wagoner gave a demo of the Joyent’s newly released offline Rails application toolkit, Slinghot. He’s had access to Slingshot for a couple weeks ahead of the public release, and shared his early impressions with us tonight. These are rough notes from his presentation.

  • A local Slingshot app is distributed as a DMG on OS X, which is a full-stack Ruby VM plus your Rails apps. Lots of files. I mean, lots.: $ find Radiant.app | wc -l -> 8087
  • Server side, Slingshot uses a plugin that generates a ‘sync’ controller. This controller has several actions:
    • sync.up
    • sync.down
    • sync.log
  • These toss XML back and forth between the server and client app, containing the changed Models and some metadata.
  • Synchronization conflicts are handled in your application’s domain. This is important to note – Slingshot is not a silver bullet for the age old offline/online synchronization problem. However, it lets you solve this problem however your application needs.

MyConfPlan Javascript Widget

I’ve hacked up a simple Javascript widget for everyone out there using myconfplan, Dr Nic Williams’ wonderful conference session planning tool. Nic recently added JSON and XML feeds to myconfplan, so this is a natural progression.

The XHTML generated by this widget is in hCalendar format, with separate events for each session. If you’re using a Microformats aware browser (or the wonderful microformats bookmarklet), you’ll be able to add my RailsConf sessions to your local calendaring app.

To display your conference session selections on your blog or website, insert these two lines of code:

<script src="http://files.jnewland.com/display_myconfplan.js" type="text/javascript"></script>
<script src="JSON_URL?callback=display_myconfplan" type="text/javascript"></script>

Replace JSON_URL with the URL to the JSON feed for one of your conferences. For example, to display my RailsConf 2007 sessions in my sidebar, I used this:

<script src="http://files.jnewland.com/display_myconfplan.js" type="text/javascript"></script>
<script src="http://myconfplan.com/users/jnewland/conferences/RailsConf2007.js?callback=display_myconfplan" type="text/javascript"></script>

Now seems like a good time to mention that I’ll be at RailsConf 2007 later this month in Portland, OR, and I couldn’t be more excited about it. You can view my session selections in the sidebar, on on myconfplan. Also, I’m staying at the Jupiter Hotel, which looks like a blast. Let me know if you’ll be there too!

Capazon 0.2 Released - Capistrano 2.0 Compatible

Capazon 0.2.0 is out. There’s only one new feature: support for Capistrano 2.0. There’s no backwards compatibility. If you’re still on Capistrano 1.4.x, please don’t upgrade. It won’t work.

To update Capazon:

  • gem install capazon

Changes

Capistrano 2.0 has support for Rake-like namespaces, so I’ve moved all tasks provided by Capazon to the ec2 namespace:

$ cap ec2:describe_images
  * executing `ec2:describe_images'
IMAGE   ami-0386636a    rbuilder-online/nuxleus-1.3-x86_9327.img.manifest.xml  099034111737     available       true
IMAGE   ami-0683666f    rbuilder-online/fedoracore6-1.0-x86_9677.img.manifest.xml       099034111737    available       true
[...]

To call these tasks from another namespace in a Capistrano recipe:

namespace :whatever do
  task :something_cool do
    [...]
    ec2.describe_images
    [...]
  end
end

Capistrano 2.0

Turns out updating extensions to work w/ Capistrano 2.0 is extremely easy. Just replace blocks like this:

Capistrano.configuration(:must_exist).load do
  task :take_over_the _world do
    [...]
  end
end

...with this:

Capistrano::Configuration.instance.load do do
  task :take_over_the _world do
    [...]
  end
end

For more on upgrading your recipes to Capistrano 2, head over to the upgrade guide on Capistrano’s new website or this post on NubyOnRails. Happy capifying!

Capazon - Capistrano Meets Amazon EC2

UPDATE: For those looking for Capistrano 2.0 support, check out Capazon 0.2.0

Just a quick note to announce Capazon 0.1.0, a Capistrano extension library to manage Amazon EC2 instances. If you are familiar with Capistrano and have an Amazon EC2 account, give it a whirl:

  • gem install capazon
  • Edit your your config/deploy.rb:
require 'capazon'

#AWS login info
set :aws_access_key_id, 'XXX'
set :aws_secret_access_key, 'X'

# Name of the keypair used to spawn and connect to the Amazon EC2 Instance
# Defaults to one created by the setup_keypair task
set :aws_keypair_name, "#{application}-capazon"

# Path to the private key for the Amazon EC2 Instance mentioned above
# Detaults to one created by setup_keypair task
set :aws_private_key_path, "#{Dir.pwd}/#{aws_keypair_name}-key"

#defaults to an ubuntu image
#set :aws_ami_id, "ami-e4b6538d"

#defaults to, um, default
#set :aws_security_group, "default"
  • $ cap describe_images
  * executing task describe_images
IMAGE   ami-0386636a    rbuilder-online/nuxleus-1.3-x86_9327.img.manifest.xml   099034111737    available       true
IMAGE   ami-08866361    rbuilder-online/test1-1.0-x86_9326.img.manifest.xml     099034111737    available       true
IMAGE   ami-1281647b    rbuilder-online/mw-tour-1.6.8-x86_9458.img.manifest.xml 099034111737    available       true
IMAGE   ami-1681647f    rbuilder-online/mw-tour-1.6.8-x86_9459.img.manifest.xml 099034111737    available       true
  • $ AWS_AMI_ID=XXXX cap run_instance

This release just scratches the surface of what I hope to accomplish with Capazon – my end goal is to provide a shared AMI as a companion to Capazon which will encapsulate some Rails deployment best practices.

Please report any bugs you may come across, and stay tuned for updates!

Tweet - Update Twitter via Quicksilver

Tweet, a Twitter Quicksilver Action, is my favorite Twitter updating vehicle as of late – except for one thing. It uses Keychain Scripting to access your saved Twitterrific username and password – which, for some ungodly reason, takes about 30 seconds on my Mac Book Pro.

I’ve made some updates to Tweet.scpt to fix the slowness:

  • Now uses the Twitter Rubygem to talk to Twitter
  • Uses the username and password stored in ~/.twitter, via the Twitter Gem
  • Uses Twitterrific’s icon, if you have it installed.

Requirements

Installation Directions

  1. Download Tweet.scpt
  2. Move Tweet.scpt to ~/Library/Application Support/Quicksilver/Actions
  3. Restart Quicksilver (Cmd+Ctrl+Q)
  4. Setup a Trigger – I use Cmd+Opt+Ctrl+T :

This, combined with Twitter Monitor, is the way I’m using Twitter most often these days. Like it or not, I think Twitter has a lot of potential. I’d love to throw some AI at the trends of public twitters – you’d see the string “coffee” grow in popularity in the AM and wane in the PM. Add Geocoding support – a way to set your ‘Location’ as well as your status – and Twitter becomes useful in another dimension. I’m interested to see what Twitter has in it’s future.

Older posts: 1 2