Saturday, August 27, 2011

Emailing with Delayed Job

We used Delayed Job to queue emails sent out to users, both to offload that blocking process and for scheduling. It has worked well so far, but recently there were some strange bugs popping up. Some emails were stuck in the queue, and the error message was about bad YAML syntax.

Delayed Job serialized objects in its handler field, and, with some user input that's not encoded properly, created incorrect YAML. For example, this could happen:
id: 1
  foo: 'here is some 'text'
  bar: 'something else'
Notice the odd number of single quotes in foo? Yeah, that's bad. Since we already had that kind of data saved, we needed another way to fix this.

Instead of having methods in the notifier.rb file like so:
def forgot_password(user)
  ...
end
We did it like this:
def forgot_password(user_id)
  user = User.find(user_id)
  ...
end
Delayed Job serialized just the user id and not the entire user object, so any potentially harmful data wasn't saved. This was more expensive since the objects had to get instantiated again, but sending out email wasn't expensive for our app, so this solution worked well.

If you ever get strange YAML syntax errors from delayed job, perhaps this method will work for you.

Friday, August 12, 2011

Attaching Events to a Disabled Submit Button

There was a form that had a few required fields, and I wanted to show a message when the user hovered over the submit button when not every field was completed. The problem was that the submit button was disabled until the fields are filled in, and I couldn't attach an event to a disabled form element.

One solution is to add an invisible element over the button.
var $disabledSubmit = $('#submit_wrapper input:disabled');
var $disabledSubmitParent = $('#submit_wrapper');
var $overlay = $('<div />');
$overlay.css({
  position: 'absolute',
  top: $disabledSubmit.position().top,
  left: $disabledSubmit.position().left,
  width: $disabledSubmit.outerWidth(),
  height: $disabledSubmit.outerHeight(),
  zIndex: 10,
  opacity: 0
});
$overlay.mouseover(this.submitHoverOver);
$overlay.mouseout(this.submitHoverOut);
$disabledSubmitParent.append($overlay);
This created an overlay over the button that handled the hiding and showing (submitHoverOver()/submitHoverOut()) of the message.

When the form was ready to submit and the button was enabled, we needed to do two things. The first was to lower the z-index of the overlay so the user can access the button.
$overlay.css('z-index', -1);
The second was to unbind the events on the overlay so we didn't continue to show the message.
$overlay.unbind();
If the user changed the data to be in a bad state, we disabled the submit button. We also needed to reattach the events and crank up the z-index of the overlay.
$overlay.css('z-index', 10);
$overlay.mouseover(this.submitHoverOver);
$overlay.mouseover(this.submitHoverOut);
Now, instead of putting up an error message or more text about required fields, the user would be directed to finish the form if he hadn't done so when he tried to submit.

Saturday, August 6, 2011

My name is Danny...and I make mistakes.

Yes, it's true. Here's what I did, and here's the reaction.



At work a few weeks ago, I was going through some callback code that sets some meta data on a model's associations. I noticed that a related flag wasn't getting set as well, that it was only set in one other specific instance with the meta data. Hmm...let's fix that, shall we? Flag added, moving on.

Star wipe to this week.

We found a bug when displaying historical data, and I quickly realized that the display was wrong because it was skipping over objects it shouldn't, objects that were flagged when they shouldn't be. Cue pants pooping.

There was a fix, and I would just need to run a script that would update the flag for all the associated models affected after the callback happens and ignore the other ones because those were the ones that explicitly have the flag set at the other, correct, time. But should I tell anyone or do I just run the script and say that I fixed the display bug? Well, WWJD (what would Jack Nicholson do)? He'd tell everyone, damn it, because he's like that. Keeping it real. Not like Chuck Norris.



Anyway, I sent out an email admitting what happened, and I included a high-level explanation of what happened along with a technical one. I explained that there is a fix and we won't lose any data, and that it would fix the current display bug but that we need to do some tests to make sure it didn't affect any other parts of the application.

My manager's response? "Hey, man, shit happens. Glad you fixed it." That's why I like working here.

Wednesday, July 13, 2011

Adding Field Separation for List Data

A Big List


MetalDetectr is effectively a list of data as specific as a user wishes to see. It will show only a list of releases a user has in his last.fm library to a list of everything on metal-archives.com. A big concern is presenting it properly, and one method is to delineate releases by whatever sort method a user wants to see. This can be by release date, by the band's name, by the release's name, or by the release's format (eg, EP, full-length, DVD).

The Algorithm


  • Start with a table row showing the earliest or most recent, depending on sort order, of the selected sort column.
  • Loop through the releases.
  • If the current release's relevant field is greater/less than the preceding one, show another table row with the current release's field value.
  • Show the release.

For example, the default sort is by US release date starting at the earliest date (and the current month so there's less noise). The list will display the current month and every album released during that month. When a release is next month, it will show next month and then all releases from that month. Continue on through the rest of the releases. If the user wants to see the list in descending order, it will show the last month first and work its way to the most current month.

The Code


First find the first value and display it in a full column span table row:
# views/releases/index,html.haml
- comparison_value = @releases.first.chain_methods(Release::FIELDS_WITH_METHODS[Release.default_sort(params[:s])])
= separator_row(comparison_value)
These two lines use the following methods:
# models/release.rb
# Sets the sort order to what's passed or us_date.
def self.default_sort(sort)
  sort || 'us_date'
end

# models/release.rb
# Takes an array of symbols and calls them on the release instance if it
# responds to them.
# Example: release.chain_methods([:us_date, :month]) => release.us_date.month
def chain_methods(methods)
  methods.inject(nil) do |memo, acc|
    target = memo ? memo : self
    target.respond_to?(acc) ? target.send(acc) : memo
  end
end

# helpers/releases_helper.rb
# Creates a row with a full colspan for the value.
def separator_row(value)
  value = Date::MONTHNAMES[value] if value.is_a?(Fixnum)
  content_tag(:tr, :class => cycle('even', 'odd')) do
    content_tag(:td, value, :class => 'separator_row', :colspan => 7)
  end
end
FIELDS_WITH_METHODS is a constant that contains a mapping of field names and methods to call on them to display properly:
# models/release.rb
FIELDS_WITH_METHODS = {
  'band' => [:band, :first, :downcase],
  'name' => [:name, :first, :downcase],
  'us_date' => [:us_date, :month],
  'euro_date' => [:euro_date, :month],
  'format' => [:format],
  nil => [:us_date, :month]
}
Then we loop through each release, updating the comparison value when we get to the next one:
# views/releases/index,html.haml
- @releases.each do |release|
  - current_value = release.chain_methods(Release::FIELDS_WITH_METHODS[Release.default_sort(params[:s])])
  - if Release.values_compared?(current_value, comparison_value, params[:d])
    - comparison_value = current_value
    = separator_row(comparison_value)
  - else
    - comparison_value = current_value
  = render release
Compare the two values based on the sort order:
# models/release.rb
# Sets the comparison operator to be greater than if the direction is nil or ascending,
# or less than if the direction is descending.
def self.comparison_operator(direction)
  (direction.nil? || direction == 'asc') ? :> : :<
end

# models/release.rb
# True if both value and comparison exist and
# if the direction is ascending:
#   true if value > comparison, false otherwise
# if the direction is descending:
#   true if value < comparison, false otherwise
def self.values_compared?(value, comparison, direction)
  value &&
  comparison &&
  value.send(
    Release.comparison_operator(direction),
    comparison
  )
end
We tried to abstract the comparisons and what's displayed so we can add new fields and only need to update the field-method mapping. There is always the possibility that a field is nil, since we don't always get all the possible data for every release, so chain_methods will call all the methods it can on a release instance until it finishes or returns nil. We could have chained a bunch of try()s together, but that didn't look right.

We also tried to get as much code out of the view as we could, and it can be improved, but it's okay for now.

Friday, July 8, 2011

Namespacing /lib Files and RSpec

I've been in an ongoing battle with RSpec to get it to properly load files in the /lib directory of a rails app. There's a class MetalArchivesFetcher wrapped in a MetalDetectr module as a namespace:

module MetalDetectr
  class MetalArchivesFetcher
    ...
  end
end

The spec file starts like this:

require 'spec_helper'
require 'metal_archives_fetcher'

describe MetalDetectr::MetalArchivesFetcher do
  ...
end

Without the require, I would receive the message, load_missing_constant': Expected /Users/danny/code/metaldetectr/lib/metal_archives_fetcher.rb to define MetalArchivesFetcher (LoadError) It felt a little off to need to require the file again because Rails already loads it in with config.autoload_paths += Dir["#{config.root}/lib/**/"] set in the config/application.rb file. I could put the require in spec_helper.rb, but it still felt strange.

I decided to remove the module namespace. That lets me remove the require line and all preceding MetalDetectr:: for every MetalArchivesFetcher call in the spec. Is this the right decision? It's definitely DRYer, but I do create tighter coupling. Jim Weirich's talk, "The Building Blocks of Modularity" (that I can't find online) does go over the trade-offs of writing code that is either more tightly or loosely coupled, and my takeaway from that is, since this file is already coupled to the application and models within it, why add an additional layer? It's more of a perceived loosening while only adding a bit more complexity. And that's usually not a good thing.

Perhaps I'll add it back in later, but for now, I'm not going to need it.

Sunday, June 26, 2011

A Real Life Github Success Story

Github has been a real treasure for developers, and I've used it both at work and for personal projects. Until now, I haven't used it to its full effect, that is, contributing.

For MetalDetectr, I wanted to allow a user to filter the list to see releases from artists he had in his last.fm library. A quick search led me to this gem, only it wasn't as fully-featured as I needed.

So I forked it.

Github made this really easy to do. Soon I had the repository in my account, cloned it locally, checked out a new branch, and I was working.

The code was clean and certainly made my life easier to get what I wanted. There was a /method_categories folder that contained the methods to do API calls to get or create information for artists, tracks, and users. I wanted to read in a user's library of artists, so I simply modeled this after the other files.

class Lastfm
  module MethodCategory
    class Library < Base
      regular_method :get_artists, [:user], [[:limit, nil], [:page, nil]] do |response|
        response.xml['artists']['artist']
      end
    end
  end
end
This created a get request call for a last.fm user, set an optional limit on the number of fetched results, and set an optional page number to scan to. Along with the API key, these fields are outlined in the last.fm api docs.

Testing worked similarly. A spec file contained the other method tests, so adding the following, plus a fixture of the xml response, was super easy.

  describe '#library' do
    it 'should return an instance of Lastfm::Library' do
      @lastfm.library.should be_an_instance_of(Lastfm::MethodCategory::Library)
    end

    describe '#get_artists' do
      it 'should get the artists\' info' do
        @lastfm.should_receive(:request).with('library.getArtists', {
          :user => 'test',
          :limit => nil,
          :page => nil
        }).and_return(make_response('library_get_artists'))
        artists = @lastfm.library.get_artists('test')
        artists[1]['name'].should eql('Dark Castle')
        artists.size.should == 2
      end
    end
  end
After adding these methods, I pushed the branch to my github repository and sent a pull request to the original repository. Again, github makes this trivially easy. Before it was accepted, I had this line in the Metaldetectr Gemfile:
gem 'lastfm', :git => 'git://github.com/dbolson/ruby-lastfm.git', :branch => 'library_get_artists'
With the pull request accepted and my code merged into the master branch, it looked like this:
gem 'lastfm'

That's all it took to contribute to open source software.

Wednesday, June 15, 2011

Metal Archives' JSON Results Parsing

Some further explanation of how to get Metal Archives' JSON data from a recent post is necessary. Through reading the markup and trial-and-error, I found the URL to receive the data I needed. Here it is:

http://www.metal-archives.com/search/ajax-advanced/searching/albums \
/?&releaseYearFrom=2011&releaseMonthFrom=1&releaseYearTo=2011 \
&releaseMonthTo=12&_=1&sEcho=0&iColumns=4&sColumns=&iDisplayStart=1& \
iDisplayLength=100&sNames=%2C%2C%2C

This returns a result set that looks like this:

{ 
 "error": "",
 "iTotalRecords": 3637,
 "iTotalDisplayRecords": 3637,
 "sEcho": 0,
 "aaData": [
  [ 
    "<a href=\"http://www.metal-archives.com/bands/037/3540277845\" title=\"037 (ES)\">037</a>",
    "<a href=\"http://www.metal-archives.com/albums/037/Los_Fuertes_Sobreviven/307703\">Los Fuertes Sobreviven</a>",
    "Full-length", 
    "May 24th, 2011 <!-- 2011-05-24 -->"  
 ],
  [ 
    "<a href=\"http://www.metal-archives.com/bands/037/3540277845\" title=\"037 (ES)\">037</a>",
    "<a href=\"http://www.metal-archives.com/albums/037/Tantas_Vidas/306172\">Tantas Vidas</a>",
    "Single", 
    "May 6th, 2011 <!-- 2011-05-06 -->"  
 ]

You'll notice the iTotalRecords field which conveniently provides the total amount to releases available. You'll also notice the the iDisplayStart parameter in the URL that lets us step through the results 100 at a time. By looping through (iTotalRecords / 100 + 1) times, incrementing iDispalyStart by i * 100, we can get a result set for all the records very quickly.

Now that we have the results, we just need a little regular expression magic to pull out all the information.

BAND_NAME_AND_COUNTRY_REGEXP = /(.+)\s{1}\(([a-zA-Z]{2})\)/
ALBUM_URL_AND_NAME_REGEXP = /"(.+)">(.+)<\/a>/
RELEASE_DATE_REGEXP = /<!--\s(.{10})\s-->/

There was a strange situation where an album didn't have a band page but displayed a message that the band didn't exist, so there is one last regular expression used to guard against a slightly alternative format for the data:

NO_BAND_REGEXP = /span.+<\/span/

The data are much easier to gather and never time-out now, so I was able to get rid of all the intermediate saving steps such as after gathering the paginated links and saving the last release searched when the site times-out. I'll probably have to add it back in to get the record label of the release since you'll notice it's absent in the JSON but it is available on the release's page.

The code to save the albums now looks like this:

agent = ::MetalArchives::Agent.new
agent.paginated_albums.each_with_index do |album_page, index|
  album_page.each do |album|
    if album[0].match(::MetalArchives::Agent::NO_BAND_REGEXP).nil?
      Release.create(
        :name => agent.album_name(album),
        :band => agent.band_name(album),
        :format => agent.release_type(album),
        :url => agent.album_url(album),
        :country => agent.country(album),
        :us_date => agent.release_date(album)
      )
    end
    CompletedStep.find_or_create_by_step(CompletedStep::ReleasesCollected)
  end
end

Quick and simple.