Updated on August 20th, 2015 – This blog post was originally written to use the Twitter public API. That API has since changed to not allow unauthenticated queries. We’ve updated this post to utilize the GitHub API instead which still allows a minimal amount of unauthenticated queries (60 an hour.) The linked jsFiddles have been updated to reflect this but you may encounter the rate limiting rather fast. Watch your console for errors indicating so.

A Backbone collection can be filled in several ways:

  1. By individually fetching or creating models and then adding them to the collection. In this situation, the collection is not responsible for any interaction with a server.
  2. By fetching a group of models using a single request made by the collection object itself. In this situation we define a url property on the collection and fetch it when we’re ready for the data.

When your collection is actually responsible for communicating with an API endpoint to fetch its models, there are several types of data that you might be interested in fetching. For example:

  1. You might have a finite set of data that you need to fetch once for your application.
  2. You might have a feed of data that is constantly updating and thus your collection periodically check for updates.

Dealing with the first scenario is quite simple. It would require defining a Collection constructor that has a url property. Calling fetch on an instance of that collection would then fetch the data and that would be sufficient.

Dealing with the second scenario is somewhat more complex for several reasons:

  1. The end point needs to be queried periodically rather than just once.
  2. The data coming back may already be in your collection (because not enough new data was generated on the server and thus the subset returned is mostly identical.)
  3. The existing collection may need to be ammended or added to.
  4. Some UI component may want to update based on new data being retrieved, but only if there is new data.

How would one go about creating this system?

Let’s create a system that watches a specific GitHub query and updates a list of repos that match it.

You can search GitHub for anything using the following end point:

https://api.github.com/search/repositories?q=language:assembly&sort=stars&order=desc

The results of your query will look as follows:

github_response.json

{
  total_count: 261,
  incomplete_results: false,
  items: [
    {
    id: 21095601,
    name: "Tetris-Duel",
    full_name: "Tetris-Duel-Team/Tetris-Duel",
    owner: {
    login: "Tetris-Duel-Team",
    id: 7956696,
    avatar_url: "https://avatars.githubusercontent.com/u/7956696?v=3",
    gravatar_id: "",
    url: "https://api.github.com/users/Tetris-Duel-Team",
    ...
    git_url: "git://github.com/Tetris-Duel-Team/Tetris-Duel.git",
    ssh_url: "git@github.com:Tetris-Duel-Team/Tetris-Duel.git",
    clone_url: "https://github.com/Tetris-Duel-Team/Tetris-Duel.git",
    svn_url: "https://github.com/Tetris-Duel-Team/Tetris-Duel",
    homepage: "",
    size: 11373,
    stargazers_count: 46,
    watchers_count: 46,
    language: "Assembly",
    has_issues: true,
    has_downloads: true,
    has_wiki: true,
    has_pages: false,
    forks_count: 6,
    mirror_url: null,
    open_issues_count: 0,
    forks: 6,
    open_issues: 0,
    watchers: 46,
    default_branch: "master",
    score: 19.616371
  }, {...}]
}

First, let’s write a Backbone Model & View that will represent a single repo:

base_repo.js

// A container for a repo object.
var Repo = Backbone.Model.extend({});

// A basic view rendering a single repo
var RepoView = Backbone.View.extend({
    tagName: "li",
    className: "repo",

    render: function() {

      // just render the repo name and description as the content of this element.
      $(this.el).html(
          '<b>' + this.model.get("name") + "</b> - " +
          this.model.get("description")
      );

      return this;
    }
});

Now, let’s create our actual collection and a quick view to render it

base_collection.js

// A collection holding many repo objects.
// also responsible for performing the
// search that fetches them.
var Repos = Backbone.Collection.extend({
    model: Repo,
    initialize: function(models, options) {
        this.language = options.language;
    },
    url: function() {
        return "https://api.github.com/search/repositories?q=language:" +
               this.language + "&sort=stars&order=desc";
    },
    parse: function(data) {

        // note that the original result contains repos inside of an items array, not at
        // the root of the response.
        return data.items;
    }
});

// A rendering of a collection of repos.
var ReposView = Backbone.View.extend({
    tagName: "ul",
    className: "repos",
    render: function() {

        // for each repo, create a view and prepend it to the list.
        this.collection.each(function(repo) {
            var repoView = new RepoView({
                model: repo
            });
            $(this.el).prepend(repoView.render().el);
        }, this);

        return this;
    }
});

We can now instantiate a Repos collection for a particular query like so:

base_assembly_repos.js

// Create a new assembly language repo collection
var assemblyRepos = new Repos([], {
    language: "assembly"
});

// create a view that will contain our repos
var assemblyReposView = new ReposView({
    collection: assemblyRepos
});

// on a successful fetch, update the collection.
assemblyRepos.fetch({
    success: function(collection) {
        $('#example_content').html(assemblyReposView.render().el);
    }
});

You can see it in this fiddle: https://jsfiddle.net/iros/kfvpcn5x/ .

Making it live

While this works, note that if we want to treat this end point as a live feed, we need to call fetch repeatedly. Instead of the above method, we can chose to do this instead:

live_repo_feed.js

// Create a new assembly language repo collection
var assemblyRepos = new Repos([], {
    language: "assembly"
});

// create a view that will contain our repos
var assemblyReposView = new ReposView({
    collection: assemblyRepos
});

// We now render this view regardless of the fact it still
// hasn't been fetched. This is because we want to bind to
// the collection and be ready to create the repo views
// as they come in.
$('#example_content').html(catReposView.render().el);

var updateRepos = function() {
    assemblyReposView.fetch({
        add: true
    });
    setTimeout(updateRepos, 1000);
};
updateRepos();

Note that we also removed the success callback. How would we actually paint new repos then? Well, conveniently every time a model is added to a collection an “add” event is fired. This allows us to subscribe to that add event instead of rendering an entire collection of models. Let’s rewrite our repos collection view accordingly:

live_repos_view.js

// A rendering of a collection of repos.
var ReposView = Backbone.View.extend({
    tagName: "ul",
    className: "repos",
    initialize: function(options) {
        // Bind on initialization rather than rendering. This might seem
        // counter-intuitive because we are effectively "rendering" this
        // view by creating other views. The reason we are doing this here
        // is because we only want to bind to "add" once, but effectively we should
        // be able to call render multiple times without subscribing to "add" more
        // than once.
        this.collection.bind("add", function(model) {

            var repoView = new RepoView({
                model: model
            });

            $(this.el).prepend(repoView.render().el);
        }, this);
    },
    render: function() {
        return this;
    }
});

You can see it in this fiddle https://jsfiddle.net/iros/Pg2aU/

Don’t forget that by adding new models to your collection, it will continue to grow in size. You may want to remove items as new ones are added, but that’s outside the scope of this article.

Reusable StreamCollection

One quick way to use this “streaming” pattern is to extend the default Backbone.Collection to allow for streaming.

stream_collection.js

// Create a StreamCollection
var StreamCollection = Backbone.Collection.extend({
  stream: function(options) {

    // Cancel any potential previous stream
    this.unstream();

    var _update = _.bind(function() {
      this.fetch(options);
      this._intervalFetch = window.setTimeout(_update, options.interval || 1000);
    }, this);

    _update();
  },

  unstream: function() {
    window.clearTimeout(this._intervalFetch);
    delete this._intervalFetch;
  },

  isStreaming : function() {
    return _.isUndefined(this._intervalFetch);
  }
});

This allows us to start the stream by simply calling:

assemblyRepos.stream({
  interval: 2000,
  add: true
});

You can see it in this fiddle hhttps://jsfiddle.net/iros/VuJVa/

The Dreaded Duplicates

Now if you let the above fiddle run for a few cycles, you might have noticed there’s quite a bit of repo duplication. This is an unfortunate side-effect of the way adding models is implemented in backbone. You would think given the nature of relational databases that a “duplicate model” would be identified by using an id attribute (since most of the time, there’s a uniqueness constraint on your data.) However, the way Backbone checks whether a model already exists in a collection is by using its cid as follows:

backbone.collection._add.snippet.js

// From Backbone.Collection's definition:
_add : function(model, options) {
  options || (options = {});
  model = this._prepareModel(model, options);
  if (!model) return false;
  var already = this.getByCid(model);
  if (already) throw new Error(["Can't add the same model to a set twice", already.id]);

Given that your models come in from the server, there is no reason why they would have a duplicate cid (seeing as on creation of a new model a unique cid is assigned to it and it’s never persisted unless you intentionally save it.)

There are two ways to work around this problem:

Overwrite the model’s cid to match its id. This will ensure duplicates are detected.

In our example this would look as follows:

cid_overwrite.js

// A container for a repo object.
var Repo = Backbone.Model.extend({
  initialize: function(attributes, options) {
    this.cid = this.id;
  }
});

While this solution works, it will still result in an error being thrown every time a duplicate model is encountered. You can always surrounded in a try/catch block where appropriate in your application. You can see that in the following fiddle if you open your console.

You can see it in this fiddle: https://jsfiddle.net/iros/4pLUR/

Alternatively, you could overwrite your collection’s add method to check for a duplicate before actually adding the model in question and only adding it if it doesn’t already exist.

In our example that would look like so:

add_duplication_check.js

// A collection holding many repo objects.
// also responsible for performing the
// search that fetches them.
var Repos = Backbone.Collection.extend({
    model: Repo,
    initialize: function(models, options) {
        this.language = options.language;
    },
    url: function() {
        return "https://api.github.com/search/repositories?q=language:" +
               this.language + "&sort=stars&order=desc";
    },
    parse: function(data) {

        // note that the original result contains repos inside of an items array, not at
        // the root of the response.
        return data.items;
    },
    add: function(models, options) {
        var newModels = [];
        _.each(models, function(model) {
            if (typeof this.get(model.id) === "undefined") {
                newModels.push(model);
            }
        }, this);
        return Backbone.Collection.prototype.add.call(this, newModels, options);
    }
});

Again, not the most elegant solution but at least we got rid of the thrown errors. You can see it in this fiddle: https://jsfiddle.net/iros/GVaWe/

Ideally, this wouldn’t be an issue and duplicates would be detected by id as well as the cid. This was in fact once fixed in the Backbone source but then subsequently reverted in this commit.

How do we solve this then??

What we’re proposing is adding a new options property to the Collection.prototype.add method:

{ unique : true }

This will guarantee that only unique models (by cid and id as a fallback) will be added to the collection if you so desire. There’s a pull request awaiting feedback and your love, if you care to dispense it: https://github.com/documentcloud/backbone/pull/808

What do you think?

Much thanks to Tim Branyen and Adam Sontag for their feedback.

Comments