Tuesday, April 3, 2012

Building Single Page Apps with ASP.NET MVC4 - Part 4 - Sorting, Filtering and Manipulating data with upshot.js

This blogpost is part of a series about building Single Page Applications using ASP.NET MVC 4 beta. When writing this post I assumed you have read the previous ones. The code snippets are just the incremental changes that are needed since the third. post I'd advise to just download the completed source code directly

  1. Single Page Applications - Part 1 - Basic Desktop application
  2. Single Page Applications - Part 2 - Advanced Desktop Application
  3. Single Page Applications - Part 3 - Basic Mobile Application
  4. Single Page Applications - Part 4 - Sorting, Filtering and Manipulation data with upshot.js
  5. Single Page Applications - Part 5 - With MVC4 RC

This post was more or less triggered by this comment on my first post in this series. James wanted to avoid creating GET functions for each single case. I already replied that he could take advantage of the OData features of WebApi but I think a dedicated post is needed since this is a topic where upshot is showing its limitations.

Updating to the latest packages

First thing I did -starting from the code from my previous post- was to upgrade all packages to the latest version. I had to rename the existing upshot.knockout.extensions.js file before updating, since I had modified it and the package manager would skip it instead of updating it. Since I'm working with C#, I used this command in the package manager console
Install-Package SinglePageApplication.CSharp

This will upgrade some dependencies and add new files such as T4 templates to the project. You will need to update some of the links to javascript files because for example knockout-2.0.0.js is upgraded and renamed to simply knockout.js. I also created a namespace 'deliveryTracker' and put my DeliveriesViewModel in it.

After the package update the UpshotContext HtmlHelper function is now available. So in my desktop and mobile views I can now replace the Javascript call to upshot.metadata with following code:

@(Html.UpshotContext(bufferChanges: true)
                  (x => x.GetDeliveriesForToday())

<script type="text/javascript">
    $(function () {
        var model = new deliveryTracker.DeliveriesViewModel();

This will generate javascript that calls upshot.metadata and also sets up a remote datasource called 'DeliveriesForToday. This means I can now remove the code that creates a new RemoteDataSource in the constructor of my DeliveryViewModel and get it from upshot.dataSources.DeliveriesForToday

/// <reference path="_references.js" />
(function (window, undefined) {
    var deliveryTracker = window["deliveryTracker"] = {};

    deliveryTracker.DeliveriesViewModel = function () {

        // ... snip

        self.dataSource = upshot.dataSources.DeliveriesForToday;

        // ... snip

    deliveryTracker.MobileDeliveriesViewModel = function () {
        //inherit from DeliveriesViewModel
        var self = this;

        // ... snip

    // ... snip ...

    window["deliveryTracker"] = deliveryTracker;

Note the calls to the ClientMapping functions which are used to indicate that I want to map the incoming data to the Customer and Delivery objects that I have created in javascript

Sorting and Filtering when fetching data

        // Initialize remote and local datasource
        self.dataSource = upshot.dataSources.DeliveriesForToday;
        self.dataSource.setFilter({ property: "IsDelivered", 
                                    operator: "==", value: false } ); //,
        self.dataSource.setSort({ property: "Customer/Name", descending: false });

The remoteDatasource is built on top of an OData service. The above request will generate this HTTP GET

GET http://localhost:61907/api/DataService/GetDeliveriesForToday?%24filter=IsDelivered+eq+false&%24orderby=Customer%2FName HTTP/1.1
X-Requested-With: XMLHttpRequest
Accept: application/json, text/javascript, */*; q=0.01
Referer: http://localhost:61907/
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
Host: localhost:61907
Connection: Keep-Alive

Note how you can sort and filter on a property of an associated entity by using a forward slash (...property: "Customer/Name"...) because this is supported by OData. The data also got filtered so the browser only received those deliveries where the IsDelivered property was set to false. Unfortunately upshot does not expose the full power of OData to the client application.

Sorting and Filtering the local data

Once data is loaded into the browser it can still be manipulated. In order to see the effect of a manipulation while data is being edited, I decided to set the 'allowRefreshWithEdits: true' property on the local datasource. This lets me sort or group the data when edits are still pending (not uploaded back to the server).
self.localDataSource = upshot.LocalDataSource({ source: self.dataSource, 
                        autoRefresh: true, allowRefreshWithEdits: true });

self.deliveries = self.localDataSource.getEntities();
self.deliveriesForCustomer = self.deliveries.groupBy("Customer");

In order to demonstrate sorting and grouping on the local data, I thought it would be fun to include the geolocation capabilities of the current browsers. If a user allows the application to read his current location, the application will sort the data by 'distance from user'.

I added latitude and longitude data to the my server-side Customer domain entity. This entity and the Delivery entity are both mapped to their javascript counterparts, like so:

deliveryTracker.Customer = function (data) {
 var self = this;

 self.CustomerId = ko.observable(data.CustomerId);
 self.Name = ko.observable(data.Name);
 self.Address = ko.observable(data.Address);
 self.Latitude = ko.observable(data.Latitude);
 self.Longitude = ko.observable(data.Longitude);

 //only exists client-side
 self.DistanceFromMe = ko.observable('unknown');

 upshot.addEntityProperties(self, "Customer:#DeliveryTracker.Models");

deliveryTracker.Delivery = function (data) {
 var self = this;

 self.DeliveryId = ko.observable(data.DeliveryId);
 self.CustomerId = ko.observable(data.CustomerId);
 self.Customer = ko.observable(data.Customer ? 
             new deliveryTracker.Customer(data.Customer) : null);
 self.Description = ko.observable(data.Description);
 self.IsDelivered = ko.observable(data.IsDelivered);

 upshot.addEntityProperties(self, "Delivery:#DeliveryTracker.Models");

It's important to note that upshot will map the Delivery object automatically but not the Customer object even when specifying it in the ClientMapping function on the UpshotContext. So whenever upshot tries to map the Delivery object, I create an observable Customer object myself. See the highlighted lines above

Now add this to the constructor of the DeliveriesViewModel:

var self = this;
var self.position
// Initialize geolocation
if (navigator.geolocation) {
 self.deliveries.subscribe(CalculateDistanceFromMe); //data is refreshed
 self.position.subscribe(CalculateDistanceFromMe);   //position is refreshed

 navigator.geolocation.getCurrentPosition(function (position) {

// Recalculate distance from deliverer when data or geolocation changes
function CalculateDistanceFromMe() {

 if (self.position() && self.deliveriesForCustomer 
            && self.deliveriesForCustomer().length > 0) {

  //calculate and update distances
                                           function (keyValuePair) {
   var customer = keyValuePair.key;
   var dist = geolocationUtils.calculateDistance(

  //sort local datasource on DistanceFromMe ascending
  self.localDataSource.setSort(function (delivery1, delivery2) {
   var value1 = delivery1.Customer().DistanceFromMe();
   var value2 = delivery2.Customer().DistanceFromMe();
   return (value1 - value2);
                self.localDataSource.refresh(); //trigger refresh with sorting

  //sort data grouped by customer
  self.deliveriesForCustomer().sort(function (left, right) {
   var value1 = left.key.DistanceFromMe();
   var value2 = right.key.DistanceFromMe();
   return (value1 - value2);

So what's happening:

  • self.position is a knockout (ko) observable that holds the current location.
  • CalculateDistanceFromMe is triggered when self.position or self.deliveries gets updated
  • calculateDistance is a function I found on HTML5 Rocks and wrapped in the 'geolocationUtils' namespace. It is called for each customer when the data gets refreshed
  • The calculated distance is stored in the client-side observable customer.DistanceFromMe property
  • Sorting on the localDataSource passes a sorting function to the setSort method on the dataSource.
  • Sorting on the deliveriesForCustomer passes a sorting function to the sort method off the knockout observable array (wrapper around javascript arrays)


Sorting is a tricky subject when mixing technologies like Microsoft is doing in this beta release of Single Page Applications. Different objects allow different features depending on which part of the abstraction leaks through

The remoteDataSource object is leaking OData conventions so when sorting or filtering you need to comply with the OData Orderby and Filter uri-conventions. For example you can easily sort on the Customer/Name property of an entityset of Delivery objects.

The localDataSource object however has its own implementation of sorting, filtering and grouping. Besides passing a sorting function, it also allows passing in a property name. So you can sort on the CustomerId of an entityset (of Delivery objects) but you cannot sort on Customer/Name

The GetEntities method on a localDataSource returns a Knockout observableArray. This object exposes a wrapper around the built-in javascript arrays and exposes the same functionality which again is different from the remote or local datasources.

What annoys me is that knockout and javascript both have a syntax for navigating associated entities, being the dot operator. If a remoteDataSource allows to sort by "Customer/Name", at least the localDataSource should expose the same interface. Also when binding data into the UI, you can easily do the following:

So navigating the associated entities using the dot operator works fine when data-binding but not when sorting, filtering, grouping, ...

As the Single Page Application are still in an early phase I suppose this is to be expected. In my opinion it's still way too soon to use upshot in its current form in any serious development.

Download the source code

You can download the code (Visual Studio 2010 project) from my Skydrive


James Hancock said...

This is a great series but everyone is just ignoring the most common element with SPA and Web API right now is reporting a list of results (i.e. select projections)

Please enlighten us in your next segment how you propose with SPAs to do a simple list of say invoices for a contact where it pulls across multiple tables with joins and returns a result set, and the client controls which columns will be pulled, not the method call itself given that Web API does not support OData Select and apparently won't for the foreseeable future even though the technology it's supposed to be replacing does support it.

And please don't do .Include on anything in the query or expect a separate method call for every permutation and combination of columns to be returned.

Include results in Select * as does all of the stuff you've currently shown us which is a good way to make a horribly performing application.

Until and unless this is fully supported, Web API is DOA as is SPA. (i.e. you need to add $select support to your odata implementation in WebAPI before V1 is released or this can't be used in production for any real world application.

Bart Jolling said...

Hi James, apparently the first release of WebApi indeed doesn't support $select for a variety of reasons but I guess you already know that :)

I do like the picture that you paint of allowing end-users to define exactly what they want to have returned to their dynamically generated screens but then upshot + WebApi is not the tool for the job. You should only consider it for situations where you can define the subset of the data already on server-side. For most applications with fixed screens, upshot's functionality is sufficient. For your kind of applications you are better of using WCF Data Services

James Hancock said...

how long until WCF Data Services is depreciated? I'd guess that it, like WCF Web API are dodos as soon as this is released just like Linq to SQL became legacy only with no new features as soon as EF was released even when EF wasn't capable of replacing Linq to SQL until V4+.

You say that most applications with fixed screens...

Except this is SPA! IT's supposed to replace Winforms, WPF and Silverlight apps. Have you noticed that every single 3rd party library for those environments has a column chooser and allows you to add columns dynamically? So any serious SPA is going to need to have this function because it's EXPECTED at this point.

Further if you use dynamic then you use odata, so now you can't do filter and order by from my understanding.

And if you make this hard coded to the Method on the server side, any change to that backend (to add or change a field or something) results in broken client code that is incredibly hard to find because of the late binding nature of jquery that is not compile-time error checkable.

Thus MS is creating a maintainence nightmare at the very least (DLL hell for the web!) and far worse for any serious SPA application that is doing LOB or other similar standard functionality.

So I'm left with the choice of using WCF Data Services which is a dead duck which I'm going to have to rewrite if I want any new functionality going forward once Web API finally supports full OData as it should out of the box, or I have to accept a noisy application that doesn't scale and is incredibly hard to maintain. (and we have no guidance on using dynamic as a response type with upshot and knockout so it's anybody's guess as to if it will even work or not)

Which would you choose?

Eran said...

How Can upshot.js work cross
url: "http://api.xxx.com/api/DataService"....."

anybody knows how to achieve this ?


Bart Jolling said...

Hi Eran,

I see some ways of doing it. If you look at my older posts you'll see I'm already manually providing a relative URL. You can just put a absolute URL instead.

Look at the constructor of the DeliveriesViewModel javascript object. This should work

providerParameters: { url: "http://api.xxx.com/api/DataService", operationName...}

Eran said...

Thanks , but the problem is jsonp support when working with upshot

Bart Jolling said...

Hi James,

As always you must choose the right tool for the job and Upshot clearly is not what you need.

I have not yet heard any rumors that upshot is going to replace Winforms, WPF, Silverlight. I'm also not aware that WebApi is replacing WCF Data Services. If have you have some links to offical documentation on this deprecation, I would greatly appreciate it if you could post the links here.

At the moment, upshot seems to be nothing more than an javascript helper library on top of WebApi. Its purpose appears to help getting your domain entities from server to client-side. This is indeed a limited use case. But if you need more functionality, you can use jQuery to talk to your WebApi services. And if WebApi is too limited for you, then just use WCF Data Services. I don't think that this is going away soon.

Bart Jolling said...

Hi Eran,

It seems a problem of jQuery not allowing cross site AJAX calls. You can solve it by adding:

jQuery.support.cors = true;

When testing my application by hosting the dataservice on a different port as the UI, the jQuery.Ajax call responded with a 'no transport' error. I found the solution here on StackOverflow