Expose

A Timeline Design For Your Blog

Yes! It's almost over. GSoC 2016 is in the final week of the journey. So in this post, I will briefly explain the work I've done along with a link to each deliverable.

Summary

For this project I have implemented several things. Listed down are the tasks that I did.
  1. JBrowse Comments Track
  2. Express back-end 
    • Host an instance of JBrowse
    • Github authentication strategy 
    • Implement route to write changes of comment track back to the data file
  3. Sails hooks to replace the functionality of the express back-end
Next I will describe the tasks and the outcomes of each of the tasks mentioned above very shorty.

JBrowse Comments Track

Following are the new technologies that were encountered while implementing the comments track.
  • dojo framework
  • dojo/xhr 
  • dojo/query
  • disqus comment platform/API
  • facebook comments
The most challenging part of this task was understanding the JBrowse code base, how the tracks, data, data stores and decorators are related, the flow of events, data structures used and how html elements are lazily rendered. 

I have added the changes to my local fork and created a pull request(#802) to the main repository. Before I was given a separate branch, I worked in the master branch of my local fork. Here are the links to the commits section of each branch.
To see the Comments Track in action, clone the main GMOD/jbrowse repository, checkout the jSocialize branch, setup JBrowse as instructed in the main Readme.md file, and then host JBrowse in any web server like apache or NodeJs. Shown below is an screenshot of the comments Track with sample data.


I have implemented my own version of ExpressJs based NodeJs web server for hosting JBrowse. More details can be found in the next section.

The main documentation can be found at the following wiki page.


Express Back-end 

The implementation of this expressJs based NodeJs server for hosting JBrowse can be found at following link.

Though we can host Jbrowse with this server, the primary requirement of this server is to provide the back-end API for writing comment track meta data back to the trackdata.json file. This is accomplished by implementing a RESTful POST endpoint. The Sails hook implementation which is explained in the next section was built to accomplish the same task since the actual server component of JBrowse is being written in Sails.js.

A secondary functionality of the express back-end server is providing an OAuth authentication strategy using passport-github2. Details on how to setup the server for this can be found in the main Readme file at the repository.

Technologies and learning materials that I came across while implementing this feature are

  • NodeJs
  • ExpressJs
  • EmbeddedJs
  • Passport / passport-github2
  • NCList data structure
  • ArrayRepr decorators 

Sails Hook

The sails hook was implemented to accomplish the task of writing comments track metadata back to the data source. Following links are related to this implementation.

The npm package can be installed to any sails core application by issuing npm i sails-hook-jbrowse command.


Future Works

Other than the above tasks which were implemented completely, I am in the process of implementing the following tasks. I am planning to finish these tasks in my free time after the GSoC program.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 Hey, this post is about a very interesting part of my project. That’s about the model details of the components associated with the comments track that I implemented. I will explain how these components are wired together and how they behave in my next blog post.
We can identify several components of this specific feature. They are as follows.
  1. CommentFeature Track
    1. Comment Box
    2. Click behavior
    3. Feature Context Menu
    4. Track Context Menu
  2. TrackData Model
  3. DISQUS Controller
  4. Express Backend
  5. Sails Hook
  6. Side Bar
Let me explain them one by one. We shall move to the first component then.
  1. CommentFeature Track
The ‘CommentFeature’ track is implemented based on the ‘HTMLFeatures’ Track. The reason for that is simple. ‘HTMLFeatures’ track has already implemented the basic requirement of the comments feature which is displaying box like html div elements on the window and then rendering them by lazily fetching required data when the user scrolls through. I implemented some customizations to extend that to CommentTrack.
The first thing that I added is a `Comment Box`. It is a html div element constructed by first sending data to the `_decorate_feature` function of the `NCList` store to decorate with `getter` and `setter` functions and then adding it to the respective block by sending to the `addFeatureToBlock` function. This div element has an attribute called `feature`, which holds all the information about the track. These data are defined in the `TrackData.json` file associated with the CommentTrack.
Then I added the click behavior. Clicking on a div element should reload the comments sidebar with the associated thread identifier. Additionally, if the div is marked as having unread comments, the div should be reverted back as read when clicked. This behavior is defined in the `_defaultConfig -> events -> click` function. 
events: {
   click: function(){
       var name = this.feature.get('name');
       reload(this.feature.get('thread_id'));
       query('#wrapper').removeClass('toggled');
       query('.sidebar-brand')[0].textContent = name;
       query('.feature.'+this.feature.get('thread_id')).removeClass('hasComments');
       this.feature.set('class_css','');
   }
}
Another option that I added is Feature Context Menu. The right click context menu items are defined under `_defaultConfig -> menuTemlate`. This accepts an array in which each array element should describe a menu option. In the CommentsTrack, each array element has three attributes called label, action and icon class. The label is the text showing in the context menu option. The action is what it does when selected. The icon class is a dijit icon and it is shown along the label. The action in each context menu option takes the form of sending a call to a method with the changed attributes of the feature along with a callback function. This callback function will update the feature associated with each comment box and redraw the track when changes happen to the feature. Here is a the code snippet for `UpdateRange` context menu option. 

{ 
    label: 'Update Range',
    action: function() {
        var start = prompt("Enter starting point",500);
        var end = prompt("Enter Ending point",1000);
        var thread = this;
        _this.updateComment({
            feature: this.feature,
            start: start,
            end: end,
            action:"update"
        },function() {
            thread.feature.set('start',start);
            thread.feature.set('end',end);
            thread.track.redraw();
        });
    },
    iconClass: 'dijitIconTask'
}
The next significant feature with regard to CommentFeature track is a `Track context menu`. Track context menu is similar to the Feature Context Menu except that actions defined under this menu are effective to the track as a whole, rather than for the individual features. In the first release of Comments track, three options are available. They are `Pin to top` option, `Delete Track` option and the `Start new thread` option. Shown below is the code snippet used to add a new option to the track context menu.

var o = this.inherited(arguments);
var track = this;
o.push.apply(o, [{
    type: 'dijit/MenuSeparator'
}, {
    label: 'Start New Thread',
    type: 'dijit/MenuItem',
    onClick: function () {
        var name = prompt("Enter title for thread");
        var start = prompt("Enter start position");
        var end = prompt("Enter End position");
        track.updateComment({
            start: start,
            end: end,
            name: name,
            action: "insert"
        }, function (res) {
            var thread_id = res.id;
            startThread('http://jbrowse.org/v2/' + thread_id, thread_id, name);
            track.newFeats.push([0, start, end, "ctgA", thread_id, name]);
            track.redraw();
        });
    }
}]);
2. TrackData Model
The next important component model that I used in CommentFeature track is called `Nested Containment List`, aka `NCList`. This is the default store associated with HTMLFeature track and therefore CommentsTrack inherits that. NCList provides mechanisms to allow nested elements for optimizing fast retrieval. However CommentFeature track does not make much use of the speed of retrieval factor in the current implementation.
NCList uses an array representation to store data and defines multiple classes to decorate these arrays later with attributes. I have defined a single class for the requirement of CommentTrack. Thus each comment array element metadata takes the following form,
[NCList Class, Start, End, Sequence Id, DISQUS Thread Identifier, Title,Class_css]
We have also added two custom attributes to the NCList model, `host` and `absCount`. The host is used to uniquely identify the host server of the JBrowse instance. The absCount is total number of created by the host in DISQUS. These two attributes are used when generating DISQUS Identifiers as postfixes to avoid identifier conflicts as described by DISQUS documentation. The rest of the attributes used have their general behavior.


3) DISQUS Controller
Talking about the next important component comes the DISQUS controller. All the DISQUS related functionalities are implemented in the controller located at `plugins/SocialFeatures/disqus_actions.js`. Following are the list of actions defined in the current implementation of comments track.
  1. disqus_config : Loads the initial configurations (Loads the identifier_1 thread by default)
  2. reload : Reloads the comment section displayed on the sidebar using the given configurations. The function uses thread identifier in the current implementation as the configuration parameter.
  3. startThread : Starts a new thread and reloads the DISQUS comments section. Uses the url, title and identifier parameters to send the call to DISQUS API.
The following code snippet shows how the startThread is implemented. I will not quote the rest here as all three functions have a similar structure.

/**
* Start a new thread and refresh disqus comment section
*/
function startThread(url,identifier,title){
  DISQUS.reset({
    reload: true,
    config: function () {  
      this.page.identifier = identifier;
      this.page.url = url;
      this.page.title = title;
    }
  });
}

 4) Express Backend
JBrowse can be hosted in any webserver like apache. However for the comment feature to work properly, that is, to make the changes permanent, we need to have a server component to write data back to the trackData.json file. The express server component implemented at https://github.com/pupudu/jBrowse-comment-server does that.
In the express backend, a single endpoint serves the update comment function. The rest endpoint accepts a post request with the old and new feature data. The generic File Stream module provided by NodeJs is used for reading and writing data to files in JSON format.

This code snippet shows the most important few lines of the endpoint which structures the data properly based on the request parameters before writing back to the file.

if(action === 'insert'){
    newId = host+dataObj.absCount;
    newThread[defaultClass.indexOf('thread_id')+1] = newId;
    list.splice(index,0,newThread);
    dataObj.featureCount++;
    dataObj.absCount++;
}
else if(action === "remove"){
    list.splice(index,1);
    dataObj.featureCount--;
}
else if(action === "update"){
    list[index] = newThread;
}
list = list.sort(Comparator);
4) Sails Hook
This component is still under construction. Details will be updated here when the implementation is finished.
5) Side Bar
Last but not least, the Sidebar can be described as the next important component. The side bar used to display DISQUS threads is a simple html structure. The styles and also the slide in and out behavior are based on CSS classes, by toggling classes to switch between the two states. This html div component is defined inside the main index.html file in which JBrowse is also instantiated.
I hope that this post would have clearly explained you about the model details of the components.

Requirement Analysis

After several discussions, my mentors and I decided to create a comments track on JBrowse to display thread links as comment boxes. Following facts were considered as the design considerations for the comments track. 

1. Specifying a region
Users should be able to highlight a region of their interest and then start a discussion specific to that region. Thus a box like element should be displayed on the window to identify the discussion. This box could additionally show the title of the associated thread. Clicking on this box should reveal the associated thread.

2. When there are multiple threads overlapping in a region, the comment boxes should not overlap and hence the boxes except one should be translated downwards to form a nested structure. 

3. User should be notified when new comments are available in a thread which is shown in the window. 

4. Users should be given the options to change attributes of the box, such as the associated range and thread title. The changes should be rendered on the window in real time without having to refresh the page again. 

Shown below is a screenshot of the comments track that we have developed as of now, considering the above design considerations. 



When it came to choosing the comment platform to be used for the JBrowse Comments Track feature, it was hard to choose between Facebook comments and DISQUS comments. After comparing the features, stability, flexibility, compatibilty and administrative efforts, DISQUS was the winner in the context of JBrowse requirement. In this post, I will briefly breakdown some facts I considered for choosing DISQUS and its features that were/will be used for the JBrowse Comments Track.

 1. Stability and Richness of Features
Since our candidate has been on the top in its field it has gone through enough to be sound and stable. Being live for a long time has resulted in Disqus to have almost all requirements that a user needs in the context of commenting. The concept of threads is supported by Disqus and it uses Urls and Identifiers to uniquely identify different threads. Basic requirements such as multi-level nested comment replies, photo comments, Login with different social media platform support and sorting are well built in Disqus. Another fancy feature seen by Disqus is that it notifies users who have the thread opened about new comments in that thread in real-time.

2. Administration
The admin panel of Disqus is clean and simple. The landing dashboard shows comment analytics as a summary.
When the site manage section is revealed, we can see that tasks such as Comment Moderation, Spam Filtering, URL Migration and Engagement analysis are just few clicks away from the menu. 


3. Disqus API
For granular level tasks such as editing posts, editing threads, getting statistics, getting notifications and thread deletion, Disqus exposes an API for web applications to communicate. In order to make use of this API, an administrator of an organization in Disqus has to create a Disqus application and get the app credentials. I have already created a Disqus app for Jbrowse and I also used this API while implementing the JBrowse Comments Track to notify a user about new comments on the Comments track itself.

This API and its endpoints could be mastered to provide more granular level features for the JBrowse Comments Track.

4. Compatibility to JBrowse
In JBrowse Comments Track, I am using a sidebar to display a thread. However, when the user switches a thread link, the sidebar should reload with the correct thread Url and Id. Unlike any other comment platforms, Disqus provides us with a function to reload the same html structure with a different thread, on just the cost of a click. This was an useful feature for JBrowse as the whole architecture of Disqus comments is based on this concept of reloading threads on comment dialog clicks.

JBrowse, according to its developers, is a fast, embeddable genome browser built completely with JavaScript and HTML5. You can download the source code and host it in any webserver. Often, apache2 is used to host JBrowse, as it is one of the most popular web servers among linux users. However for some of the functionalities of Jbrowse to work properly, JBrowse needs a server component. Therefore hosting Jbrowse in a NodeJs based web server can help to implement the backend logic required for such functions.

 There are many frameworks built on top of NodeJs to simplify the process of implementing a sound webserver easily. Two of the leading frameworks are ExpressJs and SailsJs. In this post, I will describe how to deploy JBrowse in an express based NodeJs server.  I will compare these two frameworks and describe how to lift JBrowse with SailsJs in a separate blog post. 

 Note - This process was tested in an Ubuntu 14.04 computer. You might need to do few changes if you are following this in a different operating system.

1. Creating an express application

If you are already familiar with expressJs and Nodejs, you can skip this section.

To create an express web app, first you need npm and NodeJs installed. Once you have installed these two, you can create a NodeJs application and then install express. However, you can also use the express-generator to make your life easier. To install express-generater, issue the following command,
$ npm install express-generator -g

Then to create an app with ejs as the view engine and "jBrowseExpr" as the name, issue the following command. For this tutorial and to host JBrowse, you don't need any knowledge on ejs. 
$ express jBrowseExpr --ejs

This will create a directory named jBrowse and when you go inside, you will see that express-generator has created a basic skeleton for the application. However, before you start your app, you should install the node dependencies. You can do so by,
$ npm install

Once the dependencies are installed properly, start the server by typing,
$ npm start

Then you can navigate to http://localhost:3000 to check if the server has started successfully. This page should contain the following text, if everything went smoothly so far.
Express
Welcome to Express

2. Deploying JBrowse with Express

Before deploying JBrowse inside the express app we just creted, you need to setup JBrowse in your local environment. In order to do so, follow the steps as in the official github page of JBrowse, https://github.com/GMOD/jbrowse/tree/master 

 Alternatively, you can use the unpacked version of Jbrowse without having to run the pre configurations. However this approach is not recommended unless you get stuck while following the first approach. The unpacked version of JBrowse can be found at, https://github.com/pupudu/unpacked-jbrowse 

 The next step is to move the required files inside to the express app. First move all the files inside the JBrowse to the public directory located inside the JBrowseExpr express app. We do so because the auto generated express application has already setup the public directory as its root for serving static files. 
Once you have copied all the files, create a copy of the index.html file of JBrowse inside the views directory inside the jBrowseExpr application and rename it as index.ejs. This will allow expressJs to use the index file as the root file for starting the application. When this index file is loaded, all the other files required by it are served from the public directory as static files. 
The run the npm start command again and you should see the volovx data suggestion page when you navigate to http://localhost:3000 

Congratulations! You just hosted JBrowse with NodeJs and ExpressJs





This post briefly explains about my progress towards the GSoC project following up with my progress update 1. I did a lot of experiment work and research during this time giving attention to the JBrowse code base, sailsJs, different comment platforms angularjs and nodejs.

 I followed up a tutorial published by thinkster.io to have a better understanding of the MEAN Stack since I thought that this knowledge would be useful when designing a component that works in both the backend and frontend. The tutorial can be found at https://thinkster.io/mean-stack-tutorial. I created a complete application as instructed using AngularJS, NodeJs, ExpressJs and MongoDB. To learn more about AngularJs, I followed all the topics given by W3Schools at http://www.w3schools.com/angular/default.asp. Then I jumped into another interactive tutorial from the book, AngularJs in Action. I followed about half of the book content.

Then I started learning a bit of sailsJs. I followed the get-started pack at http://sailsjs.org/get-started and currently following the tutorial at https://scotch.io/tutorials/build-a-todo-app-using-sailsjs-and-angularjs.

I also did some research about different comment platforms and found Facebook Comments and Disqus comments to be the best so far with respect to the project requirement. First I tried to add the comment option to a custom application built with expressJs. For facebook comments, I connected the FacebookApp I created earlier as the host and did the integration. For disqus, I created an account on their system and connected the disqus comment box using that account. A summary of comments and configurations can be viewed via these admin panels.

We can configure different URLs to identify different comment threads. Also, these two systems host the comments in their servers. Therefore we do not need a separate backend to store comments. We can use a backend for other custom configurations.

On being success at integrating it to a custom application, I tried to integrate it to JBrowse itself. First I did it in the localhost using the apache server as the host for JBrowse. Then I thought of hosting it public on the web. My plan was to host it on my Final Year Research Project Server given to me by my university. However, there were issues when installing JBrowse on the virtual machine.

To overcome that, I created a new repo of JBrowse with the pre-configurations done using bower and setup.sh. https://github.com/pupudu/unpacked-jbrowse. Then I cloned the repo from my virtual machine and used an expressJs application to host the JBrowse. Now it is live with the disqus comment feature at http://sid.projects.mrt.ac.lk:3000/?data=sample_data/json/volvox
A lot of css styling and design planing will be required before the final system is released.

During the first week of the community bonding period, I followed the instructions given by google and did the non technical tasks including submission of tax forms and registration for a payoneer account.

I was privileged to experience a video conference call with the JBrowse-GSoC team, Ian Holmes, Eric Yao and Saksham Saxena. Everything went well except for the few connection issues I had due to rain. It was a very nice experience and a valuable opportunity to have spoken with the mentors of the project I am assigned to. They explained a lot of interesting information about the founding of JBrowse, GMOD and the userbase of the browser.

In the meantime, I did the setup to configure JBrowse to run on apache2 server in my ubuntu development environment. I had installed and tested JBrowse on a windows system prior to the proposal submission, but not on ubuntu. During this week, I installed things such as ubuntu OS, apache2 server, nodejs, npm, bower and perl on my machine.

I followed the instructions given in the readme file at the JBrowse github repository. I faced an issue while trying to execute the ./setup.sh command. I sent a mail to the dev mailing list of JBrowse and got a solution from Sir Ian which solved the issue. I had run the 'chown' command in a wrong directory. Rest of the setup went smoothly.

Being successful at configuring it on my local machine, I tried to do the same on a remote machine hosted at sid.projects.mrt.ac.lk which is my final year project website. I could start the apache2 server on port 24 of the machine. But installing JBrowse had few issues. I am troubleshooting the issues at the moment.

I also played a little with the JBrowse codebase. I manipulated the index.html file to see how the basic html changes reflect on the final preview. I also explored the sample data sets to understand how the final preview of the site is generated.

I created a test facebook application to understand how the facebook comments plugin operates. I had to do few extra tasks to make the app public such as generating a privacy policy and updating app icons.

I also did some layout and template edits to this blog to make it more simple and easy to navigate.

I hope to be more productive in the upcoming weeks

I welcome you to the blog I created to reflect my contributions towards the project, Lightweight chat plugin for the JBrowse genome browser.

This blog is updated each week with a moderate description about the progress done during the past week. The updates could also include the challenges faced and the solutions tried and the solutions found for the challenges during the respective week.

Other than the progress reports, I would publish posts about other interesting stuff related to the project. For example, I might write a post about the Facebook Comments plugin which is a potential option for the initial phase of the proposed project.

All of you are welcome to read and share your thoughts with me regarding anything related.

Cheers!!!

Pubudu
Back to Top