Finding Records with Record Center

Record Center offers users three distinct methods for finding records—each ensuring the security and integrity of the record by only showing a user those records they have been granted access to view. Users tend to mature through these three methods as they gain more understanding of the system and the structure or records within their organization.

 
icon-rc-folders-01.png

Traditional Folder-Like Browsing

“Browse Records” allows a user to navigate through the virtual hierarchy of records defined within the system. This method feels at home to those users that have historically worked in file shares or similar environments, or those that are new to the organization and may want to “explore” to better understand the record hierarchy.

 

Basic Keyword Searching

As a user learns more about the types of records stored within the system, record hierarchies, and metadata associated with given document types, they typically migrate away from browsing and begin to adopt search as the mechanism to find records. Record Center’s generic keyword searching allows you to perform simple keyword searches, or even create your own search queries to find the content you’re looking for.

A simple keyword search for “Lease” as an example, would return any record the user has access to that contains the word “Lease” in any of the record metadata, or in the content of the record itself (provided the record has gone through an OCR/indexing process). When performing a basic keyword search, a user will typically get a larger and less targeted result set. This result set can then be further refined using standardized metadata refiners—allowing the user to incrementally reduce that result set and hone in on their targeted records.

 

Structured Searching with Record Finder

Record Finder allows a user to conduct a structured search and identify a very targeted set of records based on specific criteria. This is the most efficient way for a user to find a record, provided they know at least some of the record’s metadata. Perhaps a user knows that they’re looking for a lease that was associated with a specific business unit, specific office, and affiliated with a specific legal entity. Record Finder allows a user to plug in all known metadata and issue a search for any matching records stored within Record Center. In addition, Record Finder can be configured to support different, allowing different users access to different searching scenarios with differing search fields. Furthermore, the targeted result set returned by Record Finder is may still be refined further using the standardized metadata refiners also available through the basic keyword searching.

 

About Record Center

Record Center is your turnkey solution for enterprise-class record management. An extension of Microsoft SharePoint, Record Center arms your users and record managers with a feature-packed, intuitive solution to manage the entire life-cycle of your records. Configure, Approve and Search for records faster and easier than ever with Record Center.

 

Interested in learning more about Record Center?

Nintex Workflow Migration Considerations

Many companies are making the move from on-premises SharePoint to the cloud. There are many advantages of making the move – resource requirements are lessened, cost, features, accessibility, and more. It's great once you are there, but the biggest challenge can be the work getting there. Migration to the cloud requires careful planning, knowledge and experience to take advantage of the cloud for the different types of content. Nintex Workflows are one of the best additions to Office 365 and migrating your existing on-premises workflows successfully isn't a problem with the following considerations.

Missing Actions

Unfortunately, the cloud version of Nintex Workflows is not a 1-to-1 implementation of on-premises Nintex Workflow 2010 or 2013. The reason for this is because of Microsoft's Workflow Manager model in Office 365 which does not support all of the actions from on-premises. At the time of this writing, there are 40 out-of-the-box Nintex actions unavailable in Nintex for Office 365. Custom workflow actions are also not supported. While most of the Nintex out-of-the-box actions have been replicated, they do not necessarily work the same as it does on-premises. This is due to the architectural differences between the two environments. 

Nintex Workflow 2016 has the unique option of creating workflows that are fully compatible with Nintex Workflow for Office 365. As you begin to create a new workflow in SharePoint 2016, you can choose between creating an on-premises version with the extra 40 actions available or create an Office 365 compatible version that does not. While there are less actions available with the compatible option, it does ensure that your workflow will be compatible with Office 365 when you migrate.

SharePoint Online Limitations

There are also some key differences with SharePoint Online versus on-premises. One limitation is that a workflow file cannot not exceed 5MB in size after it is exported. When checking for a file size, know that a Nintex workflow is compressed when exported. To obtain the actual size, you will need to expand the NWP file to see what the total size is. Workflows exceeding this limit may have issues online. To reduce the risk of error, any large workflow may have to be rearchitected before moving to the cloud.  

There are many other differences with the cloud that should be known. For example, one of the issues most clients run into is the immovable 5000 item List View Threshold limit. This is the maximum number of items that can exist in a list view and keep good performance. In SharePoint on-premises, an administrator can raise the limit or set Daily Time Windows where the limits are raised. In Office 365, this limit cannot be changed and is in place 24x7. It may be necessary to create new lists to address this issue which may affect workflows. This is one example where knowledge of the limitations within SharePoint Online is paramount to a successful migration.

Other Considerations

Because of the differences in Office 365, issues can arise with workflows when implementing features. Turning on something like two-factor authentication may cause significant issues with Nintex workflows.  Knowing the limitations of Nintex's different implementations in relation to Office 365 features prevents downtime by using workarounds.  

The architecture of your workflows should be reviewed. Some actions affect performance more than others (e.g. Execute SQL, Query LDAP, etc.). Excessive looping can significantly slow down, stall, or have their execution throttled. Unfortunately, you cannot modify or increase the hardware running the workflows like you could on-premises. Also, the SharePoint Online Workflow Engine controls workflow throttling which Nintex has no control over. The only option is to increase efficiency through design with the workflow.

How Do I Get There?

All of these considerations can seem very daunting when approaching a migration to the cloud for the first time. This stress is increased since the cloud is always changing. Finding experts that are focused on this activity with experience of various requirements with solutions can help mitigate the unknowns. 

A third-party migration tool like Sharegate can greatly assist with the move. Sharegate works directly with Nintex to improve their ability to produce successful migrations. Sharegate will move all Nintex workflows, including those with actions that are not supported. Placeholders are put where the actions would normally be. The placeholders are labeled with the comments of the original action for easy identification.  This allows for a workaround to be developed within Office 365 which also helps with testing in the target environment.

Conclusion

Migrations are a complex process. The recipe for success is investigation of the existing environment, planning, knowledge of the current target environment, and a good 3rd-party tool. Experience with the cloud and tools involved is also desired. With that combination, getting your Nintex workflows to the cloud will be a success for you and your organization. 

Keys to Designing and Managing Large Repositories

The B&R team has some deep experience managing large structured document repositories within SharePoint. In some cases, those repositories were established and grew within SharePoint organically over time, while in others there were large sets of content that were migrated in from networked file shares or other ECM solutions like Documentum, OpenText, or FileNet.

Throughout SharePoint’s long history there has often been confusion between software limitations and best practices. To make matters worse in many cases there is no global agreement on what the best practices are. In our experience, many of the best practices are really guidelines that are context specific. There is no generic answer, but rather a starting point from which the requirements can then shape the most appropriate solution within the right context. In our experience, some of those key decision points are:

Structured vs. Unstructured Content

paperclips-unorganized-sm.png

Loosely categorized unstructured content stored in desperate locations across the organization.

paperclips-organized-md.png

Centrally managed structured content that has been properly categorized.

While SharePoint can be used in many ways, the first contextual decision point is structured versus unstructured content. In this post, we will be specifically focused on structured content storage repositories with content types and meta-data defined, and not unstructured repositories. This is an important differentiator for us, since the organization and usability of content in unstructured systems is radically different.

Software Boundaries and IT Capabilities

Understanding the limits of your platforms and systems is vitally important.

When thinking of the actual software and storage boundaries, SharePoint as a platform is very flexible, and continues to increase limits as modern storage and database limits increase. Here are references to the software boundaries for 2013, 2016, and Office 365.

One of the common misconceptions is that the infamous “List View Threshold” implemented in SharePoint 2010 is a boundary limiting the number of items in a list or library to 5,000. This is not a limit to the number of items in a library, but rather the number of records that can be read as part of a database query. This specific topic will be addressed as part of the System User Experience and Content Findability section.

For on-premises versions of SharePoint Server including 2010, 2013, and 2016 our focus has been on establishing system sizes that can be reliably maintained including backup and recovery procedures. This is a pretty important point because in my experience the capabilities and expectations of my clients vary widely. In some cases, they have deep experience with multi-terabyte databases and have plenty of room to work with to both backup and restore databases as needed. In other cases some customers struggle with backing up and restoring databases that are just a few hundred gigabytes due to backup technologies or lack of available working storage space. With this in mind, our initial guiding points are to look at how to isolate the structured repositories into dedicated site collections, each with a dedicated content database. The number and size of those site collections vary depending on the customer’s requirements and backup and recovery capabilities. We frequently start by advising on smaller database sizes of around 100 GB and then adjust based on their comfort levels and capabilities, but they should never exceed the Sys Admin’s ability to capture and restore a database backup.

For Office 365, Microsoft has taken ownership of the system maintenance and regular backup and recovery operations. Within, the service they have also extended the software boundaries which can make it much easier to support systems with larger repositories and fewer site collections, pushing much of the decision to the next two points relating to system usability and content findability.

System User Experience and Content Findability

The user experience of the repository is essential to its long-term success.

We will focus on looking at the processes to initially add or author content and then how it will be later discovered. Patterns and techniques that work fine in other sites or repositories can completely fail with large repositories.

While SharePoint as a platform is typically thought of in terms of collaboration and collaborative content, the scenarios for structured content in large repositories is often different. In some scenarios, the content may be comprised of scanned documents or images that are sent directly to SharePoint, while in others they could be bulk loaded electronic documents.

Unlike the collaborative scenarios, you very rarely want to add the content to the root of a SharePoint library, but either organize the content across libraries and/or sub-folders. To better handle this scenario, we will often incorporate the Content Organizer feature that Microsoft made available with SharePoint Server 2010 which offers a temporary drop off library and rules to selectively route content to another site collection, library, or folder. This rules based approach provides great automation capabilities that help to keep things properly organized, while making it much easier to add content to the system. While the Content Organizer covers most of our common scenarios, we are able to support even more advanced scenarios for automation by leveraging a workflow tool or customization when needed.

Previously, the List View Threshold feature was mentioned. While it is often discussed as a boundary or limitation, it is actually a feature intended to help maintain system performance. For SharePoint Server 2010, 2013, and 2016 it is a system setting that can be set at the web application level. The intention of this feature is to provide protection against unoptimized queries being run against the back-end SQL server. The default value of 5,000 was chosen because that is the point in which queries are processed differently by the database’s query engine, and you will start to see performance related problems. While it is safe to make small changes beyond the default limit, quickly you will experience the performance impacts the feature was designed to avoid.

The important thing to remember is that the threshold is for a given query, so the key task is to plan and implement your views to be optimized. We do this by thinking about a few key things:

Configure the Default View:

By default, SharePoint points to the All Items for the default view. Ideally, no view will be without a viable filter, but the All Items view absolutely should not be the default view in these libraries.

Column Indexes:

Key columns used to drive views or as the primary filter within your list can be indexed to improve performance. Additional information can be found here.

View Filters:

Ideally, all views will be sufficiently filtered to be below the List View Threshold of 5,000 items. This will keep view load time low.

Lookup Fields:

Avoid the use of lookup fields, as these lookup fields will require inefficient queries that perform tables scans to return content. Even smaller repositories of just a few hundred items can exceed the List View Threshold because of the query formatting.

Avoid Group By, Collapsed Option:

While the ability to group by your meta-data can be powerful, we typically instruct our clients to avoid using the option to collapse the Group By selections. The collapse option has some unexpected behavior that will result in additional database queries for each of the group by levels and values and disregard the item limits and paging configuration. It is possible to limit a view to say 30 items, but if you configure it to group by a value and collapse it by default, the first category could have 1,000 items and the system will query and load the full list, ignoring the 30-item limit. This can have severe performance implications, and is typically the primary culprit when we are asked to help troubleshoot page load performance in a specific repository.

While the ability to easily and effectively locate content has a big impact on the user experience of the system, I would argue that it is the most critical and therefore one that needs to be thought through when working within the SharePoint platform so I have broken the topic out into its own section.

If you think about SharePoint sites on a scale or continuum from the small team sites with a few libraries containing a handful of documents up to large, enterprise repositories with millions of documents it should be clear that how you find and interact with content on the two opposite ends of the spectrum needs to evolve. As systems grow larger, well beyond the minimalistic List View Threshold levels, the system needs to become more sophisticated and move away from manual browsing to content or unstructured search keyword queries to a more intelligent search driven experience.

While most systems include a properly configured search service, a much smaller percentage have it optimized in a way that it can leveraged to provide structured searches or dynamically surface relevant content. This optimization takes place at two levels; first within the search service itself, and then with customizations available in the system.

Within the Search Service we will work to identify meta-data key fields that should be established as managed properties for doing specific property searching, determine which fields need to be leveraged within a results screen for sorting and use within refinements. These changes allow us to execute more precise search queries and optimize the search results for our specific needs.

Within the site, we will then look to define the scenarios that people need to look for and find content to define structured search forms and result pages optimized for those scenarios. In some cases, they are generic to the content in the repository, while in others the scenarios are specific to a given task or role helping to simply things for specific business processes. By leveraging structured search pages, we can provide an improved user experience that dramatically reduces the time it takes to locate the relevant content as the initial search results are smaller, and then easily paired down through relevant search refiners. In addition, on common landing pages we will leverage the additional search driven web parts to highlight relevant, related, or new content as needed to support the usage scenarios.

Our Approach to Designing Record Center

As we set out to design and implement our Record Center product, we knew that it must scale to tens of millions of records both with regards to technical performance and from a user experience perspective. To accomplish this, we automated the setup and configuration process in ways to help optimize the solution for our specific purpose and use case.

While doing a product feature overview is outside the scope of this post, we are happy to report that our approach and techniques have been successfully adopted by our clients and that today the average repository size is in the hundreds of thousands of documents while still meeting performance, usability, and system maintenance goals.

Next Steps

I hope that this post provided a good overview of how to plan and maintain large repositories. It is a big topic with lots of nuances and techniques that are learned over time in the trenches. If your group is struggling with designing and managing large repositories and needs help, reach out and setup a consultation. We can either assist your team with advisement services, or help with the implementation of a robust system.

Can We Help?

Contact us today for a free consultation!

Why We Created Record Center

From time to time when I’m talking to prospective customer about Record Center, I get the question “What made you decide to build this?” It’s a great question, and the answer stems from a conversation I had with the CIO of one of our customers back in 2011. For this particular customer, we had recently implemented SharePoint Server 2010 and were in the process of migrating their Microsoft Office SharePoint Server 2007 sites and content up to 2010, while also planning for a variety of new SharePoint-based applications. When we looked across the portfolio of applications that we were going to build along with all of the other non-SharePoint-based systems they were maintaining, one thing became apparent – their records management story was non-existent. Across the organization, there were at least (that we were aware of) 11 different systems/locations that records could be stored in, and fewer than half of those had an identified business owner. To make matters worse, there was zero consistency in naming conventions, metadata, and classification schemas between the systems. It became immediately apparent that something had to be done because this was a mess from every perspective – but what exactly we were going to do was not so apparent.

You would have thought that with B&R being a SharePoint solutions provider, we would have immediately recommended SharePoint as the answer. But back in those days (and even still today), SharePoint was not known for being a great records management solution. Sure, there are site columns, managed metadata, content types, information management policies, retention policies, records declaration, and the content organizer (to name some of the features). But trying to explain how to configure and properly utilize all of these features to a site owner or business user tasked with records management for their department or group was close to impossible. The features were all over the place – library settings, site settings, site collection settings – and required various levels of access that most organizations did not want to grant. Simply put, SharePoint did not offer the end-to-end records management solution that was intuitive and easy to use; with this in mind we initially looked at alternative solutions.

brick-wall-construction.jpg

The features were all over the place and required various levels of access...

As an alternative to SharePoint, we looked at a variety of options including Documentum, FileNet, and other large systems, but the sticker shock we got when we saw not only the licensing but implementation costs (and timelines) always brought us back to SharePoint.

And that’s when Record Center was truly envisioned. When looking at the other systems, we saw the types of features and functionality they had – and we knew most of that was available in SharePoint, but was buried and difficult to implement out of the box. We knew what we had to do – take all of the disparate, but vitally important features of SharePoint that would support a records management initiative and combine them into one unified solution. And so, over the next two years, we built the application from the ground up, squeezing everything we could out of SharePoint while providing a simplified interface that provided records managers with the tools they needed to successfully perform their jobs and end users with the simplest of interfaces that allows them to find exactly what they are looking for without needing more than a few minutes of training.

In the end, Record Center became – and still is – one of the most successful IT projects in the history of our customer. And today, with the latest version of Record Center available for both SharePoint 2013 & 2016, organizations can have an accurate picture of and can easily manage the lifecycle of their records.

brick-wall-built.jpg

We knew we had to take all of the disparate features of SharePoint and combine them into one unified solution.

We know that records management isn’t a glamorous or exciting topic, but it’s a critical component to most organizational strategies for reducing liability, ensuring discoverability, and properly classifying records to meet legal and compliance requirements. Already using SharePoint? Then why implement a separate system that you must manage and maintain – use what you already have and experience a better path forward.

Interested in learning more about Record Center?

Building a REST Service with Node.JS, DocumentDb, and TypeScript

REST Services are commonly the backbone for modern applications.  They provide the interface to the back-end logic, storage, security, and other services that are vital for an application whether it is a web app, a mobile app, or both!  

For me, as recently as a years ago, my solutions were typically built with the .NET Stack consisting of ASP.NET Web API based services written in c#, various middleware plugins, and backend storage that usually involved SQL Server. Those applications could then be packaged up and deployed to IIS on premise or somewhere in the cloud such as Microsoft Azure. It's a pretty common architecture and has been a fairly successful recipe for me. 

Like many developers however in recent years I've spent an increasing amount of time with JavaScript. My traditional web application development has slowly moved from .NET server side solutions such as ASP.NET Forms & ASP.NET MVC to client side JavaScript solutions using popular JavaScript frameworks such as Angular. As I began working with the beta of Angular 2 I was first introduced to TypeScript and quickly grew to appreciate JavaScript even further. 

Spending so much time in JavaScript though I was intrigued with the idea of a unified development stack based entirely on JavaScript that wasn't necessarily tied directly to any particular platform.  I also started spending much more time with Microsoft Azure solutions and the combination of a REST based services built on Node.JS, Express, TypeScript, and DocumentDB seemed very attractive.  In my journey with that goal in mind I found I couldn't find a single all-inclusive resource that provided me what I was looking for, especially with DocumentDB,  so I worked through a sample project of my own which I'm sharing in this blog post to hopefully benefit some others on the same path.

A quick message on Azure DocumentDB

One of the foundations of this solution is Azure's DocumentDB service.  DocumentDB is one of Azure's Schema-Free NoSQL \ JSON Database offerings.  If you've had experience with MongoDB you will find DocumentDB very familiar.  In fact DocumentDB introduced protocol compatibility with MongoDB so your existing MongoDB apps can be quickly migrated to DocumentDB.   In addition to all the benefits, you might expect from a NoSQL solution you also get the high availability, highly scalable, low latency benefits of the Azure platform.  You can learn more about DocumentDB over at https://docs.microsoft.com/en-us/azure/documentdb/documentdb-introduction.

Prerequisites

Before getting started there are a couple prerequisites I suggest for the best experience while working with the source code of this project.

Visual Studio Code

If you're not using Visual Studio Code I highly encourage you to check it out. It's a fantastic free code editor from Microsoft. Grab yourself a copy at https://code.visualstudio.com/ . If you're already using another editor such as Atom or Sublime you're still good to go but you will need to make adjustments for your own debugging and development workflow for that editor.  If you're using Visual Studio Code you should have a solid "f5" debugging experience with the current configuration.

Azure DocumentDB Emulator

This project is pre-configured to use the Azure DocumentDB Emulator so no Azure Subscription is required!  The emulator is still in preview but is pretty solid and saves a lot of money for those just wishing to evaluate DocumentDB without any costs or subscriptions. More details on the emulator and links to download can be found at https://docs.microsoft.com/en-us/azure/documentdb/documentdb-nosql-local-emulator . 

If you'd like to you can also use a live instance of DocumentDB with your azure subscription but please make sure you understand the cost structure of DocumentDB as creating additional collections within a DocumentDB database has a cost involved.

Getting Started

For this sample project we'll be building a hypothetical photo location information storage service API. Our API will accept requests to create, update, delete, get photo location information.  In a future post we'll spend some more time with DocumentDB's Geospatial query features but for now we'll just keep it to simple CRUD operations.

The entire project can be found over at https://github.com/joshdcar/azure-documentdb-node-typescript.  Feel free to clone\review the code there. 

For this project we'll be making use of the following stack:

  • NodeJS - 6.10.0 (Runtime)
  • Express - 4.14.1 (Host)
  • TypeScript - 2.2.1 (Language)
  • Mocha\Chai - 2.2.39\3.4.35 (Testing)
  • Azure DocumentDB Emulator - 1.11.136.2 (Storage)

Project Setup

There are some folks with some pretty strong opinions in regards to a project's folder structure but I tend to side more with consistency than any particular ideology.

-- dist
-- src
+ -- data
 + -- LocationData.ts
 + -- LocationDataConfig.ts
 + -- LocationDocument.ts
+ -- routes
 + -- PhotoLocationRouter.ts
+ -- app.ts
+ -- index.ts
-- test
+ -- photolocation.test.ts
-- gulpfile.js
-- package.json
-- tsconfig.json

Let's break down some of these project elements:

  • dist - build destination for compiled javascript
  • src  - project source code containing our typescript. We'll be breaking down each of the source files further down in the post
  • test - test scripts
  • gulpfile.js - our build tasks
  • package.json - project metadata & NPM dependencies
  • tsconfig.json - typescript configuration file

NPM PACKAGES - PACKAGES.JSON

One of my pet peeves with sample projects, especially when I’m new to a technology,  are unexplained dependencies. With the multitude of open source tools and libraries I often find myself looking up modules to find out what they do and if I need them.  This muddies the waters for those not as familiar with the stack so I find it helpful keeping them to a minimum in my projects and explaining each of them and their purpose.

{
  "name": "azure-documentdb-node-typescript",
  "version": "1.0.0",
  "description": "A sample Node.JS based REST Service utilizing Microsoft Azure DocumentDB and TypeScript",
  "main": "dist/index.js",
  "scripts": {
    "test": "mocha --reporter spec --compilers ts:ts-node/register test/**/*.test.ts"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/joshdcar/azure-documentdb-node-typescript.git"
  },
  "keywords": [],
  "author": "Joshua Carlisle (www.joshcarlisle.io)",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/joshdcar/azure-documentdb-node-typescript/issues"
  },
  "homepage": "https://github.com/joshdcar/azure-documentdb-node-typescript#readme",
  "devDependencies": {
    "@types/chai": "^3.4.35",
    "@types/chai-http": "0.0.30",
    "@types/debug": "0.0.29",
    "@types/documentdb": "0.0.35",
    "@types/express": "^4.0.35",
    "@types/mocha": "^2.2.39",
    "@types/node": "^7.0.5",
    "chai": "^3.5.0",
    "chai-http": "^3.0.0",
    "del": "^2.2.2",
    "documentdb": "^1.10.2",
    "gulp": "^3.9.1",
    "gulp-mocha": "^4.0.1",
    "gulp-sourcemaps": "^2.4.1",
    "gulp-typescript": "^3.1.5",
    "mocha": "^3.2.0",
    "morgan": "^1.8.1",
    "ts-node": "^2.1.0",
    "typescript": "^2.2.1"
  },
  "dependencies": {
    "body-parser": "^1.16.1",
    "express": "^4.14.1"
  }
}

view rawpackage.json hosted with ❤ by GitHub

  • @types/*  - these are typescript definition files so our typescript code can interact with external libraries in a defined manner (key to intellisense in many editors)
  • Typescript - core typescript module
  • gulp/del - javascript based task runner we use for typescript builds and any extra needs. Del is a module that deletes files for us
  • gulp-sourcemaps/gulp-typescript - helper modules to allow us to use gulp to compile our typescript during builds
  • Mocha - A javascript testing framework and test runner
  • chai/chai-http - a testing assertion framework we use for creating tests. We're specifically testing out http REST requests
  • Gulp-mocha - A gulp task to help us run mocha tests in gulp (NOTE: running into issues with this one so it remains while I sort it out but we'll be running tests form npm scripts - more on this further in the post)
  • Ts-node - A typescript helper module used by Mocha for tests written in TypeScript
  • Documentdb: Azure's DocumentDB Javascript Module
  • Express - our service host
  • Body-parser - a module that helps express parse JSON parameters automatically for us.

Setting up and working with Typescript

Typescript is fairly straight forward especially for developers who are comfortable working with compiled languages.  Essentially TypeScript provides us with advanced language features that aren't yet available in the current version of Javascript. It does this by compiling typescript down to a targeted version of Javascript. For browsers application, this is typically ES5 but for Node.JS based applications we can reliably target ES6 which is newer but generally not available in most browsers.  The end result of that process though is always standard JavaScript. Typescript is not actually a runtime environment.  For .net developers, this is very much a kin to c# being compiled down to IL.   Additionally, to making debugging easier we have the concept of sourcemaps which map our generated JavaScript with the original lines of typescript code.

Where the confusion often occurs is where\when\how to compile. There are lots of options for us. For front-end UI developers webpack is a common tools. For Node.js projects, such as this, another common approach, and one we make use of in this project, is gulp. 

Configuring Typescript and Gulp

Getting Typescript to compile is typically pretty straight forward, but getting it to compile properly for debugging can be another story entirely and typically the pain point for many developers.

The first file we'll be working with is the tsconfig.json which provides compiler options to the typescript compiler.

{
  "compilerOptions": {
    "target": "es6",
    "module": "commonjs",
    "outDir": "dist"
  },
  "include": [
    "src/**/*.ts"
  ],
  "exclude": [
    "node_modules"
  ]
}

view rawtsconfig.json hosted with ❤ by GitHub

In the case of our project we wanted to target JavaScript ES6, use commonjs for our module format, and specify which directories we want the compiler to include and exclude.

@TYPES - WORKING WITH EXTERNAL LIBRARIES

Type definition files in Typescript allow TypeScript to work with external libraries that were not developed originally in typescript. Most popular JavaScript libraries have had types defined by either the project or contributions from the typescript community at large. They define the interfaces, types, functions, data types that the library exposes. The way in which TypeScript references and works with Types changed from 1.x - 2.x of typescript so you still may see references to the old way. Ensure that you're using @types/* to get the latest versions of your types from NPM and that they are in sync with the version of the library you pulled down from NPM if your library doesn't already included types. 

GULP

Gulp is a JavaScript based task runner. It has a very large community of plugins to help execute practically every conceivable task you could think of that you might need for a build. To go back to a .NET comparison Gulp is akin to the features you may find in MSBuild.

var gulp = require("gulp");
var ts = require("gulp-typescript");
var mocha = require('gulp-mocha');
var tsProject = ts.createProject("tsconfig.json");
var sourcemaps = require("gulp-sourcemaps");
var del = require("del");
var path = require("path");

/* Test Tasks
WARNING:  GULP MOCHA task is a work in progress and currently has issue.
NPM Script "Test" (package.json) currently more reliable
 */
gulp.task('test', function(){
    return gulp.src('./test/**/*.ts')
    .pipe(tsProject()).js
    .pipe(gulp.dest('.'))
    .pipe(mocha({
        reporter: 'progress'
    }))
})

/* Cleanup the DIST folder before compiling */
gulp.task('clean', function(){
    return del('dist/**/*');
})

/* Compile the Typescript */
/* IMPORTANT: The Sourcemaps settings here are important or the sourcemap url and source path in the source
maps will not be correct and your breakpoints will not hit - this is especially important for subfolders in the dist folder   */
gulp.task('compile', ['clean'], function () {
var result = tsProject.src()
                    .pipe(sourcemaps.init())
                    .pipe(tsProject()).js
                    .pipe(sourcemaps.write('.', {includeContent:false, sourceRoot: '.'})) 
                    .pipe(gulp.dest('dist'));
});

/* The default task will allow us to just run "gulp" from the command line to build everything */
gulp.task('default', ['compile']);

view rawgulpfile.js hosted with ❤ by GitHub

Gulp tasks are defined as JavaScript functions and can optionally have dependent other tasks that are executed first which is syntactically represented by the option array of functions as the second argument for a task function. In our case we have a "Compile" task that executes the typescript gulp plugin and works in conjunction with the source maps plugin to compile and output our Javascript to the dist directly.  Note that we have the "dist" directory both within the gulp task and the tsconfig task. The gulp plugin uses the base settings from the tsconfig to execute the typescript compilation process but also supports Gulps "stream" concept to output files to a desired location.

TAKE NOTE:  If you change the commands within the Compile function you may not have full debugging support within Visual Studio Code. The settings provided ensure that the sourcemaps have the correct references and paths used by Visual Studio to allow for breakpoints to be used as expected. These few lines of code took me several hours of research, trial & error for all the stars to line up. If you have problems the first place to start looking is your generate *.map files and ensure the paths provided are what you expect.

Configuring Express and your API Routes

Express is a web application framework for Node.js. Express will host our REST services and provide all the middleware plumbing we need for our service.

Index.ts

Index.ts is essentially responsible for wiring our express based application into the http pipeline within node.js. It's also the entry point for our application where we can configure options such as port to listen to. 

import * as http from 'http';
import * as debug from 'debug';

import App from './App';

debug('ts-express:server');

const port = normalizePort(process.env.PORT || 3000);
App.set('port', port);

const server = http.createServer(App);
server.listen(port);
server.on('error', onError);
server.on('listening', onListening);

function normalizePort(val: number|string): number|string|boolean{

    let port: number = (typeof val === 'string') ? parseInt(val,10): val;

    if(isNaN(port)) return val;
    else if(port > 0 ) return port;
    else return false;

}

function onError(error: NodeJS.ErrnoException): void{

     if (error.syscall !== 'listen') throw error;
  let bind = (typeof port === 'string') ? 'Pipe ' + port : 'Port ' + port;
  switch(error.code) {
    case 'EACCES':
      console.error(`${bind} requires elevated privileges`);
      process.exit(1);
      break;
    case 'EADDRINUSE':
      console.error(`${bind} is already in use`);
      process.exit(1);
      break;
    default:
      throw error;
  }
}

function onListening(): void {
  let addr = server.address();
  let bind = (typeof addr === 'string') ? `pipe ${addr}` : `port ${addr.port}`;
  debug(`Listening on ${bind}`);
}

view rawIndex.ts hosted with ❤ by GitHub

NOTE: I need to give some credit to another developer for Index.ts because I know at some point many aspects of this module were copy\paste from somewhere but unfortunately I can't recall who that was but if I do find the original source I'll make sure to update the source and this post with the developer's name.

App.ts

App.ts sets up an instance of express and provides us a location to configure our middleware with some additional plumbing such as parsing JSON, urlencoding and some basic logging.  Also key to our application is the configuration of the routes used by our API and the code that is going to handle the routes. Other possible middleware components could be authentication, authorization, and caching just to name a couple.  

this.express.use('/api/v1/photolocations', PhotoLocationRouter);
/*  Express Web Application - REST API Host  */
import * as path from 'path';
import * as express from 'express';
import * as logger from 'morgan';
import * as bodyParser from 'body-parser';
import PhotoLocationRouter from './routes/PhotoLocationRouter';

class App{

    public express: express.Application;
    
    constructor(){
        this.express = express();
        this.middleware();
        this.routes();
    }
    
    private middleware(): void{
        this.express.use(logger('dev'));
        this.express.use(bodyParser.json());
        this.express.use(bodyParser.urlencoded({extended: false}));
    }

    private routes(): void{
        let router = express.Router();
        this.express.use('/api/v1/photolocations', PhotoLocationRouter);
    }

}

export default new App().express;

view rawapp.ts hosted with ❤ by GitHub

PhotoLocationRouter.ts

The core of how our service handles request at a particular end point is managed within the router.  We define the functions that execute on a given protocol and any appropriate parameters:

this.router.get("/:id", this.GetPhotoLocation),
this.router.post("/", this.AddPhotoLocation),
this.router.put("/",this.UpdatePhotoLocation),
this.router.delete("/:id",this.DeletePhotoLocation)

For more complex applications it can be benficial to follow this seperate of routes from the app.  In the case of our sample project we're following REST standards of using Get/Post/Put/Delete appropriately for different CRUD operations.

public GetPhotoLocation(req:Request, res: Response){

         let query:string = req.params.id;
         var data:LocationData = new LocationData();

                 data.GetLocationAsync(query).then( requestResult => {
                 res.status(200).send(requestResult);

         }).catch( e => {

                 res.status(404).send({
                 message: e.message,
                 status: res.status});

         });

}

Each operation has a Request and Response object to interact with. In the request object we can access the parameters from either the URL or from the body. Our body parser middleware neatly packs up our body data into a JSON object (given the correct content-type header) and any URL paramters also get neatly extracted from the URL and packed into a property name based on the pattern provided in the route. In the above example we're able to access the "id" parameter.  The response object alllows us to respond with specific\appropriate http codes and any appropriate payload of data. 

import { Router, Request, Response, NextFunction} from 'express';
import { LocationData } from '../data/LocationData';
import { PhotoLocationDocument } from  '../data/PhotoLocationDocument';

export class PhotoLocationRouter {

    router:Router;

    constructor(){
        this.router = Router();
        this.init();
    }

    public GetPhotoLocation(req:Request, res: Response){

        let query:string = req.params.id;
        var data:LocationData = new LocationData();

        data.GetLocationAsync(query).then( requestResult => {
            res.status(200).send(requestResult);
        }).catch( e => {
                res.status(404).send({
                    message: e.message,
                    status: res.status
                });
        });

    }

    public AddPhotoLocation(req:Request, res: Response){

        var doc: PhotoLocationDocument = <PhotoLocationDocument>req.body;
        var data:LocationData = new LocationData();

        data.AddLocationAsync(doc).then( requestResult => {
            res.status(200).send(requestResult);
        }).catch( e => {
                res.status(404).send({
                    message: e.message,
                    status: res.status
                });
        });

    }

    public UpdatePhotoLocation(req:Request, res: Response){

        var doc: PhotoLocationDocument = <PhotoLocationDocument>req.body;
        var data:LocationData = new LocationData();

        data.UpdateLocationAsync(doc).then( requestResult => {
            res.status(200).send(requestResult);
        }).catch( e => {
                res.status(404).send({
                    message: e.message,
                    status: 404
                });
        });

    }

    public DeletePhotoLocation(req:Request, res: Response){

            let query:string = req.params.id;
            var data:LocationData = new LocationData();

            data.DeletePhotoLocationAsync(query).then( requestResult => {
                res.status(204).send();
                }).catch( e => {
                res.status(404).send({
                        message: e.message,
                        status: 404
                    });
            });

    };

    init(){
        this.router.get("/:id", this.GetPhotoLocation),
        this.router.post("/", this.AddPhotoLocation),
        this.router.put("/",this.UpdatePhotoLocation),
        this.router.delete("/:id",this.DeletePhotoLocation)
    }

}

const photoLocationRouter = new PhotoLocationRouter();
photoLocationRouter.init();

export default photoLocationRouter.router;

view rawPhotoLocationRouter.ts hosted with ❤ by GitHub

NOTE: I dislike peppering my blog posts with disclaimers but this configuration should be intended for development use only. Additional configuration options can\should be applied for production applications, especially with security and threat hardening. I say this because of a recent Express vulnerability that left a lot of Node.JS sites unnecessarily exposed. For a good place to start checkout Helmet over at  https://www.npmjs.com/package/helmet

Working with DocumentDB

DocumentDB is a NoSQL solution that at its core stores JSON Documents. There are various best practices around working with DocumentDB but sticking with the basic concepts we'll be creating, updating, deleting, and querying those JSON documents.  Having worked with DocumentDB recently within the .NET framework using Linq this was a bit of a departure but luckily DocumentDB supports standard SQL queries so my years of working with relational databases paid off once again!  This was great foresight from Microsoft to make use of SQL instead of implementing yet another query language (I'm pointing at you SharePoint CAML!) 

DATABASE AND COLLECTION CREATION:  This sample project has a DocumentDB database and Collection.  The code does NOT create those for you so the expectation is that you will do this step up front. The database is named photolocations and the collection locations. Due to the fact that DocumentDB Collections and Databases have a direct cost involved with them (outside the emulator of course) I'm not a fan of code that generates these entities for you. I rather have the process be explicit. 

Defining our document

As an initial step we're going to define that JSON document within our project so we can work with it in a consistent way. In addition to our own properties DocumentDBadditionally has some standard properties that are part of every document. We can think of them as system fields.  To better support these fields the DocumentDB framework requires us to implement these interfaces for new documents (such as the id) and for retrieved documents which have additional fields such as etag for optimistic concurrency checks.

import {NewDocument, RetrievedDocument} from 'documentdb';

export class PhotoLocationDocument implements NewDocument<PhotoLocationDocument>, 
                                                                                    RetrievedDocument<PhotoLocationDocument>{
    /* NewDocument Interface */
    id:string;

    /* RetrievedDocument Interface */
    _ts:string;
    _self:string;

    /* Photo Location Properties */
    name:string;
    tags: string[];
    address: {
        street: string,
        city: string,
        zip: string,
        country: string
    };
    geoLocation: {
        type: string;
        coordinates: number[];
    }

}

view rawPhotoLocationDocument.ts hosted with ❤ by GitHub

ADVICE:  To keep things simple we're exposing the DocumentDb JSON document directly from the REST Service.  There are many scenarios where you may want to have a separate model that has different\fewer properties that is returned from the REST calls based on your applications needs.

DocumentDB operations

The core heavy lifting class within DocumentDB is DocumentClient. All operations such as queries, updates, inserts, deletes are all done through the DocumentClient class.  These functions make heavy use of callbacks so to make our own data layer functions more friendly to work against we wrapped all of our calls within Promises.  Note that many of the classes and interfaces we are working with are all imported in from the DocumentDB module and provided to us through those all-important interfaces.

import {DocumentClient, SqlQuerySpec, RequestCallback, QueryError, RequestOptions, SqlParameter, RetrievedDocument} from 'documentdb'
import {LocationDataConfig } from './LocationDataConfig';
import {PhotoLocationDocument } from './PhotoLocationDocument';

export class LocationData{

    private _config:LocationDataConfig;
    private _client: DocumentClient;

    constructor(){

        this._config = new LocationDataConfig();
        this._client = new DocumentClient(this._config.host, {masterKey: this._config.authKey}, this._config.connectionPolicy); 
                
    }

    public GetLocationAsync = (id:string) => {

        var that = this;

        return new Promise<PhotoLocationDocument>((resolve, reject) => {

            var options:RequestOptions = {};
            var params: SqlParameter[] = [{name: "@id", value: id }];

            var query: SqlQuerySpec = { query:"select * from heros where heros.id=@id",
                                                            parameters: params};
                                                    
            this._client.queryDocuments(this._config.collectionUrl,query)
                        .toArray((error:QueryError, result:RetrievedDocument<PhotoLocationDocument>[]): void =>{
                            
                            if (error){ reject(error); }
                            
                            if(result.length > 0){
                                resolve(<PhotoLocationDocument>result[0]);
                            }
                            else
                            {
                                reject({message: 'Location not found'});
                            }
                        });                                                         

        });

    }

    public AddLocationAsync = (photoLocation: PhotoLocationDocument) => {

        var that = this;

        return new Promise<PhotoLocationDocument>((resolve, reject) => {

                var options:RequestOptions = {};

                that._client.createDocument<PhotoLocationDocument>(that._config.collectionUrl, photoLocation, options, 
                        (error: QueryError, resource: PhotoLocationDocument, responseHeaders: any): void => {
                            if(error){
                                reject(error);
                            }
                            resolve(resource);
                });

        });

    }

    public UpdateLocationAsync = (photoLocation: PhotoLocationDocument) => {

        var that = this;

        return new Promise<PhotoLocationDocument>((resolve,reject) =>{

            var options:RequestOptions = {};
            var documentLink = that._config.collectionUrl + '/docs/' + photoLocation.id;

            that._client.replaceDocument<PhotoLocationDocument>(documentLink, photoLocation, options, 
                        (error: QueryError, resource: PhotoLocationDocument, responseHeaders: any): void => {
                            if(error){
                                reject(error);
                            }
                            resolve(resource);
                });

        });

    }

        public DeletePhotoLocationAsync = (id:string) => {

            var that = this;

            return new Promise<PhotoLocationDocument>((resolve, reject) => {

                    var options:RequestOptions = {};
                    var documentLink = that._config.collectionUrl + '/docs/' + id;
                
                    that._client.deleteDocument(documentLink, options, 
                        (error: QueryError, resource: any, responseHeaders: any): void => {
                            if(error){
                                reject(error);
                            }
                            resolve(resource);
                    });
            });

    }

}

view rawLocationData.ts hosted with ❤ by GitHub

Unit Testing

Cards on the table I'm very new with Javascript testing frameworks but I found the test-first approach really reduced the debugging and development cycles greatly. I created the test plans first calling the service and letting them fail.  I defined all my expectations (assertions) for the service and then I developed the service until it passed all the tests. I found using Mocha \ Chai pretty straight forward and I'm looking forward to spending more time with the framework in the coming months and sharing that experience.

NOTE: I would not use my test plans for this project as a model for your own test plans. They work but they require more teardown and cleanup then I would like and I suspect introducing the concepts of mocking while working against the DocumentDB would be beneficial. For now the tests are actually performing operations against DocumentDB. I place these firmly in the "place to start" category.

Development & Debugging

Throughout the development process I made heavy use of a Chrome Plugin called postman.  Postman allows me to test various REST calls and adjust the body\headers\etc as needed. I know there are other tools such as CURL that provide the same set of features but I've found the features and user interface of Postman superior to most other solutions I've worked with adding a great bit of efficiency to my development cycle.

TIP:  Chrome has this nasty habit of automatically redirecting localhost traffic to SSL if anything else runs SSL on localhost -it's a feature called HSTS.  In our case the emulator runs under SSL on localhost so I battled this "feature" constantly. This is internal to Chrome and must be reset by going to a settings page within Chrome chrome://net-internals/#hsts and entering in "localhost" to delete the domain. Even worse this property doesn't stick and routinely gets added back in. The "fix" for this is to either run your service under SSL as well,  add a host header for your app locally so it's on a different host then localhost, or use another browser such as Firefox or IE for testing.  I know this is a safety feature but it's very annoying for us developers and I wish there was a way to disable it permanently for given domains. 

 

Parting thoughts

So we've learned the basics of building a simple REST service based on Node.JS, DocumentDB, and TypeScript. In an upcoming post I'll be building on this solution to demonstrate more features of DocumentDB, especially the GeoSpatial query features, and we're also explore adding oauth authentication to our service to safeguard access to our data.