Solving the Challenges of Record Security with Record Center

As we look back at the typical challenges of Records Management, be it the ability to implement a non-invasive content ingress and approval processes, ensuring that the entire record lifecycle is properly managed—inclusive of disposition, or ensuring that users are able to find the content they need to do their jobs in as few clicks as possible, there’s one remaining pain point that has the potential to bring your entire records management strategy to its knees… security. The structure of your records management solution drives security, but security also often drives structure, so what comes first? Thankfully, Record Center solves these problems for you and stores your records in a structure designed to support your specific security needs.

Storage vs. Security Models

When first configuring a new Record Center instance, you’re asked to select both a Storage Model and a Security Model. These two options work in concert with one another to tell Record Center at what granularity you want to secure your records, and to ensure that they are stored using a structure that supports and enables that security methodology. These options are also impacted by other configuration settings of Record Center, such as your approval model—since determining who can approve records and at what point in their record lifecycle also contains an element of record security. Fundamentally, Record Center presents these options all in a way that is more intuitive than having to manually design what an overall repository architecture looks like—one of the challenges that B&R’s Managing Director, Mike Oryszak brushed on as part of his previous Keys to Designing and Managing Large Repositories blog post.

RecordApprovals1.png

Options for Record Security

Record Center offers three separate security models that may be configured to meet your organization’s individual needs. These record security models apply after a record has been loaded into the system and processed through any necessary approvals, so it’s important to note that the selection of a specific security model does not require a user to be a consumer of record content in order to participate in the upload/ingress of content, or to perform one or more of the record’s approval stages

Entity

The entity security model provides the least granularity for record security. This is often a good fit for small teams where the number of users consuming record content is low, the organization’s corporate structure is thin, or security doesn’t need to vary between individual record types. Every record added to Record Center is assigned to a legal entity, such as B&R Business Solutions, LLC. For organizations that only have one legal entity, this field will default to a single value, but from a security perspective this effectively means that all of that entity’s records are available to the same audience, be it every employee or a smaller subset such as a compliance team. This model is also particularly useful if an organization contains many different legal entities. This is often the case in the real estate industry where different properties are often separate legal entities. Using this model, records are easily classified and secured by the entity they belong to, simplifying the ability to grant users or owners of each entity access to only that entity’s records.

The entity security model is a good fit for small teams or where security doesn’t need to vary between individual record types

Series

The series security model ensures that each individual record series created within Record Center can be individually secured. This allows you to provide granular access to specific categories of records, including all of the document types that belong to that specific series. As an example, providing a user access to a “Service Contracts” series, would give them access to all service contracts document types, which might include things like equipment leases, maintenance contracts, master service agreements, etc.

Record Center’s Metadata-based security model allows for a more dynamic implementation of record security.

Metadata

Record Center’s Metadata-based security model allows for a more dynamic implementation of record security. When using this model, an organization defines one or more record metadata field(s) that can be used to determine that record’s security. As an example, if a record type has a “Business Unit” field, and the goal is to secure records based on if they were part of the Manufacturing business unit or Corporate business unit, metadata-based security would allow an organization to define users that will have access to any record where Business Unit is set to Manufacturing. This security is applied regardless of legal entity or record series, meaning that multiple metadata-based security fields may exist on any record type. Ultimately, this allows you to define the previous Business Unit based audiences in addition to say “Office Location”, where one or more users would be granted access to all records based on a specific Office Location value.

Record Center’s Metadata-based security model allows for a more dynamic implementation of record security.

Compliance Access

In addition to the previously mentioned security models, Record Center also facilitates simple access for those users that need access to every record, such as a corporate compliance department. These users may be given access to Record Center’s “Global Record Access” group, which is applied to every record that is loaded into the system.

Conclusion

Our goal with Record Center has always been to try and simplify the otherwise daunting and complex task of designing and implementing a robust Records Management solution, be it the initial installation of a solution, designing the overall implementation, identifying an organization’s various record types, defining individual retention plans for each of those types, and ensuring that the right people can find the content they need when they need it. While Record Center’s ability to manage record security in a way that’s easy to understand is just one component of that strategy, it is vital to reducing accidental exposure, and ensuring that sensitive records are locked down to only those users that have been identified as consumers of that content.

About Record Center

Record Center is your turnkey solution for enterprise-class record management. An extension of Microsoft SharePoint, Record Center arms your users and record managers with a feature-packed, intuitive solution to manage the entire life-cycle of your records. Configure, Approve and Search for records faster and easier than ever with Record Center.

calltoaction-recordcenter.jpg

Interested in learning more about Record Center?

Information Management Policies and Complex Workflows

One of the key Enterprise Content Management (ECM) features provided by Microsoft in SharePoint Server is the Information Management Policy feature.  These policies can be used to establish multi-stage retention policies, but the scheduled nature of this feature opens it up for so much more. 

Note:  If you are not familiar with the information management policies and would like a general overview, see Plan for information management policy in SharePoint Server 2013.

For our purposes, I chose to leverage these features to support the multi-stage activity which supports scheduling each stage, defining an action to execute, and a recurrence schedule if applicable.  Using these retention schedules, we can execute scheduled activities to support business processes such as content retention, disposition, or to build a contract management solution.

Scheduling a Stage Date

The ability to schedule the start of the stage is both simple, and powerful.  You simply select one of the document’s date fields, which can be either a system field such as Created or Modified or can be a custom field as in the example below.

20170913-1.png

Action

Next, we focus on the action to take within this stage.  The configuration comes with the following actions:

  • Move to Recycle Bin:  Moves to the recycle bin for orderly removal
  • Permanent Delete:  Bypasses recycle bin and is immediately deleted
  • Transfer to another location:  Move to another site such as an archival or records site
  • Start a workflow:  Start a workflow that is associated with the content type
  • Skip to next stage:  Move directly to the next stage
  • Declare record:  Declare the item as a record in the system (in place records management)
  • Delete previous drafts:  Cleanup previous draft, minor version copies
  • Delete all previous versions:  Cleanup all previous versions

While these actions can be helpful, this is where most people start hitting the brakes.  If you have an important legal agreement or contract, you probably don’t want to just delete it or move it to a recycle bin when it is scheduled to expire.  You probably want somebody to review it and make sure it is actually no longer needed or does not need to be renewed.  For those that are familiar with the power of workflows the “Start a Workflow” action sounds great until you click that list and see an empty list of available workflows.  This is the single biggest hurdle for most people, and the point where many turn back.  Do not worry, we will come back to this shortly. 

Retention

The recurrence settings are also straight forward allowing you to repeat a stage based on a number of days, months, or years as the image below illustrates. 

20170913-2.png

Complex Workflow!

As I mentioned earlier, the “Start a workflow” action list is blank by default.  This is where our ability to implement complex workflows comes to the rescue.  These workflows can be developed using SharePoint Designer, Visual Studio, or our preferred tool Nintex Workflow.  The trick is that whatever path we choose, we need to be able to associate the workflow with the specific content type(s) for it to be available in the list of workflows within the “Start a workflow” action. 

To create a workflow that can be associated with a content type in SharePoint Server, navigate through Site Actions menu, select Nintex Workflow (2013/2016), and then Create Reusable Workflow Template as illustrated below.

20170913-3.png

We then define our workflow name, description, and associate it with a content type.

20170913-4.png

Here is an example of a Contract Review workflow we created for demo purposes.

20170913-5.png

Once our workflow is saved, we can now visit the Site Content Type Information page (Site Settings-> Site content types -> select our content type) and click the Workflow settings action under settings.

20170913-6.png

Next, we can select our workflow template and provide a unique name for the process.  For workflows that are triggered by the Info Mgt Policies, you can set the start options to enable “Allow this workflow to be manually started” and disable the new and edit options. 

20170913-7.png

Now that the workflow is associated with the content type, we can configure our Retention Policy.  From the Site Content Type Information page, select the Information management policy settings action. 

20170913-8.png

Select the “Enable Retention” option to enable the retention options and then click the “Add a retention stage” action to load the stage configuration form. 

20170913-9.png

The retention stage configuration form options were explained previously.  Define an appropriate stage schedule based on a date comparison with a date field.  The comparison can be based on days, months, or years. 

20170913-10.png

Next, select the “Start a workflow” option from the Action list and select the workflow you previously configured for the given content type. 

If applicable, configure an appropriate recurrence schedule. 

Then, click the Ok button to save your changes and continue. 

If needed, you can configure multiple stages.  For this example, you can see for the given contract content type, there is an initial stage for review.  After it progresses through the “review” stage, the second stage was configured to have a contract disposition workflow one year after expiration if the contract was not renewed as illustrated in the image below. 

20170913-11.png

Once the changes are fully saved, the document will be reviewed based on the internal process schedule and the workflow initiated. 

Single versus Multiple Stages (Multiple Workflows)

While it is possible to design and implement a single workflow that can handle the logic from the individual stages, there are some advantages to breaking the workflows down into the individual workflows for each stage.  It certainly makes the workflow easier to manage within the designer, but it also gives you more granular tracking for executions leading to clearer insights and reporting without having to build in a lot of extra actions within the workflow to break out and report on the individual stages.  Ultimately, the requirements can be fulfilled either way, but we find it easier to maintain and support with individual workflows for each stage.

 
calltoaction-general.png

Need assistance with retention and disposition workflows?

Overcoming Upload & Approval Challenges with Records Management

When we first embarked on the journey of creating Record Center, we knew there were several major pain points in the lifecycle of records management that we wanted to address. Creating a platform that would ease the burden on record managers and record approvers was one of our top priorities, and the challenges of content ingress and record approval were some of the main issues we wanted to address. 

To understand the pain of loading and approving content into a records management solution is to realize that content comes from a seemingly endless number of sources. Those sources could include other business systems or raw scans of physical documents, and could comprise any type of document an organization handles—contracts, invoices, employment agreements, non-disclosure agreements, tax documents, etc. Each of those unique document types could very well contain different metadata, different required fields, or different document formats further complicating the notion of a centralized ingress and approval process. 

Upload Content (Ingress) 

Record Center’s model of “Document Types” allows for an organization to define criteria for each type of record to be loaded into the system. Each of these document types contains its own distinct metadata requirements, approval requirements, retention plans, and disposition processes. This model allows for all records—regardless of their type—to be submitted to the same central Pending Records library, using the same process, while still ensuring that a unified record submission and approval flow may be used. 

Required Fields 

Critical to being able to easily and accurately find records is the notion of applying metadata to records. Record Center’s unique in-line preview mode allows a user submitting a record to easily see their document alongside the document’s required and optional metadata fields. This view allows them to quickly determine what fields must be completed for a given document type, and if necessary, navigate through the document in question to determine the values of those required fields. 

Approval Models 

Document Approvers and Record Managers are frequently faced with the challenge of having to approve large quantities of records. To cope with these challenges, Record Center offers several features integral to streamlining this otherwise time-consuming process. 

Unique Document Type Approvers

Each unique document type may contain its own distinct record approvers, allowing for a first stage approval by a user belonging to that document’s work center or area of expertise. This process helps to ensure that misclassified records or those with incorrect metadata are caught before entering the Record Manager’s approval queue. 

Inline Document Preview

Utilizing inline preview, a document approver and/or Record Manager can easily review a document alongside its metadata to ensure the metadata entered is accurate, and all required fields have been completed. 

Approve and View Next

As a document approver or Record Manager moves through their approval queue, they would historically have to open the document to be approved, approve that document, close it, open the next document, approve that, close it, and so on. To streamline this click-heavy process, Record Center contains a “Approve and View Next” feature, which allows the user to review, approve and open the next document in their queue with a single click. 

Bulk Approval

In the case of a bulk import where a record’s data has already been validated, Record Center includes a bulk approval option that allows a document approver or Record Manager to select multiple records at once and approve them all with a single click. 

Auto Approval

In some cases, an organization may wish to automatically approve records. The likely scenario for this is when a records management team is small, and perhaps the same people would do the content ingress as well as the approval. In this scenario, records can be configured to automatically approve after they’ve been pending approval for a set amount of time. 

 

About Record Center 

Record Center is your turnkey solution for enterprise-class record management. An extension of Microsoft SharePoint, Record Center arms your users and record managers with a feature-packed, intuitive solution to manage the entire life-cycle of your records. Configure, Approve and Search for records faster and easier than ever with Record Center. 

 
calltoaction-recordcenter.jpg

Interested in learning more about Record Center?

Finding Records with Record Center

Record Center offers users three distinct methods for finding records—each ensuring the security and integrity of the record by only showing a user those records they have been granted access to view. Users tend to mature through these three methods as they gain more understanding of the system and the structure or records within their organization.

 
icon-rc-folders-01.png

Traditional Folder-Like Browsing

“Browse Records” allows a user to navigate through the virtual hierarchy of records defined within the system. This method feels at home to those users that have historically worked in file shares or similar environments, or those that are new to the organization and may want to “explore” to better understand the record hierarchy.

 

Basic Keyword Searching

As a user learns more about the types of records stored within the system, record hierarchies, and metadata associated with given document types, they typically migrate away from browsing and begin to adopt search as the mechanism to find records. Record Center’s generic keyword searching allows you to perform simple keyword searches, or even create your own search queries to find the content you’re looking for.

A simple keyword search for “Lease” as an example, would return any record the user has access to that contains the word “Lease” in any of the record metadata, or in the content of the record itself (provided the record has gone through an OCR/indexing process). When performing a basic keyword search, a user will typically get a larger and less targeted result set. This result set can then be further refined using standardized metadata refiners—allowing the user to incrementally reduce that result set and hone in on their targeted records.

 

Structured Searching with Record Finder

Record Finder allows a user to conduct a structured search and identify a very targeted set of records based on specific criteria. This is the most efficient way for a user to find a record, provided they know at least some of the record’s metadata. Perhaps a user knows that they’re looking for a lease that was associated with a specific business unit, specific office, and affiliated with a specific legal entity. Record Finder allows a user to plug in all known metadata and issue a search for any matching records stored within Record Center. In addition, Record Finder can be configured to support different, allowing different users access to different searching scenarios with differing search fields. Furthermore, the targeted result set returned by Record Finder is may still be refined further using the standardized metadata refiners also available through the basic keyword searching.

 

About Record Center

Record Center is your turnkey solution for enterprise-class record management. An extension of Microsoft SharePoint, Record Center arms your users and record managers with a feature-packed, intuitive solution to manage the entire life-cycle of your records. Configure, Approve and Search for records faster and easier than ever with Record Center.

 

Interested in learning more about Record Center?

Keys to Designing and Managing Large Repositories

The B&R team has some deep experience managing large structured document repositories within SharePoint. In some cases, those repositories were established and grew within SharePoint organically over time, while in others there were large sets of content that were migrated in from networked file shares or other ECM solutions like Documentum, OpenText, or FileNet.

Throughout SharePoint’s long history there has often been confusion between software limitations and best practices. To make matters worse in many cases there is no global agreement on what the best practices are. In our experience, many of the best practices are really guidelines that are context specific. There is no generic answer, but rather a starting point from which the requirements can then shape the most appropriate solution within the right context. In our experience, some of those key decision points are:

Structured vs. Unstructured Content

paperclips-unorganized-sm.png

Loosely categorized unstructured content stored in desperate locations across the organization.

paperclips-organized-md.png

Centrally managed structured content that has been properly categorized.

While SharePoint can be used in many ways, the first contextual decision point is structured versus unstructured content. In this post, we will be specifically focused on structured content storage repositories with content types and meta-data defined, and not unstructured repositories. This is an important differentiator for us, since the organization and usability of content in unstructured systems is radically different.

Software Boundaries and IT Capabilities

Understanding the limits of your platforms and systems is vitally important.

When thinking of the actual software and storage boundaries, SharePoint as a platform is very flexible, and continues to increase limits as modern storage and database limits increase. Here are references to the software boundaries for 2013, 2016, and Office 365.

One of the common misconceptions is that the infamous “List View Threshold” implemented in SharePoint 2010 is a boundary limiting the number of items in a list or library to 5,000. This is not a limit to the number of items in a library, but rather the number of records that can be read as part of a database query. This specific topic will be addressed as part of the System User Experience and Content Findability section.

For on-premises versions of SharePoint Server including 2010, 2013, and 2016 our focus has been on establishing system sizes that can be reliably maintained including backup and recovery procedures. This is a pretty important point because in my experience the capabilities and expectations of my clients vary widely. In some cases, they have deep experience with multi-terabyte databases and have plenty of room to work with to both backup and restore databases as needed. In other cases some customers struggle with backing up and restoring databases that are just a few hundred gigabytes due to backup technologies or lack of available working storage space. With this in mind, our initial guiding points are to look at how to isolate the structured repositories into dedicated site collections, each with a dedicated content database. The number and size of those site collections vary depending on the customer’s requirements and backup and recovery capabilities. We frequently start by advising on smaller database sizes of around 100 GB and then adjust based on their comfort levels and capabilities, but they should never exceed the Sys Admin’s ability to capture and restore a database backup.

For Office 365, Microsoft has taken ownership of the system maintenance and regular backup and recovery operations. Within, the service they have also extended the software boundaries which can make it much easier to support systems with larger repositories and fewer site collections, pushing much of the decision to the next two points relating to system usability and content findability.

System User Experience and Content Findability

The user experience of the repository is essential to its long-term success.

We will focus on looking at the processes to initially add or author content and then how it will be later discovered. Patterns and techniques that work fine in other sites or repositories can completely fail with large repositories.

While SharePoint as a platform is typically thought of in terms of collaboration and collaborative content, the scenarios for structured content in large repositories is often different. In some scenarios, the content may be comprised of scanned documents or images that are sent directly to SharePoint, while in others they could be bulk loaded electronic documents.

Unlike the collaborative scenarios, you very rarely want to add the content to the root of a SharePoint library, but either organize the content across libraries and/or sub-folders. To better handle this scenario, we will often incorporate the Content Organizer feature that Microsoft made available with SharePoint Server 2010 which offers a temporary drop off library and rules to selectively route content to another site collection, library, or folder. This rules based approach provides great automation capabilities that help to keep things properly organized, while making it much easier to add content to the system. While the Content Organizer covers most of our common scenarios, we are able to support even more advanced scenarios for automation by leveraging a workflow tool or customization when needed.

Previously, the List View Threshold feature was mentioned. While it is often discussed as a boundary or limitation, it is actually a feature intended to help maintain system performance. For SharePoint Server 2010, 2013, and 2016 it is a system setting that can be set at the web application level. The intention of this feature is to provide protection against unoptimized queries being run against the back-end SQL server. The default value of 5,000 was chosen because that is the point in which queries are processed differently by the database’s query engine, and you will start to see performance related problems. While it is safe to make small changes beyond the default limit, quickly you will experience the performance impacts the feature was designed to avoid.

The important thing to remember is that the threshold is for a given query, so the key task is to plan and implement your views to be optimized. We do this by thinking about a few key things:

Configure the Default View:

By default, SharePoint points to the All Items for the default view. Ideally, no view will be without a viable filter, but the All Items view absolutely should not be the default view in these libraries.

Column Indexes:

Key columns used to drive views or as the primary filter within your list can be indexed to improve performance. Additional information can be found here.

View Filters:

Ideally, all views will be sufficiently filtered to be below the List View Threshold of 5,000 items. This will keep view load time low.

Lookup Fields:

Avoid the use of lookup fields, as these lookup fields will require inefficient queries that perform tables scans to return content. Even smaller repositories of just a few hundred items can exceed the List View Threshold because of the query formatting.

Avoid Group By, Collapsed Option:

While the ability to group by your meta-data can be powerful, we typically instruct our clients to avoid using the option to collapse the Group By selections. The collapse option has some unexpected behavior that will result in additional database queries for each of the group by levels and values and disregard the item limits and paging configuration. It is possible to limit a view to say 30 items, but if you configure it to group by a value and collapse it by default, the first category could have 1,000 items and the system will query and load the full list, ignoring the 30-item limit. This can have severe performance implications, and is typically the primary culprit when we are asked to help troubleshoot page load performance in a specific repository.

While the ability to easily and effectively locate content has a big impact on the user experience of the system, I would argue that it is the most critical and therefore one that needs to be thought through when working within the SharePoint platform so I have broken the topic out into its own section.

If you think about SharePoint sites on a scale or continuum from the small team sites with a few libraries containing a handful of documents up to large, enterprise repositories with millions of documents it should be clear that how you find and interact with content on the two opposite ends of the spectrum needs to evolve. As systems grow larger, well beyond the minimalistic List View Threshold levels, the system needs to become more sophisticated and move away from manual browsing to content or unstructured search keyword queries to a more intelligent search driven experience.

While most systems include a properly configured search service, a much smaller percentage have it optimized in a way that it can leveraged to provide structured searches or dynamically surface relevant content. This optimization takes place at two levels; first within the search service itself, and then with customizations available in the system.

Within the Search Service we will work to identify meta-data key fields that should be established as managed properties for doing specific property searching, determine which fields need to be leveraged within a results screen for sorting and use within refinements. These changes allow us to execute more precise search queries and optimize the search results for our specific needs.

Within the site, we will then look to define the scenarios that people need to look for and find content to define structured search forms and result pages optimized for those scenarios. In some cases, they are generic to the content in the repository, while in others the scenarios are specific to a given task or role helping to simply things for specific business processes. By leveraging structured search pages, we can provide an improved user experience that dramatically reduces the time it takes to locate the relevant content as the initial search results are smaller, and then easily paired down through relevant search refiners. In addition, on common landing pages we will leverage the additional search driven web parts to highlight relevant, related, or new content as needed to support the usage scenarios.

Our Approach to Designing Record Center

As we set out to design and implement our Record Center product, we knew that it must scale to tens of millions of records both with regards to technical performance and from a user experience perspective. To accomplish this, we automated the setup and configuration process in ways to help optimize the solution for our specific purpose and use case.

While doing a product feature overview is outside the scope of this post, we are happy to report that our approach and techniques have been successfully adopted by our clients and that today the average repository size is in the hundreds of thousands of documents while still meeting performance, usability, and system maintenance goals.

Next Steps

I hope that this post provided a good overview of how to plan and maintain large repositories. It is a big topic with lots of nuances and techniques that are learned over time in the trenches. If your group is struggling with designing and managing large repositories and needs help, reach out and setup a consultation. We can either assist your team with advisement services, or help with the implementation of a robust system.

Can We Help?

Contact us today for a free consultation!