Colleague and fellow SharePoint developer Brian McCullough asked a great question on Twitter:

This is such an important topic I thought it was worth more than a 140-character reply; hence this article. Whole books have been written on Application Lifecycle Management,and there are many facets beyond simple software deployment, but this article will focus on the baseline needs of a typical “widget” project:

• Allow reliable and repeatable movement from a development environment to test and ultimately into production potentially across dozens or thousands of SharePoint sites
• Support this not just once, but repeatedly as project versions are developed and released. If the project includes persistent data, this should be preserved (and perhaps enhanced) as part of the upgrade process

In a widget project, each “environment” may just be a couple of site collections; for example you could have site collections for dev, staging, and production all in the same tenant on farm. Since widgets run on the client side, there’s no need for them to interfere. If you’re on Office 365, you might want a First Release tenant to test your code with forthcoming updates to the platform while keeping your main staging and production environments on the standard release schedule.

Deployment in Two Parts

The first key concept is that the widget project is installed in not one but two parts:

(1) The files that are shared by all instances
(2) The files that are installed along with each instance

Old-school SharePoint “farm” solutions allowed for this. A web solution package (.wsp) would include some files to be deployed centrally, such as .dll files to put in the global assembly cache and images, css, and other supporting files that landed in the SharePoint “hive” or root folder. Then, when a feature was activated, it would copy some files, such as a .webpart file, into each site where the solution was to be used. This not only saves space, but it allows the central files to be updated in one place. If a code change is made to such a web part, for example, updating the .wsp file will update it on every site and page where the web part was used. We lost that in sandboxed solutions, where the .wsp file was installed in each site collection, leading to a need to upgrade it in every site collection when a new version came along. The nice thing was that SharePoint gave used one .wsp file containing both the shared and per-instance files.

The new SharePoint framework (SPFx) does this as well, and allows sharing the bulk of a solution among multiple SharePoint sites or even multiple farms and tenants. When an SPFx project is built, two files are produced: a bundle containing the code, HTML, and CSS to be deployed centrally (generally in a Content Delivery Network or CDN), and an .spapp file that contains the metadata needed to install each instance of the solution. The .spapp file is deployed via the App Catalog, and SharePoint uses this to determine how to display an individual instance of an SPFx web part (or, eventually, single-page application or list form).

A widget project can do the same thing. A good example is the Microsurvey sample included in the Widget Wrangler github repo. This sample is more complex than it needs to be because it was designed to show three approaches: deployment as a SharePoint Hosted Add-in, a drag-and-drop deployment (in which all files are stored in the site where they’re used), and a central deployment (which divides the files into two parts to support a simple ALM process). I regret not having provided three simpler samples and jamming all three scenarios into one, because I find that developers get confused by this. But I don’t have time right now to clean it up, so let me guide you into the sample and clarify how the ALM scenario works.

If you were to drag all the files in the SurveyApp folder into a SharePoint document library (specifically a folder called SurveyApp within the Site Assets library), the Microsurvey just works like that, and provides a survey web part and a set of forms for viewing and editing survey questions. However it’s also possible to deploy those files centrally, and just put the widget references into each site. I apologize for my naming here:

SurveyAppCentralDeploy – contains the files that must be copied to each site (I was referring to the central deployment scenario here, but it’s confusing that these are not the files that are centrally deployed. they’re copied to each site)

SurveyApp – contains the files that can be placed centrally (except for some duplicates used in the drag-and-drop scenario.)

I feel like I’ve created another SPSite/SPWeb type confusion here in my own demo, but please bear with me. If you look at the files, you can see that the per-instance files in SurveyAppCentralDeploy are very minimal and pretty much just include the Widget Wrangler tag. For example, WebPart.html.template contains the markup to host the widget, but none of the widget itself.

<div>
	<link rel="Stylesheet" type="text/css" href="%AppPath%/Survey.css" />
<table>
<tr>
<td class="webPart">
<div style="width: 400px; height: 250px;"
ng-controller="main as vm"
ng-include="'%AppPath%/main.html'"></div></td>
</tr>
</table>
<script type="text/javascript" src="%AppPath%/pnp-ww.js" 
 ww-appName="microSurvey" 
 ww-appType="Angular"
 ww-appScripts='[{"src": "https://ajax.googleapis.com/ajax/libs/angularjs/1.4.5/angular.min.js", "priority":0, "test": "false"},
 {"src": "%AppPath%/mainController.js", "priority":1, "test": "false"},
 {"src": "%AppPath%/settingsController.js", "priority":2, "test": "false"},
 {"src": "%AppPath%/listFormController.js", "priority":3, "test": "false"},
 {"src": "%AppPath%/surveyService.js", "priority":4, "test": "false"},
 {"src": "%AppPath%/spDataService.js", "priority":5, "test": "false"}
 ]'>
 </script>
</div>

This is the file that a Content Editor Web Part points at to display a Microsurvey. I’ll explain the .template bit in a later section; for now just realize that these are the files that have to be local to each site because of the way SharePoint works:

• Default.aspx – the Settings page
• ListDisplayForm.aspx – the Display form for viewing a survey question
• ListEditForm.aspx – the Edit form for editing a survey question
• ListNewForm – the New form for adding a new question
• Microsurvey.dwp – a file that gets defines a content editor web part, suitable for importing into a page or the web part gallery
• WebPart.html – the html referenced by the content editor web part

These six files are unlikely to change; the Microsurvey could turn into a megasurvey and they would stay the same. So it’s not so bad to copy them into every site. And SharePoint won’t let you define a list form, page, or web part outside the site collection, so there isn’t much choice.

These files are duplicated in the SurveyApp folder because of the drag-and-drop scenario, but they’re not really needed, and the install script won’t put them in the central location. The rest of the SurveyApp folder is where the meat of the solution is; there are two Angular controllers, two services, and a bunch of supporting files including the Widget Wrangler itself. Those are the files that are likely to need updating to fix a bug or add a new feature.

I like to tell the story of the day the BlueMetal Intranet got faster. My colleague and Widget Wrangler collaborator Julie Turner had written some widgets that were in nearly every site on our Office 365 intranet (hundreds, maybe thousands of sites). She found a more efficient way to do something, and of course it was in the centrally deployed code. She updated the files and – bang! – every site got the optimized code. Users might have noticed that the pages got a little snappier on every site. Try that with a drag-and drop or sandboxed solution – you’d be visiting every site collection to deploy the update.

Where is the central location?

I typically create a site collection just for storing these centrally hosted scripts, which I call the “scripts site”. That makes SharePoint something like a Content Delivery Network. Julie wrote a whole article on just that, where she guides you through the process of deciding where this should go, and she astutely uses SharePoint versioning to version her code. This is a really powerful concept! You can use SharePoint as a source control system. In fact, by using Approvals, and giving your test users permission to see unapproved versions, you can actually have a widget render in a tested (approved) version for most users while the testers are testing a new version, all using the same URL’s. Crazy stuff!

One thing to keep in mind, however, is that SharePoint Online is less efficient at doing this than SharePoint on premises, and that’s because the file (BLOB) caching in SharePoint is kind of useless in Office 365 where SharePoint farms have hundreds of servers. Please check out my article on caching in SharePoint Online;  see the section on Static Content.

Yet another consideration is cross-origin browser rules. If you put the files in SharePoint using the same hostname as where they’re used, you’re all set. But if the hostname of the central location is different than the sites which use it, you might run into cross-origin browser issues. These are the rules that prevent a site from reaching across domains and acting on your behalf in, say, your banking site. Most widget artifacts, such as scripts, images, and css files aren’t a problem, but anything that gets pulled in using an ajax call is. A notable example of this is an HTML template in Angular, used in ng-include or a custom directive. (In SPFx projects, it’s easy to put these in the WebPack bundle, and thus avoid the issue). Anyway if you have these files and need them work across hostnames, you’ll need to set up a CORS policy on the site that hosts the central files.

Bottom line is you can put the central files anywhere you wish. In most cases it’s not a big deal; just set up a SharePoint site or grab some space on a CDN or web server – and make sure to have locations for dev, staging, and production.

Referencing the Central Location

One thing about this whole scheme, however, is that we need a way for the widget to find the central files that provide its implementation. SPFx does that for you (the location is in a manifest in the .spapp file) – but with simple widgets you’re on your own.

There may be one location for each developer, one for each test or staging environment, and one for production, or you might have a single location like Julie’s versioned document library. In any case, you don’t want to be hard-coding the central location into your solution code.

To handle this, I use a simple template scheme in which I append “.template” onto the end of each file that references the central location. You’ll want to set your code editor to understand this; I recommend Scott Addie’s article on setting up associations in VS Code. It’s possible to set “.js.template” to indicate a JavaScript file, “.css.template” to indicate a css file, etc., so you don’t lose any of the cool editing features like code highlighting and intellisense.

Within these template files, I insert tokens to indicate the location of the shared scripts. In the Microsurvey sample, I used the token %AppPath% to indicate the location of the shared scripts. If you scroll up to the code sample, you’ll see plenty of %AppPath% tokens are used to resolve this location. While you’re at it, make additional tokens as needed to handle any other differences between dev, staging, and production, such as the name of a list, an RESTful endpoint, etc.

The install script replaces these tokens with their values in PowerShell; for example:

if ($item.Name.EndsWith(".template"))
{
# Remove .template from file name
$itemFinalFullName = $item.FullName.Replace(".template", "")
$itemFinalName = $item.Name.Replace(".template", "")

# Replace tokens in template to build file
(Get-Content $item.FullName) |
Foreach-Object {$_ -replace "%AppPath%", $spScriptPath} |
Set-Content $itemFinalName

# Upload the file
Add-SPOFile -Path $itemFinalName -Folder $spAppPath
Write-Host "Deployed file: $itemFinalName"

# Clean up the file with the tokens replaced
Remove-Item $itemFinalName
}

The Microsurvey example is kind of primitive; I’ve gotten better at this over time with my widget projects, but this should give you the essence. Generally there will be at least two install scripts; the convention I use is:

• Add-Mywidgetsolution.ps1 – Installs an instance into a site (run once per site)
• Install-Mywidgetsolution.ps1 – Installs the central files (run once for all sites)

If these scripts can’t be run repeatedly to update the solution, you might want to include some update scripts as well.

Managing Environmental Settings

So now the application has been split into shared and per-instance files, and they’re all talking to one another. But how do you manage the values that are different in each environment (dev, staging, and production)? For example, when you install your application in production, how does it know to point to the central files for production and not staging?

An obvious solution is to make the central location an argument to the installation scripts, but I feel this is error-prone. What’s the point of testing your code in staging if a typo can be introduced when installing it into production?

Another approach I’ve seen a lot is to place these values at the top of the installation script. That’s a little less error prone, but it’s still possible to make a mistake, and it sometimes requires editing the installation scripts after they’ve been tested. That really bothered me, so I came up with a simple solution, which is to isolate the per-environment values in a really simple little script that isn’t changed between versions. I usually call this script Get-Settings.ps1, because, well, it gets the settings.

Here’s an example:

#
# Settings for MicroSurvey Installer
#
[PSCustomObject]@{

ScriptSiteUrl = "https://bgtest14.sharepoint.com/sites/DemoScripts/";
ScriptLibrary = "MicroSurvey";
AppTitle = "MicroSurvey";

}

Each installation script reads the values into an object called $settings, where the values can be accessed as needed.

$settings = .\Get-Settings.ps1
# ...
$spScriptPath = $settings.ScriptSiteUrl + $settings.ScriptLibrary

Now it’s possible to leave a copy of Get-Settings.ps1 in each environment, and update the new Add, Install, Update, and other PowerShell scripts along with each new release. That way the Update-Microsurvey.ps1 script, for example, is fully tested in staging and runs exactly as it is in production.

Versioning

When I wrote the Microsurvey example I was stil learning how to do this, and its support for version updates is less than ideal. Some day I’ll fix all that, and also remove the other oddities such as separating out the various scenarios. But until then, let me just pass on some techniques I’ve learned along the way to make it possible to version your widget-based solutions.

1. Make the installation scripts handle repeated operation. For example, your installation script might create a folder to put your widget code in, or it might create a SharePoint list that’s needed by your solution. The solution can be as simple as adding “-ErrorAction SilentlyContinue” to key lines in your script, or may involve carefully checking to see what’s already there when creating something.

2. Something has to provision the storage used by your solution; this is often the installation script, but in some cases it’s useful to do it in Javascript, right inside the solution. The Microsurvey demonstrates both methods. The installation script creates the needed lists and wires up the custom forms, or, if you use the drag and drop method, visiting the default.aspx page provisions all the storage from JavaScript. This provisioning code needs to be smart enough to upgrade the storage from any previous version as well as to create everything from scratch in a new site.

3. Consider and test upgrading instances where user data is already present. For example, if I was going to add the option of having an image for each answer to the Microsurvey, I’d want to test adding that capability to a site that already had questions with text answers already in place

Postscript

This is a work in progress; I do plan to improve the Microsurvey example with some of the things I’ve learned since I wrote it. And there will be a whole new set of practices to learn for the SharePoint Framework; they haven’t yet provided a lot of guidance on ALM and versioning for SPFx solutions.

Do you have suggestions or techniques to share? Please add a comment and share!

Thanks!