This article is reposted from my old MSDN blog. Please post comments here as I am no longer able to publish or respond to them on MSDN. Thanks!

---

Microsoft is cleaning house. Now that it has to maintain SharePoint for thousands of enterprises and millions of users in Office 365, Microsoft is working to clean up all the odd and messy bits of its flagship collaboration product. In a recent training course on Microsoft Virtual Academy, Microsoft urged developers to change the way they package and deploy their code in order to clean up a mess that has been building since 2003.

In this case the problem doesn’t really affect the customizations themselves (though most existing customizations are not cloud-ready); instead, the change is with the way custom solutions are installed into SharePoint and deployed in SharePoint sites. Instructors Vesa Juvonen and Steve Walker were careful to say they aren’t deprecating anything (at least not now) – but they admitted to some design shortcomings in SharePoint’s Feature framework and encouraged everyone to adopt a different approach.

The new approach eliminates a lot of problems that affect SharePoint upgrades and migration, and that can introduce quirky behavior and broken content if everything isn’t done perfectly. That’s the good news. The other news is that where the tools for the old approach are mature and familiar to SharePoint developers, there is virtually no tooling for the new one, just a collection of code samples at this point. So adopting the new model will be more costly until better tools are available.

This article will summarize the changes and analyze their impact on SharePoint developers and customers.

The Big Change

In technical terms, Microsoft is recommending that developers stop using SharePoint’s Feature framework and list, web, and site templates in their solutions. The Feature framework was added in SharePoint 2007, and allows site administrators to activate “Features” that provision content such as site columns, content types, lists, files, web part definitions, and all sorts of other things in SharePoint. List, web, and site templates are similar, except that a whole list or site is created. All of this is enabled by an arcane set of XML schemas called CAML, or Collaborative Application Markup Language. Now, instead of defining SharePoint content in CAML, Microsoft wants everyone to start creating content programmatically using a pattern called remote provisioning.

Let’s face it, Features and Templates are flaky. Activate a feature and things “light up” in SharePoint; that’s the cool part. However when you deactivate a feature, the content it created might persist, go away, or just break. Versioning and updates are black art. If an admin forgets to deactivate a feature before uninstalling the code that supported it (and it might have been activated in thousands of sites), the feature is “orphaned” resulting in errors and upgrade headaches. And perhaps you’ve noticed that if you create a site from a template and then change the template, the site doesn’t pick up the change. Over time all these problems add up and users just blame SharePoint.

Microsoft has seen the error of their ways and wants developers to stop using CAML-based deployment and instead use a pattern called “remote provisioning” in which a remote process is used to create SharePoint content ranging from from sites to site columns. Actually this pattern isn’t new, it’s been available as long as there have been remote API’s to create content; it’s just that all the tooling and MSDN documentation pointed towards using Features and Templates instead. Here are some examples of remote provisioning:

• .NET Code running in a Provider Hosted App using a client API (CSOM or REST) to create content in SharePoint. The Patterns and Practices team chose this for their large collection of samples.
• .NET code running in a console application using a client API (CSOM or REST)
• Client-side calls made from PowerShell (here is a Codeplex project that may help)
• Client-side calls (REST or JSOM) made from Javascript in a SharePoint Hosted App
• The Mechanical Turk approach: a person manually creates content using a web browser

The remote provisioning advice has been coming from Microsoft since last summer, but the Virtual Academy training is by far the strongest in telling developers to stop using features and templates. The main focus of the course was on transitioning from full-trust “farm” solutions to cloud-ready approaches based on the “app model.” The instructors played fast and loose with the term “app model”, extending it to mean nearly any approach that runs code outside of SharePoint and avoids the Features and Template packaging. Developers would be well advised to watch the course in its entirety, and to dig into the Patterns and Practices wiki and Yammer group. The training includes many live demos and code walk-throughs on Remote Provisioning and the reasoning behind the changes.

Farm Solution Roadmap

Existing SharePoint customers may be comforted that Microsoft reiterated its plans to continue to support farm solutions for the foreseeable future, but only on premises. The instructors offered detailed advice on developing farm solutions in order to avoid the problems with Features and Templates:

• Provision content types and site columns in code rather than using Features. The big problem with this is that when content type or site column is created by a farm solution Feature, the definition is stored directly on web servers instead of in the content database. Thus if the feature is removed, or the content is connected to a farm that doesn’t have exactly the same solution and feature installed, all lists and libraries using the content types and site columns will break.
• Avoid list templates. This is awkward advice because Microsoft introduced a new list template designer in Visual Studio 2013; clearly this change in direction is a very recent one. The problem with list templates is that they are dependent on a file called schema.xml which is stored on web servers; if the solution is removed, all lists based on the templates will stop working. Instead of using list templates, build out the list in code running in a feature receiver or use remote provisioning.
• Avoid custom field types. This has been the advice for a while now; it’s too bad because custom field types were really cool (they allow you to create a new type of content in SharePoint).

What Microsoft is trying to do is remove problems in which SharePoint content is invalidated when it gets out of sync with a particular set of solutions installed on a SharePoint farm. These problems make it difficult or impossible to upgrade SharePoint, and lead to big challenges with disaster recovery, when content is restored to a new SharePoint farm from backup or in a DR replication scenario.

Sandboxed Solutions Roadmap

The future of sandboxed solutions is extremely doubtful at this point. You may recall that sandboxed solutions were officially deprecated in SharePoint 2013, but then Microsoft recanted and said that only the ability to run custom server code would be discontinued. In the class, one of the top recommendations is to avoid sandboxed solutions, not only the custom server code but sandboxed solutions of any kind. They pointed out problems with orphaned objects that are left behind when sandboxed solutions are retracted.

This is a little awkward, because Microsoft has been using sandboxed solutions in support of newly introduced features such as the Design Manager, a branding tool introduced in SharePoint 2013. Steve Walker took a hard line nonetheless, and hinted that the sandbox would eventually be shut down once and for all. (Skip to 43:40 in the second video to hear it directly.)

Branding Guidance

During the Virtual Academy class, Microsoft provided quite a bit of branding guidance. With the exception of the new Office 365 themes, there wasn’t a lot new here, but the advice bears repeating because it once again relates to issues with SharePoint upgrades.

The traditional way to brand a SharePoint site is to change its master page, but master page changes generally do not survive SharePoint upgrades. This isn’t news; Microsoft changes the look and feel in every version of SharePoint, and master pages have needed a rewrite every time. (In many cases the old master page still works, but hides all the added functionality in the new version of SharePoint).

The problem is worse in Office 365, since new versions arrive more frequently. Microsoft has already changed the master page three times since 2013; if you had written a new master page, you would have missed the improved navigation and Office 365 app launcher.

The advice is to take as light of a touch as possible. Here are the options from lightest (and least flexible) to heaviest (the very flexible master page):

1. Consider not branding your site. “You do not brand Outlook or Word, why do you need to do branding on collaboration sites?”
2. Use Office 365 Themes. Changing the theme in one place will change it on every SharePoint site as well as in other Office 365 products such as Outlook Web Access and Delve. You can include a logo, URL for clicking the logo, background color, and colors for an Office 365 theme.
3. SharePoint Themes. These affect only one SharePoint site, so the need to be changed in every site. This could be automated through a PowerShell script or custom code. You may find the SharePoint Color Palette Tool helpful in creating SharePoint themes.
4. Alternate CSS. With this strategy, a developer builds a custom style sheet that is added to every page in SharePoint. Using this technique you can change colors and fonts, and to move things around on the page.
   Microsoft began allowing the alternate CSS to be set using the client API (CSOM) in March 2014 online, and in the April 2014 CU for SharePoint 2013. The Patterns and Practices group is working with the SharePoint engineering team to lock down some consistent element ID’s and classes that will not change across new versions of SharePoint so an alternate CSS file won’t break as SharePoint is upgraded.
5. Custom Master Page. This allows major changes such as introducing responsive design or making the site “not look like SharePoint.” However there is an ongoing need to tweak or rewrite the master page as SharePoint upgrades occur.
   This is especially problematic when master pages are installed into individual site collections, which is the only option in Office 365. If the master page is in an on-premises farm solution, it can be updated centrally, but if it’s placed into each site’s content then every site collection needs to be upgraded when changes occur.

Microsoft was clear that custom master pages are still supported. They called them a “tax” however; the responsibility of keeping the master page in sync with SharePoint belongs to the customer, not to Microsoft. In Office 365 where changes are ongoing and master pages are distributed, this will be an ongoing maintenance cost for the customer.

Relationship to the SharePoint and Office 365 App Models

The instructors used the term “app model” to mean nearly any approach that runs code outside of SharePoint and avoids the Features and Template packaging, however there is an important distinction between the App models and solutions that reside within a SharePoint site!

Apps run alongside of sites – they’re isolated because the code comes from a store and was written by who knows who, so they’re less trusted. All three app models (SharePoint Hosted Apps, Provider Hosted Apps, and Office 365 Apps) provide this isolation. The isolation can be limiting; it makes many scenarios such as SharePoint branding impossible to run in an App, and it means that App Parts run within IFrames, which bring their own set of challenges.

The twist here is that rather than running the customizations in an app, the app becomes an installer that places the customization into a SharePoint site where it can run unfettered. So rather than presenting an App part, an app could upload a .webpart file to the site’s web part gallery, and install JavaScript and other files that make the web part run directly within the SharePoint site.

This idea of using an App as an installer has been around for years actually; I used to call it “content injection” but now they call it “remote provisioning.” (As far as I can tell the first thought leadership in this area was from Doug Ware; he was blogging on the topic during the SP2013 Beta. See this article and others in the series.) You probably won’t see this kind of app in the Office Store as it requires too much permission; it’s a pattern to use within an enterprise. Just keep in mind that you don’t need to use an app to do the installation; it could be a PowerShell script, a console app running in an Azure web job or Windows scheduled task, or really anything that is remote and provisions sites and content in SharePoint.

Don’t Panic

No, SharePoint is not dead, Apps are not dead, and the Earth will continue to spin on its axis for the foreseeable future.

If you build or use custom SharePoint solutions, you don’t have to change anything right now. The top-line advice from Microsoft was to move gradually to the app model and remote provisioning. But you should pay attention because the existing way of deploying content and customizations is really problematic, especially when SharePoint upgrades occur.

It’s probably OK to continue to use the old methods on existing projects; they could be converted later and the tooling is likely to improve over the next couple of years. Right now remote provisioning requires extra development work compared with the Feature framework, mainly due to the excellent tools in Visual Studio for building Features. So there’s a tradeoff between doing the extra work now and waiting for better tools to arrive. In any case, you should be aware of the new approach and try to favor it in any new customization projects.

(Cross-posted to the BlueMetal Blog)