Tuesday, March 6, 2018

Sitecore 9 configuration roles: content management, reporting, processing

Sitecore 9 configuration roles

A while ago I had to setup a few Sitecore servers with a topology as:
  • Content Delivery
  • Content Management (and all the rest)
In Sitecore 9 things have changed a little bit when setting up server roles. As we could read in the documention we have these roles at our disposal:
  • ContentDelivery
  • ContentManagement
  • Processing
  • Reporting
  • Standalone
For our CD server, the choice was easy: we set the server to the ContentDelivery role.
For our CM server, it was not that obvious. As the documentation mentioned combining roles is possible and it even mentions "ContentManagement, Processing, Reporting" as an example we though this would be a good idea. 

ContentManagement, Processing, Reporting

Unfortunately immediately after setting these roles, the server crashed. We noticed that we had to set remote settings for processing and reporting server which seemed very weird as our processing and reporting was not remote. 

I ended up asking this on SSE and also to Sitecore Support. Support logged this as a bug and gave us the solution - details on this can be found on SSE, so no need to copy them here.

With these changes, we had no need to set any remote settings. And the site worked!

But a few wise men asked me: "why are you doing this?". The only answer I had at that time was: "because I can"..  which is maybe not the best answer when explaining a server setup :)


Before Sitecore 9 and the configuration roles we would have had a setup for this CM server that is now similar to a Standalone. Because that was easy. And disabling and enabling files was not...

So why not use standalone role now? Well, we assumed that as this server is not a content delivery one we would benefit from defining the roles. But just assuming is as wicked as using roles just because you can...

Compare configs

So let's compare the resulting configs.

There were quite some differences actually. I won't go into all the details as that is probably just boring but let's focus on some main points: [SO = standalone, CMPR = ContentManagement,Processing,Reporting]
  • databases: in a SO setup, core and master are set as default databases, in a CMPR setup these are all set to web
  • tracking: in a CMPR setup entries are removed regarding RobotDetection, form dropouts, session commits, content testing,..  
  • EXM: a lot of entries removed in CMPR, especially on tracking
So without going into details, I would assume that when not running in Standalone the website running on the ContentManagement server might not be doing your tracking and testing correctly and if entrying from a mail you'll also miss out.

But.. is this important? Will it make the server more performant? 
I think not. 

So what do you need to do? Well, you have a choice..  Also consider if you are using the server as a true test environment (using an extra publishing target perhaps) you probably do want ContentDelivery enabled, meaning a standalone setup.

The easy choice is: set it to standalone.
But.. if you would have separate Processing and/or Reporting roles, would you add ContentDelivery as a role to the ContentManagement server? Probably/maybe not.. so why do that when those roles get combined?

Because it's easy... :)


Wednesday, February 7, 2018

Displaying extra data on a Sitecore 9 Goal

Goals in Sitecore 9

In our project we had goals set when a whitepaper was downloaded. This works fine, but it would be much nicer if we could show which whitepaper was actually downloaded. All the whitepapers have an identifier code, so the challenge was to add this identifier to the goal data.

Capturing custom goal data

Capturing the data is very well explained in the official documentation of Sitecore. We used the "Data" and "DataKey" properties of the goal:
  • DataKey: a key that identifies the contents of  "Data" - e.g. 'Whitepaper code'
  • Data : any data collected as part of triggering the event - in our case the whitepaper identifier
This all works fine and we can see the data in the database (xDB shards) as part of the events.

Displaying custom goal data

Data is nice, but you need a way to show it. It wasn't immediatly clear how to do this, so I asked it on Sitecore Stack Exchange.
Jarmo Jarvi pointed me in the good direction by mentioning a tremendous blog post by Jonathan Robbins. The blog post was based on Sitecore 8 - the idea behind my post here is to show the differences when doing this in Sitecore 9 and xConnect. The result will be the same:
an extra column in the goals section of the activity tab of a contact in the Experience Profile


The ExperienceProfileContactViews pipeline is where all the magic happens. Adding the extra column to the results table is identical to the description for Sitecore 8. Fetching the data (in GetGoals) however is quite different as we have to use xConnect now:
public class AddGoalDataColumn : ReportProcessorBase
  public override void Process(ReportProcessorArgs args)

public static class Schema
  public static ViewField GoalData = new ViewField("GoalData");

public class FillGoalData : ReportProcessorBase
  public override void Process(ReportProcessorArgs args)
    var resultTableForView = args.ResultTableForView;
    Assert.IsNotNull(resultTableForView, "Result table for {0} could not be found.", args.ReportParameters.ViewName);
    var i = 0;
    foreach (var row in resultTableForView.AsEnumerable())
      var goalData = args.QueryResult.Rows[i].ItemArray[4];
      if (goalData != null)
        row[Schema.GoalData.Name] = goalData;

public class GetGoals : ReportProcessorBase
  public override void Process(ReportProcessorArgs args)
    var goalsDataXconnect = GetGoalsDataXconnect(args.ReportParameters.ContactId);
    args.QueryResult = goalsDataXconnect;

  private static DataTable GetGoalsDataXconnect(Guid contactId)
    var goalsTableWithSchema = CreateGoalsTableWithSchema();
    var contactExpandOptions = new ContactExpandOptions(Array.Empty<string>())
      Interactions = new RelatedInteractionsExpandOptions("WebVisit")
        StartDateTime = DateTime.MinValue

    FillRawTable(GetContactByOptions(contactId, contactExpandOptions).Interactions.Where(p => p.Events.OfType<Goal>().Any()), goalsTableWithSchema);
    return goalsTableWithSchema;

  private static DataTable CreateGoalsTableWithSchema()
    var dataTable = new DataTable();
      new DataColumn("_id", typeof(Guid)),
      new DataColumn("ContactId", typeof(Guid)),
      new DataColumn("Pages_PageEvents_PageEventDefinitionId", typeof(Guid)),
      new DataColumn("Pages_PageEvents_DateTime", typeof(DateTime)),
      new DataColumn("Pages_PageEvents_Data", typeof(string)),
      new DataColumn("Pages_Url_Path", typeof(string)),
      new DataColumn("Pages_Url_QueryString", typeof(string)),
      new DataColumn("Pages_PageEvents_Value", typeof(int)),
      new DataColumn("Pages_Item__id", typeof(Guid)),
      new DataColumn("SiteName", typeof(string))

    return dataTable;

  private static void FillRawTable(IEnumerable<Interaction> goalsInteractions, DataTable rawTable)
    foreach (var goalsInteraction in goalsInteractions)
      foreach (var goal in goalsInteraction.Events.OfType<Goal>())
        var currentEvent = goal;
        var row = rawTable.NewRow();
        row["_id"] = goalsInteraction.Id;
        row["ContactId"] = goalsInteraction.Contact.Id;
        row["Pages_PageEvents_PageEventDefinitionId"] = currentEvent.DefinitionId;
        row["Pages_PageEvents_DateTime"] = currentEvent.Timestamp;
        row["Pages_PageEvents_Data"] = currentEvent.Data;
        row["Pages_PageEvents_Value"] = currentEvent.EngagementValue;
        var dataRow = row;
        const string index = "SiteName";
        var webVisit = goalsInteraction.WebVisit();
        var str = webVisit?.SiteName ?? string.Empty;
        dataRow[index] = str;
        if (currentEvent.ParentEventId.HasValue)
          if (goalsInteraction.Events.FirstOrDefault(p => p.Id == currentEvent.ParentEventId.Value) is PageViewEvent pageViewEvent)
            row["Pages_Item__id"] = pageViewEvent.ItemId;
            var urlString = new UrlString(pageViewEvent.Url);
            row["Pages_Url_Path"] = urlString.Path;
            row["Pages_Url_QueryString"] = urlString.Query;


  private static Contact GetContactByOptions(Guid contactId, ExpandOptions options = null)
    using (var client = SitecoreXConnectClientConfiguration.GetClient())
      if (options == null)
        options = new ContactExpandOptions(Array.Empty<string>())
          Interactions = new RelatedInteractionsExpandOptions("IpInfo", "WebVisit")

      var contactReference = new ContactReference(contactId);
      var contact = client.Get(contactReference, options);
      if (contact == null)
        throw new ContactNotFoundException($"No Contact with id [{contactId}] found");

      return contact;

One more change is the configuration. It is almost the same as in Sitecore 8, but we don't need to change the goals-query anymore - instead we have to patch the GetGoals in the same ExperienceProfileContactViews pipeline.
    <group groupName="ExperienceProfileContactViews">
          <processor patch:after="*[@type='Sitecore.Cintel.Reporting.Contact.Goal.Processors.ConstructGoalsDataTable, Sitecore.Cintel']"
             type="MyNamespace.AddGoalDataColumn, MyProject" />
          <processor patch:after="*[@type='Sitecore.Cintel.Reporting.Contact.Goal.Processors.PopulateGoalsWithXdbData, Sitecore.Cintel']"
             type="MyNamespace.FillGoalData, MyProject" />
          <processor patch:instead="*[@type='Sitecore.Cintel.Reporting.ReportingServerDatasource.Goals.GetGoals, Sitecore.Cintel']"
             type="MyNamespace.GetGoals, MyProject" />

Final step - core database

As Jonathan mentioned, we do need to add a new item in the core database to display our new column in the list of Goals. Create this item under the path /sitecore/client/Applications/ExperienceProfile/Contact/PageSettings/Tabs/Activity/Activity Subtabs/Goals/GoalsPanel/Goals and set a descriptive HeaderText and a DataField value that matches the schema name.

Your result should look like this:

I woud like to thank Jonathan Robbins for his original blog post and Jarmo Jarvi for his answer on SSE.

Monday, February 5, 2018

Display custom contact facets in Sitecore 9 Experience Profile

Custom contact facets

Creating custom facets is something that pops up quite often when customers start using Sitecore xDB and contact data. Sitecore 9 changed lots of things for developers working with xDB. The new xConnect layer is a wonderfull thing but when upgrading to Sitecore 9, the odds that you will need to rewrite some/all your code regarding xDB are not looking good.

Although.. you might/should see this as an opportunity to re-think what has been implemented before.

I want to focus on one particular part in this post. We had some custom facets implemented. Doing this with xConnect in Sitecore 9 is well documented on the official Sitecore documentation site.

Experience Profile

But when you have custom facets you usually also want to display this information in the Experience Profile. We used to do this based on some blog post from Adam Conn and/or Jonathan Robbins.
When upgrading to Sitecore 9, we had to made some small changes to this code. Especially (and obviously) the part where we actually do a query to fetch the data from xDB.

Contact View pipeline

As you can read in the previously mentioned blog posts the contact view pipeline is where alle the magic happens in 3 steps:
  1. Creating a data structure to hold the results of the query
  2. Executing the query
  3. Populating the data structure with the results of the query
The part where our changes are located can be found in the execution phase. Our processor will be in a pipeline referred to by a build-in Sitecore processor:
<group groupName="ExperienceProfileContactDataSourceQueries">
      <processor type="MyNamespace.MyDataProcessor, MyNamespace" />

<group groupName="ExperienceProfileContactViews">
    <processor type="Sitecore.Cintel.Reporting.Processors.ExecuteReportingServerDatasourceQuery, Sitecore.Cintel">
      <param desc="queryName">my-custom-query</param>
Let's focus on getting the data.

Query Pipeline Processors

The query pipeline processor is the part that executes the query - gets the data. It is (still) a processor based on ReportProcessorBase.

We will be using the model that was created with the custom facet. For the example we have:
  • custom facet: "CustomFacet"
  • property "Company" in this facet (type string)
The code for MyNamespace.MyDataProcessor:

public override void Process(ReportProcessorArgs args)
  var table = CreateTableWithSchema();
  GetTableFromContact(table, args.ReportParameters.ContactId);
  args.QueryResult = table;

private static DataTable CreateTableWithSchema()
  var dataTable = new DataTable() { Locale = CultureInfo.InvariantCulture };
    new DataColumn(XConnectFields.Contact.Id, typeof(Guid)),
    new DataColumn("Custom_Company", typeof(string))

  return dataTable;

private static void GetTableFromContact(DataTable rawTable, Guid contactId)
  string[] facets = { CustomFacet.DefaultFacetKey };
  var contact = GetContact(contactId, facets);
  var row = rawTable.NewRow();
  row[XConnectFields.Contact.Id] = contactId;

  if (contact.Facets.TryGetValue(CustomFacet.DefaultFacetKey, out var customFacet))
    row["Custom_Company"] = ((CustomFacet)customFacet)?.Company;


private static Contact GetContact(Guid contactId, string[] facets)
  using (var client = SitecoreXConnectClientConfiguration.GetClient())
    var contactReference = new ContactReference(contactId);
    var contact = facets == null || facets.Length == 0 ? client.Get(contactReference, new ContactExpandOptions(Array.Empty<string>())) : client.Get(contactReference, new ContactExpandOptions(facets));
    if (contact == null)
      throw new ContactNotFoundException(FormattableString.Invariant($"No Contact with id [{contactId}] found"));
  return contact;

When comparing this code to what we had in Sitecore 8, we have a bit more code but it all seems to make sense. Especially when  you get familiar with coding with xConnect.
What happens here in short:
  1. Create a dataTable
  2. Fetch the contact from xConnect with the necessary facet(s)
  3. Read the data from the contact and add it to the dataTable
  4. Set the dataTable in the pipeline arguments

Special thanks to Nico Geeroms who did the actual coding and testing on this one.

Monday, January 22, 2018

Sitecore Forms Send Email Campaign message

Sitecore Forms 9.0 Update-1 rev. 171219

Sitecore release it's Update-1 version of the platform. In this version they included EXM (Email Experience Manager) as out-of-the-box part of the product. No more separate module..  like they did with Forms before. This change also had some (expected) changes to the Forms module. In the initial release version there was no submit action that could send an email. We expected this to be introduced together with EXM and that was a correct assumption.

So.. we have a "Send Email Campaign message" submit action now.

Send Email Campaign message submit action

I tried to do a quick test on a vanilla install of the platform:
  1. Create an automated campaign in EXM (how-to)
  2. Test the campaign - yes, the test mail arrived perfectly.
  3. Create a form (any form will do), added a few fields and all wanted save actions, amongst which the "Send Email Campaign Message" - place it before the redirect ;)
    The save action will allow you to select a campaign - my newly created campaign was in the list so I selected that one.
  4. Add the form to a page (we had to create/generate an mvc layout for this*)
  5. Publish everything and let's try this...
Submit the form with some test data.. and.. damn..  "Failed to send email!
In the Sitecore logs I found more information:  ERROR Contact id is null.

As I am not that familiar with EXM (yet), as probably quite a lot of others, I was not aware that I could only send a mail to a known contact. Sitecore Support helped me on this one, so now I figured out that I do need to identify my contact first.
To send an email with the "Send Email Campaign Message" action, your contact needs to be identified.
Sound reasonable, but the (first) problem is that there is no out-of-the-box submit action to do this. Luckily all you need to do this yourself has been documented on the official doc site.  (still weird that they can document it, but not put it in the product...).

Documentation on the submit action could have saved me some time, so hopefully this small post will help someone.

Further usage

We did not find a way to send the form data in the message (without customizing the save action) - unless the form data is all in the contact data, which probably is not the case.

We also haven't found a solution to send the email to someone else - not the person who submitted the form. The mail will be send to the identified contact.


I must admit I was hoping Update-1 would have more impact on the Forms part of the product. I was also hoping the "Send Email" functionality would be in there. One could say it is, but without custom code is useless. Let's get our hopes up for Update-2...

* Sitecore Forms is MVC only - and the vanilla setup of Sitecore still comes with a default WebForms homepage 😞

Thursday, January 11, 2018

Sitecore Experience Accelerator SXA 1.6: Snippets

Sitecore Experience Accelerator 1.6

Together with Sitecore 9.0 update-1 we welcomed SXA 1.6.
One of the things I was really looking forward to are Snippets:
The Snippet rendering lets you create a reusable group of renderings. It is a composite rendering that consists of several renderings that can be designed separately in the Experience Editor.
Our analists asked for this a while ago as it makes life for editors potentially a lot easier. Partial designs are nice, but their content is fixed - you cannot alter anything on the page itself, not even switch datasources. With snippets you can now create your own "composite rendering" (in the end, that is indeed what it is) and reuse this on lots of pages. Unlike partials designs, you do need to place the snippet on the page yourself - it doesn't come automatically with the page design.

Create a grid based on (multiple) splitters and add other renderings in it. Save this bunch, reuse it, and be able to still adapt the content if needed. Oh yes, this will be used!

I installed SXA 1.6 on a vanilla Sitecore 9.0-1 (installing is still a piece of cake btw) and tried them out.

Creating a snippet

To add a snippet on your page, select Snippet from the Composites section in the toolbox (experience editor). Just as you would add any other rendering. Select the location -placeholder- where you want the snippet to appear and drag & drop.

You will get the screen to select the "Associated Content" or datasource. This is an item of type Snippet. You get 2 possible locations presented: a global folder for your site called Snippets and the local Data folder (which is located as a child underneath your current item). People working with Sitecore will recognize such a screen. 

If you want to reuse your snippet on other pages, make sure to put it in the global Snippets folder as your local Data one will not be available on other pages in your site. 

The screenshot shows an example after we had created a demo item in each folder.

Filling the snippet

You can add anything you want to the snippet. Just drag & drop all the desired renderings on it, enter content, add more datasources and so on. Just as if the snippet wasn't there. 

For my test, I just added a column splitter with a RichText component in the first column and a reusable RichText component in the second.

If you check the created items in the content editor you will notice that all was done as expected - the local datasources are underneath the snippet item in a Data folder and the reusable RT component datasource item was also on it's normal place in the global texts folder.

Reusing the snippet

Datasource Configuration

Before you start reusing the snippet, take a look at this section in the snippet item:

This is the datasource configuration which by default (standard values) will be set on "Do not copy".
You need to think about this one. I didn't find a way to set this in the experience editor by the way - if someone knows how to do that please share.. and if not possible the SXA team may put this on their backlog ;)

What are the options here:
  • Do not copy - use global datasource : if this is set when the snippet is reused, the datasources are not created locally but instead the snippet will refer to the original datasources. Meaning that if you change anything, it will also change on all other locations where the snippet is used. The snippets remain coupled.
  • Copy global data source to local context upon slection: if this is set when the snippet is reused, the snippet item and all the related datasources (it's children) are copied to the current item's local Data folder. Meaning that you can change all data within the snippet without affecting anything else. The snippets are not coupled anymore.
  • Ask user whether the copy of global data source to local context is required upon selection: seems obvious.. ask the user which of the above he wants.
I tried all three of them and will focus further on the last one. 

If this is set, and the user added his snippet a screen is presented to him/her as seen in the screenshot on the right. For me this was crystal clear, but I did get the comment that for a (simple) end user that might not be the case. Haven't been able to test that yet, so future will tell. Anyway, I added the snippet I created twice on a new page - once with "yes" and once with "no".

If I looked in the content editor all looked fine (at first sight). The items were created locally one - for the snippet where I selected "yes" as expected. Also, as expected, the reusable RichText component datasources were not copied - the remained in the global data folder.  In the experience editor all looked fine as well. 

Small issue
Too bad I did find a little issue though.. apparently in case of "yes" the datasource items were created fine, but were not set in the snippet rendering - so the snippet was still using the global ones. It does work when you use the "copy to local" settings, so the issue is only when letting the user choose. I created a ticket with Sitecore Support and will edit here when I get a fix. 


As I was really looking forward to it, it's a shame I found an issue - but as I'm sure the sxa team will fix this (and I will keep you posted). The feature has everthing I think I wanted so actually: Hooray for Boo.. no, snippets :)

Wednesday, January 10, 2018

Upgrading to Sitecore 9 update-1 (rev. 171219)

Notes from upgrading from a 9.0 initial release

I just upgraded a stadalone (vanilla) install of Sitecore 9.0 initial release (rev. 171002) -installed with SIF- to update-1 (rev. 171219). All went fine, no real issues but a few small tips might help...
Note that we had installed our initial version with SIF
If you have used SIF-less (a UI Wrapper for Sitecore Installation Framework) remember that your result will be the same as it uses SIF underneath.

Sitecore configuration files

When installing Sitecore with SIF, some of your parameters are set inside the Sitecore configuration files. This will give conflicts when upgrading to update-1. Luckily the Sitecore upgrade wizard is now smart enough to tell us this. I had conflicts in:
  •  App_Config\Sitecore.config: The 'dataFolder' variable has been manually modified (weird?)
  • App_Config\Sitecore\ContentSearch\Sitecore.ContentSearch.Solr.DefaultIndexConfiguration.config:  the url to our Solr instance was in here
  • App_Config\Sitecore\ContentSearch\Sitecore.ContentSearch.Solr.Index.Master.config
    App_Config\Sitecore\ContentTesting\Sitecore.ContentTesting.Solr.IndexConfiguration.config: <param desc="core">...</param> is in there and changed to $(id) instead of the name of the core - so if that doesn't match the name of the index it won't work anymore. Note that this is only changed for some indexes like master and core, not for others like web.
  • Web.config: the search provider was reset to Lucene - had to switch back to Solr

An mentioned Sitecore will warn you about these changes so no worries - just make sure that you do review the configs after the upgrade or you will get errors.

EXM install

EXM is "new" in this release (not really new, but it wasn't available in the initial release) and comes packed within the Sitecore box now. During the upgrade process you will need to deploy the EXM databases. It won't tell you how to set your SQL user though (or it assumes that we use the dbo - do not!). I tried to figure it out (see StackExchange).
My conclusion for the user in the connection string -at the moment- is: 
  • read/write rights on both databases
  • execute rights on the exm.master database

Tuesday, January 9, 2018

Custom Sitecore DocumentOptions with Solr

Almost 2 years ago I wrote a post about using custom indexes in a Helix environment. That post is still accurate, but the code was based on Lucene. As we are now all moving towards using Solr with our (non-PAAS) Sitecore setups, I though it might be a good idea to bring this topic back on the table with a Solr example this time.

(custom) indexes

I am assuming that you know about Helix, and about custom indexes. If you ever created a custom index you probably have used the documentOptions configuration section - maybe without noticing. It is used to include and/or exclude fields and templates and define computed fields. So you probably used it :)

And it wouldn't be Sitecore if we couldn't customize this...

Our own documentOptions

Why? Because we can. No..  we might have a good reason, like making our custom index definitions (more) Helix compliant. Normally your feature will not have a clue about "page" templates. But what if you want to define the include templates in your index? Those could be page templates.. or at least templates that inherit from your feature template. That is why I build my own documentOptions - to include a way to include templates derived from template-X.


So the idea now is to create a custom document options class by inheriting from the SolrDocumentBuilderOptions. We add a new method to allow adding templates in a new section with included base templates. This will not break any other existing configuration sections.

An example config looks like:
<documentOptions type="YourNamespace.TestOptions, YourAssembly">
    <include hint="list:AddIncludedBaseTemplate">
This looks very familiar - as intended. We create a new include section with the hint "list:AddIncludedBaseTemplate". The name 'AddIncludedBaseTemplate' will come back later in our code.



public virtual void AddIncludedBaseTemplate(string templateId)
  Assert.ArgumentNotNull(templateId, "templateId");
  ID id;
  Assert.IsTrue(ID.TryParse(templateId, out id), "Configuration: AddIncludedBaseTemplate entry is not a valid GUID. Template ID Value: " + templateId);
  foreach (var linkedId in GetLinkedTemplates(id))
    AddTemplateFilter(linkedId, true);
To see the rest of the code, I refer to the original post as nothing has to be changed to that in order to make it work on Solr (instead of Lucene).


To change the code from the Lucene example to a Solr one, we just had to change the base class to SolrDocumentBuilderOptions. 
We are now again able to configure our index to only use templates that inherit from our base templates. Still cool. And remember you can easily re-use this logic to create other document options to tweak your index behavior.