SolR

Introduce a (SolR) Sitecore Search Abstraction

After my previous post on Supporting Integrations, I received a few comments asking why was SolR was in the integration’s module group, as it is part of the sitecore API.

In this blog i will explain why and in more detail how to isolate a SolR integration.

Yes Sitecore Search is part of the Sitecore API, but it relies on an 3rd party system! Please read my previous post about why you need to identify, separate and isolate modules with external dependencies, as Sitecore Search API faces exactly the same challenges.

With the bonus that there are 3 supported implementations (Lucene, SolR and Azure Search) which are almost the same, but not quite!

Sitecore Search Issues

In most of the helix-based solution I have seen indexing is implemented in the framework layer which provides some helper extensions. Then each feature uses indexing module with Sitecore Search API to implement their requirements. This typically leads to the following issues:

  • Duplicated code across features
  • No clear definition of the indexing/constraints/sorting requirements for the solution.
  • Non-consistent implementation across the solution i.e. Predicate builder vs LINQ.
  • Optimization is difficult.

With each feature implement their indexing requirements, it leads to duplicated code as it feature needs to build the query to add sitecore root item, base templates, language etc. for each request, before adding the feature specific part of the query.

Therefore when fixing a bug or performance issues you must track down all the places where Search is used and then determine if they require the same fix and or the optimization.

How to abstract away the SolR Search Implementation

  • Identify the indexing requirements
    • Introduce an abstraction in the foundation layer (Indexing).
  • Create the implementation (Solr Indexing) that implements the abstraction define by Indexing in the foundation layer.
    • Address the sorting issues (i.e. different items templates have different date fields)
  • Let the features use the indexing abstractions (i.e. Course, News, Calendar, etc.)

Identify the indexing requirements

There are 3 main components to define the indexing requirements constraints, pagination & sorting.

Constraints

Constraints define what the filters can be applied to reduce the number of items that are returned. In this example it will be possible to apply the following constraints:

  • Location in tree sitecore (i.e. site specific news folder, all content, etc.)
  • Language (i.e. return items with an English language version)
  • Template, i.e. does the item inherit from a specific template (i.e. news, calendar, etc.)
  • Taxonomy – return items based on their categorization (i.e. football, skiing, etc.)

Pagination

Defines the number of search result per page and which page you require.

Sorting

Is responsible for defining what is used to sort the result items and the direction (ascending or descending), for example using date to get the 10 latest news.

If you want to sort by date, one challenge is to determine how to sort he results, as different pages will have different fields. Some pages have no date apart from created/updated, news normally has a specific news date and calendar events have start/end dates.

The SolR implementation must NOT KNOW ABOUT PAGE TYPES, I will write a blog post soon with a solution.

The following code defines the indexing requirements.

public interface IConstraint
{
    Item RootItem { get; }
    Language Language { get; }
    ID BaseTemplate { get; }
    IEnumerable<Category> Categories { get; }
}
public interface IPagination
{
    int Number { get; }
    int Size { get; }
}
public enum SortDirection
{
    Ascending,
    Descending
}

Then we need to define the result of making a search and a repository to make the search

public interface IPagedSearchResult
{
    IEnumerable<Item> Results { get; }
    IPagination Pagination { get; }
    int TotalHits { get; }
    bool HasMoreResults { get; }
}
public interface IPagedSearchResultRepository
{
    IPagedSearchResult Get([NotNull] IConstraint searchConstraint, [NotNull] IPagination pagination, SortDirection sortDirection);
}

The definition of the search result could of been type safe, i.e. return a model of type T instead of the Item, but I wanted to keep the example simple and not use a specific binding framework.

Anyway I hope this post will help, Alan

Sitecore Helix – Supporting Integrations

This blog will outline how it is possible to identify, separate and isolate dependencies on external systems, by introducing an Integrations module group. See here for more information about module groups in Helix.

But why do we need a Integrations module group?

Any feature that requires integration to achieve its purpose, will introduce additional challenges relating to stability and additional system knowledge, than a standard feature layer module.

Stability

Helix is built on a number of principles that help deal with stability, the Stable-dependencies principle and Stable-abstractions principle for more details see my blog post.

Features that rely on 3rd party systems are by nature more unstable than any other feature/foundation modules. As it is not usually possible to control when external systems change, upgrade or fail. Therefore, quality assurance, test, and automated deployment for the website cannot protect against this type of change or failure.

System knowledge

Working with a 3rd party system, such as CRM, ERP, Marketing Automation, DAM, SolR, Custom API, etc. requires additionally system specific knowledge. So it is a good idea to use abstractions to hide the system specific complexities for several reasons:

  1. The web team might not have the specific system knowledge.
  2. The web team should not be distracted by the additional complexities of all the integrations.
  3. The team responsible for implementing the integration may have limited Sitecore knowledge and should not be distracted by the rest of the web solution.
  4. It is good practice to separate modules by their responsibilities by splitting the presentation and the retrieval of data from the external system.

Integrations Module Group

The intention/purpose of the Integrations module group is to clearly define which modules have a dependency on an external system and ensure they are only responsible for that integration to the external system.

Example

This solution is responsible for selling a wide variety of courses. The customer has its own custom course catalog API and a complex legacy enrollment system.

The following diagram shows the module architecture for the solution.

In the foundation layer the following modules were introduced to provide abstractions. If you are not familiar with IoC and Abstractions see my earlier post.

  • Course Catalog
    • Defines the abstraction/system agnostic logical data module for the course catalog.
    • Acts as an abstraction between the website and Custom Catalog API.
    • It helps to focus on the ideal model that supports the business objectives.
  • Enrollment
    • Defines the abstractions to support the process of initiating attendance on a course at a specific school and shopping cart.
    • Acts as an abstraction between the website and the enrollment legacy system.
    • It helps to focus on the ideal model that supports the business objectives.

In the Integrations Group in Features layer the following modules where introduced

  • EXT Course API
    • Responsible for getting data provided by the EXT Course API.
    • Provide the implementation of the Course Catalog (foundation layer) abstractions.
    • Responsible for caching the course catalog, as the API only supports periodical batch retrieval.
  • EPOC Enrollment Management
    • Responsible for integration of the functionality provided by the EPOC Enrollment Management SDK.
    • Provide the implementation of the Enrollment (foundation layer) abstractions.

In the Features layer the following modules where introduced

  • Course
    • Responsible for the presentation of the course catalog, retrieved via the course catalog abstractions defined in the foundation layer.
  • Enrollment
    • Responsible for presentation and controlling the process of initiating attendance on a course and displaying the shopping cart, using the abstraction  defined in the foundation layer.

Additional Bonus

Once the integration code is isolated in a single module and only responsible for the integration, it is easier using dependency injection to achieve the following:

  1. Update external system, as the code to change is clearly defined and separated from the presentation and website logic.
  2. Provide the ability to support more than one version of an integration (i.e. different sites use different version)
  3. Move the integrations modules to an integrations platform, if it is the domain model for the customer business.

I hope this blog post gives you some ideas on how to isolate and remove the complexities introduced by integrations from your Sitecore solution, Alan

In my next post, I will explain why and in more detail how to isolate the SolR integration.

 

 

resolve DI

Dependency Injection with Sitecore 9 Scheduling

With this post I hope to draw attention to the ability to use dependency injection with Sitecore’s Scheduling Agents, for more information about agents see Johns blog post.

Most solutions now use DI with MVC controllers which is great, but I have noticed it is not used with agents?

Which is a pity as it is very simple to do this in Sitecore 9.0, Just add resolve =”true”.

resolve DI

Then add any dependencies to your constructor as required.

private readonly ILogger _logger;
private readonly UpdateSeatAvailabilityService _updateSeatAvailabilityService;

public UpdateSeatAvailabilityAgent(
       [NotNull]ILogger logger, 
       [NotNull]UpdateSeatAvailabilityService updateSeatAvailabilityService)
        {
            Assert.ArgumentNotNull(logger, nameof(logger));
            Assert.ArgumentNotNull(updateSeatAvailabilityService, nameof(updateSeatAvailabilityService));
            _logger = logger;
            _updateSeatAvailabilityService = updateSeatAvailabilityService;
        }

Hope this helps, Alan

Salesforce Connector – Strange Error (AlreadyExists, contact) Syncing checkbox field from Salesforce to Sitecore

Problem

We had setup the pipeline batch to sync from sitecore to Sales-force, and the from Salesforce to Sitecore and it worked.

Then we had to add a single checkbox that should be mapped from Salesforce to Sitecore, the checkbox was already setup to be sync’d from sitecore to Salesforce.

After setting it up, we got the following error whenever we changed the checkbox  value in Salesforce.


ManagedPoolThread #3 13:40:16 INFO  [Data Exchange] Submitting batch to xConnect. (pipeline: Read Contacts from Salesforce Pipeline, pipeline step: Submit Remaining Operations in xConnect Batch, pipeline step identifier: 1bb0f7dd-1d1f-454a-879e-6655399a42bf, description: Batch from thread 0, operation count: 8)
ManagedPoolThread #3 13:40:16 ERROR [Data Exchange] Exception thrown when processing a pipeline batch. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
ManagedPoolThread #3 13:40:16 ERROR [Data Exchange] Exception while submitting batch to xConnect. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
ManagedPoolThread #3 13:40:16 ERROR [Data Exchange]    at Sitecore.DataExchange.Providers.XConnect.Processors.PipelineSteps.BaseSubmitXConnectBatchStepProcessor.SubmitBatch(String batchDescription, IXdbContext client, PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
   at Sitecore.DataExchange.Providers.XConnect.Processors.PipelineSteps.SubmitXConnectBatchStepProcessor.ProcessPipelineStep(PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
   at Sitecore.DataExchange.Processors.Pipelines.PipelineProcessor.ProcessPipelineStep(PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
   at Sitecore.DataExchange.Processors.Pipelines.PipelineProcessor.ProcessPipeline(Pipeline pipeline, PipelineContext pipelineContext, ILogger logger)
   at Sitecore.DataExchange.Processors.PipelineBatches.BasePipelineBatchProcessor.ProcessPipelineBatch(PipelineBatch pipelineBatch, PipelineBatchContext pipelineBatchContext, ILogger logger)
ManagedPoolThread #3 13:40:16 ERROR [Data Exchange] One or more errors occurred. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
ManagedPoolThread #3 13:40:16 ERROR [Data Exchange]    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Sitecore.DataExchange.Providers.XConnect.Processors.PipelineSteps.BaseSubmitXConnectBatchStepProcessor.SubmitBatch(String batchDescription, IXdbContext client, PipelineStep pipelineStep, PipelineContext pipelineContext, ILogger logger)
ManagedPoolThread #3 13:40:16 ERROR [Data Exchange] START AGGREGATE EXCEPTION (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect, ID: 19, HResult: -2146233088)
 Exception thrown when processing a pipeline batch. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
One or more operations did not succeed (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
   at Sitecore.XConnect.XdbContext.&amp;amp;lt;ExecuteBatchAsyncInternal&amp;amp;gt;d__79.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.XdbContext.&amp;amp;lt;SubmitAsync&amp;amp;gt;d__74.MoveNext()
START AGGREGATE EXCEPTION (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect, ID: 20, HResult: -2146233088)
  Exception thrown when processing a pipeline batch. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
Operation #7, AlreadyExists, Contact {0b9e30fe-f081-0000-0000-056ff184830f}, NewsletterInformation (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
  Exception thrown when processing a pipeline batch. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
Operation #7, AlreadyExists, Contact {0b9e30fe-f081-0000-0000-056ff184830f}, NewsletterInformation (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
END AGGREGATE EXCEPTION (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect, ID: 20, HResult: -2146233088)

 Exception thrown when processing a pipeline batch. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
One or more operations did not succeed (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
   at Sitecore.XConnect.XdbContext.&amp;amp;lt;ExecuteBatchAsyncInternal&amp;amp;gt;d__79.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sitecore.XConnect.XdbContext.&amp;amp;lt;SubmitAsync&amp;amp;gt;d__74.MoveNext()
START AGGREGATE EXCEPTION (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect, ID: 21, HResult: -2146233088)
  Exception thrown when processing a pipeline batch. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
Operation #7, AlreadyExists, Contact {0b9e30fe-f081-0000-0000-056ff184830f}, NewsletterInformation (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
  Exception thrown when processing a pipeline batch. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
Operation #7, AlreadyExists, Contact {0b9e30fe-f081-0000-0000-056ff184830f}, NewsletterInformation (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect)
END AGGREGATE EXCEPTION (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect, ID: 21, HResult: -2146233088)

END AGGREGATE EXCEPTION (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect, ID: 19, HResult: -2146233088)

ManagedPoolThread #3 13:40:16 DEBUG [Data Exchange] Pipeline batch processor finished processing pipelines. (pipeline batch: xConnectToSalesforceAndSalesforceToxConnect, completed pipelines: 1)


Now you might of noticed that the issue is related to the contact already existing?

  • Operation #7, AlreadyExists, Contact {0b9e30fe-f081-0000-0000-056ff184830f})

Strange error message? Of course the contact already exists, as the contact is first created via the website and then sync’d to Salesforce?

Solution

The issue was related to the fact, that I had missed to add the “Newsletter Information” facet to the “Facets to Read” field on the “Resolve Contact Model by Salesforce Id from xConnect” item, see image below.

To be honest I believe that the error message was very misleading, hence this blog post. As there are 1000’s of items to associated with the Salesforce connector, it is very easy to miss a field selection and or other required item setup.

Credits

Big thanks to Paul Kravchenko, from Sitecore support, who worked very had to identify and fix this issue, which we managed after a few online meetings 🙂

Hope this helps others out there, Alan

Structured, Type Safe Settings in Sitecore

This feature seems to be overlooked, so I hope this blog post will draw more attention to this feature, and make it’s use more widespread.

In Sitecore, it is easy to map configuration settings to a C# class, whilst maintaining a structure that adheres to the helix principles, see the config below.

Then the mapped C# class can registered with IServiceCollection, so it can be injected into any class using dependency injection.

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:environment="http://www.sitecore.net/xmlconfig/environment">
<sitecore environment:require="Local">
<feature>
<salesforce>
<clientSettings type="Example.Feature.Salesforce.Infrastructure.SalesforceClientSettings, FKCC.Feature.Salesforce" singleInstance="true">
<Username>example@blog.example.com</Username>
<Password>xxxxxxx</Password>
<Token>yyyyyyy</Token>
<CacheExpiry>60</CacheExpiry>
<OrganisationId>1111111</OrganisationId>
</clientSettings>
</salesforce>
</feature>
</sitecore>
</configuration>

Previously settings used to be a long flat list of settings, which if we were lucky were grouped use prefixes in the name attribute to indicate which feature they were used by.

<setting name="Feature.Salesforce.Authentication.Username" value="xxxx@example.com" />
<setting name="Feature.Salesforce.Authentication.Password" value="BestPasswordInTheWorld" />
<setting name="Feature.Salesforce.Authentication.SfToken" value="Its a SF token" />
<setting name="Feature.Salesforce.Authentication.CacheExpiry" value="60" />

Solution

It is now very simple to map structured configuration to a type safe C# class, and it involves 4 simple steps.

Step 1 – Define C# Class

Define the C# class that stores the data defined by the settings in the config file, for this example, we will define some authentication settings for a sales force client.

namespace Example.Feature.Salesforce.Infrastructure
{
public class SalesforceClientSettings
{
public string Password { get; protected set; }
public string Username { get; protected set; }
public string Token { get; protected set; }
public int CacheExpiry { get; protected set; }
public string OrganisationId { get; set; }
}
}

Step 2 – Define the settings

It is not required, but I would recommend following the Helix principles when defining the settings structure i.e.

[Layer]/[Feature Name]/[Settings Name]

The type attribute defines which class (i.e. the one defined in step 1) to map the settings element to.

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:environment="http://www.sitecore.net/xmlconfig/environment">
<sitecore environment:require="Local">
<feature>
<salesforce>
<clientSettings type="Example.Feature.Salesforce.Infrastructure.SalesforceClientSettings, FKCC.Feature.Salesforce" singleInstance="true">
<Username>example@blog.example.com</Username>
<Password>xxxxxxx</Password>
<Token>zzzzzz</Token>
<CacheExpiry>60</CacheExpiry>
<OrganisationId>1111111</OrganisationId>
</clientSettings>
</salesforce>
</feature>
</sitecore>
</configuration>

Step 3 – Map the configuration to the C# class

Sitecore makes this so easy, using Factory.CreateObject method, which loads the configuration and maps it to the C# class.

(SalesforceClientSettings) Factory.CreateObject("feature/salesforce/clientSettings", true)

Note: Factory.CreateObject expects that configuration path, is relative to the sitecore configuration, not the complete path.

Step 4 Setup dependency injection.

Register the created class with the IServiceCollection, so we can access the class, where necessary using constructor injection.

namespace Example.Feature.Salesforce.Infrastructure
{
public class ServiceConfigurator : IServicesConfigurator
{
public void Configure(IServiceCollection serviceCollection)
{
serviceCollection.AddSingleton(provider =>
(SalesforceClientSettings) Factory.CreateObject("feature/salesforce/clientSettings", true));

}
}
}

I hope this blog posts, helps you to structure your settings in a more maintainable and coherent structure, Alan

Sitecore SIF NewSignedCertificate – The time period is invalid

Problem – The time period is invalid. 0x80630705

The client certificate for xConnect expired on my developer machine for a solution I was developing. I thought no problem I will get SIF to generate new certificates for the website and xConnect.

Unfortunately when I ran SIF, i got the following error when it was running CreateSignedCert : NewSignedCertificate.


PS>TerminatingError(New-SelfSignedCertificate): "CertEnroll::CX509Enrollment::_CreateRequest: The time period is invalid. 0x80630705 (-2140993787 PEER_E_INVALID_TIME_PERIOD)"
>> TerminatingError(New-SelfSignedCertificate): "CertEnroll::CX509Enrollment::_CreateRequest: The time period is invalid. 0x80630705 (-2140993787 PEER_E_INVALID_TIME_PERIOD)"
Install-SitecoreConfiguration : CertEnroll::CX509Enrollment::_CreateRequest: The time period is invalid. 0x80630705 (-21
40993787 PEER_E_INVALID_TIME_PERIOD)
At D:\Projects\FK.Donki\Sitecore\setup\FKCC-Install-Local-Sc-XP0.ps1:42 char:1
+ Install-SitecoreConfiguration @certParams -Verbose
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Install-SitecoreConfiguration
Install-SitecoreConfiguration : CertEnroll::CX509Enrollment::_CreateRequest: The time period is invalid. 0x80630705 (-2
140993787 PEER_E_INVALID_TIME_PERIOD)
At D:\Projects\FK.Donki\Sitecore\setup\FKCC-Install-Local-Sc-XP0.ps1:42 char:1
+ Install-SitecoreConfiguration @certParams -Verbose
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Install-SitecoreConfiguration


 

Firstly I would like to say thanks to Richard Dzien, from Sitecore as he was super quick via slack to help and identify what the problem.

The issue is caused by the fact that the Trusted root certificates, which had expired. See the image below.

Solution

Part 1 – Delete from Trusted Roots Certificates (Computer Account)

The solution is to delete the certificates form the machine account, which you can do either via the MMC certificates snap in or use power shell. Then run SIF again.

Part 2 – Delete from Trusted Roots Certificates (My User Account)

There was also a copy of the root certificates, as you can see below in my personal Certificate store. which also need to be deleted.

Part 3 – Delete from disk

In addition there can be a copy in C:\Certificates, which also need to be deleted.

Once the certificates are deleted from all locations everything worked. SIF 2 – the root certificates will expire in 10 years so no problem there, once it is released.

I hope this helps, Alan

Bonus help – Certificate not found, when calling xConnect

If you get an error that the xConnect client certificate can not be found in your sitecore log file!

But the certificate is in the store and has not expired!

This could be because the root certificate has expired.

 

 

Sitecore Update xDB Contact

Problem

We had a solution where we had to process any contacts that changed their details in the past 24 hours, for example changed their name, birthday, custom preferences, etc.

I expected I could get a list of all updated contacts (i.e. contacts where one of their facets had changed), but unfortunately, adding or updating facets does not change the contact’s Last Updated property?

According to the Sitecore support & documentation, this is not a bug, but the desired behavior?

Solution

The Contacts Last Updated property is updated, only if an identifier is added or removed.

The following code is a nasty work around that adds/removes an identifier; to force an update of the Last Updated property.

  private void TempWorkAroundToGetSitecoreToUpdateLastmodified(Contact contact, XConnectClient client)
	  {
      //when a contact facet is updated the contacts last modified is not updated?
      // see documentation WTF https://doc.sitecore.net/developers/xp/xconnect/xconnect-client-api/contacts/update-contacts.html

      // the only way to set the last modified for a contact, is to add or remove an identifier, so we have to toggle that
	    var identifierToRemove = contact.Identifiers.FirstOrDefault(x => x.Source == Constants.Source.FakeSource);
	    if (identifierToRemove != null)
	    {
	      client.RemoveContactIdentifier(contact, identifierToRemove);
      }
	    else
	    {
	      client.AddContactIdentifier(contact, new ContactIdentifier(Constants.Source.FakeSource, contact.Id.HasValue ? contact.Id.ToString() : Guid.NewGuid().ToString(), ContactIdentifierType.Known));
	    }
	  }

Each time a contact was updated, we called this code, to ensure it is possible to get a list of all contacts that have changed, within a given time span.

Hope this helps and enjoy the summer, Alan