Author Archives: Alan Coates

Is the custom Sitecore field type being used?

Took over a 18 year old sitecore solution, which had a number of custom field types and before upgrading to Sitecore 10 of course we wanted to remove any unused functionality.

Unfortunately the customer had no idea if the field types where being used as there are over 900+ sites.

Problem

How to determine if a custom field type is in use?

I thought it would be easy using the link database, the search box in the shell and or the search in content editor.

Unfortunately when I searched for the ID of the field type I found nothing.

I assumed it was because the field type is defined in the core database and used in the master database.

Solution

Therefore it was time to look into the SQL database, to try and determine if the custom fields where in use.

First step was to determine how sitecore knows the field type for a field? I hoped there was a field type column on the following tables that are responsible for storing field values.

  • SharedFields
  • VersionedFields
  • UnversionedFields

Unfortunately sitecore does not save the field type in the SQL database as a dedicated column, in fact they store the type like any other field, as a value on the field.

The field responsible for storing what type of field a field is, is the the Type field from the Template Field template. See the image below, notice the source field is set to get values from the core database.

The Type field id is “{AB162CC0-DC80-4ABF-8871-998EE5D7BA32}” and as you can see in the image below we can then restrict our search to fields values of that field.

This is when I discovered that the value stored for the field is not the Guid of the field type, but the name :-(.

So the following SQL can be used to search for any field type and determine if the field is used.

The ItemId is the Id of the field which uses the custom field type.

SELECT *
  FROM [dtu_master_upgrade].[dbo].[SharedFields] 
    where FieldId = 'AB162CC0-DC80-4ABF-8871-998EE5D7BA32' and 
          Value like '%[NAME OF CUSTOM FIELD TYPE]%'

SELECT *
  FROM [dtu_master_upgrade].[dbo].[UnversionedFields] 
    where FieldId = 'AB162CC0-DC80-4ABF-8871-998EE5D7BA32' and 
          Value like '%[NAME OF CUSTOM FIELD TYPE]%'

SELECT *
  FROM [dtu_master_upgrade].[dbo].[VersionedFields] 
    where FieldId = 'AB162CC0-DC80-4ABF-8871-998EE5D7BA32' and 
          Value like '%[NAME OF CUSTOM FIELD TYPE]%'

From the field you can get the template and you can then check if the template is being used, by using the Links button on the Navigate Menu, see below.

Hope this helps, Alan

Progress bar with progress messages for long running processes.

Sitecore has made it very easy from within the sitecore client to click a button do the following:

  • Start a long running process as a background job.
  • Showing a progress bar dialog and define the title.
  • Provides the ability to update the message as the job progresses in the background.
  • Hide the progress bar once the background job completes.

It seems to be a feature that has not been used much, so I hope this blog post will draw attention to it. I will assume that you already know how to add a custom button and execute code when it is clicked, using a customer command class, if not see here and or google it.

Step 1 – Setup

In the overridden Execute function for the custom command get the details you need, in our case the item id, language and database. Then call Client page start, which calls the Run function.

public class ExportBookCommand : Command
{
        public override void Execute(CommandContext context)
        {
            Assert.ArgumentNotNull(context, "context");
            Assert.IsNotNull(context.Items, "context items are null");
            Assert.IsTrue(context.Items.Length > 0, "context items length is 0");
            var item = context.Items[0];
            var parameters = new NameValueCollection
            {
                [Constants.Parameters.Keys.Id] = item.ID.ToString(),
                [Constants.Parameters.Keys.Language] = item.Language.ToString(),
                [Constants.Parameters.Keys.Database] = item.Database.Name
            };
            Context.ClientPage.Start(this, nameof(Run), parameters);
        }
        protected void Run(ClientPipelineArgs args)

Step 2 – Show progress bar

The Run function (see code below) is responsible for the following:

  • Do any validation and show an alert if required.
  • Build any classes your background job is going to need
    • We are exporting a book to multiple word word documents.
  • Call ProgressBox.Execute which is responsisble for the following:
    • Show the progress bar
    • Defines the title for the progress bar
    • Defines the initial message for the progress bar
    • Starts the background job and calls the function StartProcess todo the work.
      • Adds the Job to the current client context, which we use later.
protected void Run(ClientPipelineArgs args)
{
	Assert.ArgumentNotNull(args, "args");
	// Validate args, show an alert if not valid
	if(notValid(args))
	{
		SheerResponse.Alert("explain what went wrong"));
		return;
	}
    // create the data, for the background job
	var book = _bookFactory.Create(args.Parameters);
	Assert.IsNotNull(book, "Could not create book model");
	var bookArray = new object[] {book};
		// show progress bar and start background job
	ProgressBox.Execute("Job name, used as the inital message shown on the prgress bar",
		Translate.Text("The title of the progress bar"), "~/Network/32x32/server_into.png",
		StartProcess, bookArray);
}

private void StartProcess(params object[] parameters)
{
	Assert.ArgumentNotNull(parameters, "parameters");
	var book = parameters[0] as Models.Book;
	Assert.IsNotNull(book, "The first element in the object array is not a book");
	var exportBookService = new ExportBookService();
	exportBookService.ExportBook(book);
}

Step 3 – Update progress message

To update the messages for the progress bar is simple as the current client context has a Job property which makes it simple to add messages, relating to the progress of the process.

Context.Job.Status.Messages.Add($"{Translate.Text("export:progress")}{folder.Name}");

In our solution we showed a new message for each top level folder (which was a letter) we exported, see the image below, which is currently processing the letter B

I hope this blog post has helps to give the user a meaningful indication of what is happing when a process takes a long time t complete.

Sitecore and Hangfire

A while ago I wrote a blogpost on Sitecore & Azure Durable Functions, which looked at off loading CPU/Memory intensive scheduled tasks.

I recently encountered the same issue on another Sitecore solution that was pure on premise and we could not use Azure functions, but don’t worry Hangfire to the rescue.

Hangfire is an easy way to perform background processing in .NET and .NET Core applications, and it supports fire and forget jobs, delayed jobs, scheduled jobs, dependant jobs, batchs of background jobs that are created atomically and considered as a single entity. Hangfire is free for commercial usage.

Of course if a job fails, it is possible to define how long the job should wait to retry and how many times.

The sitecore solution had a few main performance issues to resolve, sending out lots of add hoc emails, and processing data from an external legacy system that was super slow.

Solution

We decided to setup and additional server with hangfire running as a.net core website. Then within the sitecore solution you need to install the Hangfire Nuget package and configure the connection string to SQL (can also chose redis if you prefer).

When we wanted to send an email, we just sent the job to Hangfire, using the code below.

var jobId = BackgroundJob.Enqueue(
    <ISendEmailService>(uis => uis.SendEmail(eMailDetails));

We also needed some recurring jobs i.e., to replace the scheduled tasks running within sitecore. We had to introduce a rest API to implement the sitecore specific functionality. But in fact, that part of the code was minimal in comparison to the integration with their custom CRM. So on the hangfire server, we added recurring job sync users from their CRM to Sitecore.

RecurringJob.AddOrUpdate<IUserService>(uis => uis.SynchronizeUsers(), synchronizeUsersTab);

Well for once not so much code, but just more an idea how to off load you sitecore CM server.

Cheeer’s, Alan

Wrong Language

We have a very big Sitecore solution with over 3000 editors, unfortunately they spend a great deal of time creating a content in the wrong language.
Easy enough to fix a single item, but when each page has a lot of renderings, which in turn have a lot of data source items (see image below), the task is not so easy and very time consuming.

Solution

Provide the ability to select a given item and move all the language versions from one language to another and then delete the source language version.

I decided to add 2 buttons, and I was lucky as there was already a Data Migration chunk I could add the buttons to.

See this blog for a detailed introduction to adding buttons to the sitecore client.

In order to know which language to move from and to, I added a source and target parameter to the move language command.

This is very easy todo, as on the “click” field for small button you can add parameters, then when sitecore calls the Execute function the CommandContext has the values in the Parameters list, see the code below, which can then identify the source and target language.

public override void Execute(CommandContext context)
{
  var currentItem = context?.Items[0];
  if (currentItem == null)
    return;

  string source = context?.Parameters["source"];
  Assert.IsNotNullOrEmpty(source, $"Command parameter:source can not be empty or null");
  string target = context?.Parameters["target"];
  Assert.IsNotNullOrEmpty(target, $"Command parameter:source can not be empty or null");
  context.Parameters.Add("item", currentItem.ID.ToString());
  context.ClientPage.Start(this, nameof(Move), context.Parameters);
}

We need some configuration and in the solution there was already a data migration settings item that could store the following:

  • Modules Folder Template id
    • So I can identify which sub item contains the data source items and don’t iterate over the entire tree.
  • List of fields to ignore
    • i.e. revision, updated, updated by, owner, security, etc.

The code to move the language versions is quite simple.

  1. Get the current item, using the source language.
  2. Get the settings item.
  3. Get the items in the modules folder, by looking for any sub items that derive from the Modules Folder Template, then adding their descendants.
  4. For each item
    1. Get all the source versions (see code below)
    2. For each version create the new target language version
      1. For each field
        1. Skip all shared fields
        2. Skip fields that are in the Fields to Exclude list
        3. Copy the field
    3. Remove all the source language version
private void Copy(
        [NotNull] Database database,
        [NotNull] Item source, 
        [NotNull] Language target,
        [NotNull] IEnumerable<Field> languageMigrationExcludedFields)
        {
            Assert.ArgumentNotNull(source, nameof(source));
            Assert.ArgumentNotNull(target, nameof(target));
            Assert.ArgumentNotNull(languageMigrationExcludedFields, nameof(languageMigrationExcludedFields));

            foreach (var sourceVersion in source.Versions.GetVersions())
            {
                if (sourceVersion == null)
                    continue;
                var targetItem = database.GetItem(source.ID, target);
                var newTargetVersion = targetItem.Versions.AddVersion();
                Copy(sourceVersion, newTargetVersion, languageMigrationExcludedFields);
                sourceVersion.Versions.RemoveVersion();
            }
        }

Hope this helps, cheers Alan

Sitecore config disappeared?

We had a Sitecore 9 solution that required the include files to be rearranged to align with a new azure deployment/provisioning setup using Terraform and to fix some duplication and inconsistencies.

Problem – Unicorn Sync Failed

When we enabled unicorn sync in the CD pipeline (see here for more details about Unicorn) the deploy failed?

Unfortunately (see image above) the configuration relating to the unicorn’s sc.variable‘s were missing?

Don’t have 2 <sitecore> elements

If there are 2 <sitecore> elements in a sitecore include file, even if they have mutely exclusive require statements Sitecore’s config merge ignores the file, without reporting an error!

Solution

There are 2 solutions to the problem

  1. Split the configuration into 2 files each with their own <sitecore> element.
  2. Merge the 2 <sitecore> elements into one element and move the require constraints to each sub element.

We decided to solution 2, see the result below.

Use patch:before=”prototypes” that way the new variables are shown at the top when showconfig.aspx is used (see image below) instead of half way down the page.

I hope this will help and no one else has to waste hours figuring this out, cheers Alan

Azure Functions – Json: Self referencing loop detected for property

Problem

Whilst developing a Rest API using Azure functions I got the error below.

 Newtonsoft.Json: Self referencing loop detected for property 'task' with type 'System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Collections.Generic.IEnumerable`1[Feature.Order.Domain.TravelCard.OrderedTravelCard],Feature.Order.Domain.TravelCard.OrderedTravelCardRepository+<Get>d__4]'. Path 'stateMachine.<>t__builder'

As we had created a lot of new models and called a number of external API’s, which had a bad habit of changing their models without notification. I assumed incorrectly that one of the swagger files for the API’s was not correct.

But after much investigation I could not detect any issue with the data returned from the external API’s and the models match correctly their definitions.

Eventually I found out what the issue was, can you see what is wrong with the following code?

Solution

See the image above that a Task<…> is returned from the OrderedTravelCardRepository.Get() function.

I had forgotten to await the asynchronous call of the function, see the fixed code below.

Simple when you know how, simply awful when you don’t!

I hope this blog post helps others avoid the time I wasted on this issue.

Where did my request body data go?

Problem

The solution had several API controllers (see image above) that expected data to be posted in the body. Then for no reason the body data was null, and therefore the controller started throwing argument null exceptions, and the front end of course stopped working.

The issue was caused by a new custom HttpRequestProcessor (see image below), which was called in the httpRequestBegin pipeline.

private void ProcessApi(HttpRequestArgs args)
{
      using (var mem = new MemoryStream())
      {                
           HttpContext.Current.Request.InputStream.CopyTo(mem);
           HttpContext.Current.Request.InputStream.Seek(0, SeekOrigin.Begin);
                
           using (var reader = new StreamReader(mem))
           {
                mem.Seek(0, SeekOrigin.Begin);
                var body = reader.ReadToEnd();

                if (string.IsNullOrEmpty(body))
                    return;

                var requestBody = JsonConvert.DeserializeObject<RequestBody>(body);
                _setLanguageService.SetLanguage(requestBody);
           }
      }            
 }

It needed the data from the request body, for some reason the setting the position back to 0, did not work?

Therefore the controller got null instead of the data contained in the body.

Solution

Therefore it was necessary to copy the stream contents into a memory stream, read the data from that stream, then deserialize the class. See the solution below.

I hope this helps, and this issue was found Sitecore 8, with a lot of customization, patches, code and modifications over the past 16 years.

Site Context for ApiControllers

Now almost every Sitecore project has rest API’s and I am always shocked when the database, language etc. is hard coded and or additional configuration is added to define default the language, database etc.

Wouldn’t it be nice if you can define the site context for the controller?

Then you can define a site declaration, and or use an existing site that each controller should use and then use the language, database etc. that is define for the site language, database etc.

Solution

The SiteContextAttribute provides the ability to define which site an ApiController should use, for example in the picture above it is setup to use the “Person” site.

    public class SiteContextAttribute : ActionFilterAttribute
    {
        protected readonly string SiteName;

        public SiteContextAttribute(string siteName)
        {
            this.SiteName = siteName;
        }

        private SiteContextSwitcher _siteContextSwitcher;
        private LanguageSwitcher _languageSwitcher;
        private DatabaseSwitcher _databaseSwitcher;

        public override void OnActionExecuting(HttpActionContext actionContext)
        {
            base.OnActionExecuting(actionContext);

            var siteContext = SiteContext.GetSite(this.SiteName);

            _siteContextSwitcher = new SiteContextSwitcher(siteContext);
            _databaseSwitcher = new DatabaseSwitcher(siteContext.Database);
            _languageSwitcher = new LanguageSwitcher(LanguageManager.GetLanguage(siteContext.Language));
        }

        public override void OnActionExecuted(HttpActionExecutedContext actionContext)
        {
            _languageSwitcher?.Dispose();
            _databaseSwitcher?.Dispose();
            _siteContextSwitcher?.Dispose();

            base.OnActionExecuted(actionContext);
        }
    }

The code gets the site name, then gets the site context and setups the language, database and site context for the controller.

For example, see below it is it possible to use the Context.Database and also the language of the item will also be correct.

I hope this helps, Alan

How SQL Index Fragmentation will kill Sitecore’s Performance

Thought I wrote a blog post about this years ago, but apparently I didn’t.

Problem

Poor index maintenance is a major cause of decreased SQL Server performance, which in turn will impact your Sitecore’s performance. The Sitecore databases contains tables with numerous entries, that get updated frequently, therefore high index fragmentation will occur.

Detecting SQL Server index fragmentation

The following script displays the average fragmentation, and as a help generates the SQL query to fix it.

SELECT OBJECT_NAME(ind.OBJECT_ID) AS TableName,
ind.name AS IndexName, indexstats.index_type_desc AS IndexType,
indexstats.avg_fragmentation_in_percent,
'ALTER INDEX ' + QUOTENAME(ind.name) + ' ON ' +QUOTENAME(object_name(ind.object_id)) +
CASE WHEN indexstats.avg_fragmentation_in_percent>30 THEN ' REBUILD '
WHEN indexstats.avg_fragmentation_in_percent>=5 THEN 'REORGANIZE'
ELSE NULL END as [SQLQuery] -- if <5 not required, so no query needed
FROM sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL, NULL) indexstats
INNER JOIN sys.indexes ind ON ind.object_id = indexstats.object_id
AND ind.index_id = indexstats.index_id
WHERE
--indexstats.avg_fragmentation_in_percent , e.g. >10, you can specify any number in percent
ind.Name is not null
ORDER BY indexstats.avg_fragmentation_in_percent DESC

Below you can see the typical result of running the script above. I was shocked as the majority of indexes on my local SQL server where over 99%.

Solution

The script above generates the SQL statements needed to defragment the affected indexes, so you can automate the defragmentation process, using SQL Server Maintenance Plans.

Anyway I hope this helps keeping you sitecore solution running at its best, Alan

Sitecore and Azure Durable Functions

In this post I will show how Azure Durable functions can complement your sitecore solution and help enhance performance.

Problem

We took over a Sitecore solution and its content management server was running very slowly and intermittently the sitecore client would be unresponsive and crash

The problem was caused a number a lot of CPU/Data/Bandwidth intensive schedule tasks that were running to retrieve a wide range of data from a number of web services, then aggregate the data and perform complicated calculations, of which a small sub set of the result were presented on the website.

Solution

As the solution was already hosted in Azure, a perfect solution was to off load the heavy lifting from the Content Management server to Azure Functions, to do the data retrieval, calculations and provide the results for the website. Firstly, a very brief overview of the pro’s and con’s of Azure functions.

Pro’s

  • Serverless execution model
  • Dynamic Scaling
  • Micro pricing
  • Security
  • Wide range of triggers
    • Https, Timer (CRON), Azure storage changes, Azure Queue, Message from Service bus, etc.

Con’s

  • Stateless
  • Execution time limit (default 5 mins, max 10)
  • Concurrency

The main challenge with Azure functions is that most of the schedule tasks could take more than 10 minutes to complete and require state management. But not to worry as Azure Durable Functions came to the rescue.

Azure Durable Functions

Durable Functions are an extension of Azure Functions and Azure WebJobs that lets you write stateful functions in a serverless environment. The extension manages state, checkpoints, and restarts for you, so it is possible to implement code that run for a long time.

In addition if an Azure function fails, for example the web request times out, you can define if the durable function should wait and retry X times, before failing. Behind the scenes, the Durable Functions extension is built on top of the Durable Task Framework, an open-source library on GitHub for building durable task orchestrations.

Advantages of Durable Functions

  • They define workflows in code. No JSON schemas or designers are needed.
  • They can call other functions either synchronously or asynchronously.
  • Output from called functions can be saved to local variables.
  • They automatically checkpoint their progress whenever the function awaits.
  • Local state is never lost, even if the process recycles or the VM reboots.
  • Easy to Unit Test
  • Can run for a very long time, in theory forever
  • Cost effective, as you do not pay for execution time whilst waiting for tasks to complete.

Here is a brief introduction to the most common Durable Functions patterns

Pattern 1 – Function chaining

Function chaining refers to the pattern of executing a sequence of functions in a particular order. Often the output of one function needs to be applied to the input of another function.

function chaining

The code below is an example of how you would achieve this

chaining code

Pattern 2 – Fan-out/fan-in

Fan-out/fan-in refers to the pattern of executing multiple functions in parallel, and then waiting for all to finish. Often some aggregation work is done on results returned from the functions. This is perfect when you want to do a lot of things in parallel, to reduce the time taken to complete the task and then aggregate/process all the results.

Below is an example of how the code could look

Pattern 3 – Monitoring

The monitor pattern refers to a flexible recurring process in a workflow – for example, polling until certain conditions are met. A regular timer-trigger can address a simple scenario, such as a periodic clean-up job, but its interval is static and managing instance lifetimes becomes complex. Durable Functions enables flexible recurrence intervals, task lifetime management, and the ability to create multiple monitor processes from a single orchestration.

An example could be instead of exposing an endpoint for an external client to monitor a long-running operation, the long-running monitor consumes an external endpoint, waiting for some state change. See the example below.

Is Replaying

One thing that catches people out is that the code is re-run from the start of the function after each await completes, therefore for example with Logging and other code you need to check for IsReplaying so you only log once.

Durable Functions – Orchestrator code constraints

There are a number code constraints, that must be adhered to when using Durable function orchestration.

  • Code must be deterministic.
    • It will be replayed multiple times and must produce the same result each time.
    • For example, no direct calls to get the current date/time, get random numbers, generate random GUIDs, or call into remote endpoints.
  • Non-deterministic operations must be done in activity functions
    • This includes any interaction with other input or output bindings. This ensures that any non-deterministic values will be generated once on the first execution and saved into the execution history. Subsequent executions will then use the saved value automatically.
  • Orchestrator code should be non-blocking.
    • For example, that means no I/O and no calls to Thread.Sleep or equivalent APIs
    • Orchestrator code must never initiate any async operation, except by using the IIDurableOrchestrationContext API.
    • For example, no Task.Run, Task.Delay or HttpClient.SendAsync.
    • The Durable Task Framework executes orchestrator code on a single thread and cannot interact with any other threads that could be scheduled by other async APIs.
  • Infinite loops should be avoided
    • Because the Durable Task Framework saves execution history as the orchestration function progresses, an infinite loop could cause an orchestrator instance to run out of memory.
    • For infinite loop scenarios, use APIs such as ContinueAsNew to restart the function execution and discard previous execution history.

Result

By migrating all the long running CPU/data/bandwidth intensive tasks to Azure Durable Functions, the performance of the Sitecore solution went from painful to fantastic.

Unfortunately it is very common that Sitecore solutions assume responsibility for task that are not the websites responsibility, but pairing with Azure functions can help mitigate this issue.

An additional benefit was that the website was isolated/protected from 3rd party system changes, as when an external system changes only the Azure functions had to be modified and deployed – therefore no down time for the sitecore solution.

Anyway I hope sitecore develops will consider Azure functions to enhance their sitecore solutions.