A while ago I wrote a blogpost on Sitecore & Azure Durable Functions, which looked at off loading CPU/Memory intensive scheduled tasks.
I recently encountered the same issue on another Sitecore solution that was pure on premise and we could not use Azure functions, but don’t worry Hangfire to the rescue.
Hangfire is an easy way to perform background processing in .NET and .NET Core applications, and it supports fire and forget jobs, delayed jobs, scheduled jobs, dependant jobs, batchs of background jobs that are created atomically and considered as a single entity. Hangfire is free for commercial usage.
Of course if a job fails, it is possible to define how long the job should wait to retry and how many times.
The sitecore solution had a few main performance issues to resolve, sending out lots of add hoc emails, and processing data from an external legacy system that was super slow.
We decided to setup and additional server with hangfire running as a.net core website. Then within the sitecore solution you need to install the Hangfire Nuget package and configure the connection string to SQL (can also chose redis if you prefer).
When we wanted to send an email, we just sent the job to Hangfire, using the code below.
var jobId = BackgroundJob.Enqueue( <ISendEmailService>(uis => uis.SendEmail(eMailDetails));
We also needed some recurring jobs i.e., to replace the scheduled tasks running within sitecore. We had to introduce a rest API to implement the sitecore specific functionality. But in fact, that part of the code was minimal in comparison to the integration with their custom CRM. So on the hangfire server, we added recurring job sync users from their CRM to Sitecore.
RecurringJob.AddOrUpdate<IUserService>(uis => uis.SynchronizeUsers(), synchronizeUsersTab);
Well for once not so much code, but just more an idea how to off load you sitecore CM server.