We had a solution that had serve performance issue when it got a lot of visitors. Sitecore was casting the following exception and SolR had a similar errors in its logs:
Unable to read data from the transport connection: The connection was closed
We identified that the problem was caused by hitting the network bandwidth in Azure!
Yes, there were a lot of visitors, but enough to hit the bandwidth limit, the customer upgraded the plan to get more network bandwidth, but still the issues continued.
But what could cause this issue?
I started to review the SolR implementation and found the issue quite quickly.
return IQueryable<Result> .Where(result => result.Date < DateTime.UtcNow) .OrderByDescending(result => result.Date) .GetResults() .Take(count) .ToList();
The Take() was made after GetResults() was called, so the entire data set is returned to Sitecore from SolR, then the take was applied to get the top 5 results.
This simple mistake was what caused all the network and performance issues.
return IQueryable<Result> .Where(result => result.Date < DateTime.UtcNow) .OrderByDescending(result => result.Date) .Take(count) .GetResults() .ToList();
It was a simple fix (in 150+ places) to move the Take before GetResults!
This is why I believe that you should always Introduce a (SolR) Sitecore Search Abstraction, please read my post on this very subject, instead of returning the IQueryable interface.
Hope this helps, Alan