Persistent "Heartbeat Error" in Sitecore Search Indexing Jobs

 We’ve been encountering a recurring issue during Sitecore search indexing jobs where they fail with the error: "Job failed due to heartbeat error." This error disrupts indexing and impacts production, creating a significant bottleneck in search functionality.

---

Key Observations

1. Threshold Limitation:

   • The issue often relates to a threshold where if more than 30% of documents fail (e.g., due to 404 errors), the entire indexing job is discarded.
   • This threshold was confirmed by Sitecore Support as a hard limit with no current configuration option to adjust it.

2. Unclear Error Reporting:

   • Errors like the "heartbeat error" provide no detailed context, making it difficult to determine whether the issue is due to resource limitations, 404 errors, or other failures.

3. Sitemap Issues:

   • Delayed sitemap refreshes sometimes lead to invalid or outdated URLs being crawled, resulting in multiple 404 errors.

4. Random Behavior:

   • In some cases, even with valid configurations and no 404s, jobs fail due to the heartbeat error without any clear cause.

---

Temporary Fixes Applied

1. Adjusting Sitemap:

   • Updated and validated the sitemap to ensure all links are functional and up to date.
   • Excluded problematic URLs and broken links to reduce 404 errors.

2. Simplified Extractor Configuration:

   • Removed unnecessary extractors to streamline the indexing process.
   • Combined document extractors to manage resources more efficiently.

3. Retry Mechanism:

   • Reran jobs after addressing sitemap and threshold-related issues, which resolved some occurrences of the error.

---

Discussed below pionts with the Sitecore Team

During our investigation and resolution efforts for the persistent "heartbeat error," we requested the Sitecore team to address the following critical questions:

1. Can the 30% Threshold Be Adjusted?

   • The current threshold causes the entire indexing job to fail if more than 30% of documents encounter issues (e.g., 404 errors). We inquired whether this threshold is configurable to accommodate different scenarios and prevent unnecessary failures.

2. Improved Error Reporting:

   • We requested more descriptive error messages to clearly identify the root cause of failures, such as whether the issue is due to the 30% threshold, resource capacity, or another specific reason.

3. Clarification on Heartbeat Error:

   • We asked whether the heartbeat error is solely tied to the threshold or if other factors, such as resource limitations or system configurations, could contribute to this issue.

4. Sitemap Handling:

   • Given the potential delays in sitemap refreshes, we asked if there are recommendations for ensuring the crawler processes the most up-to-date sitemap without being affected by temporary 404 errors.

5. Proactive Notifications:

   • We raised the need for proactive notifications for global incidents affecting crawlers, such as the ones reported in the Sitecore Status portal, to minimize downtime and ensure teams are informed promptly.

6. Future Roadmap for Improvements:

   • We requested insights into Sitecore's plans for enhancing the search platform, including:
     • Allowing configurable thresholds.
     • Improving error handling and reporting.
     • Addressing known bugs or feature requests related to crawlers and extractors.