Extracting HTML from PDFs in Sitecore Search: A Step-by-Step Implementation

Recently, I was tasked with implementing a solution to extract structured content from PDFs using Sitecore Search. PDFs, by nature, are not HTML-based, and since Sitecore Search document extractors only parse HTML, I needed a way to transform PDFs into an HTML-readable structure. Here’s how I approached and implemented this requirement effectively.

---

The Challenge

The primary goal was to extract key attributes (like titles, descriptions, and tags) from PDFs by leveraging their HTML structure. However, identifying the HTML structure of a PDF isn’t straightforward. I needed to:

1. Extract the complete HTML structure of sample PDFs.
2. Analyze their HTML patterns to configure accurate document extractors.

---

Implementation Steps

1. Creating a Dummy Attribute for HTML Extraction

To extract the entire HTML content of a PDF, I created a dummy attribute that would hold the HTML structure.

• Entity: Content
• Display Name: PDF to HTML
• Attribute Name: pdf_to_html
• Placement: Standard
• Data Type: String

This attribute acted as a placeholder for the extracted HTML content, making it easy to reference later.

---

2. Setting Up a Temporary Web Crawler

To analyze the HTML structure of PDFs, I created a temporary advanced web crawler to process sample PDF URLs.

• Source Configuration:

  • Source Name: Temp PDF to HTML
  • Connector Type: Web Crawler (Advanced)
  • Max Depth: Set to 0 to prevent the crawler from indexing linked URLs.

• Request Triggers:
  I added URLs for a few representative PDFs (covering diverse structures such as text-heavy, image-heavy, and Q&A formats) as request triggers.

This temporary crawler ensured that only the selected PDFs were processed.

---

3. Configuring the JavaScript Document Extractor

To extract the HTML structure, I configured a JavaScript document extractor with the following logic:

// Sample document extractor function to get HTML from PDF content.
function extract(request, response) {
    $ = response.body;
    return [{
        'pdf_to_html': $('html').html(), // Extracts the HTML structure of the document
        'type': "pdf" // Mandatory attribute for defining the document type
    }];
}

This function uses the html() jQuery method to fetch the root HTML structure of each PDF and assigns it to the pdf_to_html attribute.

---

4. Viewing the Extracted HTML in Content Collection

After publishing and running the crawler, I inspected the indexed content to analyze the HTML structure.

• Navigated to Content Collection in the Sitecore CEC.
• Filtered by the source Temp PDF to HTML.
• Opened individual content items to examine the PDF to HTML attribute, which displayed the full HTML structure of the PDF.

By repeating this process for multiple sample PDFs, I was able to identify patterns in the HTML content.

---

Key Insights and Learnings

1. Understand HTML Structures Early:
   Scanning diverse PDFs early helped identify common patterns and edge cases, enabling accurate document extractor configurations.

2. Iterative Testing:
   Viewing the extracted HTML in the Content Collection ensured that the configurations were capturing the desired data accurately.

3. Temporary Sources for Exploration:
   Using a temporary crawler isolated the exploration process without impacting production content or configurations.

---

Final Outcome

With the HTML structures in hand, I configured specialized document extractors tailored to extract meaningful attributes like titles, descriptions, and tags from PDFs. The solution is now scalable to handle a variety of PDF types while ensuring consistent indexing in Sitecore Search.

---

Closing Thoughts

This experience highlights the flexibility and power of Sitecore Search’s document extractors. By leveraging tools like JavaScript extractors and advanced web crawlers, extracting structured content from non-HTML formats like PDFs becomes a manageable and efficient process.