Common Web Content (Sitemap) Errors and Solutions

Tips for troubleshooting issues that may occur when you ingest a website content using the Web Content (Sitemap) connector.

A Data Lake Object and a Data Model Object Were Created, But the Search Index and Chunking Failed

When you create a new Web Content connector (Sitemap), you can enable search indexing.

Issue

Search Index or chunks weren’t created.

Solution

From App Launcher, select Data 360.
Open the Search Index tab, select the Search Index you created, and click Rebuild from the dropdown menu on the right.
Manually create a new Search Index from the Search Index tab: Click New -> Easy Setup, and select the corresponding Data Model Object.

Data Stream's Last Run Status Is Failure

The last run of the Data Stream failed.

Issue

The connector’s rate limit is 5 records (pages) per second. If your website doesn’t support such a rate limit, your connector may fail.

Solution

Ask your IT team to authorize the IP addresses listed in IP Addresses Used By Data 360 Services to avoid rate-limit blockages.

My Connector Status Is Complete. None or Not All of My Pages Were Ingested

When you create a Web Content (Sitemap) connector, your website pages are ingested into Data 360.

Issue

Website pages were not ingested.

Solution

Select the checkbox next to Ignore only Disallow in robots.txt, or Ignore robots.txt entirely. These configurations override your website’s robots.txt file. You can ignore the website’s robots.txt file only if you have rights to its contents and IT approval.
If the issue is caused by client-side rendering, where links to subpages are rendered via a script, this problem is a known limitation of the Sitemap connector. Content created from client-side rendering isn’t ingested.

My Connector Completed Successfully, But Zero Records Synced

When you create a Web Content (Sitemap) connector, Data 360 syncs your records.

Issue

Zero records synced.

Solution

Success with zero records happens when the server doesn’t allow Data 360 to crawl any page, because a robots.txt file has “disallow: /” property. To allow the connector to ingest content from your website, try removing this property from your robots.txt file and run the connector again.

Data Stream Status Shows Errors

When you create a connection, you can view your synced records in the Data Streams tab.

Issue

The data stream shows errors.

Solution

Wait for the data stream to finish its run, which can take up to 12 hours. Then access the Data Stream tab to see whether records were ingested.

If records were ingested, the connector ran successfully, but contact your Salesforce admin to start an investigation.
If records weren’t ingested, retry the connector: Click the error status, and then click Retry Now.
If data is still not being ingested, contact your Salesforce admin to start an investigation.