Check if the website you’re ingesting blocks the connector. Some websites may block the connector in two possible ways.
User-Agent Blocking: By default, the connector uses the Mozilla/5.0 user agent. Some sites may reject it. If your connection test fails, try entering a different user agent in the User Agent field.
IP Blocking: Sites using services like Cloudflare may block unauthorized IPs. Confirm with your IT or security teams that the website doesn’t block an IP address used by Data 360 listed in IP addresses used by Data 360 services.
New Data Fails to Ingest
When you ingest data using the Web Content (Sitemap) connector, Data 360 syncs it.
My Sync Completed, But No Records Are Showing on the Data Stream Tab
When you ingest content from a website using the Web Content (Sitemap) connector, your records can be viewed on the Data Stream tab.
Issue
No records were ingested.
Solution
Check the format of your ingested files. The Web Content (Sitemap) connector currently supports only HTML and PDF files. Data Cloud skips and doesn’t index other file formats.
Dynamic Content and JavaScript Rendering: The Web Content (Sitemap) connector doesn’t support content that is dynamically rendered via JavaScript at runtime.
To validate whether this is the issue, inspect the website page source.
Option 1
Right-click on the page.
From the context menu, click View Page Source (in Mac, click command+option+u).
Check if the expected content is present. If the content is only visible after the page loads in a browser and not in the raw HTML source, it’s likely that it’s JavaScript-rendered, and can’t be ingested.
Option 2
Right-click on the page.
From the context menu, click Inspect.
Click the Settings icon button on the top right.
Scroll down to Debugger and check Disable JavaScript.
Refresh the page to view what the connector can ingest.
j