You build your vector database, craft an Agentforce agent, and link your search indexes to corporate knowledge. Then during user testing, your agent misses a glaring detail from a PDF or misinterprets a conversation entirely. The common reaction is to blame the LLM. The reality? Your chunking strategy fractured the data before the model ever saw it.

If a text snippet lacks proper context when retrieved from the vector database, your agent cannot generate an accurate response. This is the structural reality of retrieval-augmented generation (RAG): garbage in, garbage out.

This post shows how pro-code chunking for Data 360 search indexes using Data 360 Code Extension functions can be used to address three common chunking failures: fragmented data tables, multispeaker dialogue, and exposed personally identifiable information (PII). For each, it compares native chunking against custom pro-code chunking and shows the impact on agent responses.

Where native chunking falls short

Data 360 offers reliable native chunking for standard content. Mechanisms like Section-aware chunking and Semantic-based passage extraction cut documents along structural headings or thematic boundaries. They prevent sentences from splitting awkwardly and keep related paragraphs together.

However, enterprise data is rarely clean. It lives in complex multipage PDFs, regulatory records, dense tables, and multispeaker transcripts. When this data is fed into a character-count or token-length splitter, semantic continuity shatters.

How to write and deploy Code Extension functions

Note: The code examples in this post are intended to illustrate the concepts described in a simple way. They are designed for representative, happy-path examples, not for the full range of possible real-world inputs. Because the implementations rely on regular expressions (regex) and other pattern-based heuristics, they can produce both false positives and false negatives. For example, a heuristic might incorrectly classify an ordinary value as an account number or fail to recognize a valid table, speaker label, timestamp, or identifier in an unexpected format. Production-grade implementations of these functions are beyond the scope of this post.

Data 360 Code Extension functions let you deploy pro-code Python scripts natively within the Salesforce trust boundary for unstructured data pipelines where the built-in chunking behavior isn’t enough.

A function is a small, serverless-style unit that takes an input, performs a computation, and returns an output, running fully sandboxed and stateless. You author and debug it locally in your favorite IDE with the Data Custom Code SDK, deploy it through the Salesforce CLI or the UI, and wire it into your pipeline at the search index creation step. A code extension function can also call a generative or predictive model when your use case needs it. To go deeper, see the developer documentation.

How we tested

To test Code Extension custom chunking capabilities and compare them with native chunking capabilities, we evaluated three different documents. You can explore the structure of a chunking function and learn how to deploy it and run it during search index creation in the developer docs.

For each use case, we followed the same methodology:

1. Chunk the same source document via two separate paths:
   
   1. Path A (Native): The standard Agentforce Data Libraries (ADL) route with default chunking configuration
   2. Path B (Code Extension): Custom Python chunking function deployed via Code Extension
2. Query the resulting vector search indexes for Path A chunks using:
   

3. Query the chunks for Path B by querying the unstructured data model object (UDMO) directly:
   

4. Compare the chunks retrieved and evaluate what an agent would answer from each.

Use case 1: fragmented data tables

Imagine you are indexing a quarterly financial report (ABC_corp_q4_fy2025_financial_report.pdf): a 5-page PDF containing an executive summary, four dense multicolumn data tables, and forward-looking guidance.

Here are a few snapshots from the financial report document:

The Problem: Standard splitters often slice text strictly by character or token limits. Tables spanning multiple columns lose the relationship between headers and data rows. When your agent retrieves a chunk containing raw numbers, it has no column context to understand what those numbers mean.

You can solve this problem by implementing a structure-aware Python function that detects table boundaries (for example, markdown pipe tables, grid tables, and HTML tables), keeps each table as a unified block, and when a table exceeds the chunk size limit, splits it into sub-chunks with column headers repeated at the top of each sub-chunk.

Sample code

Native chunks: what you get without Code Extension

Here is what native chunking produced, a wall of numbers with no structural context:

Native Chunk Example 1 (Raw Data Dump):

Native Chunk Example 2 (Fragmented Table):

The Problem: As just one example, the numbers following “AI & Automation” (42.1 35.8 22.4 17.6 87.9 82.5 28.1 410) sit in a chunk with no column headers. What is 42.1? Revenue? A margin? A growth rate? Without the header row (Segment | Q4 FY25 Rev($M) | Q3 FY25 Rev($M) | ... | Headcount), these numbers are meaningless. Your agent cannot answer even basic questions about this data.

Code Extension chunks: structure preserved

The custom function detects table boundaries, preserves the header row, and produces structured chunks:

Custom Chunk Example 1 (Executive Summary — Clean Prose):

Custom Chunk Example 2 (Table with Headers Preserved — Revenue by Segment):

Custom Chunk Example 3 (Regional Performance — Each Region Self-Contained):

Impact on agent responses

User Query: “What is the YoY growth rate for the AI & Automation segment and how does its operating margin compare to Cloud Services?”

Use case 2: multispeaker dialogue

Now consider a use case in which you’re grounding your service agents on a recorded customer escalation call: a 45-minute platform migration planning session with five participants (Sarah Chen, VP of Engineering; Marcus Webb, Solutions Architect; Priya Desai, Database Lead; Tom Nakamura, DevOps Manager; Lisa Okafor, Customer Success Director).

Source document: Globex_Migration_Escalation_Call_2025_Q4.pdf

Here is a snapshot from the source document:

The Problem: Standard chunking produces massive walls of unstructured text. Speaker boundaries are lost, timestamps disappear, and topical shifts get merged into a single chunk. The result: chunks thousands of characters long where you cannot tell who said what or when.

You can solve this by using a windowed sliding dialogue mechanism with these parameters: WINDOW_SIZE = 6 turns, STRIDE = 3 turns, MAX_CHUNK_CHARS = 1500. This helps ensure that every question and its answer are almost always co-located in at least one chunk. Adjacent chunks overlap so context is virtually never orphaned.

Sample code

Native chunks: what you get without Code Extension

Native Chunk Example (Wall of Text — No Speaker Attribution):

The Problems:

1. No speaker separation: Turns from Marcus, Sarah, Tom, and Priya are mashed together in a single paragraph
2. Timestamps embedded randomly: 00:25:46 appears midsentence with no structural break
3. Questions divorced from answers: Marcus asks “can Kevin work with the DBA team?” and Priya’s answer is buried without clear attribution
4. Topic bleed: The Oracle-to-Postgres migration capacity discussion runs directly into the SLA penalty discussion with no boundary

Code Extension chunks: windowed dialogue

The custom function parses speaker turns, then creates overlapping windows:

Custom Chunk Example 1 (Clear Speaker Attribution and Overlap):

Custom Chunk Example 2 (Decision and Response Together):

Key Difference: Each chunk now contains multiple turns from the same topical exchange. The sliding window (stride of 3) helps ensure that a question asked in one chunk is always paired with its answer. Speaker names are preserved as structural anchors.

Impact on agent responses

User Query: “What is the committed downtime window for enterprise tier customers during the database migration?”

Use case 3: PII masking and regulatory cleansing

In this use case, you’re operating in a regulated sector (for example, healthcare, banking, or public sector) where internal identifiers must be sanitized before ingestion into a vector database that agents will query.

Source document: Sample_PII_Masking_Test_Document.pdf is a five-page document with simulated clinical intake records, banking credit memos, and public sector benefits cases. It contains both standard PII and proprietary identifiers.

Here is a snapshot from the source document:

The Problem: Standard pattern recognition is not applied during chunking via the ADL route. All identifiers, standard and proprietary, pass through into the vector index unmasked.

You can address this by implementing regex-based pattern matching over the raw text inside your custom function before chunking. This cleanses SSNs, credit card numbers, phone numbers, email addresses, employer identification numbers (EINs), ICD codes, and other industry-specific formats on-the-fly. The data is sanitized before it ever reaches the chunk data model object (DMO).

Sample code

Native chunks: all PII exposed

Native Chunk Example 1 (Healthcare — Full PII Exposed):

Note: The X characters below are synthetic placeholders; they represent identifiers that are present and unmasked, not values that have already been redacted.

Native Chunk Example 2 (Banking — All Identifiers Visible):

Native Chunk Example 3 (Cross-System — Re-identification Risk):

The Problem: Every sensitive identifier sits unmasked in the vector index:

• SSNs: XXX-XX-XXXX, XXX-XX-XXXX, XXX-XX-XXXX
• Credit card number: XXXX-XXXX-XXXX-XXXX
• Email address: m.thornton@example.com
• Phone number: (XXX) XXX-XXXX
• Medical record number: MRH-XXXX-XXXXXX-X
• Internal KYC profile: KYC-IND-XX-XXXX-XXXXXX
• Payer member ID: BCBSIL-GRPXXXX-MXXXXXXX
• DHS case number: DHS-IL-XXXX-XXXX-XXXXXX-XX

When your agent retrieves these chunks, it may echo PII directly back to end users, which constitutes a compliance violation under HIPAA, PCI-DSS, and many state privacy laws.

Code Extension chunks: PII masked before indexing

The custom function applies regex-based pattern matching to sanitize sensitive data before chunking:

Custom Chunk Example 1 (Healthcare — Standard PII Redacted):

Note the ‘[SWIFT_REDACTED]’ token replacing ‘INTERNAL’: an over-broad SWIFT-code pattern matched a plain word. Also note that certain PII,  such as names, addresses, dates of birth are not handled by the example function).

Custom Chunk Example 2 (Banking — Standard PII and Financial IDs Redacted):

Custom Chunk Example 3 (Cross-System Events — SSN and EIN Masked):

Custom Chunk Example 4 (Public Sector — Government IDs Masked):

What got caught and what needs custom patterns

In this test, the Code Extension function successfully redacted:

• SSNs: XXX-XX-XXXX → [SSN_REDACTED]
• Credit card numbers: XXXX-XXXX-XXXX-XXXX → [CREDIT_CARD_REDACTED]
• Email addresses: m.thornton@example.com → [EMAIL_REDACTED]
• Phone numbers: (XXX) XXX-XXXX → [PHONE_REDACTED]
• EINs: 00-0000000 → [EIN_REDACTED]
• ICD Codes: J96.01 → [ICD_CODE_REDACTED]

Proprietary identifiers that require organization-specific patterns (these passed through, demonstrating why you must extend the pattern registry for your formats):

• MRH-XXXX-XXXXXX-X — Medical record number
• BCBSIL-GRPXXXX-MXXXXXXX — Payer member ID
• DHS-IL-XXXX-XXXX-XXXXXX-XX — DHS case number
• ILHIE-PKT-XXXXXXXX-XXXX — HIE patient token

The key insight: Code Extension gives you the framework to add any pattern your organization needs. Standard numeric identifier patterns are handled by the existing sample code. Proprietary formats are added as one-line regex entries:

code here 22

Impact on agent responses

User Query: “What was the clinical pathway for the patient admitted on October 17th?”

Compliance implications

The bottom line: without Code Extension, everything goes into the vector store unmasked. With the Code Extension function, PII formats covered by the sample patterns are masked  during search index chunking, and the extensible pattern registry gives you a path to add proprietary formats as you discover them.

Conclusion

Code Extension functions are not a replacement for native chunking; they provide an escalation path when enterprise data complexity outpaces standard algorithms. Investing in pro-code chunking ensures your search indexes remain contextually rich and drive accurate Agentforce responses.

Get started with Code Extension

What are you waiting for? Start building Code Extensions today: Enable Code Extension in Data 360 setup feature manager, install the Data Custom Code SDK and the Salesforce CLI, and start building right away. To learn more:

• Check out the Code Extension function documentation.
• Watch the Extend Data 360 with Code Extension YouTube playlist.

About the authors

Jitin Mehndiratta is a Product Management Director at Salesforce working on Data 360. He focuses on enabling developers to build intelligent data pipelines that power Agentforce experiences.

Samaritha Patlori is a Product Manager for Code Extension at Salesforce Data 360. With 7+ years across engineering and product, she is currently focused on enabling developers to run custom code securely within Data 360.