Use LLM Gateway Calls in Custom Chunking Functions

Make LLM gateway calls from custom chunking functions when your chunking behavior depends on semantic understanding. Use this approach to define chunking behavior in prompt instructions and map model responses to the required response schema.

• Set up Salesforce CLI, the Data Code Extension plugin, and the Python SDK. See Set Up Salesforce CLI for Custom Functions.
• Initialize a function package and make sure that you can run local function validation commands. See Write a Custom Chunking Function.
• Use a Salesforce org with at least one generative AI model configured.
• Configure external client app access for model API authentication. Complete Step 1 and Step 2 from Access Models API with REST.

Implement this flow inside the function(request, runtime) callable in payload/entrypoint.py: prompt construction, LLM Gateway invocation, response parsing, and chunk mapping. For more information on writing a custom function, see Write and Validate a Custom Chunking Function.

1. Read the document elements to chunk from request.input.

2. Build the prompt that you send to the model. Write your chunking instructions in a template file or directly in entrypoint.py, and insert the document text from request.input into those instructions at run time.

   For a reference implementation, see example/chunking_with_llm/entrypoint.py and example/chunking_with_llm/files in the initialized function package.

3. Build a text-generation request with GenerateTextRequestBuilder and set a supported model.

4. Call runtime.llm_gateway.generate_text(...).

5. Parse the response text, for example, by using ---CHUNK---.

6. Map each segment to a chunk (text, seq_no, chunk_type). Aggregate chunks across all elements with a continuous seq_no, and return SearchIndexChunkingV1Response(output=chunks).

Handle failed Gateway responses explicitly. Log failures and raise controlled errors. Don’t place sensitive data in prompts or logs.

1. From your function package root, confirm that your function entrypoint and test payload files are ready for local validation.

2. Configure SDK authentication for local LLM testing.

   • Configure client credentials:
     • datacustomcode configure --auth-type client_credentials
   • Provide your org domain, connected app consumer key, and consumer secret when prompted.

3. Run local validation against your function entrypoint and test the payload.

   • sf data-code-extension function run -e payload/entrypoint.py -t tests/test.json

• Write a Custom Chunking Function
• Considerations When Writing Custom Chunking Functions
• Use Predictive Model Calls in Custom Chunking Functions
• Use Custom Functions in Data 360