Preemptive entropy reduction in vector embeddings to enhance semantic search

Abstract

At least one processor may receive documents and a set of validation prompts, segment the documents into chunks, embed the chunks into a vector space, query a large language model (LLM) with the validation prompts, intercept chunk retrievals from the vector space in response to the validation prompts, map the retrieved chunks to their positions within the documents, calculate document coverage for the retrieved chunks, generate a report of the document coverage, and refine at least one of the validation prompts, document segmentation, or embedding process in response to the report to reduce entropy in the vector space.

CPC Classifications