Multi-layered method of prefiltering and ranking for text based vector retrieval

Abstract

Methods, system, and non-transitory processor-readable storage medium for a text-based vector retrieval system are provided herein. An example method includes generating, by a text-based vector retrieval system, a set of topics Tx for a document set DX, generating for each document DN a set of topics TD, comparing the set of topics TD against the set of topics TX to generate a filtered set of topics TDFinal, comparing the topics TDFinal against final topics of all other documents to generate a set DNMatching, encoding each document from the document set into a vector database with corresponding DNMatching and TDFinal as metadata, grouping vectors into N search clusters, retrieving N topics from a query string to generate TQuery topics, comparing the TQuery topics against topics in each cluster to identify matching clusters having R matching topics, and calculating similarity scores for documents in the matching clusters.

CPC Classifications