![]() ![]() Those elements are just scanned sequentially, and the search function returns the top-k smallest distances seen so far. The encoding is mainly to make the vectors more compact. The elements of inverted lists are encoded vectors (+ the corresponding vector id). In some contexts it is beneficial to use other types of quantizers, for example a GPU based quantizer, a MultiIndexQuantizer or a HNSW based quantizer. This leads to a number of centroids of the form nlist = C * sqrt (n). NOTE: As a rule of thumb, denoting by n the number of points to be indexed, a typical way to select the number of centroids is to aim at balancing the cost of the assignment to the centroids ( nlist * d for a plain k-means) with the number of distance computations performed when parsing the inverted lists (in the order of nprobe / nlist * n * C, where the constant accounts for the uneven distribution of the list and the fact that a single vector comparison is done more efficiently when done by batch with centroids, say C=10 to give an idea). The nprobe is specified at query time (useful for measuring trade-offs between speed and accuracy). The train method of the IndexIVF adds the centroids to the flat index. Typically, one would use a Flat index as coarse quantizer. Cell probe method with a flat index as coarse quantizer The query is searched in this index, and the returned vector id(s) are the inverted list(s) that should be visited. The constructor takes an index as a parameter (the quantizer or coarse quantizer), which is used to do the assignment to the inverted lists. The failure case appears when the cell of the nearest neighbor of a given query is not selected. The query is compared to each of the database vector assigned to these lists.ĭoing so, only a fraction of the database is compared to the query: as a first approximation, this fraction is nprobe/nlist, but this approximation is usually under-estimated because the inverted lists have not equal lengths.At query time, a set of nprobe inverted lists is selected.The database vectors are assigned to one of these cells thanks using a quantization function (in the case of k-means, the assignment to the centroid closest to the query), and stored in an inverted file structure formed of nlist inverted lists.The feature space is partitioned into nlist cells.We use a partition-based method based on Multi-probing. The corresponding algorithms are sometimes referred to as cell-probe methods. At each quantization stage, the size of the codebook can be refined.Ī typical way to speed-up the process at the cost of loosing the guarantee to find the nearest neighbor is to employ a partitioning technique such as k-means. Residual encodings ( IndexResidual): vectors are quantized and progressively refined by residual.PQ encoding ( IndexPQ): vectors are split into sub-vectors that are each quantized to a few bits (usually 8).8/6/4-bit integer encoding ( IndexScalarQuantizer with QT_8bit/ QT_6bit/ QT_4bit): vectors quantized to 256/64/16 levels.16-bit float encoding ( IndexScalarQuantizer with QT_fp16): the vectors are compressed to 16-bit floats, which may cause some loss of precision.no encoding at all ( IndexFlat): the vectors are stored without compression.The available encodings are (from least to strongest compression): ![]() Note that this shrinks the index and changes the numbering. They do support efficient direct vector access (with reconstruct and reconstruct_n) They don't support add_with_id (but they can be wrapped in an IndexIDMap to add that functionality). They do not store vector ids, since in many cases sequential numbering is enough. Flat indexesįlat indexes just encode the vectors into codes of a fixed size and store them in an array of ntotal * code_size bytes.Īt search time, all the indexed vectors are decoded sequentially and compared to the query vectors.įor the IndexPQ the comparison is done in the compressed domain, which is faster. The index can be constructed explicitly with the class constructor, or by using index_factory. Quantizer, d, nlists, M, nbits, M_refine, nbits_refine IVFADC+R (same as IVFADC with re-ranking based on codes) IVFADC (coarse quantizer+PQ on residuals) Optimized by using random rotation instead of random projectionsĤ and 6 bits per component are also implemented. Locality-Sensitive Hashing (binary flat index) The 8 additional bytes are the vector id that needs to be stored. Takes another index to assign vectors to inverted lists. Inverted file with exact post-verification Hierarchical Navigable Small World graph exploration The basic indexes are given hereafter: MethodĪlso for cosine (normalize vectors beforehand) How to make Faiss run faster Typical use cases and benchmarks Index IO, cloning and hyper parameter tuningįast accumulation of PQ and AQ codes (FastScan) Guidelines to choose an index Faiss indexes Faiss building blocks: clustering, PCA, quantization ![]()
0 Comments
Leave a Reply. |