How to request OpenSeaech to rank results regarding to similarity of textual document…
- User
How to request OpenSeaech to rank results regarding to similarity of textual document field to a value while the documents ids are specified in the query
- Assistant
Below is a “recipe” that works in any recent OpenSearch / Elasticsearch-compatible cluster (2.x+).
The idea is always the same:- Narrow the result set to the document IDs you care about (that part must NOT influence the score).
- Add a scoring clause that compares the chosen text field with your “reference” text (keyword, sentence, paragraph, vector, …).
- Let OpenSearch sort by the score it just calculated.
────────────────────────────────────────
- Simple BM25 lexical similarity
────────────────────────────────────────
If you only need classic keyword‐based relevance (BM25) you can use a normal
match(ormulti_match) inside abool, while the IDs sit in thefilterclause:
POST my-index/_search { "size": 20, "query": { "bool": { "filter": { // step 1: pin the docs "ids": { "values": ["doc1","doc17","doc99"] } }, "must": { // step 2: score them "match": { "body": { "query" : "refund request process", "operator": "and" // tweak as you prefer } } } } } }•
filter→ zero cost for scoring (yes/no gate).
•must→ BM25 score decides the ranking among the survivors.──────────────────────────────────────── 2. “More-Like-This” lexical similarity ──────────────────────────────────────── If your reference text is longer than a couple of words,
more_like_this(MLT) gives slightly better lexical similarity than a plainmatch:POST my-index/_search { "size": 20, "query": { "bool": { "filter": { // ID filter again "ids": { "values": ["doc1","doc17","doc99"] } }, "must": { // MLT scoring "more_like_this": { "fields" : ["body"], "like" : "This customer is asking how to get a refund for a returned product.", "min_term_freq" : 1, "max_query_terms": 12 } } } } }──────────────────────────────────────── 3. Dense-vector (semantic) similarity ──────────────────────────────────────── If you already store sentence-/document-embeddings in a
dense_vectorfield (or intend to), you’ll get much better “conceptual” matches.Mapping snippet (once, when you create the index):
"body_vector": { "type": "knn_vector", "dimension": 768, // or 384, 1024… whatever your model emits "method": { // typical Faiss/HNSW settings "name": "hnsw", "engine": "faiss", "space_type": "cosinesimil" } }3a) Preferred (OpenSearch ≥ 2.3):
knnquery with native filterPOST my-index/_search { "size": 10, "query": { "knn": { "field" : "body_vector", "query_vector" : [0.11, 0.42, …], // embedding of your reference text "k" : 10, // final top-K to return "num_candidates": 50, // candidate pool "filter": { // <-- filter lives INSIDE knn! "ids": { "values": ["doc1","doc17","doc99"] } } } } }3b) Works in any version:
script_scorewrapperPOST my-index/_search { "size": 10, "query": { "script_score": { "query": { // just the ID filter "ids": { "values": ["doc1","doc17","doc99"] } }, "script": { "source": """ cosineSimilarity(params.qv, 'body_vector') + 1.0 """, "params": { "qv": [0.11, 0.42, …] // embedding of reference text } } } } }•
+ 1.0avoids negative scores (they get clipped to 0 otherwise).
• Exactly the same principle works for dot-product or L2 distances.──────────────────────────────────────── 4. If you want all IDs back, even when similarity is 0 ──────────────────────────────────────── Suppress the “must” clause completely and only use
script_score(or useshouldwith"minimum_should_match":0).
They will all come back, purely ordered by the similarity score.──────────────────────────────────────── 5. Tips & gotchas ──────────────────────────────────────── • An
idsfilter is the most efficient way to pin specific docs. Internally it’s just a bitmap.
• >10 000 IDs? Use atermsquery on_idinstead ofids(same effect, avoids the 10k hard-limit).
• Combine multiple filters (dates, user ACLs, …) in the samebool.filterarray.
• Need re-ranking? Retrieve, say, top-100 with BM25 then run a second request with vector re-score. OpenSearch hasrescorefor exactly that.That’s essentially it:
Bool-filter your IDs ➜ attach any scoring query you like ➜ enjoy properly ranked results 🚀
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.