Reranking

Retrieval's job is to get the answer somewhere into the top 50. Reranking's job is to get it into the top 3. They're different problems, solved by different models, and knowing how they split the work lets you build search that finds the answer and puts it where the model will read it.

Suppose hybrid retrieval returns 50 candidates with recall@50 of 0.94. The answer is almost always in there, but you can only afford to pass the top 3 to 5 to the model because of cost, latency and lost-in-the-middle. If the best chunk is at rank 22, the model never sees it. A reranker scores each candidate against the query again, more carefully, and reorders the list.

Bi-encoders and cross-encoders

Your retriever is a bi-encoder: it embeds the query and each document separately and compares the vectors. That's what makes it fast, since document vectors are computed once in advance. The model never sees the query and the document together, though, so it can't judge how well one answers the other.

A cross-encoder reads the query and one document together and returns a single relevance score. It can tell whether the document answers the question or just shares its topic. It has to run once per candidate, so it's far too slow to use on a whole corpus. The standard design follows from that: retrieve widely with a fast bi-encoder (top 50 to 100), then rerank that short list with a cross-encoder and pass the top 3 to 5 on.

Measuring rank as well as presence

Recall@k is yes or no: is the gold chunk in the top k? Reranking cares about position inside the list, so you need metrics that notice rank. MRR (mean reciprocal rank) averages 1 divided by the rank of the first relevant result, so it rewards getting the answer to position 1. nDCG also rewards ranking several relevant results highly.

Recall@50 doesn't move, because reranking never adds anything retrieval missed. Recall@3 jumps from 0.61 to 0.88: 27 points of questions whose answer now reaches the model, with no change to retrieval. That's why a reranker is often the most valuable single addition to a working retriever.

Starter code

# Each query: the retriever's ordered candidate list (doc dicts) and the gold doc id.
# 'answers' = True means the doc actually answers the query (what a cross-encoder detects);
# 'topical' = shared-topic score the bi-encoder over-weights (why gold sits mid-rank).
QUERIES = [
  {"gold": "d1", "cands": [
     {"id":"d9","answers":False,"topical":0.9}, {"id":"d4","answers":False,"topical":0.8},
     {"id":"d1","answers":True, "topical":0.5}, {"id":"d7","answers":False,"topical":0.4}]},
  {"gold": "d2", "cands": [
     {"id":"d5","answers":False,"topical":0.85},{"id":"d2","answers":True,"topical":0.6},
     {"id":"d8","answers":False,"topical":0.55},{"id":"d3","answers":False,"topical":0.3}]},
  {"gold": "d6", "cands": [
     {"id":"d6","answers":True,"topical":0.7}, {"id":"d0","answers":False,"topical":0.65},
     {"id":"d1","answers":False,"topical":0.5}, {"id":"d9","answers":False,"topical":0.2}]},
]

def rank_of_gold(order, gold):
    return next(i+1 for i, c in enumerate(order) if c["id"] == gold)

def recall_at(orders, k):
    return sum(rank_of_gold(o, g) <= k for o, g in orders) / len(orders)

def mrr(orders):
    return sum(1.0 / rank_of_gold(o, g) for o, g in orders) / len(orders)

# cross-encoder: reads query+doc jointly, so it *knows* which doc answers.
def cross_encoder_score(c):
    return (1.0 if c["answers"] else 0.0) + 0.1 * c["topical"]

# TODO: build 'before' orders (as given) and 'after' orders (sorted by cross_encoder_score desc),
# then print recall@3 and MRR for each. Pair each order with its gold id.

before = [(q["cands"], q["gold"]) for q in QUERIES]
after  = [(sorted(q["cands"], key=cross_encoder_score, reverse=True), q["gold"]) for q in QUERIES]

for name, orders in (("before", before), ("after", after)):
    print(f"{name}: recall@3={recall_at(orders,3):.2f}  MRR={mrr(orders):.2f}")