Skip to content
TheChatArchiveThe Chat Archive
Mistralmistral-medium-2505· Mistral AIHugging Face LMArena Human Preference 140K

i have a custom sentence transformer model that is trained to generate both a dense…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-001’s answer.
  1. User

    i have a custom sentence transformer model that is trained to generate both a dense representation (as a single vector, with d = 1024 or 2048) via a custom pooling module, that include learnable parameters) that can be used to compute similarity with cosine similarity, and a token - level representation, as a set of vectors with len = input sequence len and d = 128), that can be used to compute similarity using maxsim.

    i would like to add a layer/s on topo of this model in order use it as a reranker, where i take the output of the model for 2 inputs, and pass those outputs to this new element to extract a similarity score
    
  2. Mistral Assistant

    To create a reranker on top of your custom Sentence Transformer model, you'll need to design a layer (or layers) that can take the outputs from your model (both the dense and token-level representations) and compute a similarity score between two inputs. Here's a step-by-step approach to achieve this:

    1. Understand the Inputs to the Reranker

    Your model outputs two types of representations for each input:

    • Dense representation: A single vector of dimension d = 1024 or 2048.
    • Token-level representation: A sequence of vectors with dimension d = 128 and length equal to the input sequence length.

    For two inputs (e.g., a query and a document), the reranker will receive:

    • Two dense vectors: dense_a and dense_b.
    • Two sets of token-level vectors: tokens_a (shape [seq_len_a, 128]) and tokens_b (shape [seq_len_b, 128]).

    2. Design the Reranker Architecture

    The reranker should combine information from both the dense and token-level representations to compute a similarity score. Here’s a possible architecture:

    Option 1: Simple Combination

    • Compute cosine similarity between the dense vectors: dense_sim = cosine(dense_a, dense_b).
    • Compute maxsim between the token-level representations: token_sim = maxsim(tokens_a, tokens_b).
    • Combine these scores with a learned weight:
      similarity_score = w1 * dense_sim + w2 * token_sim
      
      where w1 and w2 are learnable parameters.

    Option 2: Cross-Attention or Interaction Layer

    For a more sophisticated approach, you can add layers that allow the dense and token-level representations to interact:

    1. Dense Interaction:
      • Concatenate or subtract the dense vectors: dense_interaction = dense_a - dense_b or dense_interaction = torch.cat([dense_a, dense_b, dense_a * dense_b]).
      • Pass this through a feed-forward network (FFN) to get a dense similarity score.
    2. Token-Level Interaction:
      • Use cross-attention between tokens_a and tokens_b to compute a token-level similarity matrix.
      • Pool this matrix (e.g., using max pooling) to get a token-level similarity score.
    3. Combine Scores:
      • Combine the dense and token-level similarity scores using a learned weight or another FFN.

    Option 3: Transformer-Based Reranker

    If you want a more powerful reranker, you can use a small transformer layer:

    1. Flatten the token-level representations and concatenate them with the dense vectors.
    2. Pass this combined representation through a transformer layer to compute a similarity score.
      # Example pseudocode
      combined_input = torch.cat([dense_a, dense_b, tokens_a.flatten(), tokens_b.flatten()])
      similarity_score = transformer_layer(combined_input)
      

    3. Implementation Example

    Here’s a PyTorch implementation for Option 1 (simple combination):

    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    
    class Reranker(nn.Module):
        def __init__(self, dense_dim=1024, token_dim=128):
            super().__init__()
            # Learnable weights for combining dense and token similarities
            self.w_dense = nn.Parameter(torch.tensor(0.5))
            self.w_token = nn.Parameter(torch.tensor(0.5))
    
        def forward(self, dense_a, dense_b, tokens_a, tokens_b):
            # Compute dense similarity (cosine)
            dense_sim = F.cosine_similarity(dense_a, dense_b, dim=-1)
    
            # Compute token similarity (maxsim)
            # tokens_a: [seq_len_a, token_dim], tokens_b: [seq_len_b, token_dim]
            # Compute pairwise cosine similarities
            sim_matrix = torch.matmul(tokens_a, tokens_b.T)  # [seq_len_a, seq_len_b]
            token_sim = sim_matrix.max()  # maxsim
    
            # Combine similarities
            similarity_score = self.w_dense * dense_sim + self.w_token * token_sim
            return similarity_score
    

    4. Training the Reranker

    To train the reranker:

    1. Use a dataset with labeled pairs (e.g., positive and negative pairs).
    2. Define a loss function (e.g., cross-entropy or margin-based loss) to optimize the similarity scores.
    3. Train the reranker end-to-end with the rest of your model or separately.

    5. Advanced Options

    • Add non-linearities (e.g., ReLU, sigmoid) to the combined similarity score.
    • Use attention mechanisms to weigh the importance of different parts of the token-level representations.
    • Experiment with different pooling strategies for the token-level representations (e.g., mean pooling, attention pooling).

    Example with Cross-Attention (Option 2):

    class CrossAttentionReranker(nn.Module):
        def __init__(self, dense_dim=1024, token_dim=128):
            super().__init__()
            self.dense_ffn = nn.Sequential(
                nn.Linear(dense_dim * 2, dense_dim),
                nn.ReLU(),
                nn.Linear(dense_dim, 1)
            )
            self.token_attention = nn.MultiheadAttention(embed_dim=token_dim, num_heads=4)
            self.combiner = nn.Linear(2, 1)  # Combines dense and token scores
    
        def forward(self, dense_a, dense_b, tokens_a, tokens_b):
            # Dense interaction
            dense_interaction = torch.cat([dense_a, dense_b], dim=-1)
            dense_score = self.dense_ffn(dense_interaction).squeeze(-1)
    
            # Token interaction (cross-attention)
            # tokens_a: [seq_len_a, token_dim], tokens_b: [seq_len_b, token_dim]
            # Add batch dimension if needed
            if tokens_a.dim() == 2:
                tokens_a = tokens_a.unsqueeze(0)
                tokens_b = tokens_b.unsqueeze(0)
            attn_output, _ = self.token_attention(
                tokens_a.transpose(0, 1),  # [seq_len_a, batch, token_dim]
                tokens_b.transpose(0, 1),  # [seq_len_b, batch, token_dim]
                tokens_b.transpose(0, 1)
            )
            token_score = attn_output.max(dim=0).values.mean(dim=-1)  # Max pooling
    
            # Combine scores
            combined_score = self.combiner(torch.stack([dense_score, token_score], dim=-1))
            return combined_score
    

    Notes:

    • The choice of architecture depends on your data and computational resources.
    • Start with a simple approach (Option 1) and gradually increase complexity if needed.
    • Monitor performance on a validation set to avoid overfitting.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.