Human-written demonstration (OpenAssistant volunteers)· OpenAssistant (human-written)
OpenAssistant Conversations (oasst2)
how to use nn.embedding
In OpenAssistant data, “assistant” turns were written by human volunteers as demonstrations. They are not model outputs.
- User
how to use nn.embedding
- Assistant
Given below is the example of PyTorch Embedding:
import torch import torch.nn as nn import torch.nn.functional as Fun import torch.optim as opt torch.manual_seed(2) word_conversion = {"hey": 0, "there": 1} embeddings = nn.Embedding(2, 3) lookup = torch.tensor([word_conversion["hey"]], dtype=torch.long) hey_embeddings = embeddings(lookup) print(hey_embeddings) n, d, m = 2, 4, 6 embeddings = nn.Embedding(n, d, max_norm=True) Weight = torch.randn((m, d), requires_grad=True) index = torch.tensor([1, 3]) x = embedding.weight.clone() @ Weight.t() y = embedding(index) @ Weight.t() ouputt = (x.unsqueeze(1) + y.unsqueeze(2)) loss_factor = output.sigmoid().prod() loss_factor.backward() class NewModel(nn.Module): def __init__(self, embed_size, embed_dimension): super(NewModel, self).__init__() self.embed_size = embed_size self.embed_dimension = embed_dimension self.u_embeddings = nn.Embedding(embed_size, embed_dimension, sparse=True) self.v_embeddings = nn.Embedding(embed_size, embed_dimension, sparse = True) self.init_embed() def init_embed(self): initrange = 0.75 / self.embed_dimension self.u_embeddings.weight.data.uniform_(-initrange, initrange) self.v_embeddings.weight.data.uniform_(-0, 0) def forward(self, pos_u, pos_v, neg_v): embed_u = self.u_embeddings(pos_u) embed_v = self.v_embeddings(pos_v) score = torch.mul(embed_u, embed_v).squeeze() score = torch.sum(score, dim = 1) score = F.logsigmoid(score) neg_embed_v = self.v_embeddings(neg_v) negtv_score = torch.bmm(neg_embed_v, embed_u.unsqueeze(2)).squeeze() negtv_score = F.logsigmoid(-1 * negtv_score) return -1 * (torch.sum(score)+torch.sum(negtv_score)) - User
What is an nn.embedding
- Assistant
nn.embeddingis a simple lookup table that stores embeddings of a fixed dictionary and size from pytorch. An embedding is like a code book. Given the feature is an index in an array of all possible features.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →
- Human-written demonstration (OpenAssistant volunteers)Compare side by side →