Skip to content
TheChatArchiveThe Chat Archive
Human-written demonstration (OpenAssistant volunteers)· OpenAssistant (human-written)Hugging Face OpenAssistant Conversations (oasst2)

how to use nn.embedding

In OpenAssistant data, “assistant” turns were written by human volunteers as demonstrations. They are not model outputs.
  1. User

    how to use nn.embedding

  2. Assistant

    Given below is the example of PyTorch Embedding:

    import torch
    import torch.nn as nn
    import torch.nn.functional as Fun
    import torch.optim as opt
    torch.manual_seed(2)
    word_conversion = {"hey": 0, "there": 1}
    embeddings = nn.Embedding(2, 3)
    lookup = torch.tensor([word_conversion["hey"]], dtype=torch.long)
    hey_embeddings = embeddings(lookup)
    print(hey_embeddings)
    n, d, m = 2, 4, 6
    embeddings = nn.Embedding(n, d, max_norm=True)
    Weight = torch.randn((m, d), requires_grad=True)
    index = torch.tensor([1, 3])
    x = embedding.weight.clone() @ Weight.t()
    y = embedding(index) @ Weight.t()
    ouputt = (x.unsqueeze(1) + y.unsqueeze(2))
    loss_factor = output.sigmoid().prod()
    loss_factor.backward()
    class NewModel(nn.Module):
    def __init__(self, embed_size, embed_dimension):
    super(NewModel, self).__init__()
    self.embed_size = embed_size
    self.embed_dimension = embed_dimension
    self.u_embeddings = nn.Embedding(embed_size, embed_dimension, sparse=True)
    self.v_embeddings = nn.Embedding(embed_size, embed_dimension, sparse = True)
    self.init_embed()
    def init_embed(self):
    initrange = 0.75 / self.embed_dimension
    self.u_embeddings.weight.data.uniform_(-initrange, initrange)
    self.v_embeddings.weight.data.uniform_(-0, 0)
    def forward(self, pos_u, pos_v, neg_v):
    embed_u = self.u_embeddings(pos_u)
    embed_v = self.v_embeddings(pos_v)
    score = torch.mul(embed_u, embed_v).squeeze()
    score = torch.sum(score, dim = 1)
    score = F.logsigmoid(score)
    neg_embed_v = self.v_embeddings(neg_v)
    negtv_score = torch.bmm(neg_embed_v, embed_u.unsqueeze(2)).squeeze()
    negtv_score = F.logsigmoid(-1 * negtv_score)
    return -1 * (torch.sum(score)+torch.sum(negtv_score))
    
  3. User

    What is an nn.embedding

  4. Assistant

    nn.embedding is a simple lookup table that stores embeddings of a fixed dictionary and size from pytorch. An embedding is like a code book. Given the feature is an index in an array of all possible features.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.