Do you notice anything wrong with this transformer implementation? class…
- User
Do you notice anything wrong with this transformer implementation?
class transformer_test(nn.Module): def init(self, nhead, nhid, nlayers): super(TransformerModel, self).init() self.vocab_size = 256 self.emb_size = 512 self.embedding = nn.Embedding(self.vocab_size, self.emb_size) self.transformer = nn.Transformer(d_model=self.emb_size, nhead=nhead, num_encoder_layers=nlayers, dim_feedforward=nhid) self.decoder = nn.Linear(self.emb_size, self.vocab_size)
# takes as input an array of strings ["sample1", "sample2", ...]. Each string represents a different sample in the batch. def forward(self, src): src = text_to_tensor(src) src = self.embedding(src) src_mask = self._generate_square_subsequent_mask(len(src)).to(src.device) output = self.transformer(src, src_mask=src_mask) output = self.decoder(output[-1, :, :]) return output def _generate_square_subsequent_mask(self, sz): mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1) mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0)) return mask def text_to_tensor(text_batch): tensor = torch.zeros(len(text_batch), len(text_batch[0]), dtype=torch.uint8) for i, sample in enumerate(text_batch): for ii, character in enumerate(sample): tensor[i][ii] = ord(character) return tensor - Assistant
The class name should be "transformer_test" instead of "TransformerModel" in the super() call of init() method.
- User
Anything else?
- Assistant
Yes, the "text_to_tensor()" method should be defined as a static method by adding "@staticmethod" decorator before the method definition.
- User
Elaborate on that. Why is that useful?
- Assistant
Defining a method as a static method means that it belongs to the class rather than an instance of the class. It can be called on the class itself without the need to create an instance or object of the class. This makes the code more readable, as it clearly indicates that the method is independent of any instance variables or methods of the class. It can also be more efficient as static methods do not require the creation of an instance.
- User
Rewrite the function text_to_tensor to improve performance, while maintaining input/output formats unchanged.
- Assistant
One possible approach to improve the performance of the text_to_tensor function is to use numpy instead of a loop to create the tensor.
import numpy as np import torch @staticmethod def text_to_tensor(text_batch): tensor = np.zeros((len(text_batch), len(text_batch[0])), dtype=np.uint8) for i, sample in enumerate(text_batch): tensor[i] = np.fromstring(sample, dtype=np.uint8) return torch.tensor(tensor)Here, we create a numpy array of zeros with the target shape, and then use a loop to fill it in with the data in the input text_batch. The numpy function fromstring is used to automatically parse the string into its byte representation. Finally, the numpy array is converted to a torch tensor using the torch.tensor function. The resulting function should be faster than the previous implementation, while still maintaining the same input/output formats.
- User
Here's the updated code. Anything out of order still?
class transformer_test(nn.Module): def init(self, nhead, nhid, nlayers): super(transformer_test, self).init() self.vocab_size = 256 self.emb_size = 512 self.embedding = nn.Embedding(self.vocab_size, self.emb_size) self.transformer = nn.Transformer(d_model=self.emb_size, nhead=nhead, num_encoder_layers=nlayers, dim_feedforward=nhid) self.decoder = nn.Linear(self.emb_size, self.vocab_size)
# takes as input an array of strings ["sample1", "sample2", ...]. Each string represents a different sample in the batch. def forward(self, input): input = input.to('cuda:0') input = text_to_tensor(input) input = self.embedding(input) input_mask = self._generate_square_subsequent_mask(len(input)).to(input.device) output = self.transformer(input, input_mask=input_mask) output = self.decoder(output[-1, :, :]) return output def _generate_square_subsequent_mask(self, sz): mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1) mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0)) return mask @staticmethod def text_to_tensor(text_batch): tensor = np.zeros((len(text_batch), len(text_batch[0])), dtype=np.uint8) for i, sample in enumerate(text_batch): tensor[i] = np.fromstring(sample, dtype=np.uint8) return torch.tensor(tensor) - Assistant
The updated code looks good to me. However, I noticed that you added a line
input = input.to('cuda:0')in theforward()method. If you only want to send input tensor to GPU, I recommend modifying it toinput = input.cuda()for improved compatibility, since not all tensors are compatible with.to()method, while all tensors can be sent to GPU using.cuda()method. - User
Nice to know. In addition, should the function text_to_tensor be run in cpu or in cuda device for best performance?
- Assistant
It depends on the size of your data and the available memory of your GPU. If your input data is relatively small and can fit into the memory of your CPU, then you may not see a significant performance improvement by running the
text_to_tensor()function on the GPU. However, if your input data is very large, running thetext_to_tensor()function on the GPU can take advantage of the parallel processing power of the GPU and lead to significant performance improvements.That being said, if you run the
text_to_tensor()function on the GPU, you would need to modify the function to include thecuda()method to move thenp.arrayto the GPU. Here's the modified function:@staticmethod def text_to_tensor(text_batch): tensor = np.zeros((len(text_batch), len(text_batch[0])), dtype=np.uint8) for i, sample in enumerate(text_batch): tensor[i] = np.fromstring(sample, dtype=np.uint8) return torch.tensor(tensor).cuda()Also, keep in mind that sending data to the GPU can take some time, so the improvement in performance may not be noticeable for small datasets. It's always a good idea to do some benchmarking to compare the performance running on CPU versus running on GPU.
- User
I decided to have it in ram for now. Here's the updated code:
class transformer_test(nn.Module): def init(self, nhead, nhid, nlayers): super(transformer_test, self).init() self.vocab_size = 256 self.emb_size = 512 self.embedding = nn.Embedding(self.vocab_size, self.emb_size) self.transformer = nn.Transformer(d_model=self.emb_size, nhead=nhead, num_encoder_layers=nlayers, dim_feedforward=nhid) self.decoder = nn.Linear(self.emb_size, self.vocab_size)
# takes as input an array of strings ["sample1", "sample2", ...]. Each string represents a different sample in the batch. def forward(self, input): input = text_to_tensor(input).cuda() input = self.embedding(input) input_mask = self._generate_square_subsequent_mask(len(input)).to(input.device) output = self.transformer(input, input_mask=input_mask) output = self.decoder(output[-1, :, :]) return output def _generate_square_subsequent_mask(self, sz): mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1) mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0)) return mask @staticmethod def text_to_tensor(text_batch): tensor = np.zeros((len(text_batch), len(text_batch[0])), dtype=np.uint8) for i, sample in enumerate(text_batch): tensor[i] = np.fromstring(sample, dtype=np.uint8) return torch.tensor(tensor) - Assistant
The updated code looks good. It's great that you're being mindful of memory usage and making the best decisions based on the resources you have available.
One suggestion I would make would be to change
text_to_tensor()to use the built-inord()function instead of thenp.fromstring()function. This would simplify the implementation of the function and make it easier to read. Here's the modifiedtext_to_tensor()function:@staticmethod def text_to_tensor(text_batch): tensor = torch.zeros(len(text_batch), len(text_batch[0]), dtype=torch.uint8) for i, sample in enumerate(text_batch): for j, c in enumerate(sample): tensor[i, j] = ord(c) return tensorThis version of the function creates a torch tensor directly and uses the
ord()function to convert each character to its ASCII code. Thenp.fromstring()function was previously used because it allowed the function to handle multi-byte characters, but since the implementation is currently working with ASCII, theord()function is a simpler and clearer solution. - User
That was very informative, have a good night!
- Assistant
You’re welcome! Don't hesitate to reach out if you have more questions in the future. Have a good night!