paraphrase following words and make them more formal: Thanks for the comments. Following…
- User
paraphrase following words and make them more formal: Thanks for the comments. Following table provides detailed information about TEQ’s applicable layers in different models. We process linear layers whose transformation scales can be absorbed by their preceding layers (Layer Normalization, etc.). For instance, in one transformer block of OPT-6.7B, QKV layers share the same preceding layers, thus their share one set of trainable parameters. From the statistics we find that very few parameters can leverage TEQ’s training, making our method easy to use.
- Assistant
Thank you for the feedback. The table presented below offers comprehensive details regarding the applicable layers of TEQ in various models. We handle linear layers that possess transformation scales that can be assimilated by their preceding layers, such as Layer Normalization, among others. As an illustration, within a single transformer block of OPT-6.7B, the QKV layers have the same preceding layers and therefore utilize the same set of trainable parameters. Based on the statistics, we have observed that TEQ's training only requires a minimal number of parameters, thereby enhancing the user-friendliness of our approach.