maxlength = 200 def processsentence(sentence, tokenizer, addstartend=False):…
- User
max_length = 200 def process_sentence(sentence, tokenizer, add_start_end=False): sentence_text = sentence.numpy().decode('utf-8') if add_start_end: sentence_text = "[START]" + sentence_text + "[END]" tokenized_sentence = tokenizer.tokenize(sentence_text).merge_dims(-2, -1) return tokenized_sentence
def tokenize_map_fn(french_sentence, english_sentence): fr_sentence_proc = tf.py_function(func=process_sentence, inp=[french_sentence, fr_tokenizer, False], Tout=tf.int64) fr_sentence_proc_deocde = tf.py_function(func=process_sentence, inp=[french_sentence, fr_tokenizer, True], Tout=tf.int64) en_sentence_proc = tf.py_function(func=process_sentence, inp=[english_sentence, en_tokenizer, False], Tout=tf.int64) return fr_sentence_proc, fr_sentence_proc_deocde, en_sentence_proc
Assuming dataset already defined and is a tf.data.Dataset object
processed_dataset = dataset.take(127085).map(tokenize_map_fn)
To efficiently create padded and ragged tensors, first stack all elements from processed_dataset
fr_sequences, fr_sequences_decode, en_sequences = [], [], [] for fr_seq, fr_seq_decode, en_seq in processed_dataset: fr_sequences.append(fr_seq) fr_sequences_decode.append(fr_seq_decode) en_sequences.append(en_seq)
Convert lists of sequences into ragged tensors and then pad them
fr_ragged = tf.ragged.stack(fr_sequences) fr_padded = fr_ragged.to_tensor(default_value=0, shape=[None, None, max_length]) print("fr sequences YES")
fr_ragged_decode = tf.ragged.stack(fr_sequences_decode) fr_padded_decode = fr_ragged_decode.to_tensor(default_value=0, shape=[None, None, max_length]) print("sequencesdeocde Yes")
en_ragged = tf.ragged.stack(en_sequences) en_padded = en_ragged.to_tensor(default_value=0, shape=[None, None, max_length]) print("en_sequences yes")
x_train = fr_padded x2_train = fr_padded_decode y_train = en_padded
TypeError Traceback (most recent call last) Cell In[29], line 16 13 return fr_sentence_proc, fr_sentence_proc_deocde, en_sentence_proc 15 # Assuming dataset already defined and is a tf.data.Dataset object ---> 16 processed_dataset = dataset.take(127085).map(tokenize_map_fn) 18 # To efficiently create padded and ragged tensors, first stack all elements from processed_dataset 19 fr_sequences, fr_sequences_decode, en_sequences = [], [], []
File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\tensorflow\python\data\ops\dataset_ops.py:2202, in DatasetV2.map(self, map_func, num_parallel_calls, deterministic, name) 2199 if deterministic is not None and not DEBUG_MODE: 2200 warnings.warn("The
deterministicargument has no effect unless the " 2201 "num_parallel_callsargument is specified.") -> 2202 return MapDataset(self, map_func, preserve_cardinality=True, name=name) 2203 else: 2204 return ParallelMapDataset( 2205 self, 2206 map_func, (...) 2209 preserve_cardinality=True, 2210 name=name)File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\tensorflow\python\data\ops\dataset_ops.py:5400, in MapDataset.init(self, input_dataset, map_func, use_inter_op_parallelism, preserve_cardinality, use_legacy_function, name) 5398 self._use_inter_op_parallelism = use_inter_op_parallelism 5399 self._preserve_cardinality = preserve_cardinality ... File "C:\Users\ixelo\AppData\Local\Temp\ipykernel_14408\98278579.py", line 10, in tokenize_map_fn * fr_sentence_proc = tf.py_function(func=process_sentence, inp=[french_sentence, fr_tokenizer, False], Tout=tf.int64)
TypeError: Tensors in list passed to 'input' of 'EagerPyFunc' Op have types [string, <NOT CONVERTIBLE TO TENSOR>, bool] that are invalid. Tensors: [<tf.Tensor 'args_0:0' shape=() dtype=string>, <tensorflow_text.python.ops.bert_tokenizer.BertTokenizer object at 0x000001CAAA3B6FE0>, False]Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...
- Assistant
The error you are facing is because
tf.py_functionexpects all of its inputs to be TensorFlow tensors, but you are passing a non-tensor object (fr_tokenizeranden_tokenizer, which are instances of some tokenizer class) directly to this function. Thetf.py_functionwrapper allows you to run arbitrary Python code within TensorFlow operations, but all inputs to this function need to be tensors, and non-tensor objects should be wrapped or handled differently.To fix the issue, you should modify your approach to avoid passing non-tensor objects (like tokenizers) into
tf.py_function. Instead, consider two alternative strategies:Strategy 1: Move Tokenization Outside
tf.py_functionIf the tokenizer can operate on batched inputs, you could apply tokenization to the entire dataset outside of
tf.py_function. This strategy, however, might not be feasible if the tokenizer expects non-tensor Python strings or if you must usetf.py_functionfor other reasons.Strategy 2: Use Global Tokenizer Within
tf.py_functionYou can access global objects from within
tf.py_function. Instead of passing the tokenizer as an argument, ensure the tokenizer is accessible globally or within the scope whereprocess_sentenceis defined. Adjust the function to not take the tokenizer as a parameter but use it directly from the global scope.Here is how you might adjust the
tokenize_map_fnfunction:# Assuming fr_tokenizer and en_tokenizer are defined globally or in an accessible scope def process_sentence(sentence, add_start_end=False, language="fr"): sentence_text = sentence.numpy().decode('utf-8') tokenizer = fr_tokenizer if language == "fr" else en_tokenizer if add_start_end: sentence_text = "[START]" + sentence_text + "[END]" tokenized_sentence = tokenizer.tokenize(sentence_text).merge_dims(-2, -1) return tokenized_sentence def tokenize_map_fn(french_sentence, english_sentence): fr_sentence_proc = tf.py_function(func=lambda s: process_sentence(s, False, "fr"), inp=[french_sentence], Tout=tf.int64) fr_sentence_proc_decode = tf.py_function(func=lambda s: process_sentence(s, True, "fr"), inp=[french_sentence], Tout=tf.int64) en_sentence_proc = tf.py_function(func=lambda s: process_sentence(s, False, "en"), inp=[english_sentence], Tout=tf.int64) return fr_sentence_proc, fr_sentence_proc_decode, en_sentence_procKeep in mind that using
tf.py_functioncan reduce performance and might not be fully compatible with all TensorFlow optimizations or deployments (e.g., TensorFlow Lite, TensorFlow JS). If possible, consider using TensorFlow-native operations or tokenizers designed to work directly ontf.Tensorobjects for better performance and compatibility.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.