Skip to content
The Chat Archive
gpt-4-0125-preview· OpenAI WildChat-4.8M (AI2)

maxlength = 200 def tokenizeandmerge(frsentence, ensentence): frtokens =…

  1. User

    max_length = 200

    def tokenize_and_merge(fr_sentence, en_sentence): fr_tokens = fr_tokenizer.tokenize(fr_sentence.np().decode('utf-8')).merge_dims(-2, -1) fr_decode_tokens = fr_tokenizer.tokenize("[START]" + fr_sentence.np().decode('utf-8') + "[END]").merge_dims(-2, -1) en_tokens = en_tokenizer.tokenize(en_sentence.np().decode('utf-8')).merge_dims(-2, -1) return fr_tokens, fr_decode_tokens, en_tokens

    dataset = dataset.map(tokenize_and_merge, num_parallel_calls=tf.data.AUTOTUNE) dataset = dataset.padded_batch(batch_size=32, padded_shapes=([max_length], [max_length], [max_length]), padding_values=0)

    x_train = [] x2_train = [] y_train = []

    for fr_batch, fr_decode_batch, en_batch in dataset: x_train.append(fr_batch) x2_train.append(fr_decode_batch) y_train.append(en_batch)

    x_train = tf.concat(x_train, axis=0) x2_train = tf.concat(x2_train, axis=0) y_train = tf.concat(y_train, axis=0)

    print("x_train shape:", x_train.shape) print("x2_train shape:", x2_train.shape) print("y_train shape:", y_train.shape)


    AttributeError Traceback (most recent call last) Cell In[26], line 9 6 en_tokens = en_tokenizer.tokenize(en_sentence.np().decode('utf-8')).merge_dims(-2, -1) 7 return fr_tokens, fr_decode_tokens, en_tokens ----> 9 dataset = dataset.map(tokenize_and_merge, num_parallel_calls=tf.data.AUTOTUNE) 10 dataset = dataset.padded_batch(batch_size=32, padded_shapes=([max_length], [max_length], [max_length]), padding_values=0) 12 x_train = []

    File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\tensorflow\python\data\ops\dataset_ops.py:2204, in DatasetV2.map(self, map_func, num_parallel_calls, deterministic, name) 2202 return MapDataset(self, map_func, preserve_cardinality=True, name=name) 2203 else: -> 2204 return ParallelMapDataset( 2205 self, 2206 map_func, 2207 num_parallel_calls, 2208 deterministic, 2209 preserve_cardinality=True, 2210 name=name)

    File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\tensorflow\python\data\ops\dataset_ops.py:5441, in ParallelMapDataset.init(self, input_dataset, map_func, num_parallel_calls, deterministic, use_inter_op_parallelism, preserve_cardinality, use_legacy_function, name) 5439 self._input_dataset = input_dataset 5440 self._use_inter_op_parallelism = use_inter_op_parallelism -> 5441 self._map_func = structured_function.StructuredFunctionWrapper( 5442 map_func, ... File "C:\Users\ixelo\AppData\Local\Temp\ipykernel_14408\2057811764.py", line 4, in tokenize_and_merge * fr_tokens = fr_tokenizer.tokenize(fr_sentence.np().decode('utf-8')).merge_dims(-2, -1)

    AttributeError: 'Tensor' object has no attribute 'np'
    

    Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

  2. Assistant

    The error you're encountering is because you're trying to use the .np() method on a TensorFlow tensor object, which is incorrect. TensorFlow tensors don't have an .np() method. Instead, you should directly deal with the tensor object or convert it to a numpy array using .numpy() method. However, .numpy() cannot be used in a TensorFlow graph operation (such as within dataset.map which is graph-executed), as it needs to be executed eagerly.

    Since dataset.map operates in graph mode (not eager mode), you cannot use .numpy() within functions being mapped over a dataset either. Your approach needs to handle tensors directly within the TensorFlow graph.

    Here's a revised approach that doesn't attempt to convert tensors to numpy arrays within tokenize_and_merge function:

    def tokenize_and_merge(fr_sentence, en_sentence):
        # Assuming fr_tokenizer and en_tokenizer have a way to directly tokenize tf.Tensor inputs
    
        # Directly tokenize the input sentences without trying to convert to numpy arrays
        fr_tokens = fr_tokenizer.tokenize(fr_sentence).merge_dims(-2, -1)
        # Add [START] and [END] tokens manually if your tokenizer doesn't directly support them
        # This part might need custom logic based on your tokenizer handling
        fr_decode_tokens = fr_tokenizer.tokenize("[START]" + fr_sentence + "[END]").merge_dims(-2, -1)
        en_tokens = en_tokenizer.tokenize(en_sentence).merge_dims(-2, -1)
        
        # Return tokenized and potentially padded tensors
        return fr_tokens, fr_decode_tokens, en_tokens
    
    # Note: This assumes you have your dataset, tokenizer objects properly defined somewhere above.
    # Apply the function to your dataset
    dataset = dataset.map(tokenize_and_merge, num_parallel_calls=tf.data.AUTOTUNE)
    

    Adjust your tokenization logic according to how your tokenizers are implemented. If your tokenizers cannot process TensorFlow tensors directly (which is the case for many pre-built or custom tokenization functions), you might need to wrap the tokenization logic in a tf.py_function to execute it eagerly. However, use tf.py_function cautiously because it can significantly slow down the data pipeline:

    def tokenize_and_merge_wrapper(fr_sentence, en_sentence):
        # Use a wrapper function to use tf.py_function inside dataset.map
        [fr_tokens, fr_decode_tokens, en_tokens] = tf.py_function(
            tokenize_and_merge,  # The original function name
            [fr_sentence, en_sentence],  # Inputs
            [tf.int64, tf.int64, tf.int64]  # Specify the output types
        )
        # You might need to set shapes here because `tf.py_function` doesn't retain shape information
        return fr_tokens, fr_decode_tokens, en_tokens
    
    dataset = dataset.map(tokenize_and_merge_wrapper, num_parallel_calls=tf.data.AUTOTUNE)
    

    Keep in mind, this modified tf.py_function approach allows you to use arbitrary Python code (including .numpy()) inside the graph operation, but it comes at the cost of performance. From your code snippet, it's unclear how your tokenizer objects are implemented, so you might need to adapt these suggestions accordingly.