def splittext(text,minlength=15,maxlength=30): global yuyimodel 定义句末和句内标点符号 endpunc =…
- User
def split_text(text,min_length=15,max_length=30): global yuyi_model # 定义句末和句内标点符号 end_punc = '。!?;\n' inner_punc = ',、:'
# 第一步:初步分句 sentences = re.split(f'([{end_punc}])', text) # 将句子和分隔符合并 if sentences[-1] == '': sentences = sentences[:-1] sentences = [''.join(i) for i in zip(sentences[0::2], sentences[1::2] + [''])] chunks = [] for sentence in sentences: # 去除首尾空白字符 sentence = sentence.strip() if not sentence: continue # 如果句子长度在10到40之间,直接加入结果 if min_length <= len(sentence) <= max_length: chunks.append(sentence) else: # 第二步:使用句内标点符号进一步分割 tokens = re.split(f'([{inner_punc}])', sentence) tokens = [t for t in tokens if t != ''] # 去除空字符串 # 将 tokens 合并成子句 clauses = [] i = 0 while i < len(tokens): if i+1 < len(tokens): clauses.append(''.join(tokens[i:i+2])) i +=2 else: clauses.append(tokens[i]) i +=1 for clause in clauses: clause = clause.strip() if not clause: continue if len(clause) > max_length: # 第三步:使用结巴分词进一步切分 words = jieba.lcut(clause) # 将词按10到40字进行分块 start = 0 while start < len(words): size = 0 block = [] while start < len(words) and size + len(words[start]) <= max_length: block.append(words[start]) size += len(words[start]) start += 1 # 如果块的长度不足10字,尝试与后面的词合并 if size < min_length and start < len(words): while start < len(words) and size < min_length: block.append(words[start]) size += len(words[start]) start += 1 if block: chunks.append(''.join(block)) else: chunks.append(clause) # 第四步:智能拼接长度小于10的块 result = [] i = 0 while i < len(chunks): current_chunk = chunks[i] # 如果当前块长度在10到40之间,直接加入结果 if min_length <= len(current_chunk) <= max_length: result.append(current_chunk) i += 1 elif len(current_chunk) < min_length: merged = False # 计算与前一个块的语义相似度 if result: prev_chunk = result[-1] combined_length = len(prev_chunk) + len(current_chunk) if combined_length <= max_length or True: # 即使超过40也尝试合并 sentences = [prev_chunk, current_chunk] embeddings = yuyi_model.encode(sentences) sim_prev = cosine_similarity(embeddings[0], embeddings[1]) else: sim_prev = -1 else: sim_prev = -1 # 计算与后一个块的语义相似度 if i +1 < len(chunks): next_chunk = chunks[i+1] combined_length = len(current_chunk) + len(next_chunk) if combined_length <= max_length or True: # 即使超过40也尝试合并 sentences = [current_chunk, next_chunk] embeddings = yuyi_model.encode(sentences) sim_next = cosine_similarity(embeddings[0], embeddings[1]) else: sim_next = -1 else: sim_next = -1 # 比较相似度,选择更高的进行合并 if sim_prev >= sim_next and sim_prev != -1: # 合并到前一个块 result[-1] += current_chunk elif sim_next != -1: # 合并到后一个块 combined_chunk = current_chunk + chunks[i+1] result.append(combined_chunk) i +=1 # 跳过下一个块 else: # 无法合并,直接加入(可能不符合长度要求) result.append(current_chunk) i +=1 else: # 理论上不会出现长度超过40的块 # 但我们还是处理一下 # 将当前块按40个字符切分 start = 0 while start < len(current_chunk): end = start + max_length result.append(current_chunk[start:end]) start = end i +=1 return result # 合并到后一个块 combined_chunk = current_chunk + chunks[i+1] result.append(combined_chunk) i +=1 # 跳过下一个块 这里是不是可以优化?不应该直接加入到结果中,因为合并后的块还可能小于min_length - Assistant
Yes, you're absolutely correct. In the current implementation, when merging the current chunk with the next chunk (due to higher semantic similarity), the combined chunk might still be shorter than the
min_length. This means it might not satisfy the length requirements we've set, and we're directly adding it to theresultlist without further checks.Optimization:
Instead of directly appending the
combined_chunkto theresult, we should:- Check if the combined chunk meets the
min_lengthrequirement. - If not, continue merging it with subsequent chunks until it does, or until we've exhausted the chunks.
Here's how you can modify your code to incorporate this optimization:
# 合并到后一个块 combined_chunk = current_chunk + chunks[i+1] i += 1 # 跳过下一个块 # 检查合并后的块长度,如果仍然小于min_length,继续合并 while len(combined_chunk) < min_length and (i + 1) < len(chunks): i += 1 combined_chunk += chunks[i] result.append(combined_chunk) i += 1Explanation:
- Combine Current and Next Chunk: We first combine the current chunk with the next one based on the higher semantic similarity.
- Check Length: After combining, we check if the
combined_chunkis still less thanmin_length. - Continue Merging if Necessary: If it is, we enter a loop where we continue to merge with the following chunks until the
combined_chunkmeets themin_lengthor we have no more chunks to merge. - Update Index: We increment
iappropriately to ensure we don't skip or reprocess any chunks.
Updated Code Snippet:
Here's the updated section of your code with the optimization:
elif sim_next != -1: # 合并到后一个块 combined_chunk = current_chunk + chunks[i+1] i += 1 # 跳过下一个块 # 检查合并后的块长度,如果仍然小于min_length,继续合并 while len(combined_chunk) < min_length and (i + 1) < len(chunks): i += 1 combined_chunk += chunks[i] result.append(combined_chunk) i += 1 else: # 无法合并,直接加入(可能不符合长度要求) result.append(current_chunk) i += 1Notes:
- Boundary Conditions: Ensure that when incrementing
i, we don't go out of bounds of thechunkslist. - Avoid Infinite Loops: The loop should have proper exit conditions to avoid infinite looping.
- Semantic Similarity: If you want to ensure that the additional chunks you're merging are also semantically similar, you might consider also checking the similarity when merging more chunks. However, this can complicate the logic and increase computation time.
Alternative Approach:
If you prefer to keep semantic similarity checks during further merging, you can adjust the loop to:
- Calculate the similarity of the
combined_chunkwith the next chunk. - Decide whether to merge based on the similarity and length constraints.
However, for simplicity and efficiency, especially if the primary goal is to meet the
min_length, the above solution should suffice. - Check if the combined chunk meets the