处理报错 D:\data -resnet\main.py:134: FutureWarning: You are using torch.load with…
- User
处理报错 D:\data -resnet\main.py:134: FutureWarning: You are using
torch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user viatorch.serialization.add_safe_globals. We recommend you start settingweights_only=Truefor any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. model.load_state_dict(torch.load(f'{resultdatafolder}/result.model'), strict=False) - Assistant
您遇到的警告是与PyTorch的
torch.load函数有关的未来警告。这个警告是由于torch.load函数使用了默认的pickle模块来加载数据,而pickle在反序列化过程中可能会执行恶意代码。为了防止这样的安全问题,PyTorch计划在未来版本中改变weights_only参数的默认值为True,这意味着只加载权重数据。要处理这个警告,并确保您的代码在将来的版本中是安全且兼容的,您可以显式地将
weights_only参数设置为True。以下是修改后的代码示例:model.load_state_dict(torch.load(f'{resultdatafolder}/result.model', weights_only=True), strict=False)这种方式将确保您只加载模型的权重数据,而不会加载其他可能存在风险的数据对象。如果您不确定来源或不信任加载的模型文件,这将是一个更安全的选择。
如果您确实需要加载完整的模型对象,包括其他附属信息,可以考虑以下方法:
- 确认文件来源可信。
- 将需要允许的对象显式添加到“白名单”中,使用
torch.serialization.add_safe_globals函数。
请根据您的实际需求选择合适的解决方案。
- User
处理报错 E:\anaconda\envs\condaenv\python.exe "D:\data -resnet\main.py" Traceback (most recent call last): File "D:\data -resnet\main.py", line 280, in main() File "D:\data -resnet\main.py", line 44, in main train_set = MyDataset(source_root, source_classes, source_sample_ids) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\data -resnet\myDataset.py", line 15, in init self.features, self.labels, self.class2label = self.load_feature() ^^^^^^^^^^^^^^^^^^^ File "D:\data -resnet\myDataset.py", line 36, in load_feature with open(self.root, mode='r') as f: ^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: 'D:\data -resnet\train'
进程已结束,退出代码为 1
- Assistant
您分享的错误信息表明,在执行代码时尝试打开一个文件时遇到了权限问题,具体来说是
PermissionError: [Errno 13] Permission denied。这个错误通常出现的原因可能是:-
路径不正确:确保路径
'D:\\data -resnet\\train'是一个文件,而不是一个目录。open函数需要指向一个具体的文件而非目录。 -
权限问题:检查您正在运行代码的用户是否有权限读取该文件。如果没有权限,需要修改文件权限或者以管理员身份运行代码。
-
文件被锁定或正在使用:如果文件正在被其他进程使用,您可能会遇到权限问题。确保该文件在操作时未被其他程序占用。
-
路径转义问题:即使看起来路径是正确的,也可以确保路径格式无误。如果程序手动拼接路径,不妨尝试使用
os.path.join来构建路径:import os self.root = os.path.join('D:', 'data -resnet', 'train')
要解决这个问题,可以考虑以下步骤:
-
检查文件路径:确保您实际想操作的是一个文件而不是一个目录。
-
检查文件权限:右键点击文件或目录,查看其属性,并确保运行代码的用户拥有足够的读取权限。
-
使用完整文件路径:在
open函数中传入完整的文件路径,确保您要打开的对象确实是文件。
如果以上检查没有发现问题,请确认应用程序具有适当的权限执行文件读取操作,或者尝试使用管理员权限运行。这样可以确保程序能够访问所需的文件路径。如果问题依旧存在,请提供更多细节,以便进一步帮助。
-
- User
以下是我的mydataset.py
myDataset.py
import csv import os import numpy as np import torch from torch.utils.data import Dataset
class MyDataset(Dataset): def init(self, root, window_size=2048, overlap=512, mean=None, std=None): super(MyDataset, self).init() self.root = root self.window_size = window_size self.overlap = overlap self.input_dim = 2 self.features, self.labels, self.class2label = self.load_feature()
# 将features转换为NumPy数组,方便计算 self.features = np.array(self.features) # [num_samples, feature_length] if mean is None or std is None: # 在训练集上计算均值和标准差 self.mean = np.mean(self.features, axis=0) self.std = np.std(self.features, axis=0) # 防止标准差为0 self.std[self.std == 0] = 1 else: # 在测试集上,使用训练集的均值和标准差 self.mean = mean self.std = std def load_feature(self): features = [] labels = [] class2label = {} with open(self.root, mode='r') as f: reader = csv.reader(f) for row_idx, row in enumerate(reader): if len(row) < 3: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行长度不足,已跳过。") continue try: # 假设每行的前部分是特征,最后两列是 _class 和 label feature_flat = [float(x) for x in row[:-2]] _class = int(row[-2]) label = int(row[-1]) except ValueError as ve: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行数据类型错误,已跳过。原因: {ve}") continue feature_length = len(feature_flat) if feature_length % self.input_dim != 0: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行特征长度 {feature_length} 不是 input_dim {self.input_dim} 的倍数,已跳过。") continue total_steps = feature_length // self.input_dim # e.g., 8192 / 2 = 4096 # 切分窗口 window_size_steps = self.window_size // self.input_dim # 2048 / 2 = 1024 步 overlap_steps = self.overlap // self.input_dim # 512 / 2 = 256 步 step_steps = window_size_steps - overlap_steps # 1024 - 256 = 768 步 num_windows = (total_steps - overlap_steps) // step_steps for i in range(num_windows): start_step = i * step_steps end_step = start_step + window_size_steps if end_step > total_steps: break window = feature_flat[start_step * self.input_dim : end_step * self.input_dim] if len(window) == self.window_size: features.append(window) labels.append(label) # 可选:处理最后一个窗口 remaining_steps = total_steps - (num_windows * step_steps) if remaining_steps >= (window_size_steps * 0.5): # 例如,超过一半窗口长度 window = feature_flat[-self.window_size:] if len(window) == self.window_size: features.append(window) labels.append(label) # 更新 class2label if _class not in class2label: class2label[_class] = label assert len(features) == len(labels), "Features and labels length mismatch." print(f'数据加载完成,总样本数: {len(features)}') return features, labels, class2label def __len__(self): return len(self.features) def __getitem__(self, idx): feature = self.features[idx] label = self.labels[idx] try: feature = np.array(feature).reshape(-1, self.input_dim) # [seq_length, input_dim] except ValueError as ve: raise ValueError(f"错误: 无法将特征重塑为 [seq_length, input_dim]。原因: {ve}") # 应用标准化 feature = (feature - self.mean.reshape(-1, self.input_dim)) / self.std.reshape(-1, self.input_dim) feature = torch.tensor(feature, dtype=torch.float32) label = torch.tensor(label, dtype=torch.long) return feature, labelif name == 'main': train_set = MyDataset(root=r'./dataset/train.csv', window_size=2048, overlap=512) test_set = MyDataset(root=r'./dataset/test.csv', window_size=2048, overlap=512)
print(len(train_set)) print(len(test_set)) print(test_set.class2label) print(list(test_set.class2label.keys())) feature, label = next(iter(train_set)) print(feature.shape) # 应为 [seq_length, input_dim] e.g., [1024, 2] print(label) - Assistant
根据您提供的
myDataset.py代码和之前的错误信息,发生PermissionError的原因可能是您的self.root变量指向了一个目录,而不是一个CSV文件。在您的
__main__部分,代码尝试分别加载'./dataset/train.csv'和'./dataset/test.csv'。如果路径不正确(例如,指向了一个不存在的文件或目录),将在尝试打开文件时抛出权限错误。以下是一些检查和可能的解决方案:
-
检查文件路径:确保路径
'./dataset/train.csv'和'./dataset/test.csv'存在且指向具体的CSV文件,而不是目录。您可以DEBUG输出该路径确保可访问,例如:print(os.path.abspath(self.root)) # 打印绝对路径来检查路径问题 -
检查文件权限:确保运行代码的用户有权限读取这些CSV文件。在文件所在目录执行以下命令来检查权限(对Windows而言):
在文件上右键 -> 属性 -> 安全 -> 设置相关的读取权限。
-
确保路径正确使用: 使用
os.path.join来管理文件路径,确保路径组合正确。train_file_path = os.path.join('D:', 'data', 'dataset', 'train.csv') test_file_path = os.path.join('D:', 'data', 'dataset', 'test.csv') -
确认文件是否正确并非空:确保文件中有内容,并且内容格式符合您数据加载模块的预期格式。尝试用文本编辑器打开查看内容。
另外,如果误导您的是以
'D:\\data -resnet\\train'为目录,并且在此处放置train.csv文件,确保在MyDataset初始化时,全路径名(含文件名)是存在的。如果未能解决问题,请逐行检查您根路径下特定文件是否存在,或重新提供详细的目录结构与格式以便更进一步地分析问题。
-
- User
我的路径确实是指向目录的,请给我改成指向目录的
- Assistant
为了修改代码以支持从一个目录中加载多个CSV文件中的数据,您可以对
MyDataset类进行一些调整。假设每个CSV文件都遵循相同的格式,您可以在初始化MyDataset实例时将root参数指向目录,然后遍历目录中的所有CSV文件来加载数据。下面是修改后的代码示例:
import csv import os import numpy as np import torch from torch.utils.data import Dataset class MyDataset(Dataset): def __init__(self, root_dir, window_size=2048, overlap=512, mean=None, std=None): super(MyDataset, self).__init__() self.root_dir = root_dir self.window_size = window_size self.overlap = overlap self.input_dim = 2 self.features, self.labels, self.class2label = self.load_features_from_directory() self.features = np.array(self.features) # [num_samples, feature_length] if mean is None or std is None: self.mean = np.mean(self.features, axis=0) self.std = np.std(self.features, axis=0) self.std[self.std == 0] = 1 else: self.mean = mean self.std = std def load_features_from_directory(self): features = [] labels = [] class2label = {} for filename in os.listdir(self.root_dir): if filename.endswith('.csv'): file_path = os.path.join(self.root_dir, filename) file_features, file_labels, file_class2label = self.load_features_from_file(file_path) features.extend(file_features) labels.extend(file_labels) class2label.update(file_class2label) return features, labels, class2label def load_features_from_file(self, file_path): features = [] labels = [] class2label = {} with open(file_path, mode='r') as f: reader = csv.reader(f) for row_idx, row in enumerate(reader): if len(row) < 3: print(f"警告: 文件 {file_path} 中的第 {row_idx + 1} 行长度不足,已跳过。") continue try: feature_flat = [float(x) for x in row[:-2]] _class = int(row[-2]) label = int(row[-1]) except ValueError as ve: print(f"警告: 文件 {file_path} 中的第 {row_idx + 1} 行数据类型错误,已跳过。原因: {ve}") continue feature_length = len(feature_flat) if feature_length % self.input_dim != 0: print(f"警告: 文件 {file_path} 中的第 {row_idx + 1} 行特征长度 {feature_length} 不是 input_dim {self.input_dim} 的倍数,已跳过。") continue total_steps = feature_length // self.input_dim window_size_steps = self.window_size // self.input_dim overlap_steps = self.overlap // self.input_dim step_steps = window_size_steps - overlap_steps num_windows = (total_steps - overlap_steps) // step_steps for i in range(num_windows): start_step = i * step_steps end_step = start_step + window_size_steps if end_step > total_steps: break window = feature_flat[start_step * self.input_dim : end_step * self.input_dim] if len(window) == self.window_size: features.append(window) labels.append(label) remaining_steps = total_steps - (num_windows * step_steps) if remaining_steps >= (window_size_steps * 0.5): window = feature_flat[-self.window_size:] if len(window) == self.window_size: features.append(window) labels.append(label) if _class not in class2label: class2label[_class] = label return features, labels, class2label def __len__(self): return len(self.features) def __getitem__(self, idx): feature = self.features[idx] label = self.labels[idx] try: feature = np.array(feature).reshape(-1, self.input_dim) except ValueError as ve: raise ValueError(f"错误: 无法将特征重塑为 [seq_length, input_dim]。原因: {ve}") feature = (feature - self.mean.reshape(-1, self.input_dim)) / self.std.reshape(-1, self.input_dim) feature = torch.tensor(feature, dtype=torch.float32) label = torch.tensor(label, dtype=torch.long) return feature, label if __name__ == '__main__': train_set = MyDataset(root_dir=r'./dataset/train', window_size=2048, overlap=512) test_set = MyDataset(root_dir=r'./dataset/test', window_size=2048, overlap=512) print(len(train_set)) print(len(test_set)) print(test_set.class2label) print(list(test_set.class2label.keys())) feature, label = next(iter(train_set)) print(feature.shape) # 应为 [seq_length, input_dim] e.g., [1024, 2] print(label)在这个修改后的版本中,我们增加了一个方法
load_features_from_directory来遍历指定目录中的所有CSV文件,并调用另一个方法load_features_from_file来加载每个文件中的数据。确保所有CSV文件位于各自的目录中,并且这些目录被正确传递给MyDataset。 - User
这是我的旧版确定可以运行的代码,你对比一下你的修改
myDataset.py
import csv import os import numpy as np import torch from torch.utils.data import Dataset
class MyDataset(Dataset): def init(self, root, window_size=2048, overlap=512): """ 初始化数据集,切分长序列为多个子序列。
参数: - root (str): CSV 文件路径。 - window_size (int): 每个子序列的长度(特征数量)。 - overlap (int): 相邻子序列之间的重叠部分(特征数量)。 """ super(MyDataset, self).__init__() self.root = root self.window_size = window_size self.overlap = overlap self.input_dim = 2 # 根据模型输入维度确认 self.features, self.labels, self.class2label = self.load_feature() def load_feature(self): features = [] labels = [] class2label = {} with open(self.root, mode='r') as f: reader = csv.reader(f) for row_idx, row in enumerate(reader): if len(row) < 3: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行长度不足,已跳过。") continue try: # 假设每行的前部分是特征,最后两列是 _class 和 label feature_flat = [float(x) for x in row[:-2]] _class = int(row[-2]) label = int(row[-1]) except ValueError as ve: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行数据类型错误,已跳过。原因: {ve}") continue feature_length = len(feature_flat) if feature_length % self.input_dim != 0: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行特征长度 {feature_length} 不是 input_dim {self.input_dim} 的倍数,已跳过。") continue total_steps = feature_length // self.input_dim # e.g., 8192 / 2 = 4096 # 切分窗口 window_size_steps = self.window_size // self.input_dim # 2048 / 2 = 1024 步 overlap_steps = self.overlap // self.input_dim # 512 / 2 = 256 步 step_steps = window_size_steps - overlap_steps # 1024 - 256 = 768 步 num_windows = (total_steps - overlap_steps) // step_steps for i in range(num_windows): start_step = i * step_steps end_step = start_step + window_size_steps if end_step > total_steps: break window = feature_flat[start_step * self.input_dim : end_step * self.input_dim] if len(window) == self.window_size: features.append(window) labels.append(label) # 可选:处理最后一个窗口 remaining_steps = total_steps - (num_windows * step_steps) if remaining_steps >= (window_size_steps * 0.5): # 例如,超过一半窗口长度 window = feature_flat[-self.window_size:] if len(window) == self.window_size: features.append(window) labels.append(label) # 更新 class2label if _class not in class2label: class2label[_class] = label assert len(features) == len(labels), "Features and labels length mismatch." print(f'数据加载完成,总样本数: {len(features)}') return features, labels, class2label def __len__(self): return len(self.features) def __getitem__(self, idx): feature, label = self.features[idx], self.labels[idx] try: feature = np.array(feature).reshape(-1, self.input_dim) # [seq_length, input_dim] except ValueError as ve: raise ValueError(f"错误: 无法将特征重塑为 [seq_length, input_dim]。原因: {ve}") feature = torch.tensor(feature, dtype=torch.float32) label = torch.tensor(label, dtype=torch.long) return feature, labelif name == 'main': train_set = MyDataset(root=r'./dataset/train.csv', window_size=2048, overlap=512) test_set = MyDataset(root=r'./dataset/test.csv', window_size=2048, overlap=512)
print(len(train_set)) print(len(test_set)) print(test_set.class2label) print(list(test_set.class2label.keys())) feature, label = next(iter(train_set)) print(feature.shape) # 应为 [seq_length, input_dim] e.g., [1024, 2] print(label) - Assistant
感谢您提供了旧版本的代码。根据您的要求,我们需要将该代码从支持读取单个CSV文件,调整为从指定目录中读取多个CSV文件。以下是修改后的代码,其中重点在于遍历指定目录并从每个CSV文件中加载数据。
import csv import os import numpy as np import torch from torch.utils.data import Dataset class MyDataset(Dataset): def __init__(self, root_dir, window_size=2048, overlap=512): """ 初始化数据集,切分长序列为多个子序列。 参数: - root_dir (str): 存放CSV文件的目录路径。 - window_size (int): 每个子序列的长度(特征数量)。 - overlap (int): 相邻子序列之间的重叠部分(特征数量)。 """ super(MyDataset, self).__init__() self.root_dir = root_dir self.window_size = window_size self.overlap = overlap self.input_dim = 2 # 根据模型输入维度确认 self.features, self.labels, self.class2label = self.load_features_from_directory() def load_features_from_directory(self): features = [] labels = [] class2label = {} # 遍历目录中的所有CSV文件 for filename in os.listdir(self.root_dir): if filename.endswith('.csv'): file_path = os.path.join(self.root_dir, filename) file_features, file_labels, file_class2label = self.load_features_from_file(file_path) features.extend(file_features) labels.extend(file_labels) class2label.update(file_class2label) assert len(features) == len(labels), "Features and labels length mismatch." print(f'数据加载完成,总样本数: {len(features)}') return features, labels, class2label def load_features_from_file(self, file_path): features = [] labels = [] class2label = {} with open(file_path, mode='r') as f: reader = csv.reader(f) for row_idx, row in enumerate(reader): if len(row) < 3: print(f"警告: 文件 {file_path} 中的第 {row_idx + 1} 行长度不足,已跳过。") continue try: feature_flat = [float(x) for x in row[:-2]] _class = int(row[-2]) label = int(row[-1]) except ValueError as ve: print(f"警告: 文件 {file_path} 中的第 {row_idx + 1} 行数据类型错误,已跳过。原因: {ve}") continue feature_length = len(feature_flat) if feature_length % self.input_dim != 0: print(f"警告: 文件 {file_path} 中的第 {row_idx + 1} 行特征长度 {feature_length} 不是 input_dim {self.input_dim} 的倍数,已跳过。") continue total_steps = feature_length // self.input_dim # 切分窗口 window_size_steps = self.window_size // self.input_dim overlap_steps = self.overlap // self.input_dim step_steps = window_size_steps - overlap_steps num_windows = (total_steps - overlap_steps) // step_steps for i in range(num_windows): start_step = i * step_steps end_step = start_step + window_size_steps if end_step > total_steps: break window = feature_flat[start_step * self.input_dim : end_step * self.input_dim] if len(window) == self.window_size: features.append(window) labels.append(label) remaining_steps = total_steps - (num_windows * step_steps) if remaining_steps >= (window_size_steps * 0.5): window = feature_flat[-self.window_size:] if len(window) == self.window_size: features.append(window) labels.append(label) if _class not in class2label: class2label[_class] = label return features, labels, class2label def __len__(self): return len(self.features) def __getitem__(self, idx): feature, label = self.features[idx], self.labels[idx] try: feature = np.array(feature).reshape(-1, self.input_dim) except ValueError as ve: raise ValueError(f"错误: 无法将特征重塑为 [seq_length, input_dim]。原因: {ve}") feature = torch.tensor(feature, dtype=torch.float32) label = torch.tensor(label, dtype=torch.long) return feature, label if __name__ == '__main__': train_set = MyDataset(root_dir=r'./dataset/train', window_size=2048, overlap=512) test_set = MyDataset(root_dir=r'./dataset/test', window_size=2048, overlap=512) print(len(train_set)) print(len(test_set)) print(test_set.class2label) print(list(test_set.class2label.keys())) feature, label = next(iter(train_set)) print(feature.shape) # 应为 [seq_length, input_dim] e.g., [1024, 2] print(label)主要改变点:
-
root改为root_dir:因为我们现在处理的是目录而非单个文件。 -
新增
load_features_from_directory方法:该方法遍历目录中的所有CSV文件,将其传给load_features_from_file方法处理。 -
为每个CSV文件复用
load_features_from_file方法:单个文件读取逻辑保留在load_features_from_file中以便复用。
通过这些修改,程序应能从指定目录中所有CSV文件中加载并处理数据。请确保目录路径正确并包含正确格式的CSV文件。
-
- User
报错 E:\anaconda\envs\condaenv\python.exe "D:\data -resnet\main.py" Traceback (most recent call last): File "D:\data -resnet\main.py", line 280, in main() File "D:\data -resnet\main.py", line 59, in main feature, label = next(iter(train_set)) ^^^^^^^^^^^^^^^^^^^^^ StopIteration 数据加载完成,总样本数: 0 数据加载完成,总样本数: 0 class2label: {} train_set sample num: 0 test_set sample num: 0 训练集类别分布: Counter() 测试集类别分布: Counter()
进程已结束,退出代码为 1
- Assistant
从错误信息来看,
StopIteration异常是在尝试迭代数据集的时候发生的。这通常意味着数据集中没有任何样本可以迭代。在您的代码中,train_set和test_set的总样本数都是0,这也说明数据加载过程中没有成功地从CSV文件中读取和解析数据。下面是一些可能导致此问题的原因和解决方法:-
CSV文件是否存在:首先,确保指定的目录中确实存在CSV文件,并且这些CSV文件中含有数据。
-
文件路径正确性:确保传入的
root_dir路径是正确的,路径指向的是包含CSV文件的目录。您可以在代码中打印路径来确认:print(f"正在读取目录: {self.root_dir}") -
检查CSV文件内容:CSV文件格式需要符合程序的预期,例如每行至少要有3列。如果某些文件不符合这种格式,它们的内容可能会被跳过从而导致无法加载数据。
-
CSV文件扩展名:确保所有的文件扩展名是
.csv并且没有拼写错误。 -
调试输出:在读取文件时增加一些调试输出,打印遍历到的文件名,确保所有文件都被读取:
for filename in os.listdir(self.root_dir): if filename.endswith('.csv'): print(f"正在处理文件: {filename}") file_path = os.path.join(self.root_dir, filename) file_features, file_labels, file_class2label = self.load_features_from_file(file_path) features.extend(file_features) labels.extend(file_labels) class2label.update(file_class2label) -
内容是否被跳过:在
load_features_from_file函数中检查有没有频繁打印出“警告: ...”信息,因为代码逻辑设计有可能已经跳过了在某些错误情况下读取的每一行。
通过以上检查和日志打印,您可以逐步找到为什么数据集中样本数为0的原因。这错误通常是因为输入文件没有被正确地读入,可能是因为路径问题、文件内容格式不符合或是其他在文件读取部分的逻辑问题。确认这些条件之后,您可能需要进一步调试
load_features_from_file代码以确保每个CSV文件内容正确被解析为所需要的特征和标签。 -
- User
我的路径是这两个source_root = r'D:\data -resnet\train' target_root = r'D:\data -resnet\test'
- Assistant
感谢提供了具体的路径信息。以下是针对这两个路径加载数据时可能需要的检查步骤,确保程序可以正确读取路径下的CSV文件。
-
确认路径正确性和文件存在性:
- 确保路径
D:\data -resnet\train和D:\data -resnet\test确实存在并包含预期的CSV文件。 - 确保文件扩展名是
.csv,没有拼写错误。
- 确保路径
-
调试代码增强:
- 在遍历目录之前和之后打印日志,并在处理每个文件时也记录日志,例如:
print(f"正在检查目录: {self.root_dir}") for filename in os.listdir(self.root_dir): if filename.endswith('.csv'): print(f"正在处理文件: {filename}") file_path = os.path.join(self.root_dir, filename) file_features, file_labels, file_class2label = self.load_features_from_file(file_path) features.extend(file_features) labels.extend(file_labels) class2label.update(file_class2label) else: print(f"跳过非CSV文件: {filename}") -
文件检查:
- 手动检查这些文件中的数据格式是否符合您的预期,例如,确保每一行至少有3列数据,最后两列是整数类型(代表_class和label)。
-
读取和解析文件的逻辑是否正确:
- 确保
load_features_from_file函数逻辑没有导致所有数据行被跳过。比如说增加调试日志记录读取的行数和成功解析的数据行数,检查是否存在逻辑错误导致所有行均被跳过。
- 确保
-
操作系统权限:
- 确保执行程序的用户对这些目录和文件有读取权限。
如果经过这些检查发现问题依然存在,建议逐步调试程序中的每一个关键步骤,使用更多的打印信息了解程序在执行过程中所加载到的文件及其内容。同时,要确保整个数据处理过程中不存在目录遍历或条件判断的逻辑错误来准确识别和读取CSV文件。
-
- User
目录指向一直都不是csv文件,是一个文件夹,里面包括10个.mat文件,请确认
- Assistant
感谢您澄清路径中的内容为
.mat文件而非.csv文件。如果您的数据是存储在MATLAB的.mat格式中而不是CSV文件,那么加载的方法将需完全调整。在Python中,可以使用
scipy.io.loadmat函数来读取.mat文件。以下是一个示例,展示如何从目录中读取.mat文件并从中提取数据:修改后的代码示例
import os import numpy as np import torch from torch.utils.data import Dataset from scipy.io import loadmat # 需要用到scipy库来读取.mat文件 class MyDataset(Dataset): def __init__(self, root_dir, window_size=2048, overlap=512): """ 初始化数据集,切分长序列为多个子序列。 参数: - root_dir (str): 存放.mat文件的目录路径。 - window_size (int): 每个子序列的长度(特征数量)。 - overlap (int): 相邻子序列之间的重叠部分(特征数量)。 """ super(MyDataset, self).__init__() self.root_dir = root_dir self.window_size = window_size self.overlap = overlap self.input_dim = 2 # 根据模型输入维度确认 self.features, self.labels, self.class2label = self.load_features_from_directory() def load_features_from_directory(self): features = [] labels = [] class2label = {} # 遍历目录中的所有.mat文件 for filename in os.listdir(self.root_dir): if filename.endswith('.mat'): print(f"正在处理文件: {filename}") file_path = os.path.join(self.root_dir, filename) file_features, file_labels, file_class2label = self.load_features_from_file(file_path) features.extend(file_features) labels.extend(file_labels) class2label.update(file_class2label) assert len(features) == len(labels), "Features and labels length mismatch." print(f'数据加载完成,总样本数: {len(features)}') return features, labels, class2label def load_features_from_file(self, file_path): mat_data = loadmat(file_path) # 假设'features'和'labels'是.mat文件中的键(确保这些名称与实际的一致) features = mat_data['features'] labels = mat_data['labels'].flatten() # 确保labels是1D数组 # 假设_labels包含行样本的_last两列,每一行最后两列是_class和label class2label = {} # 处理每个样本 processed_features = [] processed_labels = [] num_samples = features.shape[0] for i in range(num_samples): feature_flat = features[i, :-2] # 假设 features 是矩阵形式 _class = features[i, -2] label = features[i, -1] feature_flat = feature_flat.flatten() feature_length = len(feature_flat) if feature_length % self.input_dim != 0: print(f"警告: 文件 {file_path} 中的特征长度 {feature_length} 不是 input_dim {self.input_dim} 的倍数,已跳过。") continue total_steps = feature_length // self.input_dim window_size_steps = self.window_size // self.input_dim overlap_steps = self.overlap // self.input_dim step_steps = window_size_steps - overlap_steps num_windows = (total_steps - overlap_steps) // step_steps for i in range(num_windows): start_step = i * step_steps end_step = start_step + window_size_steps if end_step > total_steps: break window = feature_flat[start_step * self.input_dim : end_step * self.input_dim] if len(window) == self.window_size: processed_features.append(window) processed_labels.append(label) remaining_steps = total_steps - (num_windows * step_steps) if remaining_steps >= (window_size_steps * 0.5): window = feature_flat[-self.window_size:] if len(window) == self.window_size: processed_features.append(window) processed_labels.append(label) if _class not in class2label: class2label[_class] = label return processed_features, processed_labels, class2label def __len__(self): return len(self.features) def __getitem__(self, idx): feature, label = self.features[idx], self.labels[idx] try: feature = np.array(feature).reshape(-1, self.input_dim) except ValueError as ve: raise ValueError(f"错误: 无法将特征重塑为 [seq_length, input_dim]。原因: {ve}") feature = torch.tensor(feature, dtype=torch.float32) label = torch.tensor(label, dtype=torch.long) return feature, label if __name__ == '__main__': train_set = MyDataset(root_dir=r'D:\data -resnet\train', window_size=2048, overlap=512) test_set = MyDataset(root_dir=r'D:\data -resnet\test', window_size=2048, overlap=512) print(len(train_set)) print(len(test_set)) print(test_set.class2label) print(list(test_set.class2label.keys())) for feature, label in train_set: print(feature.shape) # 应为 [seq_length, input_dim] e.g., [1024, 2] print(label)注意事项:
-
MAT文件键名:上面的代码假设
'features'和'labels'是每个.mat文件中的变量名称。请根据实际的文件内容更改这些名称。 -
特征和标签提取逻辑:根据
.mat文件的数据结构和存储格式(例如矩阵的行列如何对应特征和标签),调整处理代码。 -
库需求:确保安装了
scipy库来使用loadmat功能。可以用以下命令安装:pip install scipy
请确保在真实数据上根据实际情况更新键名和数据处理逻辑。通过这些调整,您可以从
.mat文件中提取并处理数据以构建数据集。 -
- User
再给你看一下我旧版的确定可以运行的mydataset.py,你对比确认一下
myDataset.py
import csv import os import numpy as np import torch from torch.utils.data import Dataset
class MyDataset(Dataset): def init(self, root, window_size=2048, overlap=512): """ 初始化数据集,切分长序列为多个子序列。
参数: - root (str): CSV 文件路径。 - window_size (int): 每个子序列的长度(特征数量)。 - overlap (int): 相邻子序列之间的重叠部分(特征数量)。 """ super(MyDataset, self).__init__() self.root = root self.window_size = window_size self.overlap = overlap self.input_dim = 2 # 根据模型输入维度确认 self.features, self.labels, self.class2label = self.load_feature() def load_feature(self): features = [] labels = [] class2label = {} with open(self.root, mode='r') as f: reader = csv.reader(f) for row_idx, row in enumerate(reader): if len(row) < 3: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行长度不足,已跳过。") continue try: # 假设每行的前部分是特征,最后两列是 _class 和 label feature_flat = [float(x) for x in row[:-2]] _class = int(row[-2]) label = int(row[-1]) except ValueError as ve: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行数据类型错误,已跳过。原因: {ve}") continue feature_length = len(feature_flat) if feature_length % self.input_dim != 0: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行特征长度 {feature_length} 不是 input_dim {self.input_dim} 的倍数,已跳过。") continue total_steps = feature_length // self.input_dim # e.g., 8192 / 2 = 4096 # 切分窗口 window_size_steps = self.window_size // self.input_dim # 2048 / 2 = 1024 步 overlap_steps = self.overlap // self.input_dim # 512 / 2 = 256 步 step_steps = window_size_steps - overlap_steps # 1024 - 256 = 768 步 num_windows = (total_steps - overlap_steps) // step_steps for i in range(num_windows): start_step = i * step_steps end_step = start_step + window_size_steps if end_step > total_steps: break window = feature_flat[start_step * self.input_dim : end_step * self.input_dim] if len(window) == self.window_size: features.append(window) labels.append(label) # 可选:处理最后一个窗口 remaining_steps = total_steps - (num_windows * step_steps) if remaining_steps >= (window_size_steps * 0.5): # 例如,超过一半窗口长度 window = feature_flat[-self.window_size:] if len(window) == self.window_size: features.append(window) labels.append(label) # 更新 class2label if _class not in class2label: class2label[_class] = label assert len(features) == len(labels), "Features and labels length mismatch." print(f'数据加载完成,总样本数: {len(features)}') return features, labels, class2label def __len__(self): return len(self.features) def __getitem__(self, idx): feature, label = self.features[idx], self.labels[idx] try: feature = np.array(feature).reshape(-1, self.input_dim) # [seq_length, input_dim] except ValueError as ve: raise ValueError(f"错误: 无法将特征重塑为 [seq_length, input_dim]。原因: {ve}") feature = torch.tensor(feature, dtype=torch.float32) label = torch.tensor(label, dtype=torch.long) return feature, labelif name == 'main': train_set = MyDataset(root=r'./dataset/train.csv', window_size=2048, overlap=512) test_set = MyDataset(root=r'./dataset/test.csv', window_size=2048, overlap=512)
print(len(train_set)) print(len(test_set)) print(test_set.class2label) print(list(test_set.class2label.keys())) feature, label = next(iter(train_set)) print(feature.shape) # 应为 [seq_length, input_dim] e.g., [1024, 2] print(label) - Assistant
感谢您提供了已知工作的旧版代码。旧代码的核心是从CSV文件中读取数据,解析特征和标签。然而,您实际使用的是MATLAB
.mat文件。因此,我将提供一个基于旧版的类似结构,但适用于.mat文件的示例。以下是如何修改代码以支持读取
.mat文件的示例:import os import numpy as np import torch from torch.utils.data import Dataset from scipy.io import loadmat # 导入scipy库来处理.mat文件 class MyDataset(Dataset): def __init__(self, root_dir, window_size=2048, overlap=512): """ 初始化数据集,切分长序列为多个子序列。 参数: - root_dir (str): 存放.mat文件的目录路径。 - window_size (int): 每个子序列的长度(特征数量)。 - overlap (int): 相邻子序列之间的重叠部分(特征数量)。 """ super(MyDataset, self).__init__() self.root_dir = root_dir self.window_size = window_size self.overlap = overlap self.input_dim = 2 # 假设每个样本都有两个维度 self.features, self.labels, self.class2label = self.load_features_from_directory() def load_features_from_directory(self): features = [] labels = [] class2label = {} for filename in os.listdir(self.root_dir): if filename.endswith('.mat'): print(f"正在处理文件: {filename}") file_path = os.path.join(self.root_dir, filename) file_features, file_labels, file_class2label = self.load_features_from_file(file_path) features.extend(file_features) labels.extend(file_labels) class2label.update(file_class2label) assert len(features) == len(labels), "Features and labels length mismatch." print(f'数据加载完成,总样本数: {len(features)}') return features, labels, class2label def load_features_from_file(self, file_path): mat_data = loadmat(file_path) # 假设'features'和'labels'是.mat文件中的键(需要根据实际'.mat'文件内容替换) features = mat_data['features'] labels = mat_data['labels'].flatten() # 确保labels是1D数组 class2label = {} processed_features = [] processed_labels = [] num_samples = features.shape[0] for i in range(num_samples): feature_flat = features[i, :-2] # 假设 features 是矩阵形式,最后两列是_class和label _class = features[i, -2] label = features[i, -1] feature_flat = feature_flat.flatten() feature_length = len(feature_flat) if feature_length % self.input_dim != 0: print(f"警告: 文件 {file_path} 中的特征长度 {feature_length} 不是 input_dim {self.input_dim} 的倍数,已跳过。") continue total_steps = feature_length // self.input_dim window_size_steps = self.window_size // self.input_dim overlap_steps = self.overlap // self.input_dim step_steps = window_size_steps - overlap_steps num_windows = (total_steps - overlap_steps) // step_steps for i in range(num_windows): start_step = i * step_steps end_step = start_step + window_size_steps if end_step > total_steps: break window = feature_flat[start_step * self.input_dim : end_step * self.input_dim] if len(window) == self.window_size: processed_features.append(window) processed_labels.append(label) remaining_steps = total_steps - (num_windows * step_steps) if remaining_steps >= (window_size_steps * 0.5): window = feature_flat[-self.window_size:] if len(window) == self.window_size: processed_features.append(window) processed_labels.append(label) if _class not in class2label: class2label[_class] = label return processed_features, processed_labels, class2label def __len__(self): return len(self.features) def __getitem__(self, idx): feature, label = self.features[idx], self.labels[idx] try: feature = np.array(feature).reshape(-1, self.input_dim) except ValueError as ve: raise ValueError(f"错误: 无法将特征重塑为 [seq_length, input_dim]。原因: {ve}") feature = torch.tensor(feature, dtype=torch.float32) label = torch.tensor(label, dtype=torch.long) return feature, label if __name__ == '__main__': train_set = MyDataset(root_dir=r'D:\data -resnet\train', window_size=2048, overlap=512) test_set = MyDataset(root_dir=r'D:\data -resnet\test', window_size=2048, overlap=512) print(len(train_set)) print(len(test_set)) print(test_set.class2label) print(list(test_set.class2label.keys())) for feature, label in train_set: print(feature.shape) # 应为 [seq_length, input_dim] e.g., [1024, 2] print(label)重要提示:
-
调整字典键:替换
mat_data['features']和mat_data['labels']的键名,以匹配您的.mat文件中的实际变量名。 -
数据布局和逻辑:根据MATLAB文件的内容和格式,可能需要修改特征和标签提取逻辑。执行前请确保这些假设逻辑(形状、列数等)和MATLAB中数据存储格式是一致的。
-
环境依赖:确保安装
scipy库来处理MATLAB文件。安装命令为:pip install scipy。
这样设置后,可以处理目录中所有的
.mat文件,与CSV文件的逻辑相同,您只需根据文件内容设定正确的处理逻辑。 -
- User
不是不是,那可能是我的理解错了,我需要你按照旧代码的核心从CSV文件中读取数据,解析特征和标签,忘掉我跟你说的.mat
- Assistant
抱歉此前的误解。根据您的需求,我们只需要从原始的以CSV文件为读取目标的数据加载逻辑中进行调整,以支持从特定的CSV文件读取数据并解析特征和标签。下面是如何在旧代码的基础上简单进行一下代码清理和确认:
import csv import os import numpy as np import torch from torch.utils.data import Dataset class MyDataset(Dataset): def __init__(self, root, window_size=2048, overlap=512): """ 初始化数据集,切分长序列为多个子序列。 参数: - root (str): CSV 文件路径。 - window_size (int): 每个子序列的长度(特征数量)。 - overlap (int): 相邻子序列之间的重叠部分(特征数量)。 """ super(MyDataset, self).__init__() self.root = root self.window_size = window_size self.overlap = overlap self.input_dim = 2 # 假设每个样本都有两个维度 self.features, self.labels, self.class2label = self.load_feature() def load_feature(self): features = [] labels = [] class2label = {} with open(self.root, mode='r') as f: reader = csv.reader(f) for row_idx, row in enumerate(reader): if len(row) < 3: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行长度不足,已跳过。") continue try: # 假设每行的前部分是特征,最后两列是 _class 和 label feature_flat = [float(x) for x in row[:-2]] _class = int(row[-2]) label = int(row[-1]) except ValueError as ve: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行数据类型错误,已跳过。原因: {ve}") continue feature_length = len(feature_flat) if feature_length % self.input_dim != 0: print(f"警告: 文件 {self.root} 中的第 {row_idx + 1} 行特征长度 {feature_length} 不是 input_dim {self.input_dim} 的倍数,已跳过。") continue total_steps = feature_length // self.input_dim # 切分窗口 window_size_steps = self.window_size // self.input_dim overlap_steps = self.overlap // self.input_dim step_steps = window_size_steps - overlap_steps num_windows = (total_steps - overlap_steps) // step_steps for i in range(num_windows): start_step = i * step_steps end_step = start_step + window_size_steps if end_step > total_steps: break window = feature_flat[start_step * self.input_dim : end_step * self.input_dim] if len(window) == self.window_size: features.append(window) labels.append(label) # 可选:处理最后一个窗口 remaining_steps = total_steps - (num_windows * step_steps) if remaining_steps >= (window_size_steps * 0.5): window = feature_flat[-self.window_size:] if len(window) == self.window_size: features.append(window) labels.append(label) # 更新 class2label if _class not in class2label: class2label[_class] = label assert len(features) == len(labels), "Features and labels length mismatch." print(f'数据加载完成,总样本数: {len(features)}') return features, labels, class2label def __len__(self): return len(self.features) def __getitem__(self, idx): feature, label = self.features[idx], self.labels[idx] try: feature = np.array(feature).reshape(-1, self.input_dim) except ValueError as ve: raise ValueError(f"错误: 无法将特征重塑为 [seq_length, input_dim]。原因: {ve}") feature = torch.tensor(feature, dtype=torch.float32) label = torch.tensor(label, dtype=torch.long) return feature, label if __name__ == '__main__': train_set = MyDataset(root=r'D:\data -resnet\train.csv', window_size=2048, overlap=512) test_set = MyDataset(root=r'D:\data -resnet\test.csv', window_size=2048, overlap=512) print(len(train_set)) print(len(test_set)) print(test_set.class2label) print(list(test_set.class2label.keys())) feature, label = next(iter(train_set)) print(feature.shape) # 应为 [seq_length, input_dim] e.g., [1024, 2] print(label)在您需要注意的具体要点:
-
数据文件路径:检查代码中的文件路径是否存在且指向了正确的CSV文件。
-
CSV数据格式:确认CSV文件的格式与代码预期一致,尤其是文件的列数和数据类型。
-
日志调试:如果所有CSV文件均符合预期但代码无法正常运行,可添加更多的日志信息来检查在哪里可能出现问题。
-
环境的准备:确保在Python环境中安装好所需的库(如PyTorch、NumPy等)。
根据此代码,您应当能从指定的CSV文件中加载数据并进行解析以供训练使用。如果还有具体问题,请继续提出以便更详细地帮助您解决。
-
- User
报错 E:\anaconda\envs\condaenv\python.exe "D:\data -resnet\main.py" Traceback (most recent call last): File "D:\data -resnet\main.py", line 280, in main() File "D:\data -resnet\main.py", line 44, in main train_set = MyDataset(source_root, source_classes, source_sample_ids) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\data -resnet\myDataset.py", line 22, in init self.features, self.labels, self.class2label = self.load_feature() ^^^^^^^^^^^^^^^^^^^ File "D:\data -resnet\myDataset.py", line 29, in load_feature with open(self.root, mode='r') as f: ^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: 'D:\data -resnet\train'
进程已结束,退出代码为 1
- Assistant
PermissionError: [Errno 13] Permission denied错误表明程序试图以读取模式打开一个路径,但由于某种原因被拒绝。这个错误通常与以下问题之一有关:-
路径指向错误:确保
self.root确实指向一个具体的文件而不是一个目录。在您的情况下,错误消息显示您正尝试打开'D:\\data -resnet\\train'作为文件,但这似乎是一个目录。 -
权限问题:确保您有权限读取该路径下的文件。如果您是以非管理员身份运行程序,您可能需要以管理员身份运行或者修改文件权限。
-
路径拼接错误:可能由于路径拼接问题指向了一个错误的位置。
要解决这个问题,请尝试以下步骤:
-
验证路径:确保您在调用
MyDataset构造函数时传递的是完整的文件路径,而不是目录。根据前面的代码示例,调用应像这样:train_set = MyDataset(root=r'D:\data -resnet\train.csv', window_size=2048, overlap=512) -
检查目录和文件名:确认路径结构是正确的,并且相应的CSV文件存在。如果在指定目录有多个文件而不是一个文件,您需要调整路径以包含文件名。
-
更改代码以遍历目录:如果您是想从一个目录中读取多个CSV文件,可以像下面这样调整代码:
def load_features_from_directory(self): # 在这里处理目录内的所有文件。 all_features = [] all_labels = [] class2label = {} for filename in os.listdir(self.root): if filename.endswith('.csv'): file_path = os.path.join(self.root, filename) features, labels, _class2label = self.load_features_from_file(file_path) all_features.extend(features) all_labels.extend(labels) class2label.update(_class2label) return all_features, all_labels, class2label -
为路径创建新测试用例:如果存储在目录中的数据文件名称多样,写一个新路径以指向一个已知测试CSV文件,这样可以隔离环境或数据问题。
以上步骤应该能帮助您找到并解决这个权限问题。确保正确传递路径和文件名是解决此类问题的关键。
-
- Assistant
如果您有其他问题或需要进一步的帮助,请随时告诉我!