convert into single line for cmd python tools/llama/build_dataset.py \ --input "data" \…
- User
convert into single line for cmd python tools/llama/build_dataset.py
--input "data"
--output "data/protos"
--text-extension .lab
--num-workers 16 - Assistant
Certainly! Here's the command converted into a single line for CMD:
python tools/llama/build_dataset.py --input "data" --output "data/protos" --text-extension .lab --num-workers 16 - User
now use that and adjust this code below to run that
public void NumMe(string condaEnvName, string pythonScript, string dataArg, int numWorkers, int batchSize, string configName, string checkpointPath) { Color Backup = Create_NPY.BackColor; Create_NPY.BackColor = Color.Red; // Normalize paths to avoid double slashes or inconsistent slashes string pythonScriptNormalized = Path.GetFullPath(pythonScript); string checkpointPathNormalized = Path.GetFullPath(checkpointPath);
// Construct the python arguments string string pythonArgs = $"{pythonScriptNormalized} {dataArg} --num-workers {numWorkers} --batch-size {batchSize} --config-name \"{configName}\" --checkpoint-path \"{checkpointPathNormalized}\""; // For Windows, activate conda and run python command in one cmd instance: string command = $"conda activate {condaEnvName} && python {pythonArgs}"; //Console.WriteLine("Running command:"); //Console.WriteLine(command); var processInfo = new ProcessStartInfo("cmd.exe", "/C " + command) { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false, CreateNoWindow = true, WorkingDirectory = @"C:\AI\fish-speech\" }; using (var process = new Process()) { process.StartInfo = processInfo; //process.OutputDataReceived += (sender, e) => { if (e.Data != null) MessageBox.Show(e.Data); }; //process.ErrorDataReceived += (sender, e) => { if (e.Data != null) MessageBox.Show(e.Data); }; process.Start(); process.BeginOutputReadLine(); process.BeginErrorReadLine(); process.WaitForExit(); //MessageBox.Show($"Process exited with code {process.ExitCode}"); Create_NPY.BackColor = Backup; }}
return the entire code back rename the void to ProtoMe.
- Assistant
public void ProtoMe(string condaEnvName, string pythonScript, string dataArg, int numWorkers) { Color Backup = Create_NPY.BackColor; Create_NPY.BackColor = Color.Red; // Normalize python script path string pythonScriptNormalized = Path.GetFullPath(pythonScript); // Construct the python arguments string according to the single line command: // python tools/llama/build_dataset.py --input "data" --output "data/protos" --text-extension .lab --num-workers 16 // Here dataArg will be used for --input, output is hardcoded as "data/protos", text extension is fixed as .lab string pythonArgs = $"{pythonScriptNormalized} --input \"{dataArg}\" --output \"data/protos\" --text-extension .lab --num-workers {numWorkers}"; // For Windows, activate conda and run python command in one cmd instance: string command = $"conda activate {condaEnvName} && python {pythonArgs}"; var processInfo = new ProcessStartInfo("cmd.exe", "/C " + command) { RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false, CreateNoWindow = true, WorkingDirectory = @"C:\AI\fish-speech\" }; using (var process = new Process()) { process.StartInfo = processInfo; // Optional: to capture real-time output you could hook these events // process.OutputDataReceived += (sender, e) => { if (e.Data != null) /* handle output */ }; // process.ErrorDataReceived += (sender, e) => { if (e.Data != null) /* handle error */ }; process.Start(); process.BeginOutputReadLine(); process.BeginErrorReadLine(); process.WaitForExit(); Create_NPY.BackColor = Backup; } } - User
now adjust this to match our new code
// Variables representing parts of your command (adjust as needed) string condaEnvName = "fish-speech"; string pythonScript = @"C:\AI\fish-speech\tools\vqgan\extract_vq.py"; string dataArg = DatasetHome + @"" + Dataset_List.SelectedItem.ToString(); int numWorkers = 1; int batchSize = 12; string configName = "firefly_gan_vq"; string checkpointPath = @"C:\AI\fish-speech\checkpoints\fish-speech-1.5\firefly-gan-vq-fsq-8x1024-21hz-generator.pth";
NumMe(condaEnvName, pythonScript, dataArg, numWorkers, batchSize, configName, checkpointPath);
- Assistant
Based on the new
ProtoMemethod signature and the simplified argument list, here is the adjusted code snippet:// Variables representing parts of your command (adjust as needed) string condaEnvName = "fish-speech"; string pythonScript = @"C:\AI\fish-speech\tools\llama\build_dataset.py"; // updated path to match new script string dataArg = DatasetHome + @"\" + Dataset_List.SelectedItem.ToString(); int numWorkers = 16; // updated to match example command ProtoMe(condaEnvName, pythonScript, dataArg, numWorkers);Notes:
ProtoMeonly takes 4 parameters now:condaEnvName,pythonScript,dataArg, andnumWorkers.batchSize,configName, andcheckpointPathare no longer needed in the newProtoMemethod and are removed.- The python script path is updated to
tools\llama\build_dataset.pyto match the new single-line command you gave earlier. numWorkersis set to 16 matching the example usage. Adjust as needed.
- User
make the code below organize the list view items in order
private void Dataset_List_SelectedIndexChanged(object sender, EventArgs e) { if (Dataset_List.SelectedItem == null) { Speaker_List.Items.Clear(); Voice_List.Items.Clear(); SelectedDataset = null; return; } Speaker_List.Items.Clear(); Voice_List.Items.Clear(); SelectedDataset = Path.Combine(DatasetHome, Dataset_List.SelectedItem.ToString()); try { if (Directory.Exists(SelectedDataset)) { foreach (var folder in Directory.GetDirectories(SelectedDataset)) { string folderName = Path.GetFileName(folder); Speaker_List.Items.Add(folderName); } } else { MessageBox.Show("Selected dataset directory does not exist.", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } } catch (Exception ex) { MessageBox.Show($"Failed to load speakers: {ex.Message}", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } }
- Assistant
To organize the
Speaker_Listitems in order (alphabetically), you should sort the folder names before adding them. Here's the adjusted code with the sorting added:private void Dataset_List_SelectedIndexChanged(object sender, EventArgs e) { if (Dataset_List.SelectedItem == null) { Speaker_List.Items.Clear(); Voice_List.Items.Clear(); SelectedDataset = null; return; } Speaker_List.Items.Clear(); Voice_List.Items.Clear(); SelectedDataset = Path.Combine(DatasetHome, Dataset_List.SelectedItem.ToString()); try { if (Directory.Exists(SelectedDataset)) { // Get directory names, sort them, then add to Speaker_List var folders = Directory.GetDirectories(SelectedDataset) .Select(folder => Path.GetFileName(folder)) .OrderBy(name => name, StringComparer.CurrentCultureIgnoreCase); foreach (var folderName in folders) { Speaker_List.Items.Add(folderName); } } else { MessageBox.Show("Selected dataset directory does not exist.", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } } catch (Exception ex) { MessageBox.Show($"Failed to load speakers: {ex.Message}", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } }Explanation:
- Used LINQ's
OrderBywithStringComparer.CurrentCultureIgnoreCasefor case-insensitive alphabetical sorting. - Sorted folder names before adding them to
Speaker_List.Items.
- Used LINQ's
- User
organize them by name like how windows file system does
- Assistant
To organize the folder names like Windows Explorer, which sorts using natural sorting (also called "logical sorting" or "human-friendly sorting")—where numeric parts are sorted numerically rather than lexicographically—you can use the Windows API
StrCmpLogicalWfunction via P/Invoke.Here's how to adjust your method to sort items using Windows' natural sort order:
using System.Runtime.InteropServices; using System.Linq; // Add this outside your class: [DllImport("shlwapi.dll", CharSet = CharSet.Unicode)] private static extern int StrCmpLogicalW(string x, string y); private void Dataset_List_SelectedIndexChanged(object sender, EventArgs e) { if (Dataset_List.SelectedItem == null) { Speaker_List.Items.Clear(); Voice_List.Items.Clear(); SelectedDataset = null; return; } Speaker_List.Items.Clear(); Voice_List.Items.Clear(); SelectedDataset = Path.Combine(DatasetHome, Dataset_List.SelectedItem.ToString()); try { if (Directory.Exists(SelectedDataset)) { var folders = Directory.GetDirectories(SelectedDataset) .Select(folder => Path.GetFileName(folder)) .OrderBy(name => name, Comparer<string>.Create((a, b) => StrCmpLogicalW(a, b))) .ToList(); foreach (var folderName in folders) { Speaker_List.Items.Add(folderName); } } else { MessageBox.Show("Selected dataset directory does not exist.", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } } catch (Exception ex) { MessageBox.Show($"Failed to load speakers: {ex.Message}", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } }Explanation:
StrCmpLogicalWis a Windows API function that compares strings using natural sort order (e.g.,"file9"<"file10").- We import it with
DllImport. - We create a custom comparer with it, which is used inside
OrderByto sort the folder names naturally. - This will replicate Windows Explorer's folder sorting behavior exactly.
Make sure your project targets Windows platform (this works only on Windows).
- User
do the same for this private void Speaker_List_SelectedIndexChanged(object sender, EventArgs e) { Voice_List.Items.Clear(); if (Dataset_List.SelectedItem == null || Speaker_List.SelectedItem == null) return; string totalPath = Path.Combine( DatasetHome, Dataset_List.SelectedItem.ToString(), Speaker_List.SelectedItem.ToString()); try { if (Directory.Exists(totalPath)) { var files = Directory.GetFiles(totalPath) .Where(f => f.EndsWith(".mp3", StringComparison.OrdinalIgnoreCase) || f.EndsWith(".wav", StringComparison.OrdinalIgnoreCase) || f.EndsWith(".flac", StringComparison.OrdinalIgnoreCase)); foreach (var file in files) { Voice_List.Items.Add(Path.GetFileName(file)); } } } catch (Exception ex) { MessageBox.Show($"Failed to load voice files: {ex.Message}", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } }
- Assistant
Sure! Here's your
Speaker_List_SelectedIndexChangedmethod updated to sort the voice files using Windows natural sorting (like Explorer), using the sameStrCmpLogicalWAPI:using System.Runtime.InteropServices; using System.Linq; // Keep this declaration somewhere in your class (only once): [DllImport("shlwapi.dll", CharSet = CharSet.Unicode)] private static extern int StrCmpLogicalW(string x, string y); private void Speaker_List_SelectedIndexChanged(object sender, EventArgs e) { Voice_List.Items.Clear(); if (Dataset_List.SelectedItem == null || Speaker_List.SelectedItem == null) return; string totalPath = Path.Combine( DatasetHome, Dataset_List.SelectedItem.ToString(), Speaker_List.SelectedItem.ToString()); try { if (Directory.Exists(totalPath)) { var files = Directory.GetFiles(totalPath) .Where(f => f.EndsWith(".mp3", StringComparison.OrdinalIgnoreCase) || f.EndsWith(".wav", StringComparison.OrdinalIgnoreCase) || f.EndsWith(".flac", StringComparison.OrdinalIgnoreCase)) .Select(Path.GetFileName) .OrderBy(name => name, Comparer<string>.Create((a, b) => StrCmpLogicalW(a, b))) .ToList(); foreach (var fileName in files) { Voice_List.Items.Add(fileName); } } } catch (Exception ex) { MessageBox.Show($"Failed to load voice files: {ex.Message}", "Error", MessageBoxButtons.OK, MessageBoxIcon.Error); } }Summary:
- Added
.Select(Path.GetFileName)to get only the file names. - Sorted the list using natural sorting with
StrCmpLogicalW. - This mirrors how Windows Explorer sorts filenames with numbers correctly.
- Added
- User
why is it complaining the file exists? Microsoft Windows [Version 10.0.22631.5189] (c) Microsoft Corporation. All rights reserved.
C:\AI\fish-speech>conda activate fish-speech
(fish-speech) C:\AI\fish-speech>python tools/llama/build_dataset.py --input C:\AI\Datasets\Fish-Speech\GirlySpeech --output C:\AI\Datasets\Fish-Speech\GirlySpeech\protos --text-extension .lab --num-workers 16 Loading C:\AI\Datasets\Fish-Speech\GirlySpeech: 1it [00:00, 250.09it/s] Loading C:\AI\Datasets\Fish-Speech\GirlySpeech: 0it [00:00, ?it/s] 2025-05-03 04:39:20.194 | ERROR | main:task_generator_folder:41 - Failed to read text C:\AI\Datasets\Fish-Speech\GirlySpeech\SPK1\ere.npy: [Errno 2] No such file or directory: 'C:\AI\Datasets\Fish-Speech\GirlySpeech\SPK1\ere.lab' Grouping C:\AI\Datasets\Fish-Speech\GirlySpeech: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 988.29it/s] 2025-05-03 04:39:20.195 | INFO | main:task_generator_folder:46 - Found 0 groups in C:\AI\Datasets\Fish-Speech\GirlySpeech, []... 0it [00:00, ?it/s] 2025-05-03 04:39:20.205 | INFO | main:main:165 - Finished writing 1 shards to C:\AI\Datasets\Fish-Speech\GirlySpeech\protos
(fish-speech) C:\AI\fish-speech>
- Assistant
The error message:
Failed to read text C:\AI\Datasets\Fish-Speech\GirlySpeech\SPK1\ere.npy: [Errno 2] No such file or directory: 'C:\\AI\\Datasets\\Fish-Speech\\GirlySpeech\\SPK1\\ere.lab'and the subsequent log indicate that the script expected to find the file
ere.labinsideSPK1folder, but it doesn't exist. Because of that, it can't process the data and ends up processing zero groups.Key points about your issue:
- The script
build_dataset.pytries to read.labfiles as the text source (you specified--text-extension .lab). - It tried to load
ere.lab(corresponding toere.npy), but that.labfile wasn't found. - It's not a "file exists" error, but rather "file does not exist" (Errno 2) for the
.labfile. - Because the text files are missing, no groups were found, so the dataset is effectively empty.
- Your output directory was created and presumably some shards were written, but they have no usable content.
What to do / Check:
-
Check that
.labfiles actually exist in your dataset folder (C:\AI\Datasets\Fish-Speech\GirlySpeech\SPK1\) for every sample (in this case,ere.lab). -
Verify file extensions and file naming conventions:
- The script uses the
--text-extension .labarg to find text files. - If your text files have a different extension (e.g.
.txtor.label), you need to change that argument accordingly. - Or ensure your text files are correctly named, matching the audio files.
- The script uses the
-
Check if your dataset extraction or preparation step created the needed
.labtranscription files. -
If you are seeing
.npyfiles but no corresponding.labfiles, it means the script depends on.labfiles but they are missing.
Why might this happen?
- You may have audio files (
.wav,.mp3, etc.) and feature files (.npy), but no text/transcription files (.lab). - Sometimes, dataset preprocessing scripts generate
.npyfeature arrays but the raw text labels are missing or in a different format. - The dataset might be incomplete or you are pointing to the wrong directory.
Quick sanity check:
Try running a simple check in Python or File Explorer for the presence of those
.labfiles:import os folder = r"C:\AI\Datasets\Fish-Speech\GirlySpeech\SPK1" print([f for f in os.listdir(folder) if f.endswith('.lab')])If this prints empty or doesn’t include
ere.lab, then your dataset is missing the required label files.
Summary
- This is a missing file error, not "file exists".
build_dataset.pyrequires.labfiles — check your dataset.- Make sure the files exist and your
--text-extensionmatches your transcription files. - After fixing that, the script should be able to process the data correctly.
If you want me to help further, please share the folder structure and file types under your dataset folders!
- The script