could you help make this more efficient?: if phylo == "mammals": for odb in…
- User
could you help make this more efficient?:
if phylo == "mammals": for odb in Constants.BUSCO_DBS_MAMMALS.values(): log.record(f"running pseudo-BUSCO for {odb} database") db = pd.read_csv(f"{path}/supply/odbs/{odb}", sep="\t") db = db.loc[db[src].dropna().index] log.record(f"number of genes in {odb} database with {src} nomenclature: {len(db)}") # Get the number of genes in the query annotation that are present in the BUSCO databases overlap = set(db[src]).intersection(set(table["t_gene"])) rs = len(overlap)/len(db) stats.append((odb, rs)) elif phylo == "aves": for odb in Constants.BUSCO_DBS_BIRDS.values(): log.record(f"running pseudo-BUSCO for {odb} database") db = pd.read_csv(f"{path}/supply/odbs/{odb}", sep="\t") db = db.loc[db[src].dropna().index] log.record(f"number of genes in {odb} database with {src} nomenclature: {len(db)}") # Get the number of genes in the query annotation that are present in the BUSCO databases overlap = set(db[src]).intersection(set(table["t_gene"])) rs = len(overlap)/len(db) stats.append((odb, rs)) - Assistant
To make the code more efficient, we can focus on reducing redundant operations, improving I/O efficiency, and streamlining the control flow. Here's a revised version of the code:
# Define a function to process the databases def process_busco_dbs(busco_dbs, phylo_group): source_genes = set(table["t_gene"]) # Compute once and use for all databases for odb in busco_dbs: log.record(f"running pseudo-BUSCO for {odb} database") db_file_path = f"{path}/supply/odbs/{odb}" # Load only the necessary column to reduce memory usage db = pd.read_csv(db_file_path, sep="\t", usecols=[src]).dropna() number_of_genes = len(db) log.record(f"number of genes in {odb} database with {src} nomenclature: {number_of_genes}") # Get the number of genes in the query annotation that are present in the BUSCO databases overlap = source_genes.intersection(db[src]) rs = len(overlap) / number_of_genes stats.append((odb, rs)) # Determine which group to process if phylo == "mammals": busco_dbs = Constants.BUSCO_DBS_MAMMALS.values() process_busco_dbs(busco_dbs, "mammals") elif phylo == "aves": busco_dbs = Constants.BUSCO_DBS_BIRDS.values() process_busco_dbs(busco_dbs, "aves")In this version, the following changes have been made:
-
Function Extraction: The common code inside the if-elif blocks has been extracted into a function
process_busco_dbs. This removes redundancy and makes the code cleaner. -
Source Gene Set: The set of source genes (
table["t_gene"]) is created once outside the loop assource_genes. This avoids the repeated conversion oftable["t_gene"]to a set for each database file. -
Selective CSV Column Loading: Instead of loading the entire CSV and then dropping N/A values, we now load only the required column(s) into memory. This can significantly save memory usage and potentially speed up I/O operations.
-
Dropping N/A Values: Since we are only interested in the
srccolumn, we can drop N/A values immediately after reading the CSV. -
Variables Initialization: External variables like
pathandsrcneed to be defined before this script (not shown here), andstatsshould be initialized as an empty list before the function is called.
These changes should lead to a more efficient execution, especially in terms of memory usage and code maintainability. Also, make sure to handle potential exceptions or errors that could occur while reading the CSV files.
-