i have following code to read my csv file which is a 27GB size file import pandas as pd…
- User
i have following code to read my csv file which is a 27GB size file
import pandas as pd
Load the dataset using pandas
df = pd.read_csv(r"C:\Users\arisa\Desktop\combined_day.csv")
and im getting following error: { "name": "MemoryError", "message": "Unable to allocate 13.7 GiB for an array with shape (6317, 290138) and data type float64", "stack": "--------------------------------------------------------------------------- MemoryError Traceback (most recent call last) Cell In[1], line 3 1 import pandas as pd 2 # Load the dataset using pandas ----> 3 df = pd.read_csv(r"C:\Users\arisa\Desktop\combined_day.csv")
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py:1026, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, date_format, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options, dtype_backend) 1013 kwds_defaults = _refine_defaults_read( 1014 dialect, 1015 delimiter, (...) 1022 dtype_backend=dtype_backend, 1023 ) 1024 kwds.update(kwds_defaults) -> 1026 return _read(filepath_or_buffer, kwds)
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py:626, in _read(filepath_or_buffer, kwds) 623 return parser 625 with parser: --> 626 return parser.read(nrows)
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\parsers\readers.py:1968, in TextFileReader.read(self, nrows) 1965 else: 1966 new_col_dict = col_dict -> 1968 df = DataFrame( 1969 new_col_dict, 1970 columns=columns, 1971 index=index, 1972 copy=not using_copy_on_write(), 1973 ) 1975 self._currow += new_rows 1976 return df
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\frame.py:778, in DataFrame.init(self, data, index, columns, dtype, copy) 772 mgr = self._init_mgr( 773 data, axes={"index": index, "columns": columns}, dtype=dtype, copy=copy 774 ) 776 elif isinstance(data, dict): 777 # GH#38939 de facto copy defaults to False only in non-dict cases --> 778 mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager) 779 elif isinstance(data, ma.MaskedArray): 780 from numpy.ma import mrecords
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\construction.py:503, in dict_to_mgr(data, index, columns, dtype, typ, copy) 499 else: 500 # dtype check to exclude e.g. range objects, scalars 501 arrays = [x.copy() if hasattr(x, "dtype") else x for x in arrays] --> 503 return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\construction.py:152, in arrays_to_mgr(arrays, columns, index, dtype, verify_integrity, typ, consolidate) 149 axes = [columns, index] 151 if typ == "block": --> 152 return create_block_manager_from_column_arrays( 153 arrays, axes, consolidate=consolidate, refs=refs 154 ) 155 elif typ == "array": 156 return ArrayManager(arrays, [index, columns])
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\managers.py:2144, in create_block_manager_from_column_arrays(arrays, axes, consolidate, refs) 2142 raise_construction_error(len(arrays), arrays[0].shape, axes, e) 2143 if consolidate: -> 2144 mgr._consolidate_inplace() 2145 return mgr
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\managers.py:1788, in BlockManager._consolidate_inplace(self) 1782 def _consolidate_inplace(self) -> None: 1783 # In general, _consolidate_inplace should only be called via 1784 # DataFrame._consolidate_inplace, otherwise we will fail to invalidate 1785 # the DataFrame's _item_cache. The exception is for newly-created 1786 # BlockManager objects not yet attached to a DataFrame. 1787 if not self.is_consolidated(): -> 1788 self.blocks = _consolidate(self.blocks) 1789 self._is_consolidated = True 1790 self._known_consolidated = True
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\managers.py:2269, in _consolidate(blocks) 2267 new_blocks: list[Block] = [] 2268 for (_can_consolidate, dtype), group_blocks in grouper: -> 2269 merged_blocks, _ = _merge_blocks( 2270 list(group_blocks), dtype=dtype, can_consolidate=_can_consolidate 2271 ) 2272 new_blocks = extend_blocks(merged_blocks, new_blocks) 2273 return tuple(new_blocks)
File c:\Users\arisa\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\internals\managers.py:2301, in _merge_blocks(blocks, dtype, can_consolidate) 2298 new_values = bvals2[0]._concat_same_type(bvals2, axis=0) 2300 argsort = np.argsort(new_mgr_locs) -> 2301 new_values = new_values[argsort] 2302 new_mgr_locs = new_mgr_locs[argsort] 2304 bp = BlockPlacement(new_mgr_locs)
MemoryError: Unable to allocate 13.7 GiB for an array with shape (6317, 290138) and data type float64" }
- Assistant
The error you're encountering is due to Python attempting to load a very large CSV file into memory all at once, which exceeds the available memory on your system. A 27GB file is substantial, and depending on the contents and structure of your CSV, the memory required to hold it in a DataFrame can easily exceed your system's capabilities, especially once you factor in the overhead of Python objects.
To work with such large datasets, you have a few options:
1. Use
chunksizeParameterPandas allows you to read a CSV file in chunks of manageable size. This approach lets you process the file piece by piece.
chunk_size = 10 ** 6 # Number of rows per chunk. Adjust based on your system's memory. chunks = pd.read_csv(r"C:\Users\arisa\Desktop\combined_day.csv", chunksize=chunk_size) for chunk in chunks: # Process each chunk here # Example: print the size of each chunk print(chunk.shape)2. Use Dask
Dask is a parallel computing Python library that scales the Python ecosystem using parallel computing that integrates well with Pandas. Dask can handle datasets that are larger than your machine's memory by breaking them into smaller pieces and processing these pieces in parallel.
from dask import dataframe as dd # This doesn't load your data yet. dask_df = dd.read_csv(r"C:\Users\arisa\Desktop\combined_day.csv") # Use Dask operations similar to Pandas. Operations are lazy and computed on call to .compute() # For example, to compute a simple aggregation: result = dask_df.sum().compute()3. Optimize Data Types
If you still need to use Pandas and wish to read in the entire dataset, you can attempt to reduce memory usage by optimizing data types. For example, if your dataset contains numeric columns with integer values, make sure they are read as the appropriate integer type (
int8,int16,int32) rather than defaulting toint64. Similarly, convert float64 columns to float32 if precision allows.4. Increase Systems Memory
This is more about your hardware capacity. For large datasets, it is often beneficial to have a system with a large amount of RAM. This might mean upgrading your current system or using a high-memory instance on a cloud platform.
5. Database Solutions
Another approach is to use database solutions designed to handle large datasets efficiently. You can import your CSV into a database (like PostgreSQL, MySQL, or SQLite) and then query chunks of it as needed without loading the entire dataset into memory.
6. Use
usecolsParameterIf you only need a subset of columns from your CSV file, you can decrease memory usage by specifying those columns using the
usecolsparameter inpd.read_csv().cols_to_use = ['Column1', 'Column2'] df = pd.read_csv(r"C:\Users\arisa\Desktop\combined_day.csv", usecols=cols_to_use)Choosing the right approach depends on your specific requirements, such as whether you need to process the data all at once, your hardware limitations, and the complexity of processing you're performing on the data.