import csv

 Reading big csv files in python

The data is split into pieces based on the size we define which can be read using for loop as shown below

Code:

# chunksize is defined here

df = pd.read_csv("path/test.csv", chunksize=10000)




for data in df:

    print(data.shape)







The above code will print the shape of chunked data




Using for loop, the chunked data can be processed similar to a normal dataframe.