This post introduces how to import CSV data using Pandas to Django models.
Python has a built-in csv library, but do not use that, it is not flexible for csv data that has both string and number based data. See the reasons below:
builtin csv module is very primitive at handling mixed data-types, does all its type conversion at import-time, and even at that has a very restrictive menu of options, which will mangle most real-world datasets (inconsistent quoting and escaping, missing or incomplete values in Booleans and factors, mismatched Unicode encoding resulting in phantom quote or escape characters inside fields, incomplete lines will cause exception). Fixing csv import is one of countless benefits of pandas. So, your ultimate answer is indeed stop using builtin csv import and start using pandas.
Do not import the data in csv file to Django models via row by row method– that is too slow.
Django (version > 1.4 ) provides bulk_create as an object manager method which takes as input an array of objects created using the class constructor.
See my example code below:
import pandas as pd
df=pd.read_csv('test_csv.txt',sep=';')
#print(df)
row_iter = df.iterrows()
objs = [
myClass_in_model(
field_1 = row['Name'],
field_2 = row['Description'],
field_3 = row['Notes'],
field_4 = row['Votes']
)
for index, row in row_iter
]
myClass_in_model.objects.bulk_create(objs)
#Note: myClass_in_model: the class (i.e., the table you want to populate data from csv) we defined in Django model.py
#Note: field_1 to filed_4 are the fields you defined in your Django model.
References:
Import csv data into django models
How to write a Pandas Dataframe to Django model
Django bulk_create function example
Changing strings to Floats in an imported .csv
