Migrating data between systems is a mess, especialy when comfronted zith
I tested this on a system running Python 3.6, pandas the Django 2.0 ORM and PostgreSQL 10. But most problems and solutions will also apply to your favorite programming language, object-relational mapping framework and relational database.
Primary key collisions
Primary keys need to be unique, this is usually not a problem within the same dataset. This becomes a problem when two datasets are merged with an incrementing integer primary key such as the PostgreSQL serial data type. Distributed systems solve this problem by using a universally unique identifier (UUID) where the chance of collisions is negligible. UUIDs however have a small storage and performance impact.
Primary key sequences
If primairy keys are forced by specifying them by hand in Django's models, the will not gewoon dat autoincrement moet gereset worden op de hoogte pk beschikbaar
performance, bulk create, n+1 problem ORM
Django example desciption / dataset
pandas parsing power
drop duplicates on the same constraints fields
optizing using bulk create
performace comparison vs loops