开发者

mysql load data infile vs using table

开发者 https://www.devze.com 2023-02-23 08:21 出处:网络
I have a lot of data (11k rows) in a 开发者_开发技巧text file. I can transform it into sql file so that I can do load data infile. However, i was wondering if I insert the data into a staging table an

I have a lot of data (11k rows) in a 开发者_开发技巧text file. I can transform it into sql file so that I can do load data infile. However, i was wondering if I insert the data into a staging table and load data from that table. What is the performance comparisiong between the two. Do you suggest a better way to do it?

Thanks,


The mysql file import it pretty quick, so unless there are concerns of transactional conflicts on a live instance, I wouldn't bother with a staging table. 11k is relatively small assuming that the table doesn't have 100 fields or any large blobs/binaries.


This is a fairly typical approach, if I'm understanding you correctly. Just load the raw data, exactly as it is in the file, into a temporary table, then use SQL statements to transfer it to the target table, and truncate or drop the temp table. This is a simple approach and sort of a poor man's ETL. For 11k records, this should be fine (11k is a pretty small table, actually, I do millions of rows regularly).

If you want the best performing approach, you could use a tool like Talend to ETL and transform the data on the way into the database, and then you eliminate the need for the temporary table.

Either approach works, and I'd suggest that performance isn't likely to be the deciding factor in how to do it, nearly as much so as whatever is the simplest approach.

0

精彩评论

暂无评论...
验证码 换一张
取 消

关注公众号