How to import large amounts of data into Rails?

To load small amounts of data, I've been using rake tasks to important data from CSVs into Rails:

desc "Import users." 
task :import_users => :environment do
    File.open("users.txt", "r").each do |line|
        name, age, profession = line.strip.split("\t")
        u = User.new(:name => name, :age => age, :profession => profession)
        u.save
    end
end

For larger files (about 50,000 re开发者_C百科cords), though, this is incredibly slow. Is there a faster way to import the data?

---

You might want to take a look at activerecord-import and check out this similar thread.

---

Without extra libraries (I agree that a bulk import with AR extensions should be quicker)(though AR:Extension skips model validations) you can add a little bit of concurrency and take advantage of a multicore machine

# Returns the number of processor for Linux, OS X or Windows.
def number_of_processors
  if RUBY_PLATFORM =~ /linux/
    return `cat /proc/cpuinfo | grep processor | wc -l`.to_i
  elsif RUBY_PLATFORM =~ /darwin/
    return `sysctl -n hw.logicalcpu`.to_i
  elsif RUBY_PLATFORM =~ /win32/
    # this works for windows 2000 or greater
    require 'win32ole'
    wmi = WIN32OLE.connect("winmgmts://")
    wmi.ExecQuery("select * from Win32_ComputerSystem").each do |system| 
      begin
        processors = system.NumberOfLogicalProcessors
      rescue
        processors = 0
      end
      return [system.NumberOfProcessors, processors].max
    end
  end
  raise "can't determine 'number_of_processors' for '#{RUBY_PLATFORM}'"
end

desc "Import users." 
task :fork_import_users => :environment do
  procs = number_of_processors
  lines = IO.readlines('user.txt')
  nb_lines = lines.size
  slices = nb_lines / procs
  procs.times do
    subset = lines.slice!(0..slices)
    fork do
      subset.each do |line|
        name, age, profession = line.strip.split("\t")
        u = User.new(:name => name, :age => age, :profession => profession)
        u.save
      end
    end
  end
  Process.waitall
end

on my machine with 2 cores and the fork version I get

real    1m41.974s
user    1m32.629s
sys     0m7.318s

while with your version:

real    2m56.401s
user    1m21.953s
sys     0m7.529s

---

You should try FasterCSV. It is quite fast and dead easy to use for me.