Bad SQLite performance on external storage in Android

I'm using the external storage for storing events in a database while they are waiting to be sent to the server.

I'm seeing really bad performance when inserting records. I know the external memory can be slow but I wanted to see some number so I wrote a small app which tests it.

Here is the code:

public static final int INSERTS = 100;

File dbFile = new File(Environment.getExternalStorageDirectory(), "test.sqlite3");
// File dbFile = new File(getFilesDir(), "test.sqlite3");
dbFile.delete();

SQLiteDatabase db = SQLiteDatabase.openOrCreateDatabase(dbFile, null);

db.execSQL("CREATE TABLE events (_id integer primary key autoincrement, event_type TEXT NOT NULL, timestamp BIGINT, data TEXT);");
db.execSQL("CREATE INDEX mainIndex ON events (event_type, timestamp ASC);");

InsertHelper helper = new InsertHelper(db, "events");

final int eventTypeCol = helper.getColumnIndex("event_type");
final int timestampCol = helper.getColumnIndex("timestamp");
final int dataCol = helper.getColumnIndex("data");

long start = System.currentTimeMillis();

String eventType = "foo", data = "bar";
long timestamp = 4711;

for(int i = 0; i < INSERTS; ++i) {
    helper.prepareForInsert();
    helper.bind(eventTypeCol, eventType);
    helper.bind(timestampCol, timestamp);
    helper.bind(dataCol, data);
    helper.execute();
}

long end = System.currentTimeMillis();

Log.i("Test", String.format("InsertHelper, Speed: %d ms, Records per second: %.2f", (int)(end-start), 1000*(double)INSERTS/(double)(end-start)));

db.close();
dbFile.delete();

db = SQLiteDatabase.openOrCreateDatabase(dbFile, null);

db.execSQL("CREATE TABLE events (_id integer primary key autoincrement, event_type TEXT NOT NULL, timestamp BIGINT, data TEXT);");
db.execSQL("CREATE INDEX mainIndex ON events (event_type, timestamp ASC);");


start = System.currentTimeMillis();
ContentValues cv = new ContentValues();

for(int i = 0; i < INSERTS; ++i) {
    cv.put("event_type", eventType);
    cv.put("timestamp", timestamp);
    cv.put("data", data);
    db.insert("events", null, cv);
}

end = System.currentTimeMillis();

Log.i("Test", String.format("Normal, Speed: %d ms, Records per second: %.2f", end-start, 1000*(double)INSERTS/(double)(end-start)));

db.close();
dbFile.delete();

The database is exactly as the one my real app is using, I tried removing the index but it made no difference.

Here are the results:

Nexus One, Internal memory

      Method | Records | Time (ms) | Records per second
-------------+---------+-----------+--------------------
      Normal |   100   |    2072   |       48.26
InsertHelper |   100   |    1662   |       60.17


Nexus One, External memory:

      Method | Records | Time (ms) | Records per second
-------------+---------+-----------+--------------------
      Normal |   100   |    7390   |       13.53
InsertHelper |   100   |    7152   |       13.98


Emulator, Internal memory:

      Method | Records | Time (ms) | Records per second
-------------+---------+-----------+--------------------
      Normal |   100   |    1803   |       55.46
InsertHelper |   100   |    3075   |       32.52


Emulator, External memory:

      Method | Records | Time (ms) | Records per second
-------------+---------+-----------+--------------------
      Normal |   100   |    5742   |       17.42
InsertHelper |   100   |    7164   |       13.96 

As you can see the emulator cannot be trusted, InsertHelper should be faster if anything.

This is开发者_C百科, of course, to be expected, the test was mostly done out of curiosity.

What have me concerned however is the bad performance on my phone when using external memory, have I missed some crucial aspect of SQLiteDatabase or is it simply so that the SD card will be slow?

I can add that in my real app I've disabled locking and it makes little difference.

---

CommonsWare is correct in his comment. Something that makes a big difference for db performance is using transactions. Wrap your insert loop in a transaction. I'm not 100% sure if it would work with the InsertHelper but you can try replacing your for loop with this:

db.beginTransaction();
try {
    for(int i = 0; i < INSERTS; ++i) {
        helper.prepareForInsert();
        helper.bind(eventTypeCol, eventType);
        helper.bind(timestampCol, timestamp);
        helper.bind(dataCol, data);
        helper.execute();
    }
    db.setTransactionSuccessful();
} finally {
    db.endTransaction();
}

---

I have some db performance issues so I used your code to measure the inserts per second on my system. But I also added wrapping in {begin,end}Transaction().

In the emulator. I got:

InsertHelper-Internal-Trans, Speed: 67 ms, Records per second: 1492.54
InsertHelper-External-Trans, Speed: 70 ms, Records per second: 1428.57
Normal-Internal-Trans, Speed: 148 ms, Records per second: 675.68
Normal-External-Trans, Speed: 152 ms, Records per second: 657.89
InsertHelper-Internal-NoTrans, Speed: 514 ms, Records per second: 194.55
Normal-Internal-NoTrans, Speed: 519 ms, Records per second: 192.68
InsertHelper-External-NoTrans, Speed: 590 ms, Records per second: 169.49
Normal-External-NoTrans, Speed: 618 ms, Records per second: 161.81

And on a Samsung Galaxy Note:

InsertHelper-External-Trans, Speed: 52 ms, Records per second: 1923.08
InsertHelper-Internal-Trans, Speed: 52 ms, Records per second: 1923.08
Normal-External-Trans, Speed: 77 ms, Records per second: 1298.70
Normal-Internal-Trans, Speed: 121 ms, Records per second: 826.45
Normal-External-NoTrans, Speed: 4562 ms, Records per second: 21.92
Normal-Internal-NoTrans, Speed: 4855 ms, Records per second: 20.60
InsertHelper-External-NoTrans, Speed: 5997 ms, Records per second: 16.68
InsertHelper-Internal-NoTrans, Speed: 8361 ms, Records per second: 11.96