Handling large amounts of telemetry

I have a project where we read large amount of telemetry data. The telemetry consists of "channels", a channel is often values from a single sensor. For instance a temperature sensor.

Each reading consists of:

A: The measured value

B: The channel id

C: The time of the measurement

We receive the telemetry a fairly high rate, up to 500 Hz (500 samples each second) and as low as 1 Hz for each channel. The number of simultaneous channels is usually about 150.

The rate of each channel can not be expected to be precise, it may vary.

开发者_JAVA百科

So I am looking for a collection/algorithm to handle the data and later display the values, the displays consists of graphs and numeric readouts.

Normally we receive the data sequentially, but there can be outages of longer durations. Also the missing data from the outages can be received later.

So long story short, I need a collection that can structure the data:

A: Must be able to access a single reading by time like Value(time).

B: Must be able to read the values sequentially from time a to b.

C: Insert new values, either at the end or in the middle.

I have made an implementation using skiplist (http://en.wikipedia.org/wiki/Skiplist). It seems to meet all the requirements, but it eats lots memory. My initial calculations showed somewhere between a factor 5 to 10 of the measured values. In practice it is higher then that.

I also considered adding the values to a database, and read them from there, but it seems slow - I don't see an SQL server inserting that many values that fast.

So, any thoughts on a collection/algorithm that can do this?

---

You might be able to use HDF5 for this.

A telemetry application for this format is mentioned here.