what are the fastest/Popular search technologies [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance. Closed 10 years ago. 开发者_开发知识库

What are the fastest search technologies apart from Relational DB Searches ?

I have a collection of Text Files from varied sources (Banks/Ledgers/Stock Markets). Each line in these Text Files is a Record. Each line can further be parsed into some DB Columns(Stock Name/Date of purchase/Owner/...). It is not necessary that each line has all the fields. If some text is not mathcing any of the fields, I store it as it is in a seperate column(columnA). If the parser fails to find any column at all I just store the Line in columnA.

I than do a DB search based on user Inputs. I find that the search becomes slow after a few million records. I want to move to an alternative storage like File System or some other Indexed storage.

What do you think would be the fastest storage technology for this kind of requirement ?

I am currently using SQLite DB. Since I am about to rewrite the entire code, I will be more than happy if any other storage technology can give me faster results.

---

1. Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
2. Lucene.Net is a source code, class-per-class, API-per-API and algorithmatic port of the Java Lucene search engine to the C# and .NET platform utilizing Microsoft .NET Framework.

---

Relational databases aren't great at text searching. Perhaps you could try something more suited to the task like solr. That said, it seems like you're putting a lot of data in SQLite (it might be a bit "lite" for your purpose). If you switched to something like MySQL or Postgres you might get the speed you need without much of a rewrite.