Starting with Data Mining

I have started learning Data Mining and wish to create a small project in C++/Java that allows me to utilize a da开发者_如何学Pythontabase, say from twitter and then publish a particular set of results (for eg. all the news items on a feed). I want to know how to go about it? Where should I start?

---

This is a really broad question, so it's hard to answer. Here are some things to consider:

1. Where are you going to get the data? You mention twitter, but you'll still need to collect the data in some way. There are probably libraries out there for listening to twitter streams, or you could probably buy the data if someone is selling it.

2. Where are you going to store the data? Depending on how much you'll have and what you plan to do with it, a traditional relational database may or may not be the best fit. You may be better off with something that supports running mapreduce jobs out-of-the box.

Based on the answers to those questions, the choice of programming languages and libraries will be easier to make.

If you're really set on Java, then I think a Hadoop cluster is probably what you want to start out with. It supports writing mapreduce jobs in Java, and works as an effective platform for other systems such as HBase, a column-oriented datastore.

If your data are going to be fairly regular (that is, not much variation in structure from one record to the next), maybe Hive would be a better fit. With Hive, you can write SQL-like queries, given only data files as input. I've never used Mahout, but I understand that its machine learning capabilities are suited for data mining tasks.

These are just some ideas that come to mind. There are lots of options out there and choosing between them has as much to do with the particular problem you're trying to solve and your own personal tastes as anything else.

---

If you just want to start learning about Data Mining there are two books that I particularly really enjoy:

Pattern Recognition and Machine Learning. Christopher M. Bishop. Springer.

And this one, which is for free:

http://infolab.stanford.edu/~ullman/mmds.html

---

Good references for you are

AI course taught by people who actually know the subject,Weka website, Machine Learning datasets, Even more datasets, Framework for supporting the mining of larger datasets.

The first link is a good introduction on AI taught by Peter Norvig and Sebastian Thrun, Google's Research Director, and Stanley's creator (the autonomous car), respectively.

The second link you get you to Weka website. Download the software - which is pretty intuitive - and get the book. Make sure you understand all the concepts: what's data mining, what's machine learning, what are the most common tasks, and what are the rationales behind them. Play a lot with the examples - the software package bundles some datasets - until you understand what generated the results.

Next, go to real datasets and play with them. When tackling massive datasets, you may face several performance issues with Weka - which is more of a learning tool as far as my experience can tell. Thus I recommend you to take a look at the fifth link, which will get you to Apache Mahout website.

It's far from being a simple topic, however, it's quite interesting.

---

I can tell you how I did it.

1) I got the data using twitter4j.

2) I analyzed the data using JUNG. You have to define a class representing edges and a class representing vertices. These classes will contain the attributes of the edges and vertices.

3) Then, there is a simple function to add an edge g.addedge(V1,V2,edgeFromV1ToV2) or to add a vertex g.addVertex(V).

The class that defines edges or vertices is easy to create. As an example :

`public class MyEdge {

int Id;

}`

The same is done for vertices. Today I would do it with R, but if you don't want to learn a new programming language, just import jung which is a java library.

---

Data mining is broad fields with many different techniques; classification, clustering, association and pattern mining, outlier detection, etc.

You should first decide what you want to do and then decide wich algorithm you need.

If you are new to data mining, I would recommend to read some books like Introduction to Data Mining by Tan, Steinbach and Kumar.

---

I would like to suggest you to use python or R for data mining process. Doing work with java or c , it bit difficult in the sense you need to do a lot coding