(509) 720-7677

Big Data

As we often say, Big Data is trendy, but after the hype dies down it will remain an important tool. There certainly are large gold nuggets to be had in most commercial or industrial enterprises, but it is foolish to try to dig up every mountain to find them. The wise prospector gets a good map first!

Another common error is to assume that gathering tons of data into Hadoop or some other NoSQL data store constitutes a decent entre` into the “Big Data” arena. It doesn’t. Data is not information, and if you think about it, data is meaningless without context.

Finally, and perhaps most importantly, “Big Data” is a commitment to the scientific method. For example, let’s say that your business intelligence tool reveals that data point B tends to rise when data point A rises. It is faulty logic to assume that A directly influences B… correlation does not imply causality.

The way to avoid faulty–and probably costly–decisions based on flawed interpretations is to experiment. That is, you test the hypothesis that increasing A will also cause B to increase. The other aspect to this is to watch for side effects; increasing A may well increase B as well, but there may be an undesirable drop in C.

If some company comes swooping in to provide one-shot Big Data implementations, be skeptical. It is a process.