LetUs start here:

Are you into consuming security products or exploring data? Do you feel that you need a security appliance for everything? Do you say Rgive me the dataS or Rgive me out-of-the-box content, canned rules, signaturesS? Do you just want to be shown Rwhat you need to knowS or are you willing to figure what you need to know from the data you have? Would you rather learn Rwhat your data is trying to tell youS or Rwhat latest stuff the vendors have on saleS? At this point, if you just want Ra boxS, the path of big data analytics is not for you. Analytic mindset seems to determine the success of a big data initiative for security more than anything else. Those organizations that succeed with using big data for security are all subscribers to this view. They all state that for the foreseeable future, there will be no Rboxed security big data analyticsS products (except for some narrow and specific problems solved by specific tools).

Along the same line, somebody asked me one of those days RDo I need to toss out my SIEM and buy Ra big data productS? P NO, SILLY!!! You need to try using your SIEM to actually analyze the data inside itI. If you analyze the data inside your SIEM to its maximum potential, then you may need to look beyond that into other tools and approaches. But start from data exploration, not from tool replacement!

Therefore, the best analytics Rstarter packS is the one you can do on the data and tools you have. If you have RDBMS full of logs, flows or context data P start there. Leverage the data you have collected to make better decisions; use traditional BI tools on that database to see what emerges (some of the current Tbig data for securityU champions started like that). In fact, if all you have is Excel and bunch of exported reports P well, start exploring there!

The evolution then continues like this: ask questions of the data you have -> get a useful answer P> become more data driven P> gather more data P> ask more useful questions.

Organization then start to naturally Rthink data firstS: new threat pops up? LetUs go into our data and see what is up, then create new analytic approaches to detect and investigate it P rather than start whining Rwhat tool do I buy next?S No amount of Hadoop will give you big data analytics without a mindset. As I found out, this mindset and data curiosity is most important; by the way, mindset importance is also well-established for doing indicator hunting and anomaly detection, such as using network forensics and ETDR tools (also see Alert-driven vs Exploration-driven Security Analysis).

So, go and build your own data analytic discipline! Build analytic-centric and data-centric mindset P rather than buy or download any particular big data technology. Start data driven P not tool-driven (and, yes, Hadoop is a tool too P and the one often hard to implement, operate and utilize, especially in the absence of clarity of purpose or your goals). You cannot solve a mindset problem by buying technology; you need a mindset for leveraging data differently.

The only path is to shift the thinking, learn to be data-centric and data-driven and then solve problems that call for bigger data. Such culture change has to happen for the big data approaches to become pervasive across the industry. And yes, this includes willingness to explore, follow leads, and occasionally arrive at dead ends and algorithms that donUt work.

In fact, most of my questions about the particular algorithms aimed at those few (REALLY few!) organization that do advanced analytics on large-scale security data resulted in no single list of Rtop useful algorithms.S Machine learning (ML), Bayesian, clustering, various data mining and text mining methods were mentioned, but none were highlighted as Rmust use.S What was a must? Again, it was a mindset and willingness to dip into a toolbox of algorithms to throw at dataI

Finally, some quick tips:

Got a SIEM? Go beyond vendor reports, run those queries direct to backend, extract and visualize. Got a little other data relevant to security? Try open source mining tools, write scripts to analyze and profile data, and look at the data and see what it is trying to tell you I To summarize, while conceptually, security is becoming a big data analytics problem, practically, it wonUt become that for you if you keep investing in prevention and buying boxes.

There you have it! Now, GO EXPLORE YOUR DATA!

Datum: 18-11-2013 Anton Chuvakin Bron: Gartner Key word: online analytical processing Content: about what is Big Data