What is data mining? How analytics uncovers insights

The applications of data mining

Data mining can be applied to a variety of applications in virtually every industry.

• Retailers can deploy data mining to better identify which products people are likely to purchase based on their past buying habits, or which goods are likely to sell at certain times of the year. This can help merchandisers plan inventories and store layouts.
• Banks and other financial services providers can mine data related to their clients’ accounts, transactions, and channel preferences to better meet their needs. They can also gather then analyzed data from their websites and social media interactionsto help increase the loyalty of existing customers and attract new ones.
• Manufacturing companies can use data mining to look for patterns in the production process, so they can precisely identify bottlenecks and flawed methods and find ways to increase efficiencies. They can also apply knowledge from data mining to the design of products, and make tweaks based on feedback from customer experiences.
• Educational institutions can benefit from data mining such as analyzing data sets to predict the future learning behaviors and performance of students, and then using this knowledge to make improvements in teaching methods or curricula.
• Health care providers can mine and analyze data to determine better ways of delivering care to patients and cutting costs. With the help of data mining, they can predict how many patients they will need to care for and what type of services those patients will need. In the life sciences, mining can be used to glean insights from massive biological data, to help develop new medicines and other treatments.
• In multiple industries, including health care and retail, you can use data mining to detect fraud and other abuses—much more quickly than with traditional methods for identifying such activities.

The key components of data mining

The process of data mining includes several distinct components that address different needs:

• Preprocessing. Before you can apply data mining algorithms, you need to build a target data set. One common source for data is a data mart or warehouse. You need to perform preprocessing to be able to analyze the data sets.
• Data cleansing and preparation. The target data set must be cleaned and otherwise prepared, to remove “noise,” address missing values, filter outlying data points (for anomaly detection) to remove errors or do further exploration, create segmentation rules, and perform other functions related to data preparation.
• Association rule learning (also known as market basket analysis). These tools search for relationships among variables in a data set, such as determining which products in a store are often purchased together.
• Clustering. This feature of data mining is used to discover groups and structures in data sets that are in some way similar to each other, without using known structures in the data.
• Classification. Tools that perform classification generalize known structures to apply to new data points, such as when an email application tries to classify a message as legitimate mail or spam.
• Regression. This data mining technique tis used to predict a range of numeric values, such as sales, housing values, temperatures, or prices when given a particular data set.
• Summarization. This technique provides a compact representation of a data set, including visualization and report generation.

Dozens of vendors provide data mining software tools, some offering proprietary software and others delivering products via open source efforts.

Among the key vendors that offer proprietary data-mining software applications are Angoss, Clarabridge, IBM, Microsoft, Open Text, Oracle, RapidMiner, SAS Institute, and SAP.

Organizations that provide open source data mining software and applications include Carrot2, Knime, Massive Online Analysis, ML-Flex, Orange, UIMA, and Weka.

The risks and challenges of data mining

Data mining comes with its share of risks and challenges. As with any technology that involves the use of potentially sensitive or personally identifiable information, security and privacy are among the biggest concerns.

At a fundamental level, the data being mined needs to be complete, accurate, and reliable; after all, you’re using it to make significant business decisions and often to interact with the public, regulators, investors, and business partners. Modern forms of data also require new kinds of technologies, such as for bringing together data sets from a variety of distributed computing environments (aka big data integration) and for more complex data, such as images and video, temporal data, and spatial data.

Getting the right data and then pulling it together so it can be mined isn’t the end of the challenge for IT. The cloud, storage, and network systems need to enable high performance of the data mining tools. And the resulting information from the data mining needs to be presented clearly to the wide range of users expected to act on and interpret it. You’ll need people with skills in data science and related areas.

From a privacy standpoint, the idea of mining information that relates to how people behave, what they buy, what websites they visit, and so on can set off concerns about companies gathering too much information. That affects not just your technological implementation but your business strategy and risk profile.

Beyond the ethics of tracking individuals so thoroughly, there are also legal requirements about how data can be gathered, identified to a person, and shared. The United States’ Health Insurance Portability and Accountability Act (HIPAA) and the European Union’s General Data Protection Directive (GDPR) are among the best known.

In data mining, the initial act of preparation itself, such as aggregating and then rationalizing data, can disclose information or patterns the might compromise the confidentiality of the data. Thus, it’s possible to inadvertently run afoul of ethical concerns or legal requirements.

Data mining also requires data protection every step of the way, to make sure data is not stolen, altered, or accessed secretly. Security tools include encryption, access controls and network security mechanisms.

Data mining is a key differentiator

Despite these challenges, data mining has become a vital component of the IT strategies at many organizations that seek to gain value from all the information they’re gathering or can access. This drive will no doubt accelerate with ongoing advancements in predictive analytics, artificial intelligence, machine learning, and other related technologies.