DATA SCIENCE

 WHAT IS DATA SCIENCE ?

Data Science is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from structured and unstructured data, and apply knowledge and actionable insights from data across a broad range of application domains. Data science is related to data mining, machine learning and big data. 

Data science is a "concept to unify statistics, data analysis, informatics, and their related methods" in order to "understand and analyze actual phenomena" with data.It uses techniques and theories drawn from many fields within the context of mathematics, statistics, computer science, information science, and domain knowledge. 

What is exploratory data analysis? 

Exploratory data analysis (EDA) is used by data scientists to analyze and investigate data sets and summarize their main characteristics, often employing data visualization methods. It helps determine how best to manipulate data sources to get the answers you need, making it easier for data scientists to discover patterns, spot anomalies, test a hypothesis, or check assumptions. 

  

EDA is primarily used to see what data can reveal beyond the formal modeling or hypothesis testing task and provides a provides a better understanding of data set variables and the relationships between them. It can also help determine if the statistical techniques you are considering for data analysis are appropriate. Originally developed by American mathematician John Tukey in the 1970s, EDA techniques continue to be a widely used method in the data discovery process today. 

  

Why is exploratory data analysis important in data science? 

The main purpose of EDA is to help look at data before making any assumptions. It can help identify obvious errors, as well as better understand patterns within the data, detect outliers or anomalous events, find interesting relations among the variables. 

Data scientists can use exploratory analysis to ensure the results they produce are valid and applicable to any desired business outcomes and goals. EDA also helps stakeholders by confirming they are asking the right questions. EDA can help answer questions about standard deviations, categorical variables, and confidence intervals. Once EDA is complete and insights are drawn, its features can then be used for more sophisticated data analysis or modeling, including machine learning. 

  

Exploratory data analysis tools 

Specific statistical functions and techniques you can perform with EDA tools include: 

  

Clustering and dimension reduction techniques, which help create graphical displays of high-dimensional data containing many variables. 

Univariate visualization of each field in the raw dataset, with summary statistics. 

Bivariate visualizations and summary statistics that allow you to assess the relationship between each variable in the dataset and the target variable you’re looking at. 

Multivariate visualizations, for mapping and understanding interactions between different fields in the data. 

☆K-means Clustering is a clustering method in unsupervised learning where data points are assigned into K groups, i.e. the number of clusters, based on the distance from each group’s centroid. The data points closest to a particular centroid will be clustered under the same category. 

Predictive models, such as linear regression, use statistics and data to predict outcomes. 

Types of exploratory data analysis 

There are four primary types of EDA: 

  

1.} Univariate non-graphical. This is simplest form of data analysis, where the data being analyzed consists of just one variable. Since it’s a single variable, it doesn’t deal with causes or relationships. The main purpose of univariate analysis is to describe the data and find patterns that exist within it. 

2.}Univariate graphical. Non-graphical methods don’t provide a full picture of the data. Graphical methods are therefore required. Common types of univariate graphics include: 

Stem-and-leaf plots, which show all data values and the shape of the distribution. 

Histograms, a bar plot in which each bar represents the frequency (count) or proportion (count/total count) of cases for a range of values. 

Box plots, which graphically depict the five-number summary of minimum, first quartile, median, third quartile, and maximum. 

3.}Multivariate nongraphical: Multivariate data arises from more than one variable. Multivariate non-graphical EDA techniques generally show the relationship between two or more variables of the data through cross-tabulation or statistics. 

Multivariate graphical: Multivariate data uses graphics to display relationships between two or more sets of data. The most used graphic is a grouped bar plot or bar chart with each group representing one level of one of the variables and each bar within a group representing the levels of the other variable. 

Other common types of multivariate graphics include: 

  

Scatter plot, which is used to plot data points on a horizontal and a vertical axis to show how much one variable is affected by another. 

●Multivariate chart, which is a graphical representation of the relationships between factors and a response. 

●Run chart, which is a line graph of data plotted over time. 

●Bubble chart, which is a data visualization that displays multiple circles (bubbles) in a two-dimensional plot. 

●Heat map, which is a graphical representation of data where values are depicted by color. 

Exploratory Data Analysis Tools 

Some of the most common data science tools used to create an EDA include: 

  

■Python: An interpreted, object-oriented programming language with dynamic semantics. Its high-level, built-in data structures, combined with dynamic typing and dynamic binding, make it very attractive for rapid application development, as well as for use as a scripting or glue language to connect existing components together. Python and EDA can be used together to identify missing values in a data set, which is important so you can decide how to handle missing values for machine learning. 

■R: An open-source programming language and free software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing. The R language is widely used among statisticians in data science in developing statistical observations and data analysis. 

 

What is the EDA process in Data science?  

INTRODUCTION 

There are several models that data can be fit into for a thorough analysis. But before you do so, you have to determine which model is an ideal fit for the data at hand. For this reason, you end up exploring the data, its shape, characteristics and come up with a summary that describes the current state of data at hand and whether the data needs further processing before it can be modeled with statistical and scientific techniques. This exploration of data, usually with the help of descriptive statistics, visualization tools, and presentation techniques, make up for what we call Exploratory Data Analysis or EDA in data science. 

EDA in data science is quite like the service advisor doing a rough inspection of your car, asking a few preliminary questions, setting expectations and then taking the car in for service. It is one of the first things done with the data, so it is a critical phase, as many inferences and consequent actions depend on this exploration. 

 

DEFINITION 

The initial analysis of data supplied or extracted, to understand the trends, underlying limitations, quality, patterns, and relationships between various entities within the data set, using descriptive statistics and visualization tools is called Exploratory Data Analysis (EDA). EDA will give you a fair idea of what model better fits the data and whether any data cleansing and massaging might be required before taking the data through advanced modelling techniques or even put through Machine Learning and Artificial Intelligence algorithms. 

WHAT IS EDA?

EDA can be quite extensive and time-consuming depending on what and how much data you have. Unfortunately, there is no structured way to perform EDA, although there are a few techniques that will give you the best results out of EDA. Of the many outcomes of EDA, the important ones that one should try to get from the data are, 

• •Detect outliers and anomalies 

• •Determine the quality of data 

• •Determine what statistical models can fit the data 

• •Find out if the assumptions about the data, that you or your team started out with is correct or way off. 

• •Extract variables or dimensions on which the data can be pivoted. 

• •Determine whether to apply univariate or multivariate analytical techniques. 

EXAMPLES OF EDA IN DATA SCIENCE 

Ok, time for some examples, that might give you an idea about what EDA really entails and what are you looking for, what questions are you trying to answer.  

Example 1: Missing data 

With data comes a lot of anomalies. One of them is missing data. Although the overall data might be good, there are columns within the data set that might be missing values. This can skew your results and not provide an accurate model for further use. 

One great way to identify missing values visually is the use of the missing package in Python. This is obviously for a large data set. This gives you a graphic story of how much data is missing and on which variables. 

Without going into the details of the coding, the above graphic was achieved with a single line of code. The white lines in the above graph indicate missing values. 

Example 2: Summary statistics 

Another example here is of summary statistics that give you a fair idea of your numeric data. 

As shown above, the columns are features of the data set, and the statistics on the left column describe each. 

Example 3: Outliers 

Outliers are data that lie on the extreme or even outside the spectrum of values that a variable should normally hold, thereby giving you a hint or an opportunity to explore. 

You can immediately notice an outlier, where it shows a good percentage of customers are buying more than 50 products. An investigation can be initiated and, in many cases, they turn out to be resellers. This can be seen as an opportunity to develop a B2B relationship with the resellers and grow it as a separate vertical in the business. 

TECHNIQUES OF EDA IN DATA SCIENCE 

There are broadly two categories of EDA, graphical and non-graphical. These two are further divided into univariate and multivariate EDA, based on interdependency of variables in your data. 

Univariate non-graphical: Here, the data features a single variable, and the EDA is done in mostly tabular form, for example, summary statistics. These non-graphical analyses give you a statistic that indicates how skewed your data might be or which is the dominant value for your variable if any. 

Univariate graphical: The EDA here, involves graphic tools like bar charts and histograms to get a quick view of how variable properties are stacked against each other, whether there is a relationship between these properties and whether there is any interdependency among these properties. 

Multivariate non-graphical: Non-graphical methods like crosstabs are used to depict the relationship between two or more variables. Statistical values like correlation coefficient indicate if there are a possible relationship and the measure of correlation.

Multivariate graphical: 

A graphical representation always gives you a better understanding of the relationship, especially among multiple variables. 

TOOLS 

The most commonly used software tools to perform EDA are Python and R. 

Both enjoy massive community support and frequent updates on packages that can be used to EDA. Let’s look at the various graphical instruments that can be used to execute an EDA. 

Box plots 

Box plots are used where there is a need to summarize data on an interval scale like the ones on the stock market, where ticks observed in one whole day may be represented in a single box, highlighting the lowest, highest, median and outliers.   

Heatmap 

Heatmaps are most often used for the representation of the correlation between variables. Here is an example of a heatmap. 

As you can see from the chart, there is a strong correlation between density and residual sugar and absolutely no correlation between alcohol and residual sugar. 

Histograms 

The histogram is the graphical representation of numerical data that splits the data into ranges. The taller the bar, the greater the number of data points falling in that range. A good example here is the height data of a class of students. You would notice that the height data looks like a bell curves for a particular class with most the data lying within a certain range and a few of outside these ranges. There will be outliers too, either very short or very small.  

CONCLUSION 

Exploratory Data Analysis is essential in the analysis of massive data sets, to be able to ensure that you have the right data for the chosen statistical model. You certainly would not want to figure out at a later stage that the data is not a good fit for the statistical model you are trying to build. A sound EDA must be performed before any data mining, data analysis, or data modeling occurs.