Covariance and Correlation: Formula, Examples, X^T*X and X*X^T

Covariance vs Correlation: Difference, Formula, Examples and Comparison

Introduction

Covariance and correlation are two important statistical concepts used to measure the relationship between two variables.

They help answer questions such as:

• Does advertising increase sales?
• Does study time affect exam marks?
• Are height and weight related?

Although both measure relationships between variables, covariance and correlation have important differences in interpretation and usage.

What is Covariance?

Covariance measures the direction in which two variables change together.

$$ Cov(X,Y)= \frac{\sum(X_i-\bar X)(Y_i-\bar Y)} {n} $$

Interpretation of Covariance

Positive Covariance

When:

$$ Cov(X,Y)>0 $$

Both variables increase or decrease together.

Negative Covariance

$$ Cov(X,Y)<0 div="">

One variable increases while the other decreases.

Zero Covariance

$$ Cov(X,Y)=0 $$

There is no linear relationship between variables.

What is Correlation?

Correlation measures both the direction and strength of the relationship between two variables.

$$ r= \frac{Cov(X,Y)} {\sigma_X\sigma_Y} $$

Correlation always lies between:

$$ -1\leq r\leq1 $$

Correlation Value	Meaning
+1	Perfect positive relationship
0	No linear relationship
-1	Perfect negative relationship

Covariance vs Correlation: Main Differences

Feature	Covariance	Correlation
Purpose	Shows direction	Shows direction and strength
Range	-∞ to +∞	-1 to +1
Units	Has units	No units
Scale Effect	Affected by scale	Not affected by scale
Standardized	No	Yes

Numerical Example: Covariance and Correlation

Consider:

$$ X=\{1,2,3,4,5\} $$ $$ Y=\{50,55,65,70,80\} $$

Step 1: Calculate Mean

$$ \bar X=3 $$ $$ \bar Y=64 $$

X	Y	X-X̄	Y-Ȳ	Product
1	50	-2	-14	28
2	55	-1	-9	9
3	65	0	1	0
4	70	1	6	6
5	80	2	16	32

Covariance

$$ Cov(X,Y)=\frac{75}{5} $$ $$ \boxed{Cov(X,Y)=15} $$

Correlation

$$ r= \frac{15} {\sqrt2 \times \sqrt{114}} $$ $$ \boxed{r\approx0.99} $$

The result shows a very strong positive relationship between study hours and exam marks.

Real-Life Applications

Applications of Covariance

• Stock market return analysis
• Economic studies
• Weather and energy consumption analysis

Applications of Correlation

• Machine learning feature selection
• Medical research
• Business analytics
• Predictive modelling

Final Conclusion

Covariance and correlation are related statistical tools but they provide different information.

• Covariance tells whether two variables move together.
• Correlation tells how strongly two variables are related.
• Covariance depends on measurement units.
• Correlation is standardized and always ranges between -1 and +1.

$$ Correlation= \frac{Covariance} {Standard\ Deviation_X \times Standard\ Deviation_Y} $$

In practical data analysis, correlation is usually preferred because it allows comparison between different datasets.

Understanding XXᵀ and XᵀX in Correlation and Covariance

Many statistics and machine learning books use expressions like XXᵀ and XᵀX when explaining covariance, correlation, and data matrices.

These expressions can look confusing at first, but they are simply matrix multiplications used to calculate relationships between data points and variables.

1. What Does XXᵀ Mean?

Consider a row vector:

$$ X= \begin{bmatrix} 1&2&3 \end{bmatrix} $$

The transpose of X is:

$$ X^T= \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} $$

Now multiply:

$$ XX^T= \begin{bmatrix} 1&2&3 \end{bmatrix} \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} $$

This becomes:

$$ XX^T=1^2+2^2+3^2 $$ $$ =1+4+9 $$ $$ \boxed{XX^T=14} $$

This is the dot product of the vector with itself.

2. What Does XᵀX Mean?

Now consider X as a column vector:

$$ X= \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} $$

Its transpose is:

$$ X^T= \begin{bmatrix} 1&2&3 \end{bmatrix} $$

Multiplication gives:

$$ X^TX= \begin{bmatrix} 1&2&3 \end{bmatrix} \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} $$ $$ =1^2+2^2+3^2 $$ $$ \boxed{X^TX=14} $$

Important: For a single vector, $$ XX^T=X^TX $$ The difference appears when X is a matrix containing multiple variables.

3. How XᵀX is Used in Correlation

Correlation can be calculated using:

$$ r= \frac{X^TY} {\sqrt{X^TX}\sqrt{Y^TY}} $$

The data must first be centered by subtracting the mean.

Numerical Example

Consider two variables:

$$ X= \begin{bmatrix} 1\\ 2\\ 3 \end{bmatrix} $$ $$ Y= \begin{bmatrix} 2\\ 4\\ 6 \end{bmatrix} $$

Step 1: Calculate XᵀY

$$ X^TY= \begin{bmatrix} 1&2&3 \end{bmatrix} \begin{bmatrix} 2\\ 4\\ 6 \end{bmatrix} $$ $$ =1(2)+2(4)+3(6) $$ $$ =2+8+18 $$ $$ \boxed{X^TY=28} $$

Step 2: Calculate XᵀX

$$ X^TX= 1^2+2^2+3^2 $$ $$ =1+4+9 $$ $$ \boxed{X^TX=14} $$

Step 3: Calculate YᵀY

$$ Y^TY= 2^2+4^2+6^2 $$ $$ =4+16+36 $$ $$ \boxed{Y^TY=56} $$

Step 4: Calculate Correlation

$$ r= \frac{28} {\sqrt{14}\sqrt{56}} $$ $$ r=1 $$

Result: $$ \boxed{r=1} $$ This indicates a perfect positive correlation between X and Y.

4. Why Do Books Use XXᵀ?

In machine learning, data is often stored as a matrix.

Example:

$$ X= \begin{bmatrix} x_1&y_1\\ x_2&y_2\\ x_3&y_3 \end{bmatrix} $$

Here each row represents an observation.

When we calculate:

$$ X^TX $$

we get:

$$ \begin{bmatrix} \sum x^2&\sum xy\\ \sum xy&\sum y^2 \end{bmatrix} $$

The term:

$$ \sum xy $$

is the important part used in covariance and correlation.

5. Simple Memory Trick

Expression	Meaning
XᵀX	Relationship between variables/features
XXᵀ	Relationship between observations/samples
Covariance	Uses centered data products
Correlation	Normalized covariance

• XXᵀ and XᵀX are matrix operations.
• They are used as building blocks for covariance calculations.
• Correlation is obtained by normalizing covariance.
• The difference between XXᵀ and XᵀX depends on whether samples are stored as rows or columns.

$$ Correlation= \frac{Covariance} {Standard\ Deviation_X \times Standard\ Deviation_Y} $$