CHAPTER 5
SPECIAL TOPICS IN MATRIX ALGEBRA

This chapter extends the algebra matrices to some more advanced topics in linear algebra that have great deal of applications in statistics, econometrics, and solutions of dynamic models that apply difference and differential equations.  The main topics covered in this chapter are rank and linear dependence, solution of system of equations, eigenvalues and eigenvectors, and quadratic forms.  

TOOL BOX
Mathematica Commands			Function
Solve[(({{a, b}, {c, d}})).(({{x}, {y}}))==(({{e}, {f}})), {x, y}]		Solves the system of simultaneous equations for x & y.
Simplify[expr]					Simplifies the "expr" in the argument
Eigenvalues[A]				Gives Eigenvalues of matrix A
Eigenvectors[A]				Gives eigenvectors of A
Eigensystem[A]				Gives eigenvalues and the corresponding eigenvectors of A

I.   Rank and Linear Dependence   
Consider the set of homogenous equations Ax = 0, where A is a  square matrix of mxm coefficients, and x is a vector of mx1 unknowns.  If A is nonsingular, Det[A] ≠ 0, the only solution to Ax = 0 will be a trivial solution of x = 0.  The m columns of A then are said to be linearly independent, the rank of A is m,  r(A) = m, and A is said to be of full rank.  It follows that the m rows of A are also linearly independent.  

Example:  Solve (({{1, 3, -1}, {0, 1, 2}, {4, 1, 2}}))(({{x1}, {x2}, {x3}})) = (({{0}, {0}, {0}}))  for x1, x2 , and x3.
Solution:  Solving the three equations simultaneously results in x1 = x2 = x3 = 0.  This implies that the coefficients matrix (A) is non-singular and the A matrix has a full-rank, r(A) = 3.  
Using Mathematica,         
Clear[A,  x,  x1,  x2,  x3]   
A = (({{1, 3, -1}, {0, 1, 2}, {4, 1, 2}}));		x = (({{x1}, {x2}, {x3}}));
Det[A]
28
Solve[(({{1, 3, -1}, {0, 1, 2}, {4, 1, 2}})). (({{x1}, {x2}, {x3}})) == (({{0}, {0}, {0}})), {x1,  x2,  x3}]
{{x1 -> 0, x2 -> 0, x3 -> 0}}
In this example, Det[A] = 28≠ 0 and the only solution to the system of equations is x1 = x2 = x3 = 0.  The columns (and the rows) of A are linearly independent and A has the full rank of r(A) = 3.   However, if  DetA] = 0, then Ax = 0 would have a nontrivial solution and the m columns (or rows) of A would be linearly dependent and the rank of A would be less than 3, r(A) < 3.

Example:  Given that A = (({{2, 3, 4}, {-1, 0, 1}, {1, 1, 1}})), find the solution of Ax = 0
Solution:  Solving the equations simultaneously results in x1 = x3 and x2 = -2x3.  The determinant of matrix A is zero and A is a singular matrix, with a rank less than 3.  The equations provide a set of solutions for xs not all equal to zero.  For example, for x3 = 1, x1 = 1 and x2 = -2.  Also, any other scalar multiple of these values will be a solution to the equation.  Furthermore, the inspection of the matrix shows that any pair of vectors is linearly independent. Hence, the rank of matrix A is two.  
Using Mathematica,
Clear[A,  x,  x1,  x2,  x3] 
A = (({{2, 3, 4}, {-1, 0, 1}, {1, 1, 1}})); 		x = (({{x1}, {x2}, {x3}}));
    
    Det[ A]

0
Solve[ A.x == 0, {x1,  x2,  x3}]
{{x1 -> x3, x2 -> -2 x3}}
In general, the rank of a matrix is defined as the maximum number of linearly independent columns or rows.

II.  Solution of System of Equations: The Solve[(({{a, b}, {c, d}})).(({{x}, {y}}))==(({{e}, {f}})), {x, y}] Command
If A is a nonsingular matrix, the set of equations Ax = h has the solution x = (A^(-1))h.  This solution yields an easy way of finding the x vector, called Cramer's Rule. Every (Subscript[x, i]) is found as the ratio of two determinants, the numerator of which is the determinant of A matrix in which the (i^th) column is replaced with the h vector, and the denominator is Det[A].  

Example:  Find the solution of the system of equations (({{2, 3, -1}, {1, 2, 1}, {1, -5, 3}}))(({{x1}, {x2}, {x3}})) = (({{-2}, {3}, {16}})).
Solution:  To find (Subscript[x, 1]), replace the first column of A with h and find the determinant of the new matrix.  (Subscript[x, 1 ])is found by dividing this determinant by Det[A].  That is, x1 = ((Det[(({{-2, 3, -1}, {3, 2, 1}, {16, -5, 3}}))])/(DEt[(({{2, 3, -1}, {1, 2, 1}, {1, -5, 3}}))])) = (46/23) = 2.   To find (Subscript[x, 2]), replace the second column of A with h and find the determinant of the new matrix.  (Subscript[x, 2 ])is found by dividing this determinant by Det[A].  That is, x2 = ((Det[(({{2, -2, -1}, {1, 3, 1}, {1, 16, 3}}))])/(Det[(({{2, 3, -1}, {1, 2, 1}, {1, -5, 3}}))])) = ((-23)/23) = -1.  To find (Subscript[x, 3]), replace the third column of A with h and find the determinant of the new matrix.  (Subscript[x, 3 ])is found by dividing this determinant by Det[A].  That is, x3 = ((Det[(({{2, 3, -2}, {1, 2, 3}, {1, -5, 16}}))])/(Det[(({{2, 3, -1}, {1, 2, 1}, {1, -5, 3}}))])) = (69/23) = 3.

Cramer's rule provides a general rule for solving system of equations and is useful in derivation of some economic concepts that deals with the solution of system of equations  (in Chapter 10 we use the rule to derive the Slutsky equation.)  
To solve the same system of equations using Mathematica, use the Solve command as below,
Clear[A, B,  x]
A = (({{2, 3, -1}, {1, 2, 1}, {1, -5, 3}}));    x = (({{x1}, {x2}, {x3}}));        B = (({{-2}, {3}, {16}}));

Solve[A . x == B, {x1,  x2,  x3}] 
{{x1 -> 2, x2 -> -1, x3 -> 3}}
An alternative way of solving the same system of equations is to use the Solve by listing the equations as follows
Solve[{2  x1 + 3  x2 - x3 == -2,  
            x1 + 2  x2 + x3 == 3,  
            x1 - 5  x2 + 3  x3 == 16},  {x1, x2, x3}]
{{x1 -> 2, x2 -> -1, x3 -> 3}}

Example: Aggregate Demand (AD) Aggregate Supply (AS) Model
In a simple AD-AS model of the economy, AD and AS are expressed as functions of price level (P) and interest rate (r):  AD = 100 -.2P -.8r and AS = 18 + .45P + 1.2r.  Solve the system for the equilibrium condition in terms of r and find the equilibrium output and price levels for r = 5% and r = 10%.
Solution:  The system can be written as (({{AD}, {AS}})) = (({{- .2, - .8}, {.45, 1.2}}))(({{P}, {r}})) + (({{100}, {18}})).   Since at the equilibrium point AD = AS, set AD = AS and solve for P in terms of r.  That is, P = 126.15 - 3.08r.  Substitute for P in the AD equation to find AD in terms of r only.  That is, AD = 74.77 - .18r.  Using these two equations, AD and P can be found for r = 5% and r = 10%.  For r = 5%, the equilibrium output (AD) is 74.76 and the equilibrium price level is 126.  For r = 10%, the equilibrium output is 74.75 and the equilibrium price level is 125.85.  
Using Mathematica, 
Clear[AD, AS, p, r]
sol = Solve[(({{AD}, {AD}})) ==(({{- .2, - .8}, {.45, 1.2}})).(({{p}, {r}}))+(({{100}, {18}})),{AD, p}]
{{AD -> 100.` - 
              0.3076923076923077` (1.` (-18.` - 1.2` r) - 
                    1.` (-100.` + 0.8` r)) - 0.8` r, 
          p -> 
            1.5384615384615383` (1.` (-18.` - 1.2` r) - 
                  1.` (-100.` + 0.8` r))}}
Simplify[sol]
{{AD -> 74.76923076923077` - 
              0.18461538461538463` r, 
          p -> 126.15384615384615` - 
              3.0769230769230766` r}}
sol /. r ->  .05
{{AD -> 74.75999999999999`, p -> 125.99999999999997`}}
sol /. r ->  .10
{{AD -> 74.75076923076924`, p -> 125.84615384615383`}}

Do More Practice:
1- Solve the preceding problem for r = 15% and 20%.
2- Make a list of paired observations on r and p for the r values of 5%, 10%, 15%, and 20%.  List plot the relationship between r and p.  Comment on the slope of the plot.  What does it imply on the relationship between interest rate and inflation?
3-  Make a list of paired observations on r and AD for the r values of 5%, 10%, 15%, and 20%.  List plot the relationship between r and AD.  Comment on the slope of the plot.  What does it imply on the relationship between interest rate and output (AD)? 

Problems
1.  Find the rank of the following matrices
               
        A = (({{1, 0, 1}, {0, 2, 3}, {-1, 0, 2}})) 		B = (({{1, 2, 1}, {3, 0, 5}, {2, 10, 0}}))
              
        C =(({{1, 0, 2}, {0, 2, 1}, {2, 2, -1}})) 		D = (({{2, 2, -6}, {0, 1, 3}, {1, 3, 3}}))
               
        E = (({{1, 65, -3}, {-2, -13, 6}, {1, 65, -3}})) 	F = (({{1, 0, 0}, {0, 1, 0}, {0, 0, 1}}))
2.  Solve the following systems of equations
        a)   2x + 3y = 8
             -x + 2y = 3      
             2x + 3y +  z =  8
        b)    x -  y + 2z = -1
              x + 2y - 3z =  5
              x - 2y +  z = 1     
        c)   2x + 3y -  z = 3
              x +  y + 2z = 6
              x +  y + 2z =  5
        d)   2x -  y +  z =  2
              x + 4y + 5z = 11
3.  Demand and supply for a product are given as (Subscript[Q, d]) = 125 - 2P + .03y and (Subscript[Q, s]) = 5 + P, respectively.  (Subscript[Q, d]) is the quantity demanded, which is expressed in terms of the price of the product (P) and income (Y).  (Subscript[Q, s]) is the quantity supplied.  
a) Solve the market model for the equilibrium quantity and price in terms of income.  
b) Find the quantity demanded at the income levels of 500, 1000, 1500, and 2000.  
c)Make a list plot of the relationship between income and quantity for the income levels mentioned in (b). 
d) Comment on the relationship between income and quantity demanded.  
e) Find the income elasticity of demand between income level of 1000 and 1500.  (Note: income elasticity of demand is define as the percent change in quantity demanded over percent change in income, (Subscript[ξ, y]) = ((ΔQ  %)/(ΔY  %))).

III. Eigenvalues and Eigenvectors:  The Eigenvalues[A]  & Eigenvectors [A] Commands
The eigenvalue problem is defined as that of finding values of a scalar k such that Ax = kx, where A is some nxn matrix, k is the eigenvalue of A, and x , (x ≠ 0), is the eigenvector associated with k.  The eigenvalue problem is the basic tool for diagonalizing matrices, finding solutions of differential equations, and solving many other problems in calculus and matrix algebra.  

Equation Ax = kx can be written as (A - kI)x = 0 and has a nontrivial solution if (A - kx) is singular, or Det[A - kI] = 0.  This equation yields a polynomial in the unknown k, which may be solved for k.  The solution of this equation will render the eigenvalue(s) k and the corresponding eigenvector(s) x.  Alternative names of eigenvalues and eigenvectors are characteristic roots and characteristic vectors and latent roots and latent vectors.

Example: Find the eigenvalues and the eigenvectors of A =(({{1, 1}, {4, 1}})).     
Solution:   Equation  (A - kI)x = ((({{1, 1}, {4, 1}})) - (({{k, 0}, {0, k}}))) (({{x1}, {x2}})) =  (({{0}, {0}})) will have nontrivial solution if the determinant of ((({{1, 1}, {4, 1}})) - (({{k, 0}, {0, k}}))) = 0.  This will render a polynomial, (((1  -  k))^2) - 4 = 0, the solution set of which is (Subscript[k, 1]) = -1 and (Subscript[k, 2]) = 3.  To find the eigenvectors corresponding to these two eigenvalues, substitute for the value of  k in the equation  (A - kI)x = ((({{1, 1}, {4, 1}})) - (({{k, 0}, {0, k}}))) (({{x1}, {x2}}))  =  (({{0}, {0}})) and solve for x1 and x2.  For k1 = -1, the equation yields (({{2, 1}, {4, 2}}))(({{x1}, {x2}})) =  (({{0}, {0}})), the solution of which is 2x1 + x2 = 0.  For any arbitrary value of x1 there is a value for x2.  For example, if x1 = -1, then x2 = 2.  Therefore, the eigenvector corresponding to k1 = -1 is x1 = (({{1}, {-2}})), or any scalar multiple of x1.  
  For k2 = 3, the equation (A - kI)x = ((({{1, 1}, {4, 1}})) - (({{k, 0}, {0, k}}))) (({{x1}, {x2}}))  =  (({{0}, {0}})) yields (({{-2, 1}, {4, -2}}))(({{x1}, {x2}})) =  (({{0}, {0}})), the solution of which is 2x1 - x2 = 0. For an arbitrary value of x1, say x1 = 1, x2 = 2.  Therefore, the eigenvector corresponding to k2 = 3 is x2 = (({{1}, {2}})), or any scalar multiple of x2.  Note that, the elements of eigenvector are not independent of each other and one element is always selected arbitrarily.  Therefore, if x satisfies equation (A - kI)x = 0 for given value of k, so does cx,  where c is any constant.
In Mathematica, to find the eigenvalues and eigenvectors use the commands Eigenvalues[A] and Eigenvectors[A] to find the eigenvalues and eigenvectors, respectively, or use the command Eigensystem[A] to find the eigenvalues and Eigenvectors together, with one command.  For the same example above the solution using Mathematica is, 
Clear[A]   
A =(({{1, 1}, {4, 1}})); 
Eigenvalues[A]

{-1, 3}
Eigenvectors[A]
{{-1, 2}, {1, 2}}
Alternatively, you may use the command "Eigensystem" to get the eigenvalues and eigenvectors.
Eigensystem[A]
{{-1, 3}, {{-1, 2}, {1, 2}}}

Example: Find the eigenvalues and the eigenvectors of B = (({{0, 1}, {1, 0}})).
Solution:  Follow the steps in the preceding example by forming (B - kI)x and setting Det[B - kI] = 0.  This will yield ((({{0, 1}, {1, 0}})) - (({{k, 0}, {0, k}}))) (({{x1}, {x2}}))  = (({{0}, {0}})) or  (k^2) - 1 = 0, the solution of which is (Subscript[k, 1]) = 1 and (Subscript[k, 2]) = -1.   These are the eigenvalues of B.  To find the eigenvectors corresponding to (Subscript[k, 1]) and (Subscript[k, 2]), form (B - kI)x = 0.  For (Subscript[k, 1]) = 1, this yields (({{-1, 1}, {1, -1}}))(({{x1}, {x2}})) = (({{0}, {0}})) or x1 = x2.  For an arbitrary value of x1 = 1, x2 = 1.  Therefore, the eigenvector corresponding to (Subscript[k, 1]) = 1 is (({{1}, {1}})), or any constant c multiple of this vector, c(({{1}, {1}})).  For (Subscript[k, 2]) = -1, (B - kI)x = 0 yields (({{1, 1}, {1, 1}}))(({{x1}, {x2}})) = (({{0}, {0}})) or x1 = -x2.  For an arbitrary value of x1 = 1, x2 = -1.  Therefore, the eigenvector corresponding to (Subscript[k, 2]) = -1 is (({{1}, {-1}})), or any constant d multiple of this vector, d(({{1}, {-1}})).  The eigensystem of the B matrix is {{1, -1}, {{1, 1}, {1, -1}}, where the first bracket ({1, -1}) represents eigenvalues and the remaining two brackets are the corresponding eigenvectors.  
Using Mathematica, the eigensystem is,
Clear[B]     
B = (({{0, 1}, {1, 0}}));
Eigensystem[B]   
{{-1, 1}, {{-1, 1}, {1, 1}}}

Symmetric Matrices and Orthogonal Eigenvectors:  If the matrix A is a symmetric matrix, the corresponding eigenvectors will be orthogonal.  Two vectors, x and y are said to be orthogonal if their dot product is zero, x.y = y.x = 0, and x.x ≠  0, y.y ≠  0.  

Example:  Show that the eigenvectors of the symmetric matrix A = (({{1, 1}, {1, -1}})) are orthogonal.
Solution:  Following the steps of the previous examples, the eigenvalues of the matrix are found to be (Sqrt[2]) and -(Sqrt[2]).  The corresponding eigenvectors are x1 = (({{1 - (Sqrt[2])}, {1}})), and x2 = (({{1 + (Sqrt[2])}, {1}})), respectively.  Since x1'.x2 = x2'x1 = 0, x1'x1 = 4 - 2(Sqrt[2]), and 4 + 2(Sqrt[2]), the two eigenvectors are orthogonal.
Using Mathematica, the problem can be solved as,
Eigenvectors[ (({{1, 1}, {1, -1}}))]
{{1 - (Sqrt[2]), 1}, {1 + (Sqrt[2]), 1}}
The matrix has two eigenvectors, x =  {1 - (Sqrt[2]), 1} and y =  {1 + (Sqrt[2]),  1}.   The dot product of the two vectors is 
Clear[x1,  x2]
x1  =  {1 - Sqrt[2],  1};    x2  =  {1 + Sqrt[2],  
            1};
N[Dot[x1, x2]]
N[Dot[x1,  x1]]
N[Dot[x2, x2]]
-2.220446049250313`*^-16
1.17157287525381`
6.82842712474619`
Sine x1'.x2 = 0, x1'x1 ≠ 0, and x2'.x2 ≠ 0, x1 and x2 are orthogonal.

Example:  Find the eigenvalues and the eigenvectors of the symmetric matrix A = (({{1, 2}, {2, 4}})) and verify that the eigenvectors are orthogonal.                                   
 Solution:   Form the eigenvalue equation (A - kI)x = ((({{1, 2}, {2, 4}})) - (({{k, 0}, {0, k}}))) (({{x1}, {x2}})) =  (({{0}, {0}})) and solve for ks.  This equation will have nontrivial solution if the determinant of ((({{1, 2}, {2, 4}})) - (({{k, 0}, {0, k}}))) = 0.  The determinant renders the polynomial k(k - 5) = 0, the solution set of which is (Subscript[k, 1]) = 0 and (Subscript[k, 2]) = 5.  To find the eigenvectors corresponding to these two eigenvalues, substitute for the value of  ks in the equation (A - kI)x = ((({{1, 2}, {2, 4}})) - (({{k, 0}, {0, k}}))) (({{x1}, {x2}}))  =  (({{0}, {0}})) and solve for x1 and x2.  For k1 = 0, the equation yields (({{1, 2}, {2, 4}}))(({{x1}, {x2}})) =  (({{0}, {0}})), the solution of which is x1 + 2x2 = 0.  For an arbitrary value of x2 = 1, x1 = -2.  Therefore, the eigenvector corresponding to k1 = 0 is x1 = (({{-2}, {1}})), or any scalar multiple of x1.   For k2 = 5, the equation (A - kI)x = ((({{1, 2}, {2, 4}})) - (({{5, 0}, {0, 5}}))) (({{x1}, {x2}}))  =  (({{0}, {0}})) yields (({{-4, 2}, {2, -1}}))(({{x1}, {x2}})) =  (({{0}, {0}})), the solution of which is 2x1 - x2 = 0.  For an arbitrary value of x1, say x1 = 1, x2 = 2.  Therefore, the eigenvector corresponding to k2 = 5 is x2 = (({{1}, {2}})), or any scalar multiple of x2.  
To test for the orthogonality of x1 and x2, fine x1'.x2, x1'.x1, and x2'.x2.  Since x1'.x2 = x2'.x1 = 0 and x1'.x1 = x2'.x2 = 5 ≠ 0, the two eigenvectors are orthogonal to each other.  
To solve the same problem using Mathematica,
Clear[A,  x1,  x2]   
A = (({{1, 2}, {2, 4}}));
         
Eigensystem[(({{1, 2}, {2, 4}}))]
{{0, 5}, {{-2, 1}, {1, 2}}}
The matrix has two eigenvectors x1 = (({{-2}, {1}})) and x2= (({{1}, {2}})).   The two vectors x1 and x2 are orthogonal if x1'.x2 = x2'.x1 = 0,  x1'.x1 ≠ 0, and x2'.x2 ≠  0.  The following operations show that the two vectors are orthogonal.
 x1 = {-2, 1};   x2  =  {1,  2};
Dot[x1,  x2]
Dot[x2,  x1]
Dot[x1,  x1]
Dot[x2,  x2]
0
0
5
5

Normalized Eigenvectors :  We may normalize eigenvectors of a matrix by setting their length at unity. The length of a vector x = {x1,  x2} is L = (Sqrt[(x1^2)  +  (x2^(2 ))]).  To normalize the vector x, set L = 1 and solve for x1 and x2.  

Example:  Normalize the eigenvectors of the matrix A = (({{1, 2}, {2, 4}})) of the preceding example.
Solution:  The two eigenvectors of the A matrix are x1= {-2, 1} and x2= {1, 2}.  Let us denote the elements of x1 as x1 and x2. To normalize x1, set its length to unity.  That is, x12 + x22 = 1.  But, x1 and x2 are not independent of each other and the relation x1 = -2x2 always holds between the two (this is the relation that was found in solving the equation of eigenvalues for eigenvectors).  Therefore, we can find x1 and (Subscript[x, 2]) by solving  the two equations x1 = -2x2 and 2((Subscript[x, 2])^2) + ((Subscript[x, 2])^2) = 1 for (Subscript[x, 1]) and (Subscript[x, 2]).  This gives two solutions for (Subscript[x, 2]) which are  (Subscript[x, 2]) = (1/(Sqrt[5])) and ((-1)/(Sqrt[5])).  The corresponding solutions for (Subscript[x, 1]) are (Subscript[x, 1])= ((-2)/(Sqrt[5])) and (2/(Sqrt[5])), respectively.  Therefore, the The two sets of normalized vectors corresponding to x1 = {-2, 1} are {((-2)/(Sqrt[5])), (1/(Sqrt[5]))} and {(2/(Sqrt[5])), ((-1)/(Sqrt[5]))}. 
The same procedure follows for the x2 =  {1,  2}.  Denoting the elements of the vector x2 by x1 and x2, set its length to unity, (x1^2) + (x2^2) = 1.  Also, x1 and x2 are not independent of each other and that x2 = 2x1 always holds between the two elements of the vector.  Therefore, solve the two equations together to find x1 and x2.  The two set of normalized vectors corresponding to x2 =  {1,  2} are {((-1)/(Sqrt[5])), ((-2)/(Sqrt[5]))} and {(1/(Sqrt[5])), (2/(Sqrt[5]))}.  
Mathematica my help with solving the system equations for finding the normalized vectors.  Use the following commands to solve the equations.  To find the normalized vectors corresponding to x1 = {-2, 1},
Clear[x1,  x2]
 Solve[{Sqrt[x1^2  +  x2^2 ] ==  1,  
            x1  ==  -2 x2},  {x1,  x2}]  
{{x1 -> -(2/(Sqrt[5])), x2 -> (1/(Sqrt[5]))}, {x1 -> (2/(Sqrt[5])), 
          x2 -> -(1/(Sqrt[5]))}}
To find the normalized vectors corresponding to x2= {1, 2},
Clear[x1,  x2]
Solve[{Sqrt[x1^2  +  x2^2 ] == 1,  
              x2  == 2  x1},  {x1,  x2}]

{{x1 -> -(1/(Sqrt[5])), 
          x2 -> -(2/(Sqrt[5]))}, {x1 -> (1/(Sqrt[5])), 
          x2 -> (2/(Sqrt[5]))}}
Note:  Normalized vectors of a symmetric matrix have the property that they are orthonormal.  Two vectors x and y are orthonormal if x'.y = y'.x = 0 and x'.x = y'.y = 1

Example: Verify that the normalized vectors of the preceding example are orthonormal.
Solution: Take one of the normalized vectors corresponding to x1, say {((-2)/(Sqrt[5])), (1/(Sqrt[5]))} and call it xn.  Take one of the normalized vectors corresponding to x2, say {((-1)/(Sqrt[5])), ((-2)/(Sqrt[5]))}, and call it yn.   If xn and yn are orthonormal, the following relations should hold between the two xn'.yn = yn'.xn = 0 and xn'.xn = yn'.yn = 1.  The dot products of the vectors confirm that these relations in fact hold between xn and yn.  
These relationships can also be checked using Mathematica as follows.  
x  =  {-2/Sqrt[5],  
            1/Sqrt[5]};      y  =  {-1/Sqrt[5],  -2/
              Sqrt[5]};
N[Dot[x,  y]]
N[Dot[y, x]]
N[Dot[x, x]]
N[Dot[y,  y]]
0.`
0.`
1.`
1.`
The same can be proved for the second set of normalized vectors {(2/(Sqrt[5])), ((-1)/(Sqrt[5]))} and {(1/(Sqrt[5])), (2/(Sqrt[5]))}.  

Diagonalizing Symmetric Matrices:  In general a symmetric matrix A of order nxn has n eigenvalues, possibly not all distinct. Corresponding to these roots are a set of n orthonormal eigenvectors. A matrix X, formed by using these orthonormal eigenvectors as its columns, will have the property that X'.X = I. The product X'.A.X produces a diagonal matrix with eigenvalues of A displayed on its principal diagonal.  Therefore, any symmetric matrix A can be diagonalized by pre and post multiplying it by a matrix with columns that are normalized eigenvectors of matrix A.           

Example: Find the matrix X that will diagonalize A = (({{1, 4}, {4, 1}})).  Find the diagonalized matrix.
Solution:  The matrix has two distinct eigenvalues -3 and 5.  Corresponding to these eigenvalues are two eigenvectors x = {-1, 1} and y = {1, 1}, respectively.  To normalize the eigenvectors, set the length of each vector to unity.  For x = {-1, 1} this means that the sum square of the elements is equal to one, x21 + x22 = 1.  Also, since x1 = -x2 always hold between x1 and z2.  To find x1 and x2, solve the two equations together.  The two set of solutions for x are {((-1)/(Sqrt[2])), (1/(Sqrt[2]))} and {(1/(Sqrt[2])), ((-1)/(Sqrt[2]))}.
To find the solutions for y, follow the same procedure and solve for y1 and y2 using  y21 + y22 = 1 and  y1 =  y2.  The two set of solutions for y are {((-1)/(Sqrt[2])), ((-1)/(Sqrt[2]))} and {(1/(Sqrt[2])), (1/(Sqrt[2]))}.  The orthonormal matrices formed by the normalized eigenvectors are,  (({{((-1)/(Sqrt[2])), ((-1)/(Sqrt[2]))}, {(1/(Sqrt[2])), ((-1)/(Sqrt[2]))}}))   and  (({{(1/(Sqrt[2])), (1/(Sqrt[2]))}, {((-1)/(Sqrt[2])), (1/(Sqrt[2]))}})).  These are the X matrices, each with the properties that X'.X = I and X'.A.X is a diagonal matrix with the eigenvalues in the principal diagonal and zeroes everywhere else, (({{-3, 0}, {0, 5}})).  These can be confirmed by multiplying the relevant matrices as in the equations X'.X = I and X'.A.X.
Mathematica can be use to facilitate the calculation involved as follows:  To find the eigenvalues and the eigenvectors, use the command "EigenSystems" as follows,
Eigensystem[(({{1, 4}, {4, 1}}))]
{{-3, 5}, {{-1, 1}, {1, 1}}}
To solve the system of equations for finding the normalized eigenvectors for x use the following commands,
Clear[x1,  x2]
Solve[{x1^2  +  x2^2  == 1,  x1  == -x2},  {x1,  
            x2}]
{{x1 -> -(1/(Sqrt[2])), x2 -> (1/(Sqrt[2]))}, {x1 -> (1/(Sqrt[2])), 
          x2 -> -(1/(Sqrt[2]))}}
the two set of solutions for x are {((-1)/(Sqrt[2])), (1/(Sqrt[2]))} and {(1/(Sqrt[2])), ((-1)/(Sqrt[2]))}.  
To solve the system of equations for finding the normalized eigenvectors of y, use the following commands,
Clear[v1,  v2]
Solve[{y1^2  +  y2^2  ==  1,  y1  == y2},  {y1,  
            y2}]
{{y1 -> -(1/(Sqrt[2])), 
          y2 -> -(1/(Sqrt[2]))}, {y1 -> (1/(Sqrt[2])), 
          y2 -> (1/(Sqrt[2]))}}
The two set of solutions for y are {((-1)/(Sqrt[2])), ((-1)/(Sqrt[2]))} and {(1/(Sqrt[2])), (1/(Sqrt[2]))}.   To test the properties X'.X = I and X'.A.X is a diagonal matrix with the eigenvalues in the principal diagonal and zeroes everywhere else use the following commands.  
MatrixForm[Transpose[(({{-(1/(Sqrt[2])), -(1/(Sqrt[2]))}, {(1/(Sqrt[2])), -(1/(Sqrt[2]))}}))].(({{-(1/(Sqrt[2])), -(1/(Sqrt[2]))}, {(1/(Sqrt[2])), -(1/(Sqrt[2]))}}))]
                   
(({{1, 0}, {0, 1}}))
diag = Transpose[(({{-(1/(Sqrt[2])), -(1/(Sqrt[2]))}, {(1/(Sqrt[2])), -(1/(Sqrt[2]))}}))].(({{1, 4}, {4, 1}})) .(({{-(1/(Sqrt[2])), -(1/(Sqrt[2]))}, {(1/(Sqrt[2])), -(1/(Sqrt[2]))}}));
MatrixForm[N[diag]]

(({{-3.`, 0.`}, {0.`, 5.000000000000001`}}))

Problems
1  Find the eigenvalues and the eigenvectors of the following matrices                           
       a)  A = (({{1, 1}, {1, 3}}))   			b)   A = (({{1, 3}, {3, 1}}))                                                                                    

                 
       c)  A = (({{0, 0, 1}, {0, 0, 1}, {1, 1, 0}}))    		d) A  = (({{2, 1, 0}, {0, 1, 0}, {1, 0, 1}}))
                
2.  Show that if A is a singular matrix,  k = 0  is an eigenvalue, and conversely. 
3.   Show that if  k  is an eigenvalue of A, 1/k  is an eigenvalue of A-1.
4.   Diagonalize the following matrices
       a) A = (({{2, 1}, {2, 2}}))   		b)  A = (({{9, 3}, {3, 1}}))      		c)  A = (({{3, 0, 3}, {0, 3, 4}, {3, 4, 3}}))           
                                                             
IV. Quadratic Forms and Positive Definite Matrices 
As well as their application in statistics, quadratic forms and tests of positive definiteness of matrices are frequently used in deciding the critical points (maxima, minima, inflection points) of multi-variate functions.  If x denotes a nx1 vector and A is a nxn symmetric matrix, then x'Ax is a scalar and is called a quadratic form. For example, if  x = (({{x1}, {x2}})) and  A = (({{a11, a12}, {a21, a22}})) hen, x'Ax = a11(x1^2) + (a12 + a21)x1x2 + a22(x2^2).

Clear[x1,  x2]
Transpose[(({{x1}, {x2}}))].(({{a11, a12}, {a21, a22}})).(({{x1}, {x2}}))
{{x1 (a11 x1 + a21 x2) + x2 (a12 x1 + a22 x2)}}

The restriction that matrix A be symmetric is not a serious one, because any non-symmetric matrix can be transformed to symmetric one by averaging the off-diagonal elements.  For example, the quadratic forms of the non-symmetric matrix A = (({{1, 3}, {5, 2}})) and the symmetric matrix A* = (({{1, 4}, {4, 2}})) are the same.  However, A* is derived from A by averaging the off-diagonal elements.  To test this, let's assume that x = (({{1}, {1}})) and find x'Ax = x'A*x = 11.
Using Mathematica,
Transpose[(({{1}, {1}}))].(({{1, 3}, {5, 2}})).(({{1}, {1}}))
	
{{11}}
Transpose[(({{1}, {1}}))].(({{1, 4}, {4, 2}})).(({{1}, {1}}))

{{11}}
both products result in the same value, 11.  

A quadratic form is said to be positive definite and the matrix A is said to be positive definite if and only if X'AX > 0 for all X ≠  0. A quadratic form and the associated  matrix are said to be positive semi-definite if X'AX ≥ 0 for all X ≠  0.

Another way of testing for positive definiteness of quadratic forms is by finding the eigenvalues of their associated matrices.  A matrix A is said to be positive definite if and only if all its eigenvalues are positive.  It can also be proved that a  positive definite matrix has a determinant that is positive.

Example:  Show that the quadratic form x2 + 2xy + 2y2 is positive definite.
Solution: The corresponding matrix of the coefficients is (({{1, 0}, {2, 2}})).  The eigenvalues of this matrix are 1 and 2.  Since both eigenvalues are positive, the quadratic form and its associated matrix are positive definite.
Using Mathematica,
Eigenvalues[(({{1, 0}, {2, 2}}))]
 
{1, 2}
Note that, to form the matrix associated with the quadratic form ax2 + bxy + cy2 the coefficients of the squared variables (x2 and y2) form the diagonal elements and the coefficients of the cross product (xy) form the off-diagonal elements.  

Example:  Test for the positive definiteness of the quadratic form (x^2)+ 3(y^2)- (z^2)+ 5xy - 2xz + 5yz.
Solution:  Write the quadratic form in a matrix product form and find the eigenvalues of the corresponding matrix.  That is,  
(x^2)+ 3(y^2)+ (z^2)+ 5xy - 2xz + 7yz = (({{x, y, z}}))(({{1, 5, -2}, {0, 3, 7}, {0, 0, 1}}))(({{x}, {y}, {z}})).  The eigenvalues of the associated matrix is 1, 1, and 3.  Since all the eigenvalues are positive, the matrix and its corresponding quadratic form are positive definite.
To find the eigenvalues of the matrix, you can use Mathematica as,
Eigenvalues[(({{1, 5, -2}, {0, 3, 7}, {0, 0, 1}}))]
{1, 1, 3}

Problems
1.  Write the corresponding coefficient matrices of the following quadratic forms and transform the matrices into symmetric matrices
         a)  x2 + 3xy + y2      
         b) 3x2 + 2xy + y2
         c) 3x2 + 2zy + y2 + 2z2
  2.   Show that the following quadratic forms are positive definite:
        a)  3x2 + 2xy + 3y2
        b)  2x2 +  xy + 3y2
        c)   x2 + 4xy +2y2 + 3z2