Proof: For the proof, see [5]. Proof: [5]. D takingB A in 0 The following theorem gives some additional useful properties properties of pseudoinverses. Use Theorem 4. This page intentionally intentionally left left blank blank This page Chapter Chapter 5 5 Introduction to Introduction to the the Singular Singular Value Decomposition Value Decomposition In this this chapter chapter we we give give aa brief brief introduction introduction to to the the singular value decomposition decomposition SVD.
We We In singular value show that matrix has an SVD SVD and and describe describe some show that every every matrix has an some useful useful properties properties and and applications applications of this this important important matrix matrix factorization. The The SVD plays aa key key conceptual and computational of SVD plays conceptual and computational role throughout throughout numerical and its applications.
Note: The rest rest of the proof proof follows [24, Ch. Denote the the set as an exercise. AV2 0. Then from 5. D 0 Definition Definition 5. From the proof of Theorem 5. The columns of right singular of V are called called the right singular vectors vectors of of A and are the orthonormal orthonormal eigenvectors of of AT A1A. Remark 5. Note that V Remark 5. U and V can be be interpreted interpreted as changes changes of basis in both the domain domain and co-domain co-domain spaces spaces with respect to has aa diagonal diagonal matrix matrix representation.
Remark Remark decomposition is not unique. For example, an examination of the proof proof of Theorem Theorem 5. Computing T AA T is numerically poor in finite-precision arithmetic. Better algorithms exist that work AA directly on A via a sequence of orthogonal orthogonal transformations; transformations; see, e. Example 5. Let A R nxn " be symmetric symmetric and positive definite. Let V V be an orthogonal orthogonal matrix of eigenvectors A, i.
SVD of A. Introduction to the Singular Value Value Decomposition 5. Using Theorem 5. The singular vectors vectors satisfy satisfy the the relations relations 3. Part Part 4 4 of theorem provides provides aa numerically numerically superior superior method method for Remark 5.
Note Note that that each each subspace on, for example, reduction reduction to row or on, for example, to row or column subspace requires requires knowledge of the The relationship subspaces is is summarized summarized knowledge of the rank rank r.
The relationship to to the the four four fundamental fundamental subspaces nicely in Figure 5. The the dyadic decomposition 5. Then Then TheoremS. Some Basic Properties 5. Figure 5. Furthermore, Furthermore, ifif we be as defined then be as defined in in Theorem Theorem 5. However, However, aa simple simple if we we insist that the the singular values be be ordered from largest largest to to smallest.
Introduction to to the Singular Value Decomposition Chapter 5. A "full be similarly similarly constructed. Recall Recall the the linear linear transformation transformation T used in in the the proof proof of of Theorem Theorem 3. From 3. In rank. In other U is A by by row row transformations. Both Both compressions compressions are are analogous analogous to to the the so-called where so-called row-reduced row-reduced echelon form form which, which, when when derived by aa Gaussian Gaussian elimination elimination algorithm implemented in in echelon derived by algorithm implemented finite-precision arithmetic, arithmetic, is is not not generally generally as as reliable reliable aa procedure.
In column V is an orthogonal orthogonal transformation A by by column I;olumn transformations. Such compression is is analogous to the the that "compresses" "compresses" A Such aa compression analogous to Exercises Exercises 41 41 so-called column-reduced column-reduced echelon echelon form, form, which which is not generally generally aa reliable reliable procedure procedure when when so-called is not performed by by Gauss transformations in in finite-precision For details, see, for for performed Gauss transformations finite-precision arithmetic.
For details, see, example, [7], [7], [11], [11],[23], [23],[25]. Let X E IRmxn. Prove Prove Theorem Theorem 5. Determine an SVD of A. Let A be symmetric but indefinite. Determine SVDs the matrices matrices 5. Determine SVDs of of the 5. Do A and have the the same they have have the the same same rank? This page intentionally intentionally left left blank blank This page Chapter 6 6 Chapter Linear Equations Equations Linear In this this chapter we examine uniqueness of In chapter we examine existence existence and and uniqueness of solutions solutions of of systems systems of of linear linear equations.
General General linear linear systems systems of of the form 6. Theorem 6. There There exists exists aa solution to 6. There exists a solution to 6. There There exists exists aa unique unique solution to to 6.
There There exists exists at most one one solution solution to to 6. A are are linearly linearly independent, independent, i. Linear Equations Proof: The The proofs proofs are are straightforward straightforward and and can can be be consulted consulted in in standard standard texts texts on on linear Proof: linear algebra. Note Note that that some parts of of the the theorem theorem follow follow directly directly from from others.
For example, to to algebra. Therefore, we we prove 6, note always aa solution solution to system. Therefore, must have have the the case case of of aa nonunique nonunique solution, A is not I-I, which implies implies rank A rank A 6. Note Note that that the the results results of of solutions system 6.
Proof: follows essentially range Proof: The The subspace subspace inclusion inclusion criterion criterion follows essentially from from the the definition definition of of the the range of aa matrix.
The The matrix matrix criterion Theorem 4. Furthermore, all solutions of are of this form. Proof: To To verify verify that that 6. That all solutions arc of this seen as follows. Let Let Z arbitrary solution That all solutions are of this form form can can be be seen as follows. Z be be an an arbitrary solution of of 6. AZ AZ B. Then Then we we can can write write 6. Remark 6. Remark Remark 6. A of the the matrix linear equation equation Theorem 6. Proof: The first equivalence is immediate from Theorem 6.
Suppose A E"x". Find all solutions of the homogeneous system Ax — 0, 0. R" is arbitrary. Example 6. Consider Consider the system of linear first-order difference equations Example 6,8 46 46 Equations Chapter 6. The The general general solution solution of of 6. In linear linear system system theory, theory, this this is is called called controllability. Again from Theorem Theorem 6. The standard conditions conditions with analogues for continuous-time models The above are standard with analogues for continuous-time models i.
There There are are many many other other algebraically algebraically equivalent equivalent conditions.
Example We now now introduce Example 6. We introduce an an output output vector vector Yk yk to to the the system system 6. We We can can then then pose some new new questions questions about about the overall system that are are dual to reachability reachability and and controllability. The The answers answers are are cast cast in in terms terms that that are are dual dual in in the the linear linear algebra algebra sense sense as as well.
As aa dual we have have the of suffice to determine uniquely uniquely Jt dual to to controllability, controllability, we the notion notion of 0? The fundamental fundamental duality duality result from linear linear system system theory is the uniquely nl The A.
A the notion A compact compact matrix matrix criterion criterion for for uniqueness uniqueness of of solutions solutions to to 6. Such a criterion CC Example 7. T Definition 7. Inner Inner product Product Spaces Spaces 55 55 It is easy easy to to check check that, that, with with this this more more "abstract" of transpose, transpose, and It is "abstract" definition definition of and if if the the T i, y th j th element element of of A A is is a aij, then the the i, i, y th j th element element of of A AT is ap.
However, However, the definition above allows us us to to extend the concept concept of of transpose transpose to to the the case case of of weighted weighted inner inner definition above allows extend the mxn products in the following way. Q and. Note, too, too, from from part part 22 of of the the definition, definition, that that x, must be be real real for for all all x.
Note, x , xx must x. Remark 7. Note from parts 22 and and 3 3 of of Definition Definition 7. The Euclidean Euclidean inner inner product product of x, y E is given given by by Remark 7. V, IF F endowed is called Definition 7. If inner product product space. Example T 1. Note other choices choices are since by of the function, space.
Again, other choices choices are possible. Again, other are possible. Definition V be inner product V, we or Definition 7. Let Let V be an an inner product space. This This is - , -. The is called norm. In In In case called the the next next section, section, other on these spaces are are defined. Vector Vector Norms Norms 57 57 Theorem 7. For x, x, yy E product is 1. For For x, x, yy eE C", en, an an inner inner product product is by 2. IR is Definition 7. This seen readily from the illus This is is called called the the triangle triangle inequality, inequality, as as seen readily from the usual usual diagram diagram illus two vectors vectors in in ]R2.
It the remainder this section to state for complexRemark It is is convenient convenient in in the remainder of of this section to state results results for complexvalued vectors. The specialization specialization to the real real case case is is obvious. The to the A vector said to Definition 7.
Definition 7. A vector space space V, V, IF F is is said to be be aa normed normed linear linear space space if if and and only only ifif there exists exists aa vector vector norm norm II. Example Example 7. HOlder norms, p-norms, are by 1. Product Spaces, Norms 2. Some weighted weighted p-norms: p-norms: 2.
Let Let x, x, yy E Fhcorem 7. A particular particular case the Holder HOlder inequality A case of of the inequality is is of of special special interest. Theorem 7. Let C". Then Theorem 7. Then with equality are linearly dependent.
Since Since is definite matrix, matrix, its must be be nonnegative. Matrix Matrix Norms Norms 7. Similar Similar remarks remarks apply apply to to the the unitary unitary invariance invariance of of norms norms of of real real are not unitarily unitarily invariant. All norms are equivalent; there exist 7.
All norms on on en C" are equivalent; i. For For xx EG en, C", the the following following inequalities inequalities are are all all tight bounds; i. Finally, we Finally, we conclude conclude this this section section with with aa theorem theorem about about convergence convergence of of vectors. ConConvergence of of aa sequence sequence of of vectors to some some limit vector can can be converted into into aa statement vergence vectors to limit vector be converted statement about numbers, i.
Ee en. Matrix Norms Norms Matrix In this section we we introduce introduce the the concept concept of of matrix norm. As As with with vectors, vectors, the for In this section matrix norm. The The using matrix norms nearness of of former the latter to make make sense former notion notion is is useful useful for for perturbation perturbation analysis, analysis, while while the latter is is needed needed to sense of "convergence" vector space xn ,, IR is "convergence" of of matrices.
Attention Attention is is confined confined to to the the vector space IRm Wnxn R since since that that is what arises arises in in the majority of of applications.
Extension Extension to to the complex case case is is straightforward what the majority the complex straightforward and and essentially essentially obvious. As with vectors, this is called the triangle inequality. Let A Ee lR,mxn. Example 7. Let A A E e lR,mxn.
The "maximum column sum" norm is 1. The Schattenp-norms Example 7. For example,. The The norm norm II. The The spectral radius of of A is the scalar by P. Let Let 9. Let , where e IR R"n are are nonzero.
Definition 9. If A is root of of multiplicity of n A , we we say say that eigenvalue of of A of algebraic multiplicity m. Thll The minimal K""" is of IPll. Moreover, Moreover, itit can can also also be be 9. Fundamental 9. In particular, shown that aa A A. Unfortunately, algorithm, is numerically unstable. Example 9.
The above definitions are illustrated below for a series of matrices, each 4 4, i. We denote the geometric multiplicity by g. What What is is true true is is that that the the independent eigenvalue 0. The following theorem is is useful useful when when solving solving systems of linear linear differential differential equations. The following theorem systems of equations. Then Theorem 9.
If " is with eigenvalues eigenvalues A. It course, to have aa version version of which It is is desirable, desirable, of of course, to have of Theorem Theorem 9.
It It is is necessary necessary first first to to consider consider the the notion notion of of Jordan Jordan canonical form, form, from from which such aa result is then then available available and and presented in this chapter. Theorem x I. Jordan Canonical Canonical Form Form 9. Real Jordan Canonical Form: Xi, With For nontrivial Jordan blocks, the situation is only a bit more complicated. The The characteristic characteristic polynomials polynomials of of the the Jordan Jordan blocks blocks defined defined in in Theorem Theorem 9.
The characteristic polynomial polynomial of Theorem 9. The characteristic of aa matrix is the the product elementary divisors. The minimal of aa matrix divisors of of divisors. The minimal polynomial polynomial of matrix is is the the product product of of the the elementary elementary divisors highest degree corresponding to to distinct distinct eigenvalues. Then n 1.
Theorem 9. Thus, 1. From From Theorem 9. Again, from from Theorem 9. D 0 Example 9. Suppose A e E lR. Suppose A E7x7 is A. Determination Determination of JCF 9. Knowing TT A. The straightforward straightforward case case is, of course, course, when when Ai X,- is is simple, simple, i.
The more interesting and difficult case occurs when Ai is of algebraic multiplicity multiplicity greater than one. A Ee C"nxn or R" ]R. Let A ". Remark Remark 9. An analogous definition holds for a left left principal vector of degree k. Eigenvalues Eigenvectors Chapter Eigenvalues and and Eigenvectors synonymously with "of "of degree k.
The phrase "of "of grade k" is often often used synonymously 3. Principal vectors are sometimes also called generalized generalized eigenvectors, eigenvectors, but the latter different meaning in Chapter A right or left principal vector of degree kk is associated with a Jordan block J; ji of dimension k or larger.
The first column yields the equation Ax! Thus, Thus, the definition of principal vector is satisfied. First, determine all eigenvalues of A eE R" " nxn. Then for each distinct X A perform the following: c 1. I associated This step finds all the eigenvectors i. The number of of A -— XI. For example, if if X. If multiplicity of rank A — If the algebraic multiplicity of principal vectors still need need to be computed XA is greater than its geometric multiplicity, principal from succeeding steps.
Determination Determination of of the the JCF 9. JCF 87 3. Continue Continue in in this this way until the the total total number number of of independent independent eigenvectors eigenvectors and and principal 4. Unfortunately, this this natural-looking can fail fail to to find find all vectors. For For Unfortunately, natural-looking procedure procedure can all Jordan Jordan vectors.
Attempts Attempts to to do do such such calculations calculations in in finite-precision finite-precision floating-point floating-point arithmetic arithmetic generally prove prove unreliable. There There are are significant significant numerical numerical difficulties difficulties inherent inherent in in attempting generally attempting to compute compute aa JCF, JCF, and and the the interested interested student student is is strongly strongly urged urged to to consult consult the the classical classical and and very to very readable MATLAB readable [8] [8] to to learn learn why.
Symbolic kxk Theorem 9. Then Theorem Theorem 9. Theorem Principal vectors Jordan blocks indeTheorem 9. Principal vectors associated associated with with different different Jordan blocks are are linearly linearly independent. Example Let Example 9. Eigenvalues and Eigenvectors 1 A yields A-- 11 x? For For the the sake sake of of definiteness, defmiteness, we we consider below below the case of of aa single single Jordan but the the result clearly holds any JCF. It It is is thus thus natural to expect expect an an associated direct direct sum decomposition of of jH.
Such Such aa decomposition decomposition is is given given in in the the following associated sum decomposition following theorem. Suppose A Ee R" jH. Then Then with A-i, AI, Note that dimM A Definition 9. Let Definition 9. Eigenvalues Eigenvalues and and Eigenvectors If Suppose X block diagonalizes A, i. Suppose A Ee ]Rnxn. S is A-invariant A -invariant if if and only only if ifSS The Jordan Jordan canonical canonical form form is is aa special special case case of of the above theorem.
If A A has The the above theorem. If has distinct distinct eigenvalues Ai as in Theorem 9. We would then then get get aa block block diagonal diagonal example by Vi. We would representation for blocks rather structured Jordan blocks.
Other Other representation for A A with with full full blocks rather than than the the highly highly structured Jordan blocks. We could also use other block diagonal decompositions e. Finally, we return to the problem of developing a formula A formula for ee'l AA in the case that A x T nxn is not necessarily diagonalizable.
Equivalently, partition Equivalently, partition 9. Matrix Sign 91 91 compatibly. Then compatibly. A called the the matrix matrix sign function. It It is is aa generalization generalization of of the the sign or signum signum of of aa scalar. A survey of the matrix sign function and some of its applications can be found in [15]. Suppose A E e C" " has no eigenvalues on the imaginary axis, and let be Jordan canonical canonicalform form for for A, with with N N containing containing all all Jordan Jordan blocks blocks corresponding corresponding to to the the be aa Jordan in the the left left half-plane half-plane and and P P containing containing all all Jordan Jordan blocks blocks corresponding corresponding to eigenvalues of eigenvalues of A in to eigenvalues in eigenvalues in the the right right half-plane.
Eigenvalues Eigenvalues and and Eigenvectors Eigenvectors Chapter where the negative and positive positive identity matrices are of of the same dimensions as N and p, P, respectively.
There are are other other equivalent equivalent definitions definitions of of the sign function, function, but but the one given There the matrix matrix sign the one given here is is especially especially useful useful in in deriving deriving many of its its key key properties. The matrix sign function does not generally generally lend itself itself to reliable computation on a finite-wordlength digital computer. In fact, its reliable numerical calculation calculation is an interesting topic in its own right.
We state state some some of the more properties of matrix sign sign function function as as theorems. We of the more useful useful properties of the the matrix Their Their straightforward proofs are left left to the exercises. Then the following following hold: 1. S is diagonalizable with eigenvalues equal to del.
A Ee Cnxn ,. Let A have distinct distinct eigenvalues AI, Show Show that that vv can can be be expressed expressed uniquely uniquely as as aa linear linear combination combination arbitrary of the right eigenvectors. Find the appropriate expression expression for v as a linear combination of the left eigenvectors as well.
Prove that all of 2. Prove that all eigenvalues eigenvalues of aa skew-Hermitian matrix must be pure imaginary. A Ee C" rc nxn is Hermitian. Let A be an an eigenvalue eigenvalue of A with with corresponding 3. Suppose Suppose A " is Hermitian. Let A be of A corresponding right eigenvector x. Show that also aa left left eigenvector eigenvector for for A.
Show that xx is is also A. Prove Prove the the same same result result if A A is skew-Hermitian. Determine all possible JCFs for A. JCFs for A. Determine the eigenvalues, eigenvalues, right eigenvectors eigenvectors and and right principal vectors if if necessary, and real JCFs of the following matrices: a 2 -1 ] 0 ' [ 1 6. If The following following results results are are typical typical of of what what can can be be achieved under aa unitary unitary similarity.
This This is is proved proved in in Theorem Theorem The The answer is given Theorem x Normal matrices matrices include include Hermitian, Hermitian, matrix if matrix if and A. Theorem Then Then there exist n. In The proof induction upon noting noting that X proof is completed easily by induction that the 2,2 -block The construction can actually be performed quite easily by means of Householder Householder or Givens transformations transformations as in the proof proof of the following general general result.
Proof: Xk]. Construct aa sequence Householder matrices matrices also known HI, Xk are orthonormal. X i U2 - kk columns of V2. We illustrate the construction of the necessary Householder matrix for kk — For simplicity, simplicity, we consider the real case. Basic Canonical Canonical Forms Thus, Further details on Householder matrices, including the choice of sign and the complex case, consulted in standard numerical linear linear algebra can be consulted standard numerical algebra texts such as [7], [7], [11], [11], [23], [23], [25].
The real version of Theorem T nxn Theorem Then there there exists an Let eE E have A with the obvious analogue Note that Theorem In fact, A in The following pair of theorems form the theoretical theoretical foundation of the double-Francisdouble-FrancisQR algorithm used to compute matrix eigenvalues in a numerically stable and reliable way. Canonical Forms x Theorem Let ". Proof: The proof of of this this theorem theorem is is essentially essentially the the same same as that of of Theorem lO.
However, However, the the next next theorem theorem shows shows that that every every to xn A eE W IRnxn is also also orthogonally orthogonally similar similar i. A A quasi-upper-triangular matrix is is block block upper upper triangular triangular with with 1 matrix. Let e R" ". Definition The The triangular T in The quasi-upper-triangular matrix matrix S in Theorem Theorem Example Its Its real real JCF JCF is is 1 -1 5 0 0 0 n n Note corresponding first Note that that only only the the first first Schur Schur vector vector and and then then only only if if the the corresponding first eigenvalue eigenvalue if U orthogonal is is an an eigenvector.
However, However, what what is is true, true, and and sufficient for virtually virtually is real real if is U is is orthogonal sufficient for all applications applications see, see, for for example, example, [17] , is that that the the first first k Schur vectors span span the the same all [17] , is Schur vectors same Ainvariant subspace the eigenvectors corresponding to to the the first first k eigenvalues along the the invariant subspace as as the eigenvectors corresponding eigenvalues along diagonal of of T or S.
While every every matrix matrix can can be be reduced reduced to to Schur Schur form or RSF , RSF , it it is is of of interest interest to to know While form or know when we we can go further further and reduce aa matrix matrix via via unitary unitary similarity to diagonal diagonal form.
The when can go and reduce similarity to The following following theorem theorem answers answers this this question. A C"nxn " is is unitarily unitarily similar Theorem A matrix matrix A A eE c similar to to a a diagonal diagonal matrix matrix ifif and and only only if if H H H A is is normal normal i. Definite Matrices Then It It is then a routine exercise to show that T T must, in fact, be diagonal.
Wnxn is Definition A O. Remark If If a matrix is neither neither definite nor semidefinite, semidefinite, it is said to be indefinite. H nxn Theorem Proof: U be a unitary matrix that diagonalizes diagonalizes A However, numerical procedures procedures for computing such an equivalence directly via, say, Gaussian or elementary row and column operations, are generally unreliable. The numerically preferred equivalence is, of course, the unitary unitary equivalence known as the SVD.
However, the SVD is relatively expensive to compute and other canonical forms exist that are intermediate between l0. Two such forms are stated here. They are more stably computable than lOA Many similar results are also available. Transformations and Congruence Equivalence Equivalence Transformations and Congruence x Theorem Then exist Theorem Orthogonal Decomposition. Proof: For the proof, proof, see Proof: For the see [4].
Proof: For the proof, proof, see D 0 Remark When A has has full column rank rank but but is "near" aa rank rank deficient deficient matrix, Remark When A full column is "near" matrix, various rank rank revealing decompositions are can sometimes detect such such various revealing QR QR decompositions are available available that that can sometimes detect phenomena at considerably less less than than aa full Again, see see [4] phenomena at aa cost cost considerably full SVD. Again, [4] for for details.
The aa congruence. Note Note that that aa congruence congruence is is aa similarity similarity if if and and only only if ifXX is is unitary. Note that that congruence preserves the the property property of of being being Hermitian; Hermitian; i.
It of interest to ask ask what what other properties of of aa matrix matrix are are then X XHH AX It is is of interest to other properties preserved under under congruence. It turns out the principal principal property property so so preserved preserved is is the the sign sign preserved It turns out that that the of of each each eigenvalue. H x nxn Definition Then Theorem of Inertia. Proof: For For the the proof, proof, see, for example, p. D Proof: see, for example, [21, [21, p. D Theorem Theorem We following.
Chapter By Theorem Theorem Note the symmetric Schur complements of A or D in the theorem. One final canonical form to be mentioned is the rational A A matrix matrix A A E Xn" is said to be nonderogatory ifits Definition e lR M" is said to be if its minimal minimal polynomial polynomial and characteristic characteristic polynomial polynomial are are the same or; Jordan canonical canonical form and the same or, equivalently, equivalently, if if its its Jordan form has only one block block associated each distinct has only one associated with with each distinct eigenvalue.
Companion matrices matrices also also appear appear in in the the literature literature in in several several equivalent equivalent forms. To To Companion illustrate, consider the the companion matrix illustrate, consider companion matrix l0. This matrix matrix is is aa special special case case of of aa matrix matrix in lower Hessenberg form. Using Using the the reverse-order reverse-order identity P given by 9.
In fact, the inverse of aa nonsingular nonsingular companion matrix is in companion companion form. Canonical Canonical Forms Forms Chapter with with aa similar similar result result for for companion companion matrices matrices of of the the form form If If a companion matrix of the form Then A in in Explicit Explicit formulas formulas for for all all the right and left singular singular vectors can Remark Physics for scientists and engineers. Excel for Scientists and Engineers.
Probability and Statistics for Engineers and Scientists. Electronics and Communications for Scientists and Engineers. Algebraic geometry for scientists and engineers. Patent Fundamentals for Scientists and Engineers. Management for Engineers, Scientists and Technologists. Excel for Scientists and Engineers: Numerical Methods. Patent law for scientists and engineers. Handbook of Mathematics for Engineers and Scientists. Finite fields for computer scientists and engineers.
Numerical methods for scientists and engineers. Fortran for scientists and engineers. Fortran 95, for scientists and engineers. Catastrophe theory for scientists and engineers. Recommend Documents.
0コメント