西安网站建设 翼驰,wordpress商城模板源码,做电商网站必需知道qc,网站开发定制案例展示文章目录 1. ML中为什么需要矩阵求导2. 向量函数与矩阵求导初印象3. YX 拉伸术3.1 f(x)为标量#xff0c;X为列向量3.2 f(x)为列向量#xff0c;X 为标量3.3 f(x)为列向量#xff0c;X 为列向量 4. 常见矩阵求导公式4.1 Y A T X YA^TX YATX4.2 Y X T A X YX^TAX YXTAX 1… 文章目录 1. ML中为什么需要矩阵求导2. 向量函数与矩阵求导初印象3. YX 拉伸术3.1 f(x)为标量X为列向量3.2 f(x)为列向量X 为标量3.3 f(x)为列向量X 为列向量 4. 常见矩阵求导公式4.1 Y A T X YA^TX YATX4.2 Y X T A X YX^TAX YXTAX 1. ML中为什么需要矩阵求导 简洁 用方程式表示如下 y 1 w 1 X 11 w 2 X 12 (1) y_1w_1X_{11}w_2X_{12}\tag{1} y1w1X11w2X12(1) y 2 w 1 X 21 w 2 X 22 (2) y_2w_1X_{21}w_2X_{22}\tag{2} y2w1X21w2X22(2) 转换成矩阵表示如下 Y X W (3) YXW\tag{3} YXW(3) Y [ y 1 y 2 ] , X [ x 11 x 12 x 21 x 22 ] , W [ w 1 w 2 ] (4) Y\begin{bmatrix}y_1\\\\y_2\end{bmatrix},X\begin{bmatrix}x_{11}x_{12}\\\\x_{21}x_{22}\end{bmatrix},W\begin{bmatrix}w_{1}\\\\w_{2}\end{bmatrix}\tag{4} Y y1y2 ,X x11x21x12x22 ,W w1w2 (4) 快速 当使用python 中的numpy 库时候在相对于 for 循环Numpy 本身的计算提速相当快 源代码
import time
import numpy as npif __name__ __main__:N 1000000a np.random.rand(N)b np.random.rand(N)start time.time()c np.dot(a,b)stop time.time()print(fc{c})print(vectorized version: str(1000*(stop-start))ms)c 0start1 time.time()for i in range(N):c a[i]*b[i]stop1 time.time()print(fc{c})print(for loop: str(1000*(stop1-start1))ms)times1 (stop1-start1)/(stop-start)print(ftimes1{times1})结果
c250071.8870070607
vectorized version: 6.549358367919922ms
c250071.88700706122
for loop: 265.43641090393066ms
times140.52861303239898# 向量化居然比单独的for循环快40倍2. 向量函数与矩阵求导初印象
标量函数:输出为标量的函数 f ( x ) x 2 ⇒ x ∈ R → x 2 ∈ R f(x)x^2\Rightarrow x\in R\rightarrow x^2 \in R f(x)x2⇒x∈R→x2∈R f ( x ) x 1 2 x 2 2 ⇒ [ x 1 x 2 ] ∈ R 2 → x 1 2 x 2 2 ∈ R f(x)x_1^2x_2^2\Rightarrow \begin{bmatrix}x_1\\\\x_2\end{bmatrix}\in R^2\rightarrow x_1^2x_2^2 \in R f(x)x12x22⇒ x1x2 ∈R2→x12x22∈R向量函数:输出为向量或矩阵的函数 1 输入标量输出向量 f ( x ) [ f 1 ( x ) x f 2 ( x ) x 2 ] ⇒ x ∈ R , [ x x 2 ] ∈ R 2 f(x)\begin{bmatrix}f_1(x)x\\\\f_2(x)x^2\end{bmatrix}\Rightarrow x\in R,\begin{bmatrix}x\\\\x^2\end{bmatrix} \in R^2 f(x) f1(x)xf2(x)x2 ⇒x∈R, xx2 ∈R2 2 输入标量输出矩阵 f ( x ) [ f 11 ( x ) x f 12 ( x ) x 2 f 21 ( x ) x 3 f 22 ( x ) x 4 ] ⇒ x ∈ R , [ x x 2 x 3 x 4 ] ∈ R 2 × 2 f(x)\begin{bmatrix}f_{11}(x)xf_{12}(x)x^2\\\\f_{21}(x)x^3f_{22}(x)x^4\end{bmatrix}\Rightarrow x\in R,\begin{bmatrix}xx^2\\\\x^3x^4\end{bmatrix} \in R^{2\times2} f(x) f11(x)xf21(x)x3f12(x)x2f22(x)x4 ⇒x∈R, xx3x2x4 ∈R2×2 3 输入向量输出矩阵 f ( x ) [ f 11 ( x ) x 1 x 2 f 12 ( x ) x 1 2 x 2 2 f 21 ( x ) x 1 3 x 2 3 f 22 ( x ) x 1 4 x 2 4 ] ⇒ [ x 1 x 2 ] ∈ R 2 , [ x 1 x 2 x 1 2 x 2 2 x 1 3 x 2 3 x 1 4 x 2 4 ] ∈ R 2 × 2 f(x)\begin{bmatrix}f_{11}(x)x_1x_2f_{12}(x)x_1^2x_2^2\\\\f_{21}(x)x_1^3x_2^3f_{22}(x)x_1^4x_2^4\end{bmatrix}\Rightarrow \begin{bmatrix}x_1\\\\x_2\end{bmatrix} \in R^2,\begin{bmatrix}x_1x_2x_1^2x_2^2\\\\x_1^3x_2^3x_1^4x_2^4\end{bmatrix} \in R^{2\times2} f(x) f11(x)x1x2f21(x)x13x23f12(x)x12x22f22(x)x14x24 ⇒ x1x2 ∈R2, x1x2x13x23x12x22x14x24 ∈R2×2总结 矩阵求导的本质 d A d B 矩阵 A 中的每个元素对矩阵 B 中的每个元素求导 \frac{\mathrm{d}A}{\mathrm{d}B}矩阵A中的每个元素对矩阵B中的每个元素求导 dBdA矩阵A中的每个元素对矩阵B中的每个元素求导
3. YX 拉伸术
3.1 f(x)为标量X为列向量
标量不变向量拉伸YX中Y前面横向拉X后面纵向拉 d f ( x ) d x , Y f ( x ) 为标量 X [ x 1 x 2 ⋮ x n ] 为列向量 \frac{\mathrm{d}f(x)}{\mathrm{d}x},Yf(x)为标量X\begin{bmatrix}x_1\\\\x_2\\\\\vdots\\\\x_n\end{bmatrix}为列向量 dxdf(x),Yf(x)为标量X x1x2⋮xn 为列向量 f ( x ) f ( x 1 , x 2 , . . . . , x n ) 为标量 f(x)f(x_1,x_2,....,x_n)为标量 f(x)f(x1,x2,....,xn)为标量标量 f ( x ) f(x) f(x)不变向量X 因为在YX拉伸术中在Y后面所以向量X纵向拉伸实际上就是将多元函数的偏导写在一个列向量中 d f ( x ) d x [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}x}\begin{bmatrix}\frac{\partial f(x)}{\partial x_1}\\\\\frac{\partial f(x)}{\partial x_2}\\\\\vdots\\\\\frac{\partial f(x)}{\partial x_n}\end{bmatrix} dxdf(x) ∂x1∂f(x)∂x2∂f(x)⋮∂xn∂f(x)
3.2 f(x)为列向量X 为标量 f ( x ) [ f 1 ( x ) f 2 ( x ) ⋮ f n ( x ) ] ; X 为标量 f(x)\begin{bmatrix}f_1(x)\\\\f_2(x)\\\\\vdots\\\\f_n(x)\end{bmatrix};X 为标量 f(x) f1(x)f2(x)⋮fn(x) ;X为标量
标量不变向量拉伸YX中Y前面横向拉X后面纵向拉 d f ( x ) d x [ ∂ f 1 ( x ) ∂ x ∂ f 2 ( x ) ∂ x … ∂ f n ( x ) ∂ x ] \frac{\mathrm{d}f(x)}{\mathrm{d}x}\begin{bmatrix}\frac{\partial f_1(x)}{\partial x}\frac{\partial f_2(x)}{\partial x}\dots\frac{\partial f_n(x)}{\partial x}\end{bmatrix} dxdf(x)[∂x∂f1(x)∂x∂f2(x)…∂x∂fn(x)]
3.3 f(x)为列向量X 为列向量 f ( x ) [ f 1 ( x ) f 2 ( x ) ⋮ f n ( x ) ] ; X [ x 1 x 2 ⋮ x n ] 为列向量 f(x)\begin{bmatrix}f_1(x)\\\\f_2(x)\\\\\vdots\\\\f_n(x)\end{bmatrix};X\begin{bmatrix}x_1\\\\x_2\\\\\vdots\\\\x_n\end{bmatrix}为列向量 f(x) f1(x)f2(x)⋮fn(x) ;X x1x2⋮xn 为列向量
第一步先固定Y 将 X 纵向拉 d f ( x ) d x [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}x}\begin{bmatrix}\frac{\partial f(x)}{\partial x_1}\\\\\frac{\partial f(x)}{\partial x_2}\\\\\vdots\\\\\frac{\partial f(x)}{\partial x_n}\end{bmatrix} dxdf(x) ∂x1∂f(x)∂x2∂f(x)⋮∂xn∂f(x) 第二步看每一个项 ∂ f ( x ) ∂ x 1 \frac{\partial f(x)}{\partial x_1} ∂x1∂f(x),其中f(x)为列向量 x 1 x_1 x1为标量那么可以看出要进行 Y 横向拉 ∂ f ( x ) ∂ x 1 [ ∂ f 1 ( x ) ∂ x 1 ∂ f 2 ( x ) ∂ x 1 … ∂ f n ( x ) ∂ x 1 ] \frac{\partial f(x)}{\partial x_1}\begin{bmatrix}\frac{\partial f_1(x)}{\partial x_1}\frac{\partial f_2(x)}{\partial x_1}\dots\frac{\partial f_n(x)}{\partial x_1}\end{bmatrix} ∂x1∂f(x)[∂x1∂f1(x)∂x1∂f2(x)…∂x1∂fn(x)]第三步 将每项整合如下 d f ( x ) d x [ ∂ f 1 ( x ) ∂ x 1 ∂ f 2 ( x ) ∂ x 1 … ∂ f n ( x ) ∂ x 1 ∂ f 1 ( x ) ∂ x 2 ∂ f 2 ( x ) ∂ x 2 … ∂ f n ( x ) ∂ x 2 ⋮ ⋮ … ⋮ ∂ f 1 ( x ) ∂ x n ∂ f 2 ( x ) ∂ x n … ∂ f n ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}x}\begin{bmatrix}\frac{\partial f_1(x)}{\partial x_1}\frac{\partial f_2(x)}{\partial x_1}\dots\frac{\partial f_n(x)}{\partial x_1}\\\\\frac{\partial f_1(x)}{\partial x_2}\frac{\partial f_2(x)}{\partial x_2}\dots\frac{\partial f_n(x)}{\partial x_2}\\\\\vdots\vdots\dots\vdots\\\\\frac{\partial f_1(x)}{\partial x_n}\frac{\partial f_2(x)}{\partial x_n}\dots\frac{\partial f_n(x)}{\partial x_n}\end{bmatrix} dxdf(x) ∂x1∂f1(x)∂x2∂f1(x)⋮∂xn∂f1(x)∂x1∂f2(x)∂x2∂f2(x)⋮∂xn∂f2(x)…………∂x1∂fn(x)∂x2∂fn(x)⋮∂xn∂fn(x)
4. 常见矩阵求导公式
4.1 Y A T X YA^TX YATX f ( x ) A T X ; A [ a 1 , a 2 , … , a n ] T ; X [ x 1 , x 2 , … , x n ] T , 求 d f ( x ) d X f(x)A^TX;\quad A[a_1,a_2,\dots,a_n]^T;\quad X[x_1,x_2,\dots,x_n]^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X} f(x)ATX;A[a1,a2,…,an]T;X[x1,x2,…,xn]T,求dXdf(x)
由于 A T 1 × n , X n × 1 , 那么 f ( x ) 为标量即表示数值 A^T1\times n,Xn\times1,那么f(x)为标量即表示数值 AT1×n,Xn×1,那么f(x)为标量即表示数值标量不变向量拉伸YX中Y前面横向拉X后面纵向拉 f ( x ) ∑ i 1 N a i x i f(x)\sum_{i1}^Na_ix_i f(x)i1∑Naixi d f ( x ) d X [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}X}\begin{bmatrix}\frac{\partial f(x)}{\partial x_1}\\\\\frac{\partial f(x)}{\partial x_2}\\\\\vdots\\\\\frac{\partial f(x)}{\partial x_n}\end{bmatrix} dXdf(x) ∂x1∂f(x)∂x2∂f(x)⋮∂xn∂f(x) 可以计算 ∂ f ( x ) ∂ x i \frac{\partial f(x)}{\partial x_i} ∂xi∂f(x) ∂ f ( x ) ∂ x i a i \frac{\partial f(x)}{\partial x_i}a_i ∂xi∂f(x)ai可得如下 d f ( x ) d X [ a 1 a 2 ⋮ a n ] A \frac{\mathrm{d}f(x)}{\mathrm{d}X}\begin{bmatrix}a_1\\\\a_2\\\\\vdots\\\\a_n\end{bmatrix}A dXdf(x) a1a2⋮an A结论 当 f ( x ) A T X 当f(x)A^TX 当f(x)ATX d f ( x ) d X A \frac{\mathrm{d}f(x)}{\mathrm{d}X}A dXdf(x)A
4.2 Y X T A X YX^TAX YXTAX f ( x ) X T A X ; A [ a 11 a 12 … a 1 n a 21 a 22 … a 2 n ⋮ ⋮ … ⋮ a n 1 a n 2 … a n n ] ; X [ x 1 , x 2 , … , x n ] T , 求 d f ( x ) d X f(x)X^TAX;\quad A\begin{bmatrix}a_{11}a_{12}\dotsa_{1n}\\\\a_{21}a_{22}\dotsa_{2n}\\\\\vdots\vdots\dots\vdots\\\\a_{n1}a_{n2}\dotsa_{nn}\end{bmatrix};\quad X[x_1,x_2,\dots,x_n]^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X} f(x)XTAX;A a11a21⋮an1a12a22⋮an2…………a1na2n⋮ann ;X[x1,x2,…,xn]T,求dXdf(x) f ( x ) ∑ i 1 N ∑ j 1 N a i j x i x j f(x)\sum_{i1}^N\sum_{j1}^Na_{ij}x_ix_j f(x)i1∑Nj1∑Naijxixj
标量不变YX拉伸术X纵向拉伸 d f ( x ) d X [ ∂ f ( x ) ∂ x 1 ∂ f ( x ) ∂ x 2 ⋮ ∂ f ( x ) ∂ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}X}\begin{bmatrix}\frac{\partial f(x)}{\partial x_1}\\\\\frac{\partial f(x)}{\partial x_2}\\\\\vdots\\\\\frac{\partial f(x)}{\partial x_n}\end{bmatrix} dXdf(x) ∂x1∂f(x)∂x2∂f(x)⋮∂xn∂f(x) ∂ f ( x ) ∂ x i [ a i 1 a i 2 … a i n ] [ x 1 x 2 ⋮ x n ] [ a 1 i a 2 i … a n i ] [ x 1 x 2 ⋮ x n ] \frac{\partial f(x)}{\partial x_i}\begin{bmatrix}a_{i1}a_{i2}\dotsa_{in}\end{bmatrix}\begin{bmatrix}x_1\\\\x_2\\\\\vdots\\\\x_n\end{bmatrix}\begin{bmatrix}a_{1i}a_{2i}\dotsa_{ni}\end{bmatrix}\begin{bmatrix}x_1\\\\x_2\\\\\vdots\\\\x_n\end{bmatrix} ∂xi∂f(x)[ai1ai2…ain] x1x2⋮xn [a1ia2i…ani] x1x2⋮xn d f ( x ) d X [ a 11 a 12 … a 1 n a 21 a 22 … a 2 n ⋮ ⋮ … ⋮ a n 1 a n 2 … a n n ] [ x 1 x 2 ⋮ x n ] [ a 11 a 21 … a n 1 a 12 a 22 … a n 2 ⋮ ⋮ … ⋮ a 1 n a 2 n … a n n ] [ x 1 x 2 ⋮ x n ] \frac{\mathrm{d}f(x)}{\mathrm{d}X}\begin{bmatrix}a_{11}a_{12}\dotsa_{1n}\\\\a_{21}a_{22}\dotsa_{2n}\\\\\vdots\vdots\dots\vdots\\\\a_{n1}a_{n2}\dotsa_{nn}\end{bmatrix}\begin{bmatrix}x_1\\\\x_2\\\\\vdots\\\\x_n\end{bmatrix}\begin{bmatrix}a_{11}a_{21}\dotsa_{n1}\\\\a_{12}a_{22}\dotsa_{n2}\\\\\vdots\vdots\dots\vdots\\\\a_{1n}a_{2n}\dotsa_{nn}\end{bmatrix}\begin{bmatrix}x_1\\\\x_2\\\\\vdots\\\\x_n\end{bmatrix} dXdf(x) a11a21⋮an1a12a22⋮an2…………a1na2n⋮ann x1x2⋮xn a11a12⋮a1na21a22⋮a2n…………an1an2⋮ann x1x2⋮xn 已知 A , A T A,A^T A,AT表示如下 A [ a 11 a 12 … a 1 n a 21 a 22 … a 2 n ⋮ ⋮ … ⋮ a n 1 a n 2 … a n n ] ; A T [ a 11 a 21 … a n 1 a 12 a 22 … a n 2 ⋮ ⋮ … ⋮ a 1 n a 2 n … a n n ] A\begin{bmatrix}a_{11}a_{12}\dotsa_{1n}\\\\a_{21}a_{22}\dotsa_{2n}\\\\\vdots\vdots\dots\vdots\\\\a_{n1}a_{n2}\dotsa_{nn}\end{bmatrix}\quad;A^T\begin{bmatrix}a_{11}a_{21}\dotsa_{n1}\\\\a_{12}a_{22}\dotsa_{n2}\\\\\vdots\vdots\dots\vdots\\\\a_{1n}a_{2n}\dotsa_{nn}\end{bmatrix} A a11a21⋮an1a12a22⋮an2…………a1na2n⋮ann ;AT a11a12⋮a1na21a22⋮a2n…………an1an2⋮ann 综上所述如下 当 f ( x ) X T A X f(x)X^TAX f(x)XTAX时 d f ( x ) d X A X A T X ( A A T ) X \frac{\mathrm{d}f(x)}{\mathrm{d}X}AXA^TX(AA^T)X dXdf(x)AXATX(AAT)X
