本节主要内容

word representation
word2vec

How to represent word?

Problems with resources like WordNet

1.Great as a resource but missing nuance​​​​​​​
2 Missing new meanings of words

• Impossible to keep up-to-date

3.Subjective
4.Requires human labor to create and adapt
5.Can’t compute accurate word similarity

Problem with words as discrete symbols(one-hot vector)

1.No natural notion of similarity for one-hot vectors
2.Vector dimension is too large

Word2vec

Word2Vec:objective function

求极大似然
For each position $t=1,\dots ,T$t=1,…,T, predict context words within a
window of fixed size m, given center word $w_j$wj​.
极大似然公式：
$Likelihood=L\left(\theta \right)=\prod _{t=1}^T\prod _{-m\le j\le m}P(w_{t+j}\mid w_t;\theta )$Likelihood=L(θ)=t=1∏T​−m≤j≤m∏​P(wt+j​∣wt​;θ)
损失函数：
$J\left(\theta \right)=-\frac{1}{T}logL\left(\theta \right)=-\frac{1}{T}\sum _{t=1}^T\sum _{-m\le j\le m}logP(w_{t+j}\mid w_t;\theta )$J(θ)=−T1​logL(θ)=−T1​t=1∑T​−m≤j≤m∑​logP(wt+j​∣wt​;θ)

如何计算 $P(w_{t+j}\mid w_t;\theta )$P(wt+j​∣wt​;θ)?

定义符号如下，

• vw​whenwisacenterword
• uw​whenwisacontextword

则
$P\left(o\mid c\right)=\frac{exp\left(u_o^T{v}_c\right)}{\sum _{w\in V}exp\left(u_w^T{v}_c\right)}$P(o∣c)=w∈V∑​exp(uwT​vc​)exp(uoT​vc​)​

令 $f\left(\theta \right)=logP\left(o\mid c\right)$f(θ)=logP(o∣c) ，求偏导

$\begin{array}{cc}{\displaystyle \frac{\partial f\left(\theta \right)}{\partial v_c}}& {\displaystyle =\frac{\partial }{\partial v_c}log\frac{exp\left(u_o^T{v}_c\right)}{\sum _{w\in V}exp\left(u_w^T{v}_c\right)}}\\ {\displaystyle }& {\displaystyle =\frac{\partial }{\partial v_c}logexp\left(u_o^T{v}_c\right)-\frac{\partial }{\partial v_c}log\sum _{w\in V}exp\left(u_w^T{v}_c\right)}\end{array}$∂vc​∂f(θ)​​=∂vc​∂​logw∈V∑​exp(uwT​vc​)exp(uoT​vc​)​=∂vc​∂​logexp(uoT​vc​)−∂vc​∂​logw∈V∑​exp(uwT​vc​)​
前一部分
$f_1\left(\theta \right)=\frac{\partial }{\partial v_c}logexp\left(u_o^T{v}_c\right)=u_o$f1​(θ)=∂vc​∂​logexp(uoT​vc​)=uo​
后一部分
$\begin{array}{cc}{\displaystyle f_2\left(\theta \right)}& {\displaystyle =\frac{\partial }{\partial v_c}log\sum _{w\in V}exp\left(u_w^T{v}_c\right)}\\ {\displaystyle }& {\displaystyle =\frac{1}{\sum _{w\in V}exp\left(u_w^T{v}_c\right)}\sum _{x\in V}exp\left(u_x^T{v}_c\right)\cdot u_x}\\ {\displaystyle }& {\displaystyle =\sum _{x\in V}\frac{exp\left(u_x^T{v}_c\right)}{\sum _{w\in V}exp\left(u_w^T{v}_c\right)}\cdot u_x}\\ {\displaystyle }& {\displaystyle =\sum _{x\in V}P\left(x\mid c\right)\cdot u_x}\end{array}$f2​(θ)​=∂vc​∂​logw∈V∑​exp(uwT​vc​)=w∈V∑​exp(uwT​vc​)1​x∈V∑​exp(uxT​vc​)⋅ux​=x∈V∑​w∈V∑​exp(uwT​vc​)exp(uxT​vc​)​⋅ux​=x∈V∑​P(x∣c)⋅ux​​
$f\left(\theta \right)=f_1\left(\theta \right)-f_2\left(\theta \right)=u_o-\sum _{x\in V}P\left(x\mid c\right)\cdot u_x$f(θ)=f1​(θ)−f2​(θ)=uo​−x∈V∑​P(x∣c)⋅ux​
根据概率论的知识， ${\sum }_{x=1}^{V}p\left(x\mid c\right)\overrightarrow{u_x}$∑x=1V​p(x∣c)ux​ ​正是 $\overrightarrow{u_o}$uo​ ​对应的期望向量的方向，而 $\frac{\partial f}{\partial \overrightarrow{v_c}}$∂vc​ ​∂f​这个梯度则是把当前的 $\overrightarrow{u_o}$uo​ ​向其期望靠拢的话，需要的一个向量的差值，这与 $\frac{\partial f}{\partial \overrightarrow{v_c}}$∂vc​ ​∂f​的定义刚好一致。