简而言之：\\ 换成 \right.\\ , & 换成 &\left.

来个例子就知道了：

原本的公式是：

\begin{align}\label{up_critic}
L_Q(\theta) & = \mathbb{E}\left[\frac{1}{2}(Q_\theta(\mathcal{S}_{k,t}^m, {A}_{k,t}^m) - ({R}_{k,t}^m  \\ \nonumber
&+ \gamma Q_{\bar{\theta}}(\mathcal{S}_{k,t}^{m+1}, {A}_{k,t}^{m+1}) - \log \pi_\phi({A}_{k,t}^{m+1}|\mathcal{S}_{k,t}^{m+1})))^2\right],
\end{align}

将\和第二个&改掉就行了：

\begin{align}\label{up_critic}
L_Q(\theta) & = \mathbb{E}\left[\frac{1}{2}(Q_\theta(\mathcal{S}_{k,t}^m, {A}_{k,t}^m) - ({R}_{k,t}^m  \right.\\ \nonumber
&\left.+ \gamma Q_{\bar{\theta}}(\mathcal{S}_{k,t}^{m+1}, {A}_{k,t}^{m+1}) - \log \pi_\phi({A}_{k,t}^{m+1}|\mathcal{S}_{k,t}^{m+1})))^2\right],
\end{align}

结果如下：

若要将公式编号放在第二行则：

\begin{align}\label{up_actor}
L_\pi(\phi)  = \mathbb{E}_{\mathcal{S}_{k,t}^m \sim \mathcal{B}}&\left[\mathbb{E}_{{A}_{k,t}^m\sim\pi_\phi}\big[\alpha\log(\pi_\phi({A}_{k,t}^m|\mathcal{S}_{k,t}^m)) \right. \notag \\ 
&\left.- Q_\theta(\mathcal{S}_{k,t}^m, {A}_{k,t}^m)\big]\right].
\end{align}