Engel's theorem

Let ${\displaystyle {\mathfrak {gl}}(V)}$ be the Lie algebra of the endomorphisms of a finite-dimensional vector space V and ${\displaystyle {\mathfrak {g}}\subset {\mathfrak {gl}}(V)}$ a subalgebra. Then Engel's theorem states the following are equivalent:

1. Each ${\displaystyle X\in {\mathfrak {g}}}$ is a nilpotent endomorphism on V.
2. There exists a flag ${\displaystyle V=V_{0}\supset V_{1}\supset \cdots \supset V_{n}=0,\,\operatorname {codim} V_{i}=i}$ such that ${\displaystyle {\mathfrak {g}}\cdot V_{i}\subset V_{i+1}}$; i.e., the elements of ${\displaystyle {\mathfrak {g}}}$ are simultaneously strictly upper-triangulizable.

Note that no assumption on the underlying base field is required.

We note that Statement 2. for various ${\displaystyle {\mathfrak {g}}}$ and V is equivalent to the statement

For each nonzero finite-dimensional vector space V and a subalgebra ${\displaystyle {\mathfrak {g}}\subset {\mathfrak {gl}}(V)}$, there exists a nonzero vector v in V such that ${\displaystyle X(v)=0}$ for every ${\displaystyle X\in {\mathfrak {g}}.}$

This is the form of the theorem proven in #Proof. (This statement is trivially equivalent to Statement 2 since it allows one to inductively construct a flag with the required property.)

In general, a Lie algebra ${\displaystyle {\mathfrak {g}}}$ is said to be nilpotent if the lower central series of it vanishes in a finite step; i.e., for ${\displaystyle C^{0}{\mathfrak {g}}={\mathfrak {g}},C^{i}{\mathfrak {g}}=[{\mathfrak {g}},C^{i-1}{\mathfrak {g}}]}$ = (i+1)-th power of ${\displaystyle {\mathfrak {g}}}$, there is some k such that ${\displaystyle C^{k}{\mathfrak {g}}=0}$. Then Engel's theorem gives the theorem (also called Engel's theorem): when ${\displaystyle {\mathfrak {g}}}$ has finite dimension, ${\displaystyle {\mathfrak {g}}}$ is nilpotent if and only if ${\displaystyle \operatorname {ad} (X)}$ is nilpotent for each ${\displaystyle X\in {\mathfrak {g}}}$. Indeed, if ${\displaystyle \operatorname {ad} ({\mathfrak {g}})}$ consists of nilpotent operators, then by 1. ${\displaystyle \Leftrightarrow }$ 2. applied to the algebra ${\displaystyle \operatorname {ad} ({\mathfrak {g}})\subset {\mathfrak {gl}}({\mathfrak {g}})}$, there exists a flag ${\displaystyle {\mathfrak {g}}={\mathfrak {g}}_{0}\supset {\mathfrak {g}}_{1}\supset \cdots \supset {\mathfrak {g}}_{n}=0}$ such that ${\displaystyle [{\mathfrak {g}},{\mathfrak {g}}_{i}]\subset {\mathfrak {g}}_{i+1}}$. Since ${\displaystyle C^{i}{\mathfrak {g}}\subset {\mathfrak {g}}_{i}}$, this implies ${\displaystyle {\mathfrak {g}}}$ is nilpotent. (The converse follows straightforwardly from the definition.)

We prove the following form of the theorem:[2] if ${\displaystyle {\mathfrak {g}}\subset {\mathfrak {gl}}(V)}$ is a Lie subalgebra such that every ${\displaystyle X\in {\mathfrak {g}}}$ is a nilpotent endomorphism and if V has positive dimension, then there exists a nonzero vector v in V such that ${\displaystyle X(v)=0}$ for each X in ${\displaystyle {\mathfrak {g}}}$.

The proof is by induction on the dimension of ${\displaystyle {\mathfrak {g}}}$ and consists of a few steps. (Note the structure of the proof is very similar to that for Lie's theorem, which concerns a solvable algebra.) The basic case is trivial and we assume the dimension of ${\displaystyle {\mathfrak {g}}}$ is positive.

Step 1: Find an ideal ${\displaystyle {\mathfrak {h}}}$ of codimension one in ${\displaystyle {\mathfrak {g}}}$.

This is the most difficult step. Let ${\displaystyle {\mathfrak {h}}}$ be a maximal (proper) subalgebra of ${\displaystyle {\mathfrak {g}}}$, which exists by finite-dimensionality. We claim it is an ideal and has codimension one. For each ${\displaystyle X\in {\mathfrak {h}}}$, it is easy to check that (1) ${\displaystyle \operatorname {ad} (X)}$ induces a linear endomorphism ${\displaystyle {\mathfrak {g}}/{\mathfrak {h}}\to {\mathfrak {g}}/{\mathfrak {h}}}$ and (2) this induced map is nilpotent (in fact, ${\displaystyle \operatorname {ad} (X)}$ is nilpotent). Thus, by inductive hypothesis, there exists a nonzero vector v in ${\displaystyle {\mathfrak {g}}/{\mathfrak {h}}}$ such that ${\displaystyle \operatorname {ad} (X)(v)=0}$ for each ${\displaystyle X\in {\mathfrak {h}}}$. That is to say, if ${\displaystyle v=[Y]}$ for some Y in ${\displaystyle {\mathfrak {g}}}$ but not in ${\displaystyle {\mathfrak {h}}}$, then ${\displaystyle [X,Y]=\operatorname {ad} (X)(Y)\in {\mathfrak {h}}}$ for every ${\displaystyle X\in {\mathfrak {h}}}$. But then the subspace ${\displaystyle {\mathfrak {h}}'\subset {\mathfrak {g}}}$ spanned by ${\displaystyle {\mathfrak {h}}}$ and Y is a Lie subalgebra in which ${\displaystyle {\mathfrak {h}}}$ is an ideal. Hence, by maximality, ${\displaystyle {\mathfrak {h}}'={\mathfrak {g}}}$. This proves the claim.

Step 2: Let ${\displaystyle W=\{v\in V|X(v)=0,X\in {\mathfrak {h}}\}}$. Then ${\displaystyle {\mathfrak {g}}}$ stabilizes W; i.e., ${\displaystyle X(v)\in W}$ for each ${\displaystyle X\in {\mathfrak {g}},v\in W}$.

Indeed, for ${\displaystyle Y}$ in ${\displaystyle {\mathfrak {g}}}$ and ${\displaystyle X}$ in ${\displaystyle {\mathfrak {h}}}$, we have: ${\displaystyle X(Y(v))=Y(X(v))+[X,Y](v)=0}$ since ${\displaystyle {\mathfrak {h}}}$ is an ideal and so ${\displaystyle [X,Y]\in {\mathfrak {h}}}$. Thus, ${\displaystyle Y(v)}$ is in W.

Step 3: Finish up the proof by finding a nonzero vector that gets killed by ${\displaystyle {\mathfrak {g}}}$.

Write ${\displaystyle {\mathfrak {g}}={\mathfrak {h}}+L}$ where L is a one-dimensional vector subspace. Let Y be a nonzero vector in L and v a nonzero vector in W. Now, ${\displaystyle Y}$ is a nilpotent endomorphism (by hypothesis) and so ${\displaystyle Y^{k}(v)\neq 0,Y^{k+1}(v)=0}$ for some k. Then ${\displaystyle Y^{k}(v)}$ is a required vector as the vector lies in W by Step 2. ${\displaystyle \square }$

• Lie's theorem
• Heisenberg group