Annihilator varieties of distinguished modules of reductive Lie algebras

We provide a micro-local necessary condition for distinction of admissible representations of real reductive groups in the context of spherical pairs. Let G be a complex algebraic reductive group, and H ⊂ G be a spherical algebraic subgroup. Let g, h denote the Lie algebras of G and H, and let h
⊥ denote the orthogonal complement to h in g ∗ . A g-module is called h-distinguished if it admits a non-zero h-invariant functional. We show that the maximal G-orbit in the annihilator variety of any irreducible h-distinguished g-module intersects h ⊥. This generalizes a result of Vogan [Vog91].
We apply this to Casselman-Wallach representations of real reductive groups to obtain information on branching problems, translation functors and Jacquet modules. Further, we prove in many cases that as suggested by [Pra19, Question 1], when H is a symmetric subgroup of a real reductive group G, the existence of a tempered H-distinguished representation of G implies the existence of a generic H-distinguished representation of G. Many of the models studied in the theory of automorphic forms involve an additive character on the unipotent radical of the subgroup H, and we devised a twisted version of our theorem that yields necessary conditions for the existence of those mixed models. Our method of proof here is inspired by the theory of modules over W-algebras. As an application of our theorem we derive necessary conditions for the existence of Rankin-Selberg, Bessel, Klyachko and Shalika models. Our results are compatible with the recent Gan-Gross-Prasad conjectures for nongeneric representations [GGP20]. Finally, we provide more general results that ease the sphericity assumption on the subgroups, and apply them to local theta correspondence in type II and to degenerate Whittaker models.