Invariants of the adjoint coaction and Yetter-Drinfeld categories

Let H be a Hopf algebra over a field k. We study O(H), the subalgebra of invariants of H under the adjoint coaction, and prove that it is closely related to questions about the antipode and the integral. It may differ from C(H), the subalgebra of cocommutative elements of H. In fact, we prove that if H is unimodular then C(H)=O(H) is equivalent to assuming that the antipode is an involution. We prove that if H is a semisimple Hopf algebra over an algebraically closed field then O(H^*) is a symmetric Frobenius algebra containing the left integral of H^*. This enables us to prove that if H is also cosemisimple then C(H^*),C(H) are all separable algebras. It has been recently shown by Etingof and Gelaki (On finite-dimensional semisimple and cosemisimple Hopf algebras in positive characteristic, preprint) that in this situation S²=id and hence O(H)=C(H). In characteristic 0 semisimple Hopf algebras are cosemisimple and O(H^*) and C(H^*) coincide (and equal the so-called "character ring"). In positive characteristic O(H)≠C(H) in some cases, and O(H) may be a more natural object. For example, quasitriangular Hopf algebras are endowed with an algebra homomorphism between O(H^*) and the center of H. We show that if this homomorphism is a monomorphism then H is factorizable (a notion connected to computing invariants of 3-manifolds). We prove that if (H,R) is factorizable and semisimple then it is cosemisimple and so C(H^*) and C(H) are separable algebras. We apply these results to the associated Yetter-Drinfeld category.