Hecke Triangle Groups, Transfer Operators and Hausdorff Dimension

We consider the family of Hecke triangle groups $$ \Gamma _{w} = \langle S, T_w\rangle $$ Γ w = ⟨ S , T w ⟩ generated by the Möbius transformations $$ S : z\mapsto -1/z $$ S : z ↦ - 1 / z and $$ T_{w} : z \mapsto z+w $$ T w : z ↦ z + w with $$ w > 2.$$ w > 2 . In this case, the corresponding hyperbolic quotient $$ \Gamma _{w}\backslash {\mathbb {H}}^2 $$ Γ w \ H 2 is an infinite-area orbifold. Moreover, the limit set of $$ \Gamma _w $$ Γ w is a Cantor-like fractal whose Hausdorff dimension we denote by $$ \delta (w). $$ δ ( w ) . The first result of this paper asserts that the twisted Selberg zeta function $$ Z_{\Gamma _{ w}}(s, \rho ) $$ Z Γ w ( s , ρ ) , where $$ \rho : \Gamma _{w} \rightarrow \mathrm {U}(V) $$ ρ : Γ w → U ( V ) is an arbitrary finite-dimensional unitary representation, can be realized as the Fredholm determinant of a Mayer-type transfer operator. This result has a number of applications. We study the distribution of the zeros in the half-plane $$\mathrm {Re}(s) > \frac{1}{2}$$ Re ( s ) > 1 2 of the Selberg zeta function of a special family of subgroups $$( \Gamma _w^N )_{N\in {\mathbb {N}}} $$ ( Γ w N ) N ∈ N of $$\Gamma _w$$ Γ w . These zeros correspond to the eigenvalues of the Laplacian on the associated hyperbolic surfaces $$X_w^N = \Gamma _w^N \backslash {\mathbb {H}}^2$$ X w N = Γ w N \ H 2 . We show that the classical Selberg zeta function $$Z_{\Gamma _w}(s)$$ Z Γ w ( s ) can be approximated by determinants of finite matrices whose entries are explicitly given in terms of the Riemann zeta function. Moreover, we prove an asymptotic expansion for the Hausdorff dimension $$\delta (w)$$ δ ( w ) as $$w\rightarrow \infty $$ w → ∞ .