Answer by Roman for Definite double integral in Mathematica

(leaving out the global $R^2$ multiplier)

First step: definite integration over φ:

A = Integrate[Sin[ϑ], {φ, ArcSin[1/(Tan[ϑ]*Tan[β])],       Pi - ArcSin[1/(Tan[ϑ]*Tan[β])]}] // FullSimplify

2 ArcCos[Cot[β] Cot[ϑ]] Sin[ϑ]

Second step: indefinite integration over ϑ:

B[β_, ϑ_] = Integrate[A, ϑ, GenerateConditions -> False] // FullSimplify

(lengthy output)

Third step: insert integration boundaries. I think that here's the tricky bit that stumps the automatic integration you've been running for hours. Manually taking the limit at the lower boundary (implicitly assuming that $\pi/2-\beta<\alpha$):

J[α_, β_] = B[β, α] - Limit[B[β, ϑ], ϑ -> π/2 - β, Direction -> "FromAbove"]

(lengthy output)

Here's a numerical example to suggest that this result gives the same answer as a numeric integration and as your result formula:

With[{α = 1.3, β = 0.3},  {J[α, β],    NIntegrate[2 ArcCos[Cot[β] Cot[ϑ]] Sin[ϑ], {ϑ, π/2 - β, α}],π (1 - Cos[α]) + 2 (Cos[α] ArcSin[Cot[β] Cot[α]] - ArcTan[Cos[β]/Sqrt[Sin[β]^2 - Cos[α]^2]])}]

{0.0170522 - 6.66134*10^-16 I, 0.0170522, 0.0170522}