I don't think it's the distance that's the problem, but how much screen space the object takes up. (Which is inherently the distance)

The more pixels an object takes up, the more pixels that need to be computed with that material. If the material is expensive to calculate, it will take longer to render. With SSS, each traced ray that hits the material scatters. This creates many more rays that it needs to calculate, so you kind of get an exponential complexity increase.