This has been bugging me for a while now. I understand from this answer why gaussian kernels are effective. But I can't wrap my head around the intuition of why the infinite dimensional feature map 𝜙(𝑥) in the gaussian kernel is not subject to problems arising in this infinite dimensional space. I know that the "feature mapped" vector is never directly computed, but the inner product for example is evaluated and used as a measure of similarity between vectors.
But let's assume we are working with data sampled from a hypersphere, if in higher dimensions, the data becomes subject to problems like the concentration of measure phenomenon where everything shrinks to near zero volume and sampled points becomes randomly distributed/orthogonal on average, how would a gaussian kernel benefit in this setting since it would be taking the input data and mapping it to infinite dimensional space where in theory, all points should be uniformly close together around the center of the sphere and orthogonal on average, thus providing no insight for classification/clustering?
Or does the gaussian kernel map these points to a different geometrical structure in infinite dimensional Hilbert space? If so, why is this not then subject to other problems that arise due to the curse of dimensionality?