We have seen in recent years a need for regularization of complicated feature spaces: Vector fields, orientation fields, color perceptual spaces, the structure tensor and Diffusion Weighted Images (DWI) are few examples. In most cases we represent the feature space as a manifold. In the proposed formalism, the image is described as a section of a fiber bundle where the image domain is the base space and the feature space is the fiber. In some distinguished cases the feature space has algebraic structure as well. In the proposed framework we treat fibers which are compact Lie-group manifolds (e.g., O(N), SU(N)). We study here this case and show that the algebraic structure can help in defining a sensible regularization scheme. We solve the parameterization problem of compact manifold that is responsible for singularities anytime that one wishes to describe in one coordinate system a compact manifold. The proposed solution defines a coordinate-free diffusion process accompanied by an appropriate numerical scheme. We demonstrate this framework in an example of S1 feature space regularization which is known also as orientation diffusion.