I have written an R package, grassopt, for optimization over Grassmann manifolds. A point in a Grassmann manifold (or Grassmannian) is a hyperplane of a given dimension in a higher-dimensional vector space. The Grassmann manifold Gr(p,n) consists of all p-hyperplanes in a n-dimensional vector space. The optimization method is an adaptation of Newton's method to Grassmann manifolds (Edelman, Arias and Smith 1998). A more accessible and detailed description of the method is here.

R package, source, manual.