The investigation started by considering an exactly rank-deficient problem, i.e., a problem with a dependency among the parameters. It turns out that such a problem can be formulated as a nonlinear minimum norm problem. To solve this optimization problem two regularization methods are proposed: A Gauss-Newton Tikhonov regularized method and a minimum norm Gauss-Newton method. It is shown that both regularization methods converge to a minimum norm solution of the nonlinear problem. Its convergence rate depends solely on curvatures in the function space and in the parameter space.
When turning to the almost rank-deficient case, we face some problems that are completely different from those in the exactly rank-deficiency case. Instead of solving an optimization problem that has a well-defined solution, we now deal with a regularization problem. Our numerical approach is based on linear algebra and nonlinear optimization techniques that enables us to develop a convergence theory based on second order information for Gauss-Newton-like methods applied to truncated and Tikhonov regularized problems.
The determination of the weights in the training of artificial feed-forward neural networks gives very ill-conditioned nonlinear least squares problems for which Tikhonov regularization is often suggested. For small and medium size problems the Gauss-Newton method is applied explicitly to the Tikhonov regularized problem. It is shown that the proposed method exhibits far better theoretical properties than a Levenberg-Marquardt method. This is confirmed by numerical results. For large-scale problems we propose implementations using automatic differentiation combined with a conjugated gradient method for the approximation of the Gauss-Newton direction.
Quasi-Newton methods are important for nonlinear least squares problems where the nonlinearities dominate. We propose methods that exclusively update the nonlinearities. Using this approach certain very nonlinear and also ill-conditioned problems can be solved efficiently.
A curvilinear search is proposed. It is intended for nonlinear systems of equations, but is also useful for nonlinear least squares as well if the curvatures orthogonal to the tangent plane are small.
The following papers are part of the thesis: