Image morphology holds some useful answers. I'm currently working on a project trying to detect mountain ridges and valleys and have a few half-baked methods ready to go that will produce several useful results. The one below follows the method described in an old paper (DOI: 10.1016/j.geoderma.2003.10.008 ). I highly suggest you check out the topotoolbox as well, it has a great function called excesstopography that essentially removes large scale gradients from a DEM, making it easier to find valleys and ridges that run up or down a larger hill/mountain.
function [SCHv,SCHr,SCHhs,SCHc] = Valley_SCH(DEM,rad,thr)
%VALLEY_SCH finds valleys and ridges from a DEM
% Method follows that published by Schmidt and and Hewitt 2004
% ( DOI: 10.1016/j.geoderma.2003.10.008 )
%
% ---INPUT---
% DEM
% The digital elevation model
% rad
% The radius of the structural element
% thr
% The threshold for "noise removal"
%
% ---OUTPUT---
% SCHv
% Valley map
% SCHr
% Ridge map
% SCHhs
% "Hillslopes" map (flat areas and smooth areas)
% SCHc
% Combined map to show all four categories.
wb=waitbar(0,'Applying Schmidt method - Valleys...');
DEM(isnan(DEM))=0;
SE=strel('disk',rad,0);
VD=imbothat(DEM,SE);
VD=VD-thr;
VD(VD<0)=0;
waitbar(0.5,wb,'Applying Schmidt method - Ridges...')
HH=imtophat(DEM,SE);
HH=HH-thr;
HH(HH<0)=0;
waitbar(0.9,wb,'Applying Schmidt method - Finalising...')
%Valleys
SCHv=zeros(size(DEM));
SCHv(and(VD>0,HH==0))=1;
temp=VD./(HH+VD);
SCHv(and(VD>0,HH>0))=temp(and(VD>0,HH>0));
%Ridges
SCHr=zeros(size(DEM));
SCHr(and(HH>0,VD==0))=1;
temp=HH./(HH+VD);
SCHr(and(HH>0,VD>0))=temp(and(HH>0,VD>0));
%Hillslopes
SCHhs=zeros(size(DEM));
SCHhs(and(HH==0,VD==0))=1;
temp=2*min(HH,VD)./(HH+VD);
SCHhs(and(HH>0,VD>0))=temp(and(HH>0,VD>0));
%Combined
SCHc=int8(zeros(size(DEM)));
SCHc(and(VD>0,HH==0))=3;
SCHc(and(HH>0,VD>0))=2;
SCHc(and(HH==0,VD==0))=1;
SCHc(and(HH>0,VD==0))=0;
close(wb)
end
