The killing could either push the conflict towards United Nations peace negotiations or make it an "even more vicious war", Mattis told reporters.
Yemen has been embroiled in civil war since early 2015 when Houthi rebels took over the capital of Sanaa and President Abed Rabbo Mansour Hadi fled to the southern city of Aden.
The intervention by Ahmed Ali, a former leader of the elite republican guard once seen as a likely successor to his father, gives the anti-Houthi movement a potential figurehead after a week of fighting that saw the Houthis rout Saleh's supporters in the capital.
Saleh's body, which had appeared in a video by the militias with a gaping head wound, was taken to a rebel-controlled military hospital. He was killed by the rebels on Monday, leaving his followers in disarray.
The Houthis' top leader, Abdul-Malek al-Houthi, said Saleh paid the price for his "treason", accusing him of betraying their alliance to side with the Saudi-led coalition. "He got what he deserved", Ali Akbar Velayati, an aide to Iran's supreme leader, was quoted as saying by the semi-official Tasnim news agency.
Speaking before a gathering of supporters, al-Houthi says: "We knew that they were deceived. we hadn't hoped for what happened".
Preserve net neutrality: Letter to the editor
A tweet that shows a supposed glimpse of a future without net neutrality is making the rounds online, but it isn't exactly true. Protests are scheduled Thursday in Madison and cities all across the nation to demonstrate public support for net neutrality.
Some 41 Yemeni journalists have been held captive by Houthi fighters at the television station in Sana'a, Al Jazeera reported today.
The killing of Yemen's ex-President Ali Abdullah Saleh by the country's Shiite rebels as their alliance crumbled has thrown the almost three-year civil war into unpredictable new chaos. The Arab League's general secretariat condemned the Iran-aligned Houthi movement which killed Saleh as a "terrorist organisation" and demanded that the global community view it as such.
It also shatters hopes by Yemen's Saudi-backed government that Saleh's recent split with the Iranian-backed rebels, known as Houthis, would have weakened them and given the government and the Saudi coalition backing a chance for a turning point in the stalemated war that has brought humanitarian disaster.
However, in late November, the tensions between the former allies escalated and resulted in clashes that have already claimed lives of hundreds of people, including the ex-president himself.
The Saleh-Huthi alliance had been fraught since its inception in 2014, when the two ended decades of enmity and joined ranks to capture Sanaa from Hadi's government.
At least 234 people have been killed and more than 400 wounded in fighting in Sanaa since the beginning of the month, the International Committee of the Red Cross (ICRC) said on Tuesday.