Julie Bishop retained the deputy leadership.
Mr Dutton has now resigned his powerful position as Home Affairs minister and will move to the backbench, in keeping with the tradition of cabinet solidarity.
Dutton has consistently shied away from admitting his desire to take on Turnbull, with Liberal MP Christopher Pyne telling Sunrise host Kochie just hours before the party room voted, that Dutton remained loyal to the Liberal leader.
Neither Turnbull or Dutton have spoken yet following the vote.
Turnbull won the vote 48 to 35.
While Coalition marginal seat holders have publicly supported Mr Turnbull, some privately lobbied in favour of Mr Dutton to improve the party's chances in Queensland.
The leadership talk was fueled by an Ipsos poll published in Fairfax newspapers, which showed support for Turnbull's Liberal-National coalition falling four points to 45 percent, 10 points behind opposition party Labor and enough for a crushing electoral defeat.
Kane finally scores in August as Spurs beat Fulham
Pochettino watched his side beat Fulham 3-1 at Wembley to make it two wins out of two in the new Premier League season. Fulham lost to Crystal Palace 2-0.
Just seven of his Liberal Party colleagues would need to switch to end what is now an unviable leadership internally and by virtue of that, a dishonest product to be put before the electoral market place.
Small Business Minister Craig Laundy warns that would go down like a lead balloon.
Mr Abbott said: 'It's no way to run a government, making absolute commitments on Tuesday and breaking them on Friday'.
Australia has gone through an extraordinary period of political instability since Prime Minister John Howard lost power in 2007 after more than 11 years in office.
"I don't actually expect a challenge today, but we'll wait and see", he told the ABC on Tuesday.
Energy policy, Pauline Hanson and Tony Abbott are all contributing factors to Turnbull's woes, Professor Jennifer Curtin says.
Dutton also watered down speculation of a challenge.
Labor frontbencher Anthony Albanese said the prime minister should call an election, if he survives the week. Mr Abbott told reporters outside Parliament House on Monday evening.