The world's biggest brokerage said board member Alberto Cribiore will serve as interim non-executive chairman.
Mr Cribiore, founder of private equity firm Brera Capital, will chair a search committee to find a permanent successor to Mr O'Neal. The committee will look inside and outside the company.
Mr O'Neal's tenure comes to an end after he misjudged the company's exposure to risky subprime loans, triggering an $US8.4 billion ($9.4 billion) write-down in the third quarter.
Merrill posted a net loss of $2.5 billion for the quarter, much worse than Mr O'Neal had forecast.
Mr O'Neal, 56, became CEO of Merrill Lynch in 2002, but he lost the confidence of the board after the massive write-down.
The company said in a statement that Mr O'Neal and the board had agreed that a change in leadership would best enable Merrill Lynch to move forward.
Mr O'Neal, the first African-American to run a major Wall Street company, also hurt his cause when he floated a merger with US bank Wachovia Corp without the board's knowledge, the New York Times reported on Friday.
Mr Cribiore said in a statement that Merrill co-presidents Ahmass Fakahany and Greg Fleming, a star investment banker, will continue to serve at the company. They are also co-chief operating officers. © 2007 Australian Broadcasting Corporation