A yellow excavator pushed away concrete slabs to search though the wreckage of the 21-floor building in Lagos's wealthy Ikoyi residential and business district, AFP correspondents at the scene said.
Rescue officials said many workers were caught inside the building when it crumbled, though they could not confirm the number of people trapped inside.
Ibrahim Farinloye of the National Emergency Management Agency (NEMA) said four people had been rescued so far and four bodies recovered from the site.
Wisdom John, 28, a bricklayer, said he escaped with just a few cuts because he had been on the ground floor when the building collapsed into a pile of concrete, its floors sandwiched together.
"There was more than 50 working today and the manager too," he said, sitting in an ambulance getting treated. "We just ran out."
The Ikoyi area is one of the wealthier residential and business districts in Lagos, Nigeria's densely populated major commercial city.
Dozens of angry local residents and workers had gathered to help out soon after the collapse, many crying and voicing frustration over the slow pace of the rescue efforts.
Lagos State police commissioner Hakeem Olusegun Odumosu said it was still too early to determine the cause of the collapse.
"Many workers are trapped under the rubble," said Femi Oke-Osanyintolu, general manager of Lagos State emergency management agency.
Four other construction workers at the site told AFP dozens of their colleagues were inside when the building crumbled.