EU foreign policy chief Josep Borrell said the ministers had agreed that relations with Moscow "cannot be business as usual" and the agreement should be "fully suspended".
Following the informal meeting in Prague, he said this would make it "more difficult" and "longer" for Russian nationals to get visas.
"It will significantly reduce the number of new visas issued by the EU member states," he said.
In a concession to eastern EU members who had pushed for a tougher approach, Borrell said that countries bordering Russia "can take measures at a national level to restrict entry into the European Union".
But he said any measures would have to conform with rules for the EU's Schengen common travel zone and he emphasised it was important that members of Russian civil society should continue to be able to travel to the EU.
Ahead of the meeting, Poland and the three Baltic states of Estonia, Latvia and Lithuania had said they were considering barring Russian travellers if the EU as a whole failed to do so.
In a joint statement obtained by AFP, the four countries had said that suspending the visa facilitation agreement was "a necessary first step".
"But we need (to) drastically limit the number of visas issued, above all tourist visas, to decrease the flow of Russian citizens into the European Union and the Schengen area," they added.
They said the move should contain exceptions "for dissidents as well as other humanitarian cases".
"Until such measures are in place on the EU level, we... will consider introducing on the national level temporary measures of visa ban, or restricting border crossing for Russian citizens holding EU visas," reads the statement written in English.
For an optimal experience, please
