The Barbarian players left their hotel bubble – despite their team protocols – to have dinner at a restaurant in London and another day to have a few drinks.
The RFU said that the players leaving the hotel has compromised the bubble environment.
“We are incredibly disappointed to be calling a halt to this fixture,” said RFU chief executive Bill Sweeney. “We know how much fans were looking forward to seeing the teams play.
“However, our priority is to protect the health and safety of the England squad and the other international teams they will go up against this autumn.”
A few among the 12 players have stood down from the fixture, along with a number of other Saracens players. Robshaw, Richard Wigglesworth, Sean Maitland and Jackson Wray are said to have stood down.
Robshaw expressed his “deepest apologies” for compromising the bubble.
“A huge effort went into conducting this match in a safe fashion and it was irresponsible of me to break the protocols which are put in place to protect players, staff and the public,” he said.
“I understand that my actions have ultimately contributed to the cancellation of Sunday’s match and I am sincerely remorseful for my role in undoing all the amazing work that went into trying to make it happen.
“I promise that I will learn from this mistake and ensure something like this never happens again.”