Speaker John Bercow has described the abuse and harassment of MPs outside Parliament “a type of fascism” and called for a change of policing policy.
He said recent incidents, including Tory MP Anna Soubry being verbally abused on Monday, were “intolerable”.
More than 60 MPs have called on police to improve their response to abusive protesters outside Parliament.
The Metropolitan Police have said they are ready to “deal robustly” with any instances of criminal harassment.
Deputy Assistant Commissioner Laurence Taylor said the force was continuing to assess whether any crimes had been committed, following a third-party report of a public order offence on College Green, opposite the Houses of Parliament.
“Our role is to facilitate peaceful protest and balance the needs and rights of all those present, including protesters, MPs and members of the public,” he said.
Ms Soubry was shouted at – including being called a liar and a Nazi – during live TV interviews on BBC News and Sky.
The former minister was later called “scum” and jostled as she tried to re-enter the Palace of Westminster.
She criticised police for not intervening and called for the protesters to be prosecuted under public order laws.
College Green – the area opposite Parliament – is regularly used by media to interview politicians, as well as being a popular site for protesters to gather.
Ms Soubry told ITV’s Good Morning Britain programme on Tuesday that a small group was “roaming around Westminster intimidating people going about their lawful business”.
And although she anticipated some criticism and abuse as an MP, authorities should act when this “crossed the line”.
The cross-party group of over 60 MPs who have signed the letter – which includes those both for and against Brexit – said many of the concerns had been “repeatedly raised” with officers and senior policing staff.
“We write to express our serious concerns about the deteriorating public order and security situation in and around the exterior of the Parliamentary estate including College Green,” the letter, co-ordinated by Labour MP Stephen Doughty, read.
“After months of peaceful and calm protests by groups representing a range of political views on Brexit, an ugly element of individuals with strong far-right and extreme right connections – which your officers are well aware of – have increasingly engaged in intimidatory and potentially criminal acts targeting Members of Parliament, journalists, activists and members of the public.”
The letter said there appeared to be a “lack of co-ordination in the response from the police and appropriate authorities”.
It said that it was “utterly unacceptable for Members of Parliament, journalists, activists and members of the public to be subject to abuse, intimidation and threatening behaviour and indeed potentially serious offences while they go about their work”.
The abuse received by Ms Soubry was widely condemned.
Ms Soubry, MP for Broxtowe and supporter of a fresh Brexit referendum, said she objected to being called a Nazi.
Sky News presenter Kay Burley said the “increasingly vile, aggressive and intimidating” abuse had forced her to change her own route to Parliament and she now had to have security protection.
She told BBC Radio 5 Live she had been interviewed three times by the police about the situation, but the protesters knew their rights and what they could and couldn’t get away with.
But she added: “How far does it have to escalate before the police have to take it seriously?”
Labour’s Mary Creagh said the “really vile, misogynistic thuggery” that had been seen was not an isolated incident.
She pointed to the murder of MP Jo Cox, who was killed in her West Yorkshire constituency by right-wing extremist Thomas Mair in June 2016.
Commons Speaker John Bercow said he was “concerned” about a “pattern of protest” targeting female MPs and journalists and called it a “type of fascism”.
In his letter to the Met Police chief on Tuesday, he said he recognised it was “a difficult job striking the balance between allowing peaceful protests and intervening when things turn sour”.
But he added: “It’s one thing demonstrating from a distance with placards, or calling out slogans – and another, where the protester invades the personal space of a member, subjects him or her to a tirade of menacing, racist, sexist and misogynistic abuse, and follows them back to their place of work.”
Brexit Secretary Stephen Barclay said the “appalling scenes” outside Parliament, where Ms Soubry had been “disgracefully treated”, showed how divisive the Brexit process had been.
“There is a balance always to be struck between defending the right of freedom of speech and ensuring that there are boundaries around that,” he told BBC Radio 4’s Today programme.
Labour MP Jess Phillips, who has previously spoken out against online abuse, told the BBC’s Victoria Derbyshire programme that some protesters were “organised right-wing groups” trying to “scare our politicians into making decisions based on fringe views”.
“People deserve to be safe at work,” the Birmingham Yardley MP said. “I didn’t come here to be bullied by far-right bullies, far-left bullies, or anyone, we came here to do what we felt was best.”
Also on Monday, political commentator Owen Jones published a video on Twitter that he had recorded while being followed and shouted at by a group of protesters outside Parliament.
Last month, a video emerged of prominent Brexiteer Michael Gove being accosted by a protester dressed as Santa as he walked to Parliament.
Ms Soubry has been in touch with the parliamentary authorities responsible for security.
Mr Bercow said he was aware of protests in recent weeks around the Palace of Westminster “involving aggressive and threatening behaviour towards members by assorted groups that have donned the yellow vests seen in France” – a reference to last year’s “gilet jaune” anti-government demonstrations.
No 10 said the incident was “unacceptable” and MPs “should be free to do their jobs without any form of intimidation”.
A Downing Street spokesman said there were laws dealing with public order offences and cases of harassment and threatening behaviour.
The BBC and other broadcasters have set up temporary studios on College Green ahead of the Commons vote on Theresa May’s Brexit deal, expected on 15 January.
The BBC’s assistant political editor Norman Smith said some MPs had expressed unease privately about being interviewed there, given the frequency and vehemence of the protests.