By Matt Scuffham and Kirstin Ridley

LONDON, Feb 6 (Reuters) - Royal Bank of Scotland will pay $612 million to U.S. and British authorities to settle allegations it manipulated benchmark interest rates, and regulators warned there is more to come in the global investigation.

RBS became the third bank to pay fines in the Libor scandal.

The British bank, which is 82 percent-owned by the state after the world's costliest bank bailout in 2008, said on Wednesday it was cutting bonuses to help pay for the fine, in a bid to avoid a public backlash.

The bank fears the scandal will embolden critics who want it to further shrink its profitable investment bank and focus on basic lending at home.

"What happened at RBS and other banks is totally unacceptable," Britain's finance minister, George Osborne, told reporters.

Britain's Financial Services Authority (FSA) signalled more large fines were in the offing.

"The size and scale of our continuing investigations remains significant," said Tracey McDermott, director of enforcement and financial crime at the FSA.

More than a dozen banks and brokerage firms, including JP Morgan, Deutsche Bank AG and Citigroup Inc , are being investigated by regulators over the manipulation of benchmark interest rates such as the London interbank offered rate, known as Libor, and Euribor, which are used to price trillions of dollars' worth of loans.

For their roles in the Libor scandal, Switzerland's UBS AG agreed in December to pay penalties of $1.5 billion, and Britain's Barclays Plc has paid $453 million.

Deutsche Bank has suspended five traders in connection with alleged manipulation of Euribor, a source familiar with the matter said on Wednesday.

Recent reports have raised the possibility of other banks resolving liability through a group settlement, but in an interview on Wednesday, a top U.S. Justice Department official shot down that idea.

"Criminal cases are not resolved in a group setting," said Lanny Breuer, the head of the department's criminal division. "We are going to go after each individual financial institution."

PURE MANIPULATION

Investigators said they discovered hundreds of attempts by at least 21 RBS employees in London, Singapore and Tokyo to manipulate Libor. RBS traders aided dealers at other banks, including UBS, to rig the rates.

The abuse at RBS occurred from at least 2006 until late 2010 - after some of the traders learned of the probe into Libor.

The FSA criticised RBS for seating derivatives traders next to people who submitted Libor rates and said the bank's systems and controls were flawed as recently as March 2012.

Like their peers at Barclays and UBS, RBS staff were blatant about what they were doing in internal chatrooms, according to extracts of exchanges released by investigators.

A Swiss-franc trader at RBS told someone submitting rates to Libor that if he submitted them in the way he wanted, he would "come over there and make love to you."

A manager said "pure manipulation" was at work.

The U.S. Department of Justice said the near 300-year-old bank was guilty of a "stunning abuse of trust".

RBS is paying 87.5 million pounds ($137 million) to the FSA, $150 million to the U.S. Department of Justice and $325 million to the U.S. Commodity Futures Trading Commission, which regulates trade in derivatives.

A unit of the bank in Japan also pleaded guilty in the United States to one count of criminal wire fraud.

The parent company avoided criminal liability in the United States, meaning it can retain its banking licence there and avoid a fire sale of its U.S. business, Citizens Bank.

As part of its deferred prosecution agreement, the bank was forced to admit and accept responsibility for its misconduct.

In a 41-page statement of facts, for example, the bank admitted its employees treated requests for favourable submissions in a routine, casual manner.

In one colourful example, a trader in an electronic chat in 2009 asked for a lower submission, then told the submitter he was "like a whores drawers" in acknowledging he often passed on requests for interest rates that went up and down.

Lawyers expect a wave of civil lawsuits, potentially costing banks tens of billions of dollars. It is unclear how many individuals will be prosecuted.

U.S. prosecutors have filed criminal charges against two former employees of UBS. Days before the Swiss bank was fined, British police and anti-fraud officers arrested one of the traders later charged in the United States along with two employees of British brokerage firm RP Martin.

The U.S. Justice Department's Breuer said such inquiries continued. "As of today there are no individuals, but as you can tell, and we've demonstrated time again, this is an ongoing investigation," he said.

A SOAP OPERA

RBS Chief Executive Stephen Hester wants to re-establish the bank as a "normal" lender, shrinking the balance sheet by 700 billion pounds and cutting thousands of jobs as he jettisons the previous management's ambition for global domination.

Speaking to reporters in London, a visibly emotional Hester failed on three occasions to give a direct answer to questions on whether he had considered resigning over the Libor scandal.

"This has been a soap opera for four years," he said. "The people who sit in judgment of us can dismiss us at any time if they feel that the bad bits get to be bigger than the good bits."

Taxpayers are sitting on a loss of close to 16 billion pounds on their RBS stake, frustrating politicians.

Britain's influential parliamentary commission on banking standards has summoned Hester and RBS's chairman, Philip Hampton, to discuss the Libor scandal and the future of the bank on Monday.

RBS said all but six of the 21 staff implicated had either been fired or had already left the bank. The remainder were being disciplined.

RBS will cut 300 million pounds from its bonus pool, including clawing back awards from previous years, to pay the U.S. fines. The UK penalty will be donated to charitable causes, including supporting soldiers and their families, the government said.

John Hourican, head of RBS's investment bank, is leaving at the end of April after it was discovered the manipulation went on after he took charge. He had no involvement in, or knowledge of, the misconduct, RBS said. Hourican will receive a year's salary but forgo share awards.

"Libor is the railroad tracks on which our banking system runs," Laura Willoughby, chief executive of consumer group Move Your Money, said of the rate rigging. "RBS and other banks have shattered trust in the very foundations of our financial system."

RBS shares finished up 1.36 percent at 342.1 pence on Wednesday. 

($1 = 0.6382 British pounds) 

(Additional reporting by Laura Noonan, Andrew Osborn, Tim Castle, Myles Neligan, Douwe Miedema and Aruna Viswanatha)

Follow us on Twitter @ReutersLegal | Like us on Facebook