By Steve Slater and Matt Scuffham and Aruna Viswanatha

LONDON/WASHINGTON, Nov 5 (Reuters) - A U.S. fine for violating federal anti-money laundering laws could cost HSBC Holdings significantly more than $1.5 billion and is likely to lead to criminal charges as well, Europe's biggest bank said on Monday.

HSBC said the U.S. investigation had damaged the bank's reputation and forced it to set aside a further $800 million to cover a potential fine for breaches in anti-money laundering controls in Mexico and other violations. The provision was on top of $700 million it put aside in July

"It could be significantly higher," Chief Executive Stuart Gulliver told reporters on a conference call, saying the latest provision was based on discussions with the various U.S. authorities involved in the probe.

The timing of any settlement is in the hands of regulators and is likely to involve the filing of corporate criminal and civil charges, the bank said.

Any settlement with U.S. prosecutors is likely to come in the form of a deferred prosecution agreement, which would allow the company to avoid a formal indictment. Deferred prosecutions have become an increasingly popular way for federal prosecutors to penalize companies without running the risk of forcing them out of business by indicting them.

A U.S. Senate report in July criticised HSBC for letting clients shift potentially illicit funds from countries such as Mexico, Iran, the Cayman Islands, Saudi Arabia and Syria. HSBC had warned earlier in the year it could face criminal or civil charges as part of the investigation.

The London-based bank has said the issue was "shameful and embarrassing" after a report by Congress' Permanent Subcommittee on Investigations criticised a "pervasively polluted" culture and said HSBC's Mexican operations had moved $7 billion into its U.S. operations between 2007 and 2008. Volume that large suggested it included illegal narcotics proceeds, the report said.

"The report undoubtedly caused considerable reputational damage to HSBC. The extent to which that has resulted in loss of business is hard to measure, but it has undoubtedly damaged our brand," Gulliver said.

He said a number of staff had left the firm as a result of the investigation and a number had their pay clawed back.

HSBC shares closed down 1.3 percent at 618 pence, slightly weaker than a fall in the European bank index.

"The money laundering provision is a concern, particularly given the uncertainty on what the final figure might be," said Richard Hunter, head of equities at stockbroker Hargreaves Lansdown.

The U.S. usually resolves corporate criminal cases by imposing fines and requiring changes to a compliance program but dismisses the charges if all requirements are met. The Justice Department has entered into 27 such agreements so far this year, according to statistics compiled by the law firm Gibson, Dunn & Crutcher.

HSBC said a deferred agreement was not certain, however, and authorities have "substantial discretion" to do what they want.

The size of the fine expected by HSBC also dwarfs every other similar case, including the previous record set by ING Bank N.V., which agreed in June to forfeit $619 million to resolve allegations that it illegally moved money on behalf of sanctioned entities in Cuba and Iran.

HSBC's problems involve not just sanctions issues but also major apparent lapses in money laundering controls, as stated in the Senate subcommittee report.

"What it reflects is the gravity of the wrongdoing that HSBC is at least implicitly admitting," said Jimmy Gurule, a former top Treasury official who is now a law professor at the University of Notre Dame. "This case has multiple dimensions and layers of wrongdoing, it transcends economic sanctions."

Reuters detailed in a July Special Report how money-laundering lapses continue to persist at HSBC, with managers allegedly focused on clearing paperwork fast rather than uncovering risks, highlighting the extent of problems at the bank.

It is unclear whether other pending inquiries into money-laundering issues including at JPMorgan Chase & Co have the potential to be similar in scope, but U.S. banks have generally avoided some of the larger penalties imposed on foreign banks for sanctions or money laundering violations.

The issue is another blow for the reputation of British banks after rival Barclays was fined $450 million in June for rigging Libor interest rates. The industry has also had to set aside more than 12 billion pounds to compensate UK customers for mis-selling insurance products.

Gulliver said it would take time to clean up the mess.

"There's a whole series of things that came from probably a decade in the 2000 to 2008-09 period that have surfaced now that the industry needs to sort out, remediate, and make sure doesn't happen again.

"It will take a chunk of time to clean the system and then it will take a little bit longer than that for trust to be restored more fully," he said.

COST CUTS, JOB CUTS

HSBC reported an underlying profit - after stripping out the impact of disposals and changes in the value of its own debt - in the July-September quarter of $5.0 billion, up from a revised $2.2 billion a year earlier.

It was helped by a bigger-than-expected drop in losses from bad debts and a solid performance by its investment bank arm.

Underlying operating expenses rose by 16 percent in the quarter from a year ago due to higher compliance and regulatory costs, which it said amounted to $200 million to $300 million.

Gulliver is well into a three-year restructuring plan to streamline the bank and he said he expects to surpass his target of cutting annual costs by $3.5 billion, after driving through $3.1 billion of savings already.

But subdued revenue growth and the higher compliance costs left its underlying cost/income ratio at 63.7 percent in the third quarter, well above his 48-52 percent target. Gulliver said hitting that was "proving challenging" but he remained committed to delivering it by the end of 2013.

Return on equity, a key measure of profitability, dropped to 5.8 percent in the third quarter, down from 14.6 percent in the previous quarter and well below Gulliver's 12-15 percent target.

HSBC took another $357 million charge for mis-selling payment protection insurance in Britain, lifting the total amount set aside to $2.1 billion. The bank said it paid out $1 billion in compensation.

Gulliver said more job cuts were likely before the end of 2013 at his bank. Over the past two years, HSBC has cut almost 30,000 jobs. 

(Additional reporting by Sarah White)

Follow us on Twitter @ReutersLegal | Like us on Facebook