By Neha Dimri and Tanya Agrawal and Aruna Viswanatha

Nov 9 (Reuters) - Payment transfer company MoneyGram International Inc agreed to forfeit $100 million and admitted it aided in wire fraud and failed to maintain an effective anti-money laundering program, according to court documents filed on Friday.

Between 2003 and 2009, the U.S. Justice Department said, the company processed thousands of transactions for its own agents who often tricked victims into wiring them money by posing as relatives or promising large cash prizes.

MoneyGram received thousands of complaints by customers who were victims of the fraud, but the company failed to terminate the agents that it knew were involved, the Justice Department said.

The company also admitted it failed to maintain an effective anti-money laundering program in violation of the Bank Secrecy Act (BSA).

The case is one of the first big cases from a new crackdown against money laundering by banks and other financial institutions from the department's Money Laundering and Bank Integrity unit. While money-laundering charges have more often been brought as tag-along charges to other types of cases in the past, the new unit has been focused on the institutions that are required to take steps to catch and combat the potential laundering of funds.

Justice Department officials told Reuters in August they planned to bring more cases under the BSA against a broader spectrum of financial institutions than the department had in the past.

Earlier this week, HSBC Holdings, Europe's biggest bank, said it had set aside $1.5 billion to settle expected charges that it violated U.S. anti-money laundering laws.

TIPPED TO A LOSS

MoneyGram agreed to retain a monitor and create an independent compliance and ethics committee on its board, and the Justice Department agreed to defer and drop the charges if the company complied with terms of the settlement.

Customers reported fraud that added up to at least $100 million, the Justice Department said, and the money from the settlement will be used to compensate the victims.

The settlement tipped the company to a third-quarter loss of $54.8 million.

"Since 2009, we've created a new culture at the company and have taken numerous steps to enhance our global compliance and anti-fraud programs," Chief Executive Pamela Patsley said in a statement.

The company had already set aside $30 million in the second quarter related to the settlement and took an additional $70 million charge in the third quarter.

MoneyGram, which has 284,000 global money transfer agent locations in 196 countries and territories, said it has created two new executive-level positions to combat fraud.

MoneyGram customers use its third-party agents among other methods to transfer money across the world.

Last week, Western Union Co, the world's largest payment transfer company, cut its full-year forecast as rising competition and weak markets weighed on its business.

MoneyGram's quarterly loss of 77 cents per share compares with a profit of 22 cents per share a year earlier, when the company reported earnings of $15.8 million.

Revenue rose 5 percent to $338.6 million.

MoneyGram shares, which have lost 22 percent this year up to Thursday's close, were up 1.7 percent to $14.02 in Friday afternoon trading.

Follow us on Twitter @ReutersLegal | Like us on Facebook