NEW YORK, March 12 (Reuters) - In the wake of revelations that the New York City Police Department secretly monitored Muslim communities, civil rights organizations are questioning the legality of the anti-terrorism operation.

While U.S. Attorney General Eric Holder said Thursday that the Justice Department is "reviewing" concerns about the program, legal experts say a court challenge based on claims of racial or religious profiling would face high hurdles.

Instead, they say, any case likely would come down to a single paragraph in a longstanding court order that governs the NYPD's surveillance of political activity.

The key paragraph, part of the so-called "Handschu guidelines," sets conditions for NYPD officers who visit public places or events during anti-terrorism investigations. It prohibits them from keeping records of their observations unless the information is related to "potential unlawful activity" -- a ban that critics say the NYPD has ignored.

According to reports in the Associated Press, the NYPD kept tabs on Muslim neighborhoods in New York City and surrounding areas by sending undercover officers into mosques, meetings of college campus groups and local businesses and keeping records of what they found.

"There's a very strong suggestion that they are going into the Muslim community, chatting up folks and maintaining records on individuals, which we think runs afoul of the Handschu decree," said Arthur Eisenberg, legal director of the New York Civil Liberties Union and one of the lawyers for the original plaintiffs that brought the Handschu case.

City and police officials have defended the practices as legal.

"Anyone who intimates that it is unlawful for the police department to search online, visit public places, or map neighborhoods has either not read, misunderstood, or intentionally obfuscated the meaning of the Handschu guidelines," said Police Commissioner Raymond Kelly during a recent speech at Fordham Law School.

DECADES-OLD LITIGATION

The Handschu guidelines were established by a Manhattan federal judge in 1985, amid a long-running court battle between a group of political activists and the NYPD.

The original case, Handschu v. Special Services, was filed in 1971 by a coalition of political groups in response to widespread police surveillance during the turmoil of the 1960s. Barbara Handschu, the lead plaintiff, is a lawyer who represented the Young Lords of Harlem and the Black Panthers, among other groups.

In 2003, U.S. District Judge Charles Haight issued modified guidelines after the city asked him to lessen the restrictions to facilitate anti-terrorism efforts in the wake of the Sept. 11, 2001, attacks.

The revised order specifically addressed how the NYPD could conduct anti-terrorism investigations. It included the critical provision that limits the records police officers can keep of their observations in public places.

The order allows the plaintiffs to return to court if they believe the guidelines have been violated.

Last fall, after the first news reports on the surveillance operation appeared, lawyers for the plaintiffs filed a motion in Manhattan federal court seeking discovery of NYPD files related to the program. In court papers, they claimed the surveillance appeared to violate the Handschu guidelines because reports suggested the NYPD kept dossiers on law-abiding individuals.

Lawyers for the plaintiffs are negotiating the terms of document production with city attorneys.

Police officials point to a section of Handschu that authorizes the NYPD to "prepare general reports and assessments...for purposes of strategic or operational planning."

City lawyer Peter Farrell, who represents the city in the Handschu case, said in an email that the guidelines "make clear that the NYPD may visit public places and go online -- just as the general public does." He did not specifically comment on whether the NYPD could maintain records of their observations.

Farrell also said the Handschu guidelines apply only to "political activity," and suggested that certain NYPD actions described in news reports -- visiting local businesses, for instance -- could be outside Handschu jurisdiction altogether.

This is not the first time that civil rights groups have cried foul with respect to Handschu compliance. Critics raised concerns about the NYPD's surveillance during the 2004 Republican National Convention and the Occupy Wall Street protests, for instance.

REPORTS 'DISTURBING'

While the Handschu guidelines provide the most likely basis for challenging the police surveillance of Muslims in court, other potential avenues include claims that the NYPD violated a New York law prohibiting racial profiling, as well as the U.S. Constitution. But those may prove difficult to substantiate.

A state statute prohibits police from using race, religion or ethnicity as the "determinative factor in initiating law enforcement action against an individual."

Police and city officials have said their actions were prompted by legitimate leads, not merely by religious identity.

Faiza Patel, co-director of the Liberty and National Security Program at NYU's Brennan Center for Justice, said it may be challenging to demonstrate that the NYPD used religion as a "determinative" factor, rather than simply as one of many.

Constitutional claims also would face an uphill climb.

If police have a legitimate law enforcement purpose, the U.S. Constitution does not preclude monitoring a specific religious community, said Stephen Schulhofer, a professor at New York University School of Law and a criminal justice expert.

"The reality is that much of what the ordinary person would consider racial profiling is not legally prohibited by the Constitution," he said.

However, Schulhofer and other experts said that if police actions produced a chilling effect on free speech or religious practices -- much as the Handschu plaintiffs argued in the 1970s -- that could be a violation of the First Amendment.

In addition, if the targeting creates a "stigma" for Muslims, that could violate the equal protection clause of the 14th Amendment, experts said.

Omar Mohammedi, president of the Association of Muslim American Lawyers, has called on New York Attorney General Eric Schneiderman to investigate whether the NYPD violated Muslims' constitutional rights, including the First Amendment.

"We can show that people are not attending prayers because they are afraid," he said. Schneiderman has not said whether his office will look into the operation.

Meanwhile, U.S. Attorney General Eric Holder said Thursday that the Justice Department was reviewing concerns raised by civil rights groups about the surveillance.

If the Justice Department's Civil Rights Division launches a full investigation, it could bring a lawsuit against the NYPD or reach an agreement to change city policy.

"There are various components within the Justice Department that are actively looking at these matters," Holder said. "I think at least what I've read publicly, again, just what I've read in the newspapers, is disturbing."

(Reporting by Joseph Ax; additional reporting by Joan Gralla, Edith Honan and James Vicini)

Follow us on Twitter: @ReutersLegal