WASHINGTON, March 22 (Reuters Legal) - A federal court has rejected a class action settlement between Google Inc, authors and publishers that would have paved the way for the search giant to begin selling the millions of out-of-print books it has scanned in recent years.

New York Judge Denny Chin said the deal gave Google a significant competitive advantage and "would simply go too far" in giving it the power to "exploit" digitized copyrighted works by selling subscriptions to them online without permission.

The proposed agreement would have paid every author at least $60 per title and permitted publishers and authors to split in sales revenue of online editions and related advertising.

Google has scanned at least 12 million books in what it has said was an effort to provide easier access to the world's knowledge. It is carrying out the scans in partnership with major libraries around the world, including the New York Public Library and Stanford Library.

It was sued in 2005 by the Authors Guild and Association of American Publishers for violating copyright laws, but reached a settlement by agreeing to pay $125 million to people whose copyrighted books have been scanned, and to locate and share revenue with the authors who have yet to come forward.

Still, some critics contended that the deal gave Google an unfair competitive advantage. Chin, who was promoted to the Second Circuit Court of Appeals while the Google case was before him in district court, agreed.

"The ASA (amended settlement agreement) would give Google a significant advantage over competitors, rewarding it for engaging in wholesale copying of copyrighted works without permission," Chin wrote in rejecting the deal.

He urged Google and authors and publishers to amend the settlement to only include books whose copyright owners have agreed to the arrangement, rather than a blanket deal that would require authors to "opt out" should they not want their scanned books to be sold online.

Speaking for the publishers, John Sargent, chief executive of Macmillan publishing group, said he hoped the deal could be renegotiated.

"The publisher plaintiffs are prepared to enter into a narrower settlement along those lines to take advantage of its groundbreaking opportunities. We hope the other parties will do so as well," said Sargent in an email statement.

Chin rejected the settlement "without prejudice," meaning a revised pact could be submitted.

The Authors Guild's President Scott Turow also noted Chin's push for an "opt-in" deal and said it hoped to talk to Google and the publishers group to "arrive at a settlement within the court's parameters that makes sense for all parties."

OTHER OPPOSITION ALSO REMAINS

Critics of the proposed settlement also include Amazon.com Inc., which sells the Kindle digital reader that would not be compatible with Google's library, and Microsoft Corp. Sony Corp which makes an e-reader compatible with Google's software, and advocates for the blind are among those who favor the pact.

The Justice Department has looked into the deal, including at a group of books described as "orphan works" because they are still in copyright but the rights holder cannot be found. The settlement gives Google the right to market these works while establishing a fiduciary organization to administer them. Proceeds from the orphan works would be held to allow time to locate the author, and eventually disbursed to literacy organizations.

The Justice Department was pleased with Chin's decision, said spokeswoman Gina Talamona, who said the settlement "created concerns regarding antitrust, class certification and copyright issues."

The French government had opposed the deal but reached an agreement with Google in early 2010 to allow French works to be scanned without surrendering control of copyright. The German government opposes the deal.

The ruling comes as Google Co-founder Larry Page is set to take the CEO reins in April from Eric Schmidt. Page is an early champion of digital books and personally scanned many of the works himself early on in the project.

Analysts said the setback would have little impact on Google's financials, since any revenue would have paled in comparison to Google's main Internet search business.

"This is clearly disappointing, but we'll review the court's decision and consider our options," Google's managing counsel, Hillary Ware said in an emailed statement.

Since the settlement, Google launched in December an electronic bookstore with three million books, with permission from the relevant publishers.

Under the agreement, authors and publishers would register works and be paid for books and other publications the search giant puts online. The settlement covered books that were out of print but still under copyright.

The case is The Authors Guild et al v. Google, Inc, U.S. District Court, Southern District of New York, No. 05-08136. (Reporting by Grant McCool and Diane Bartz of Reuters; Additional reporting by Jeff Roberts of Reuters Legal)