The ruling by a three-judge panel of the 9th Circuit Court of Appeals in San Francisco orders the National Oceanic and Atmospheric Administration and the National Marine Fisheries Service to prepare a full environmental impact statement.
The statement would be a more exhaustive review of the hunts' effect than the much shorter environmental assessment.
The appeals panel said the Commerce Department agencies failed to comply with the National Environmental Policy Act when their assessment determined there was no significant impact.
"Having reviewed the environmental assessment prepared by the government agencies and the administrative record, we conclude that there are substantial questions remaining as to whether the tribe's whaling plans will have a significant effect on the environment," said Judge Marsha S. Berzon, writing for the panel.
The court also said the whale hunts violate the Marine Mammal Protection Act, which requires that any marine mammal whose numbers have fallen 60 percent be designated as depleted.
The threshold for protection under the MMPA is less than that required for listing as threatened or endangered under the Endangered Species Act, which has stricter requirements for species protection.
This is the second time an environmental assessment by the fisheries service has been rejected by the appeals court.
In 1999, a circuit panel found the agency violated the environmental policy act when it approved a hunt-management plan before completing its study of the hunt's impact.
"Clearly, we're disappointed," said NMFS spokesman Brian Gorman in Seattle, adding that the agency would likely follow through with the court's order.
The agencies have 90 days to file an appeal with the U.S. Supreme Court.
The tribe's lawyer, John Arum, said he had not completed reading the court's opinion and could not immediately comment on the decision.
Calls to Makah tribal Chairman Nathan Tyler were not immediately returned.
Friday's decision was especially important for animal welfare groups who had appealed an August ruling by U.S. District Court Judge Franklin D. Burgess in Tacoma that they failed to prove the agencies' assessment of the hunts' impacts was arbitrary or capricious.
"We are extremely excited about the courts decision," said Michael Markarian, president of the Fund for Animals. "This is the second time the court has told the U.S. government that they failed to prepare an adequate environmental study of the Makah whale hunt."
The lawsuit filed by the New York-based Fund for Animals, the Washington, D.C.-based Humane Society of the United States and others contends that Makah whaling would endanger public safety. It also contends whaling would harm resident gray whales - those which linger in Washington's Strait of Juan de Fuca while the bulk of the population migrates between Alaska and Mexico.
Makah whaling rights are guaranteed by their 1855 treaty.
After a seven-decade break, the triben at the tip of Washington's Olympic Peninsula moved to resume whaling after gray whales were removed from the Endangered Species List in 1994. Tribal whalers have killed one whale, in May 1999.
In May, the International Whaling Commission authorized the tribe to continue whaling, allowing a take of up to four gray whales annually for five years.