Eight people were rescued about 40 minutes after the crash of the C-2 "Greyhound" transport aircraft on Wednesday afternoon, the Navy said.
They were taken aboard the USS Ronald Reagan aircraft carrier and were in good condition.
The C-2A twin-propeller plane came down about 925km southeast of Okinawa as it was bringing passengers and cargo from Japan to the USS Ronald Reagan aircraft carrier, the Navy said in a statement.
The cause wasn't clear but the crash would be investigated, the Navy said.
The Reagan was participating in a joint exercise with Japan's Maritime Self-Defence Force when the plane crashed.
The aircraft carrier was leading the search and rescue efforts along with Japan's naval forces.
The ships and aircraft had searched more than 320 nautical miles as of Thursday morning, the Navy said.
Japan's Defence Ministry said the crash site is about 150km northwest of Okinotorishima, a Japanese atoll.
The Navy said it had notified next of kin that the three sailors were "whereabouts unknown" but it would delay releasing their identities publicly for three days due to policy.
In Washington, the White House said President Donald Trump had been briefed on the crash. Trump said in a tweet: "We are monitoring the situation. Prayers for all involved."
The @USNavy is conducting search and rescue following aircraft crash. We are monitoring the situation. Prayers for all involved. https://t.co/1AxmegKqvu
- Donald J. Trump (@realDonaldTrump) November 22, 2017
The November 16-26 joint exercise in waters off Okinawa has been described by the Navy as the "premier training event" between the US and Japanese navies, designed to increase defensive readiness and interoperability in air and sea operations.
The Navy's Japan-based 7th Fleet has had two fatal accidents in Asian waters this year, leaving 17 sailors dead and prompting the removal of eight top Navy officers from their posts, including the 7th Fleet commander.
The USS John S McCain and an oil tanker collided near Singapore in August, leaving 10 US sailors dead.
Seven sailors died in June when the USS Fitzgerald and a container ship collided off Japan.
The Navy has concluded that the collisions were avoidable and resulted from widespread failures by the crews and commanders, who didn't quickly recognise and respond to unfolding emergencies.
A Navy report recommended numerous changes to address the problems, ranging from improved training to increasing sleep and stress management for sailors.