The 31-year-old man was taking part in anti-terrorism training in waters around the Garden Island Naval Base, near the Sydney Opera House, when the shark attacked.
"The attack occurred very quickly. Our diver punched the shark, I believe, a couple of times. The shark then disappeared very quickly," Australian Fleet Commander Rear Admiral Nigel Coates told reporters.
"It was all over, I'm told, in a few seconds."
There have been at least four other shark attacks in Australia this summer, one of them fatal, sparking a global media frenzy of "Jaws" proportions. Last month, there were three attacks in just two days.
In the latest case, the diver was swimming on the water surface when the attack happened. He was treated on a navy boat, before being rushed ashore and to a nearby hospital for surgery.
Coates said the man was in a serious condition.
"The patient said he saw something big and grey underneath him just before the attack," an ambulance spokesman said, adding the man had severe injuries to his right hand, buttocks and down to his knee on one leg.