An ancient shark that lived 300 million years ago was likely the most dangerous predator in its marine environment, and it looks like its fearsomeness wasn't just reserved for prey from other species.

A new study suggests that Orthacanthus sharks turned to cannibalism when food became scarce, with evidence of baby shark teeth being found in poop fossils unearthed from an old Canadian coalfield.

"There is already evidence from fossilised stomach contents that ancient sharks like Orthacanthus preyed on amphibians and other fish, but this is the first evidence that these sharks also ate the young of their own species," said palaeontologist Aodhán Ó Gogáin from Trinity College Dublin in Ireland.

Finding juvenile teeth in ancient shark poop wouldn't have been enough on its own to suggest that Orthacanthus resorted to cannibalism when other food sources ran low. After all, it's possible that some other carnivorous marine species was feeding on the baby sharks, and excreted the undigested teeth in their droppings.

But what most likely settles the identification issue is the shape of the fossil poop that the researchers found.

Orthacanthus was a 3-metre-long shark with a dorsal spine, an eel-like body, and what's known as tricuspid teeth – a crown with three cusps.

It also had a distinctive corkscrew-shaped rectum, and the coprolites (fossil poop) that the researchers discovered were indeed spiral-shaped – making it hard to avoid the grisly conclusion that these sharks ate their own young (a practice known as filial cannibalism).

182220377-Orthacanthus-2Aodhán Ó Gogáin/Trinity College Dublin

In the image above, you can see a thin section of the Orthacanthus poop fossil the researchers found in the Minto Coalfield of New Brunswick, Canada, with the black box highlighting the juvenile shark teeth in the coprolite.

"We don't know why Orthacanthus resorted to eating its own young. However, the Carboniferous Period [roughly 360 to 300 million years ago] was a time when marine fishes were starting to colonise freshwater swamps in large numbers," said one of the researchers, Howard Falcon-Lang from Royal Holloway, University of London in the UK.

"It's possible that Orthacanthus used inland waterways as protected nurseries to rear its babies, but then consumed them as food when other resources became scarce," he added.

In any case, eating your own young is a relatively rare event in the animal world, for obvious enough reasons.

"There's cannibalism and then there's specifically filial cannibalism. And that is relatively unusual," Falcon-Lang told Jonathan Webb at BBC News. "We generally find it in rather stressed ecosystems, where for whatever reason, food is running scarce. Obviously, it's evolutionarily a bad move to eat your own young unless you absolutely have to."

Scientists think that Orthacanthus sharks hunted in a range of shallow waters toward the end of the Carboniferous Period, so long ago that Europe and North America lay on the equator. Back then, the species was equally at home in the ocean and the swamps of the steamy jungle environments, called 'Coal Forests' – as the remnants have now become compacted into coal seams over millennia.

"Orthacanthus was probably a bit like the modern day bull shark, in that it was able to migrate backwards and forwards between coastal swamps and shallow seas," said Ó Gogáin in a press release. "This unusual ecological adaptation may have played an important role in the colonisation of inland freshwater environments."

The findings are reported in Palaeontology.