We might have to add a brand new category of star to the textbooks: an advanced mathematical model has revealed a certain ultracompact star configuration could in fact exist, when scientists had previously thought it impossible.

This model mixes the repulsive effect of quantum vacuum polarisation – the idea that a vacuum isn't actually empty but is filled with quantum energy and particles – with the attractive principles of general relativity.

The calculations are the work of Raúl Carballo-Rubio from the International School for Advanced Studies in Italy, and describe a hypothesis where a massive star doesn't follow the usual instructions laid down by astrophysics.

"The novelty in this analysis is that, for the first time, all these ingredients have been assembled together in a fully consistent model," says Carballo-Rubio.

"Moreover, it has been shown that there exist new stellar configurations, and that these can be described in a surprisingly simple manner."

Due to the push and pull of gigantic forces, massive stars collapse under their own weight when they run out of fuel to burn. They then either explode as supernovae and become neutron stars, or collapse completely into a black hole, depending on their mass.

There's a particular mass threshold at which the dying star goes one way or another.

But what if extra quantum mechanical forces were at play? That's the question Carballo-Rubio is asking, and he suggests the rules of quantum mechanics would create a different set of thresholds or equilibriums at the end of a massive star's life.

Thanks to quantum vacuum polarisation, we'd be left with something that would look like a black hole while behaving differently, according to the new model. These new types of stars have been dubbed "semiclassical relativistic stars" because they the result of both classical and quantum physics.

One of the differences would be that the star would be horizonless – like another theoretical star made possible by quantum physics, the gravastar. There wouldn't be the same 'point of no return' for light and matter as there is around a black hole.

The next step is to see if we can actually spot any of them – or rather spot any of the ripples they create through the rest of space. One possibility is that these strange types of stars wouldn't exist for very long at all.

"It is not clear yet whether these configurations can be dynamically realised in astrophysical scenarios, or how long would they last if this is the case," says Carballo-Rubio.

Interest in this field of astrophysics has been boosted by the progress scientists have been making in detecting gravitational waves, and it's because of that work that it might be possible to find these variations on black holes.

The observatories and instruments coming online in the next few years will give scientists the chance to put this intriguing hypothesis to the test.

"If there are very dense and ultracompact stars in the Universe, similar to black holes but with no horizons, it should be possible to detect them in the next decades," says Carballo-Rubio.

The research has been published in Physical Review Letters.