Being a baby is a whole lot of fun, for the most part – we get waited on hand and foot, we spend our days eating and sleeping, and we don't have to worry about any responsibilities.

But why can't we remember anything of these early years once we grow up? It's a problem that's been puzzling experts for years.

More than a century ago, world-renowned psychotherapist Sigmund Freud coined the phrase "infantile amnesia" as he tried to figure out what was going on in our minds at this tender age.

And considering how formative those years are, you would expect more memories to stick. We experience the world for the first time, we learn how to walk and talk (so some information must be retained), and yet most of us can't remember anything before the age of about three or four.

There's no fixed, widely accepted answer to the question, but as Zaria Gorvett reports at BBC Future, numerous studies show that the time of our earliest recollections can vary from person to person and country to country.

Those cultural differences could help explain some of the mystery. In a study of 256 people from the US and China, those from the States had memories that were more elaborate and more personal, whereas the participants from China had memories that were briefer and more factual.

On average the memories from the US study subjects started six months earlier, suggesting perhaps that the ways our individual cultures frame recollections and value the past may have a significant impact on childhood memories.

Then there's the role of speech, which some (but not all) psychologists think is important. "Language helps provide a structure, or organisation, for our memories, that is a narrative," Emory University psychologist Robyn Fivush told the BBC.

It may be that as babies we just don't have the necessary mental equipment to store and organise memories properly, a hypothesis strengthened by the famous case of Henry Molaison.

After a botched operation left him with a damaged hippocampus, he was unable to remember any new events that happened to him, though he still had temporary short-term memory and could learn new skills.

Sound like a baby to you? Molaison's unfortunate experience suggests infant brains have yet to develop a hippocampus that functions as it does in adults.

We know that neurons continue to be added to our brains in our early years, and it's possible that when this building process has finished, memories can start to form.

Another hypothesis is that our sense of self hasn't properly developed when we're babies, and because we don't really know who we are, we can't figure out how to identify relevant memories and file them away. Like the other possibilities, though, it's difficult to prove.

To complicate the issue, the experiences we have as babies obviously affect the type of kids we become, so it might be that some kind of memory of them floats around in our subconscious, even if we can't specifically remember the events themselves.

The search for an explanation continues, despite the challenges of collecting hard data. If only babies could talk.