Five generations of Charlotte Moore's family have lived in the same house, crammed with old papers and possessions. Now her sons are adding to the clutter ...

'Is there anyone in the last 200 years of our family history who hasn't been touched by at least one of the following: madness/brilliance/disease/early death/adultery/eccentricity/ornithology?" asked my brother Rowan in a recent email. He and I, and our elder brother Charles, spent our formative years in the shadowy company of our dead relations. We grew up at Hancox, the Tudor house where my sons and I live today. It has been in the family since 1888; continuous ownership, a strong instinct for preservation and lack of money for home "improvements" has turned it into a timewarp.

As children, we took for granted the family portraits looking down on us, the drifts of Victorian letters, the battered lead toys we played with. If any of us needed something – a science overall, riding boots, a tent – the response was not to buy it, but to open a cupboard and dig one out. Sometimes, I yearned for new things. But on the whole I liked living with the past, and I still do.

The walls of the bedroom that my brothers and I shared bore evidence of earlier occupants in the form of drawings and scribbled messages, and we added graffiti of our own. Drawing on walls is a family habit. In my son Sam's bedroom, a frieze of waterbirds survives, chalked there by my great-uncle Gillachrist in his sunny Edwardian boyhood, a few years before the first world war abruptly ended his life.

Our daily handling of the things that had belonged to three earlier generations made our dead forebears real to us. We read their books, we dressed up in their clothes, we used spoons with their names engraved on them. Our genetic inheritance of certain characteristics was reinforced by the shared environment; they had shaped the place, but equally the place had shaped them, and us. And the effect continues. I asked my 12-year-old, Jake, what he liked about living here. He said he liked the labels, peeled off our Sunday bottle of wine and pasted on to his bedroom door by his Uncle Charles in the early 70s. "They're funny," he said. "I like the history. I like the bells to call the servants, and the fact that there aren't any servants to call."

Hancox is a timber-framed hall house. Since the 1480s, each successive century has seen the addition or subtraction of a floor, a wing, a chimney, a rendering. The result is a living history of local vernacular, hopelessly incoherent or charmingly jumbled, depending on your taste. I, of course, love the warps, the wiggles, the absence of symmetry. The building materials – the rosy bricks, the russet hanging tiles, the massive oak beams, even the iron for hinges and latches – would have come from within a five-mile radius, which is why the house feels like a natural outcrop of the hummocky Sussex Wealden landscape in which it sits.

I'd always known that the first family member to live here was Milicent Ludlow, a cousin, who later became my great-grandfather's second wife. (His first wife, Amy, was Milicent's first cousin.) I'd grown up with an image of Milicent as a comically intransigent old lady in a wide-brimmed hat who knocked down a policeman in her car (she never mastered gear changes) and enquired: "What are you doing down there, my good man?" It only dawned on me relatively recently that when Milicent moved into Hancox she was only 20, and single. What made this Victorian orphan, who had inherited her parents' money, not only take on a large and ramshackle house but set about enlarging it still further, remodelling the gardens, and – most surprising of all – appoint herself as manager of its farm?

I decided to find out more. I'm a professional writer, but I'm also the mother of three sons, two of them autistic. I had written a book, George and Sam, about living with autism. Gathering the material for that was easy – it was the fabric of my life. I had done a little investigation into family history, to help explain where my sons came from. That whetted my appetite and I decided to write about Hancox. George and Sam's needs limit my opportunities for far-flung exploration and adventure, but I realised I had all the materials at hand to make a journey of a different kind, a journey into the world of the people who had once lived under the same roof.

All families are interesting. Every human life is potentially full of drama. What's different in our case is that so much of the evidence survives. My family have a reverence for the written word; we find it difficult to throw away anything with handwriting on it. Some old letters, chewed by mice, have been preserved in plastic bags, in the hope that one day someone will feel up to tackling these complicated and fragile jigsaws. We also tend to endow inanimate objects with human qualities, which makes them hard to dispose of. Under a pile of mothy tea-cosies I found a pair of secateurs returned from the ironmongers with a brown label attached, dated 1956 and inscribed "Not Worth Repair". I attribute their survival to a hopeful feeling that someone, some day, might know how to mend them, combined with a sense of loyalty towards the dear old secateurs, which had worked so hard.

Another trait is a disregard for physical comfort and a reduced awareness of how things might look to outsiders. A lumpy horse-hair mattress, a rattling, leaky window, a wonky table propped up with a couple of books – such things are tolerated, even enjoyed. Old shoeboxes crammed with letters, dusty rows of battered notebooks, trunks full of unsorted personal documents – these do not offend our visual sense, any more than do handle-less saucepans or rust stains on old sheets or doorhandles tied on with bits of string.

I discovered that not only am I surrounded by the things my ancestors used and loved, but also that I had access to their thoughts, their feelings, their turns of phrase. In every drawer, on every shelf, there are letters, diaries, sketchbooks, photographs, household bills, school reports, exercise books, children's drawings. As I investigated, the lives of the long dead flared into vivid reality. We had, of course, grown up with a fund of family anecdotes. I knew about Uncle Ben the explorer who survived a shipwrecked Arctic winter by eating polar bears, Aunt Barbara the copper-haired feminist whose husband would spring naked out of the rhododendrons and frighten the maids, beautiful Amy who died of tuberculosis, my great-grandfather Norman who would walk 50 miles at a stretch with his hawk on his wrist. These characters had played an active part in my childhood imaginings. Now, I discovered how all the stories linked together. The narrative that emerged from the cobwebs was a remarkably coherent one, a detailed panorama of Victorian and Edwardian family life, full of humour and bathos as well as drama, passion and tragedy.

My project was, necessarily, intrusive. Reading personal documents inevitably exposes things that the writer would rather not have made public. Describing the behaviour of people who are not here to explain themselves is bound to be unfair. I can only justify it by pointing out that people are free to destroy their letters and diaries; if they don't, one day they will be read. The characters I have attempted to re-animate are long dead. I soon realised that to carry the narrative into the realm of living memory would be wrong.

When I came to put it all together, I sat at my bedroom window and thought about what my great-grandparents would notice if they could see it now. The view encompasses about half the garden and some of the farm. The great cathedral of a barn, the coach house, the stable, the brewhouse, the dairy – all are still in place, though the coach house shelters lawnmowers, not carriages, the hunters are long gone from the stable, and the brewhouse will never again be used for making beer. The farm, intensely cultivated 100 years ago with a different crop in every field, employed seven men; now two part-timers use it for sheep and bullocks. My great-grandparents would be dismayed by the dereliction of what was once the kitchen garden, where our pigs now rootle under the few survivors of Milicent's dozens of espaliered fruit trees, but they would, I hope, be reassured by the survival of some of the plants they put in, gifts of their friend Ellen Willmott, the great Edwardian gardener. The yew topiary, the drifts of narcissi, the berberis darwiniae, the clumps of brilliant orange monbretia, all thrive.

This view, like every aspect of the place, is rich in memories for me. As I look at it now, I think of Sam, running along the tops of the Tudor walls and never faltering, of George pulling bees out of the lavender bushes to examine them and never getting stung, of Jake and his friends trundling by, piled on to a mobility scooter they had found somewhere. Further back in time, I think of making "nests" with Rowan in the soft warm grass clippings in the immense wheelbarrow, of racing our hobby horses up and down the lawn, of making dens out of piles of furniture dragged out of the house. I remember my mother's horror when I ate the lords-and-ladies berries that still grow by the garden walls.

I look at the wobbly brick paths and remember the hard winter of 1963 – a very small child, I ran along these paths while Charles shot at me with his toy pistol, banks of shovelled snow towering above our heads. I can recall the deliciousness of the stale crusts lying in the sun on these same paths, thrown out for the hens but picked up by me. I remember Rowan falling on to a jagged baked bean can dropped by the dustman on the path, and rather envying him the drama of the dash to the hospital and the impressive bandage. Rowan toppled backwards over the parapet wall by the boiler house – we were always told not to sit on that wall – and there was another drama; had he broken his collar bone? Years later, nine-year-old Sam wrenched the bricks out of the paths and threw them over this parapet wall. My cousin Tom lovingly and painstakingly pieced them back together, and Sam moved on to pulling tiles off the lean-to roof.

Now this view is also animated by scenes I never saw, people I never met, almost as clearly as my real memories. My grandfather teetering along on his pennyfarthing, Milicent playing tennis in her immense "picture hat", my great-grandfather's "butterfly bed", the teak seat where he sat star-gazing with Milicent's best friend, who was also his mistress; doomed Gillachrist and his girl cousins dancing to a gramophone on the lawn in the summer of 1914 ... I feel I'm living not so much in the past as in parallel with the past.

Jake, I notice, has inherited the family tendency to hang on to things. Among his treasures is an unusually shaped toenail clipping (his own), taped to a piece of cardboard and labelled and dated for the benefit of posterity. At one time it was difficult to get him to throw even a sweet wrapper away. He loves stories about his ancestors, just as I did. George and Sam, though, have no historical sense; they live for the moment. However, though they don't know it, they too share experiences with the dead. They have grown up with the same sounds, smells, sights, textures. The scents of woodsmoke and leatherbound books, the particular creak of hinges and floorboards, the scuffle of starlings under the hanging tiles, the way low-lying winter sun seems to set fire to the dining-room panelling – they know all the small sensory details that have been familiar to five generations.

At one time, the sense of continuity I experience would have been commonplace. Now, it's unusual. Very few people live in the house where they grew up. Many, I imagine, would shudder at the idea. The urge to start afresh, to arrange things just the way you want them, to redecorate without reference to the feelings of the living or the dead – I can understand all that. Living the way I do has its own responsibilities and limitations. If I didn't have the taste for it, it would be nothing but a drag. But for me Thomas Hardy's poem Old Furniture, with its sympathy for "each shiny familiar thing", its vision of "hands behind hands" fingering the furniture, makes perfect sense:

"I know not how it may be with others

Who sit amid relics of householdry

That date from the days of their mothers' mothers,

But well I know how it is with me"

Driving up to Hancox, especially at night, it looks gloomy, lowering, "spooky". I'm often asked whether it is haunted. No: neither I nor, as far as I know, anyone else, has seen a bona fide ghost here, or sensed a malign or restless presence. Yet it is not quite accurate to say that Hancox is inhabited only by the living. The lives of those who went before me still influence me, the family traits resurface every day. On the larder shelf is an old bottle, stoppered with a rag. At the bottom lies something blackish and coiled. I happen to know that this is a grass snake, used by my great-uncle Gillachrist in a friendly prank to give the cook a "turn". As keeper of this information, how could I ever justify throwing it away?

Hancox: A House and a Family by Charlotte Moore will be published by Viking/Penguin on 1 July, £20. To order a copy for £15.99 (including UK mainland p&p), go to guardian.co.uk/bookshop or call 0330 333 6846