Is the demolition of the Heygate estate in south London the welcome end of a misguided experiment? Or is it the push for regeneration that is flawed?

Look for my white Ford Focus estate," Adrian Glasspool tells me. "It'll be the only parked car." It's a useful tip because, after wandering haphazardly round the now almost deserted Heygate estate next to the Elephant and Castle in south London, it is the car – spotted from one of the walkways – that leads me to his maisonette.

Glasspool is one of just 11 householders left on an estate of 1,260 dwellings completed in 1974, and an articulate critic of what he believes has been the unnecessary destruction of the Heygate.

The estate comprises half a dozen huge, grey, monolithic blocks confronting the busy roads around the Elephant and, between them, groups of three- and four-bedroomed maisonettes such as Glasspool's.

"Welcome to failed utopia," he says, when I eventually reach him. He's being ironic – even its most ardent fans would be hard pressed to call the Heygate utopia, though now in its abandoned state you can hear birds twittering, and squirrels come scampering up to you looking for food. But nor does he think the estate deserves to die. "There's something beautifully simplistic about these blocks," he says.

"They're not very pretty and they have become unfashionable, but they're structurally sound and functional. Just because they're a bit grey doesn't mean people can't live here happily."

Southwark council has spent the past 10 years talking about regenerating the Heygate and the past three or four emptying the estate – "decanting" to use the horrible developers' euphemism – its residents, the great majority of them council tenants but with a smattering of leaseholders who exercised their right to buy.

Glasspool, who bought his flat here in 1997, argues that a tightly knit community, with many residents who had been here from the beginning in 1974, has been destroyed and scattered to distant parts of the borough. He says one elderly woman, long decanted, still comes back to walk her dog.

Glasspool calls the destruction of the Heygate an example of "environmental determinism". "It's part of the same discourse that was being bandied around in the 1960s," he argues. "Then it was said that the tenement buildings needed to be demolished because they didn't create an environment where people could live happily. It was precisely what is being said now." He believes the idea that the estate was "blighted" by crime and drugs was part invention – the product of an excitable media and of film-makers who liked to use the Heygate as a set for gritty realist dramas – and part self-fulfilling prophecy, as the council neglected maintenance and replaced long-term tenants with short-term licensees, who tended to be more disruptive.

"Suddenly the place was being labelled a problem estate," he says. "This is all part of this regeneration discourse. Because there's nothing wrong with the buildings, they have to find an excuse to regenerate the place, ie knock it down and replace it."

I hear a similar story from another resident, the sole remaining occupant of one of the huge blocks, alone up on the tenth floor of a building in which every other flat has been sealed. He doesn't want to be named because he is still arguing the terms of his departure with Southwark, but has lived here since 1996, bought his flat in 2000 and has spent the past eight years wondering when he would have to leave.

"It's been a long road," he says with a sigh. A cold one, too: the communal heating system has ceased to work, and he is reduced to warming his flat with small convector heaters. "But the harder things get, the more determined you become." Like Glasspool, he doesn't understand why the estate had to go. "There are a lot of bright, enthusiastic, imaginative architectural students who could do something amazing with it – a coat of paint, lighting. And there must be professional architects who would be interested in it as a social project. But it's not about that; it's all about the gentrification of the area. They've chosen to knock this estate down because it's in a prime location."

You hear this argument again and again from local residents and community activists. The Elephant is convenient for central London and has terrific Tube links; as the recent building of a 40-plus storey block close by has shown, people will pay large sums for executive flats. The Heygate site will be worth a fortune when Lend Lease, Southwark's chosen developer, has cleared out the detritus of the 1970s. The 1,260 council homes will be replaced by 3,300 dwellings, mostly for private sale but with 25% set aside for "affordable" housing. When the Elephant has its new shopping centre and more user-friendly road system, this will be a highly desirable place to live. Rather too desirable, critics of the regeneration scheme argue, for the council tenants who used to live here.

Southwark accepts the regeneration project has been problematic. "It hasn't been plain sailing," says councillor Fiona Colley, cabinet member for regeneration. "There should have been new homes built for residents before they moved. That didn't happen. But they've got the right to return." In reality, though, few will come back to the social housing on the new site. Demolition will not be finished until 2014, and who knows when the new development will be completed? Much will depend on the speed with which the property market recovers.

Colley accepts there will be a degree of yuppiefication, and believes a combination of different types of housing on the same site has social advantages. But she denies that was the principal motive in knocking down the estate, insisting it had reached the end of its natural life. "These blocks are really difficult to maintain. It was seen as a model estate when it was built, but it hasn't stood the test of time." She says Southwark remains committed to social housing and would finance the regeneration itself if it could, but that successive governments have made that financially impossible, forcing councils to rely on private finance.

Naturally, the man who designed the Heygate, Tim Tinker, disagrees with Colley's assertion that the estate is obsolete after less than 40 years. Southwark didn't know who the lead architect on the project had been; nor did the Twentieth Century Society, which campaigns on behalf of post-1914 architecture. But eventually I tracked him down and took him back to the estate – his first visit for a decade – to which he devoted seven years of his professional life when he worked for Southwark in the 1960s and 70s.

He is a little anxious about our reasons for photographing him on the estate – do we want to stitch him up? – but cheers up when we avoid placing him next to graffiti proclaiming "Welcome to Hell" on one of the walkways. "I don't think it was in any sense a failed estate," he tells me as we survey his decayed handiwork. "There are failed estates, but this wasn't one of them. The hardware – what we provided in concrete and brick – was relatively OK. The problem was there wasn't the software to run the damn thing. There was a huge influx of new housing [in the 60s and early 70s], and management never really understood what they had."

Tinker also believes the council wants to knock the Heygate down because of its central location. "There weren't any problems [with the estate] until relatively recently, but the council eyed it as an opportunity. Councils always go for big-bang, new-build solutions, as opposed to looking after what they've got. Now, sometimes big-bang solutions are right, but quite often they're not and the net gain is limited."

For someone whose principal architectural legacy is about to be knocked down, the 75-year-old Tinker is remarkably philosophical.

"It's the past," he says. "People buying their houses wasn't an issue then. The idea in those days was that local authority housing should be for all. It wasn't only for the people who'd fallen under Mr Cameron's 'big society'. There was a clear feeling that local authority housing should avoid that stigma."

Tinker may not have built utopia, but there was a degree of utopianism in what he was trying to create, even in the face of growing cash constraints in the early 70s which meant the estate had to be system-built, using huge factory-made concrete slabs that could be pieced together with less on-site labour than in conventional building. "Utopian is a dangerous word," he says, "but if you're working in local-authority housing you're bound to have a utopian view. What's the point of doing it otherwise? You look back now and ask why people were enamoured with modern architecture, and I would suggest it was to do with light, sunlight. At that time these inner-city areas were extremely nasty, smoky, dirty places. The Elephant was still pretty bad, with tanneries and God knows what else." The flats he designed were light and airy, and the now despised walkways were created to keep people away from cars, which back in the late 60s when the estate was planned were just on the point of becoming ubiquitous.

He says the early tenants responded to those utopian intentions. "I used to go into the flats for the regular defects inspections," he recalls, "and it was always interesting to see what people had done to their flats or maisonettes. People did amazing things inside."

Council housing was seen as a natural mode of living, a much broader spectrum of people lived in it than was later the case, and the community he says he deliberately set out to create worked. (Worked in every sense – as ever, joblessness and the resulting dependency and personal chaos are the elephant in the room, even when discussing rooms in the Elephant.) Only later, when people were only housed if they scored highly on an index of deprivation or social challenge and the council lacked the resources to deal with the attendant problems, did the "blight" begin. Software, not hardware; people, not buildings; politics, not aesthetics.

When I paid my initial visit to the Heygate, on the day a demolition team was taking possession of the first completely empty part of the estate, I thought I was going to be writing about the death of a modernist, misguidedly idealist experiment in collective living. This forbidding estate, built with the best of intentions to house the labouring masses, had – so the thesis went – gone downhill in the 1980s and 90s because estates such as this were inherently flawed.

That view was forcefully put in Utopia on Trial, an influential book published in 1985 by Alice Coleman, who was head of the Land Use Research Unit at King's College London.

"Why should utopia have been such an all-pervading failure, when it was envisaged as a form of national salvation?," she wrote. "It was conceived in compassion, but has been born and bred in authoritarianism, profligacy and frustration. It aimed to beautify the urban environment, but has been transmogrified into the epitome of ugliness . . . The brave new utopia is essentially a device for treating people like children, first by denying them the right to choose their own kind of housing, and then by choosing for them disastrous designs that create a needless sense of social failure. It is the utopians who should be experiencing the sense of social failure. They have had their day – 40 long years of it – and it has become increasingly clear that their social engineering has not worked."

All this was music to the ears of the Thatcherites, who were at the high water mark of their power when Coleman's book appeared. It confirmed their view that the social engineering practised by the architects who had built the Heygate and hundreds of modernist estates like it across the country had been a disaster; that council housing was synonymous with crime; that owner-occupation was socially desirable, so tenants should be given the right to buy council properties. The aesthetic argument existed to serve a political view of housing, and the gathering assault on modernism, functionalism, brutalism became a cover for politicians of both right and left who wanted to withdraw from public provision.

"Council houses and council estates became unfashionable, and the people who lived in them became unfashionable as well," says Jerry Flynn, spokesman for the Elephant Amenity Network and a former resident of the Heygate. "Now there's a real stigma attached to being a council tenant, which my mother [a tenant on the Heygate] was feeling towards the end of her life." Flynn, like Glasspool a teacher, refused to buy the Thatcherite mantra that owner-occupiers would inherit the earth and is still a council tenant, now living in Bermondsey. "I live in a council house," he tells me, "partly through choice and partly through poverty."

Flynn says the Heygate's nightmarish reputation was exaggerated. "Up until 1999, it did not have a reputation for being a bad place to live. But as regeneration progressed, the estate's reputation dropped.

Almost overnight we woke up to be told we were living on one of the worst estates in Britain. It was a combination of laziness on the part of journalists, and public policy that decided estates such as the Heygate bred antisocial behaviour and crime, and needed to be broken down. The buzzphrase now is 'mixed and cohesive communities'."

Translated, that means you can't have a lot of poor people living together because they can't be trusted to live in a civilised way.

"It's patronising," says Owen Hatherley, author of Militant Modernism and A Guide to the New Ruins of Great Britain, and defender of both 60s brutalism and the mass housing it characterised. "Housing associations tell people we'll 'pepperpot' you with some stockbrokers and that'll make everything OK. Then you'll somehow become more cultured through osmosis. That's one of the reasons I bang on about the aesthetic qualities of 60s buildings – to try to remind people that these estates were attempts to house a hell of a lot of people and in quite a dignified way." Hatherley sees the rejection of social housing and the deification of owner-occupation in explicitly political terms: as an attempt, for the most part successful, to create a conservative, property-obsessed society. Nothing that is organised collectively or communally can be allowed to work. Modernism had to be attacked aesthetically because it was dangerous politically.

Hatherley, who grew up on a council estate in Southampton, doesn't blame Margaret Thatcher alone for the change. He says its roots lay in the individualism of the 1960s, when gentrification and the fixation on the traditional home began. Estates such as the Heygate, Robin Hood Gardens in Poplar, Thamesmead in south-east London and even the iconic Park Hill in Sheffield, which was granted listed status by English Heritage in 1998 to ward off those who wanted it demolished, were out of kilter with the times from the moment they were born. Created by architects who had come of age in the war years and had a commitment to collectivism, they were introduced into a world that was already suspicious of such idealism, and for whom monolithic building smacked of the Soviet Union and Big Brother. Thatcher caught that mood and gave it political momentum. Brutalism was doomed to suffer a brutal fate.

The Robin Hood Gardens estate just east of the City of London, which was designed by the famous husband-and-wife architectural team Alison and Peter Smithson and completed in 1972, has become a cause célèbre for defenders of utopian mass housing. Tower Hamlets council is determined to demolish it, but the Twentieth Century Society has been fighting a vigorous campaign to save a development which Richard Rogers, in a book recently published by the society, calls "magical" and likens to a Nash terrace. This is not the first comparison which springs to mind on the grey, cold morning I visit. The two long concrete blocks, which surround a hilly patch of green space, are now rundown and pockmarked, though still inhabited, mainly by families of Bangladeshi origin.

Feeling sorry for Tinker, who worked with the Smithsons early in his career and says they were excellent teachers, I ask the society why they're campaigning to keep Robin Hood Gardens but letting the Heygate go without a struggle. "The regeneration of the Elephant and Castle has been under way for a long time, and pragmatically we didn't think we'd have a hope in hell of winning on the Heygate," explains the society's director, Catherine Croft. The Smithsons' fame, their importance as architectural theoreticians and their central role in the evolution of brutalism have encouraged the society to fight to save the couple's only completed public housing project. So far it has been losing the battle, but it hasn't given up yet.

Aman Dalvi, corporate director of development and renewal at Tower Hamlets, is furious that the architectural lobby, which made an unsuccessful bid to get the estate listed in 2009, has slowed the planned regeneration. "They don't have to live there," he says. "They just drive past and have this concept of what was done 30 years ago.

Has any one of them come up to me and given me a solution as to how I could regenerate that estate, and where I am going to get the money?

Not a single person has done that, but I still have to find solutions for the residents who live there."

Dalvi intends to replace the 250 flats with 1,700 homes split 50-50 between private and affordable housing. Again the emphasis will be on what New Labour liked to call "building sustainable communities". "I would not countenance any homes being built which are just for social housing," he says. "You have to have a mixture of housing for sale and social housing." There will be a greater density of building, more variation in the size of homes – he says the two- and three-bedroomed flats in Robin Hood Gardens are too small for many of the families who live there – and a balance between people who are working and people who aren't. "These are all things that help you manage the estate better," he says. "Any contractor can come and redevelop an estate, but the question is will residents still want to live on the estate in 10 years' time? If they don't want to live there in 10 years' time, you've failed. How do you get it right? You have to learn from the mistakes of the past."

It is a plausible argument, well put by the immensely experienced Dalvi. But so far it does not seem to have convinced the lobbyists.

Last December, the Observer's architecture critic, Rowan Moore, made a scathing attack on the two rival regeneration schemes for the area, which will be renamed Blackwall Reach. "Both schemes are generic developers' fare," he wrote, "with homes heaped into blocks and towers to make enough money to pay for the demolition of the old buildings, with a notional bit of landscaping supplied as a palliative. There is no sense of place, no distinctiveness. The gridded elevations are not more charming than the Smithsons', and all trace of dignity or grandeur will be gone. The streets in the air, which for all their faults allow some kind of neighbourliness, will be replaced by lift lobbies."

He also wondered, given the reliance on a private developer and the current state of the market, when it would be completed. "To achieve this doubtful utopia, all existing residents, with their networks of neighbours and friendships, will be decanted elsewhere, probably never to return. The principal mistake of the 1960s, which was to throw out everything, good and bad, will be repeated. And there is no guarantee that the schemes shown will be completed soon, or ever ... Developers tend to build such places bit by bit, only moving to the next stage when they have sold everything already built."

How intriguing that Moore should employ the term "utopian" against the apostles of regeneration, who see themselves correcting the utopianianism of the 1960s. The utopia now dreamed of is the sustainable, cohesive community in which rich and poor live side by side in harmony. It is probably doomed to go the way of 60s utopianism, and for much the same reasons. It reflects the reality of a decade ago – easy credit, booming economy, obsession with home ownership. Now, as the economy seizes up, the housing market stalls, and the government caps housing benefit and seeks to drive rents in social housing up to 80% of the local market rate, people are going to be squeezed out of these shiny developments. In the absence of mass social housing of the old type, it is by no means clear where they will go.

Anne Power, professor of social policy at the London School of Economics, says creating mixed communities by planning is impossible.

"There are regeneration plans unravelling all over London at the moment because the finances just don't stack up," she says. "If they're going to sell enough at the high end, they won't put in the low end; and if they make it work at the low end, they won't be able to sell the high end."

Power, who says the demolition of the Heygate has served only to exacerbate the housing shortage in Southwark, makes the obvious – but strangely ignored – point that the volume of social housing should reflect general economic conditions. "If you think low income is vanishing, then it makes sense to reduce the amount of social housing that you've got. If you think low income isn't vanishing – and all the evidence would suggest it's not – then it makes no sense to get rid of social housing unless you've got an alternative for low-income people." Her question is a stark one. "What are you going to do about low-cost housing in a city that relies entirely on low-paid, low-skilled jobs for almost everything that its rich clientele depends on? Its hotels, its restaurants, its nurseries, its transport system, its street cleaning, everything depends on low-paid workers, and we are going to create one hell of a terrible society if we don't recognise that."

There will, she says, never be a repeat of the housebuilding boom of the 1950s, 60s and early 70s because the cost is prohibitive. Her solution is to be pragmatic. Instead of knocking down estates such as the Heygate, and replacing one form of utopianism with another, she favours adapting the infrastructure that already exists – bringing derelict and unused property back into use, converting empty commercial buildings for residential use, making sure every council property is occupied – and plugging the gaps with small-scale developments. Property developers like grand designs; people just want somewhere to live.

"It's ridiculously expensive to knock estates down," says Dickon Robinson, former development director at the Peabody Trust housing association. "After they've knocked them down, they're still paying for them. They were all built on the basis of a 60-year payback period. These are not dysfunctional buildings. If you invest in them, they will be perfectly fine. There's been this vogue recently for this kind of approach, which says, 'This is an awful estate, we give up, we can't manage it, what we're going to do is knock it down, redevelop it at three times the density and fill it up with owner-occupiers who will be a good example to these feckless local authority tenants.'"

He finds the approach naive as well as patronising, because many of the flats are bought not by owner-occupiers but by investment companies on a buy-to-let basis, and their tenants often lead even more chaotic lives than people in social housing.

Like Power, he favours a more flexible approach. "It's unsustainable to build such robust and structurally sound properties, and then take them away after 30 or 40 years. We have to build on a much longer cycle, and if necessary we have to be prepared to allow the people in those buildings to change. Let them evolve. If you have the right kind of buildings, uses can swill backwards and forwards. Supposing someone wanted to convert part of the Heygate estate into office suites; great, let's do it. It's the organic city, as opposed to the tidy-minded planned city that says, 'You were once social housing, so you will always be social housing.'"

It's a lovely idea, but one that has come too late to save the Heygate, where the last few residents – and presumably the squirrels, too – will shortly be moving on.