The 'Russian avant garde' created the 20th-century's most intensive art and architectural movement. Its paintings survive, but its buildings rot

The "Russian avant garde", it's usually called, though the artists themselves didn't use the term; they were known as the futurists, then productivists, and most consistently, constructivists. Even the "Russian" is a misnomer – the individuals in question were frequently Ukrainian, Latvian, Belarussian, Georgian. "Soviet" doesn't quite work either, as they emerged slightly before the October revolution, out of the futurist cafés and cabarets of the mid-1910s.

What they created was probably the most intensive and creative art and architectural movement of the 20th century, a sourcebook so copious that there's scarcely any movement since that wasn't anticipated by something tried and discarded between 1915 and 1935 – from abstraction, pop art, op art, minimalism, abstract expressionism, the graphic style of punk and post-punk, to brutalism, postmodernism, hi-tech and deconstructivism. But the people making this work largely didn't consider themselves to be artists; they even used the term as an insult. They wanted to destroy art altogether, not as a sulky nihilistic gesture, but because they thought they'd created something better to put in its place. They are currently almost ubiquitous, but they nearly disappeared from the historical record – something almost accidentally documented in the Royal Academy show Building the Revolution: Soviet Art and Architecture, 1915-35.

The bulk of the artworks in the show come from the collection of George Costakis, a Greek diplomat resident in Moscow from the 1940s until the 1980s. He created what has been called a "futurist ark", buying up drawings, paintings and sketches by artists who were dead, discredited, forgotten, prohibited, or who had moved on to the very different "socialist realism" prescribed from the 1930s onwards. Until Costakis's collection went public, there was only a vague idea that something extraordinary had happened in the former Russian empire – perhaps a couple of mentions of Kasimir Malevich or Alexander Rodchenko, usually in connection with the German artists they had inspired.

Costakis's work was aided from the 1970s on by the archaeological research of the Soviet historian Selim Khan-Magomedov and the late English architectural writer Catherine Cooke; it's no exaggeration to say that without this small group of people, the current prominence of the "Russian avant garde", which has featured in seemingly dozens of exhibitions on the heroic era of modernism over the last decade, would have been impossible. This is at least in part because it was equally useless to both sides in the cold war. For the west, with its CIA-sponsored abstract expressionism, the claim that Bolshevism led inevitably to the suppression of individual creativity was hard to square with this unprecedented visual flowering; while the Soviet bloc still clearly felt there was something dubiously Trotskyite about these internationalist, cosmopolitan art movements.

In Building the Revolution's catalogue, an essay by Jean-Louis Cohen outlines the close connections these artists and architects had with various western trends, from the Bauhaus to Le Corbusier, who was invited to Moscow to design a gargantuan office block for the Union of Co-operatives, which is still standing. No doubt this counted against them when the Soviet Union took a sharp rightwards turn towards nationalism and autarchy in the 1930s. Yet there's often a tendency to act as if the constructivists were themselves "western" in the cold war sense – that they were typical creative types who couldn't be encompassed into the "system". To paraphrase the title of a book on architect Konstantin Melnikov, they were "solo architects in a mass society", alternately either naive aesthetes or individualists who wouldn't bend to serve the new masters, whose suppression by the monolithic state was inevitable. This conception of the heroic subversive artist was one rejected by the constructivists throughout their existence, so it's an enduring irony that it is so often applied to them.

In the early days of the revolution, especially during the civil war of 1918-21, the futurists decorated the public spaces where the new power was promulgated and celebrated – the painter Nathan Altman created a temporary futuristic redesign of the Palace Square in St Petersburg, architect Nikolai Kolli symbolised the struggle with a public sculpture of a red wedge breaking a white block, while in the small provincial town of Vitebsk, the Unovis group maintained a constant barrage of quasi-abstract propaganda. The last is best represented in the exhibition by El Lissitzky's 1919 Rosa Luxemburg, a monument to the murdered communist leader in the form of polygonal forms flying around a central red circle. The futurists' paper Art of the Commune had direct state support, and though the leadership were ambivalent – Lenin was baffled and irritated by the futurists, Trotsky critically sympathetic – there was no suggestion of their being suppressed.

At every step, the artists developed their art specifically according to how useful it might be for socialism. In the early 1920s they staged an exhibition of the "First Working Group of Constructivists". A well-known photograph of this show features a series of seemingly abstract sculptures, often considered a precursor to later "kinetic art". The constructivists themselves considered this work as a precursor to going into the factories and producing useful objects, which some of them soon did, with mixed results. The intention was to move from the utopian to the quotidian (and back) – after designing the famous Monument to the Third International (a model of which sits in the grounds of Burlington House for the duration of the exhibition) sailor and Bolshevik supporter Vladimir Tatlin's next utopian project was designing a more functional stove.

Much of the Costakis collection dates from the early 1920s, when the new state was recovering from a vicious civil war, an international blockade and foreign military intervention, and facing total economic collapse. The proletariat that had participated in the revolution had been effectively wiped out, with the cities emptying and the heavy industry of St Petersburg destroyed; one delegate at a Bolshevik conference sarcastically congratulated the party on being the vanguard of a non-existent class.

Their only solution to rejuvenate the economy was to encourage small-time traders and the peasants who made up 80% of the population; the constructivists had other ideas. The drawings we see in the exhibition express the desire for a totally urban and industrialised landscape – skyscrapers, giant machine halls, mechanised bodies. Even the abstract art, the non-objective "suprematism" pioneered by the young propagandists of Vitebsk, often evokes the rectilinear precision of engineering drawings as much as it does the free play of the imagination. This was at least on some level a collective fantasy of efficiency, a dream of industry, in a country whose already fragile toehold in the 20th century had just been forcibly rescinded. When this work met western eyes, from the 1922 Russische Ausstellung in Berlin onwards, it was interpreted by people who found the industrial landscape familiar and normal. They missed the element of dreaming – but then the Soviets were often in equally furious denial of that themselves.

The manifestos of the new industrial artists, like Alexei Gan's Constructivism or Nikolai Tarabukin's From the Easel to the Machine, were unromantic, utilitarian. The flourishing of creativity happened because each competing faction of the avant garde was utterly committed and fanatical, not because of anything-goes pluralism. The most radical conceived of art as something that must abolish itself in order to become truly useful to the new society they fervently believed was being built. There wouldn't be "artists" in the old sense anymore – the Moscow art school Vkhutemas aimed instead at educating a polymathic engineer-artist-sociologist. The first casualty was painting, and the notion of the exhibition in museum or gallery, where connoisseurs drift around a collection of individual, unreproducible art works. Former painters delved into textile design, photography, book design and, most of all, architecture.

The Costakis collection shows the temporary propaganda kiosks by the Latvian Bolshevik Gustav Klutsis that were the result of this impulse. The second part of the exhibition shows the real buildings that came later, in the second half of the 1920s. The documentation here comes from two sources. One is the Moscow Shchusev Museum of Architecture's collection of historical photographs; the other is English photographer Richard Pare's archive of contemporary captures of these buildings in a usually parlous state, previously collected in his excellent 2008 book The Lost Vanguard. What these two collections have in common is their reminder of the circumstances and context of the period, something too often lost when we gaze longingly at the utopian blueprint.

In the Shchusev collection's image of the 1926 headquarters for the Soviet newspaper Izvestia, you can see the old Russia that the Bolsheviks feared would overwhelm them crowding round the building, hostile – the clean lines abutted by squat Tsarist pallazos, crenellations and Orthodox domes. Look at Pare's photographs of the same landscapes and you find that old Russia won that battle. Buildings that purport to be steel turn out to be straw; precise little machines for living in are dwarfed by Stalin's gothic skyscrapers and their ultra-kitsch post-Soviet imitations; advertising is ruthless and ubiquitous, covering every available surface. The depth of their defeat is measured here. In art, the avant garde survives; in everyday life, across the Russian Federation and the Commonwealth of Independent States, its works rot.

Given the political defeat of all that its members believed in, they would perhaps have preferred their utopian buildings not to survive. What is unavoidable in any close examination of the constructivists was just how passionately and sincerely they believed in the communist project. They often faced a similar fate to other true believers in the 1930s – Alexei Gan and Gustav Klutsis were among the "purged". Perhaps the fascination that the 1920s still retains, however dimly we perceive it in such different circumstances, is the promise of another communism, unlike the one that committed suicide in 1989 – a communism of colour, democracy and optimism rather than a monochrome despotism; an analogue to the recent return of interest in the aesthetics of social democracy, whether council housing or the BBC Radiophonic Workshop. That's as maybe. What is certain is that the constructivists would not have thanked us for our wistful, apolitical interest.