Temporary shops and restaurants were once a way for artists to subvert empty urban spaces. Now, they're just as likely to be part of a corporate marketing strategy

In a dark, dank nightclub beneath some railway arches, with the clatter and chug of trains overhead, I am having a minor Proustian moment. This London club was last open in the late 1990s, and its smell sends me straight back to that era, my student days: to Britpop and Blur, late-teenage clinches, 70p for a vodka and Coke. The aroma is strong, sour, specific, but it won't linger here for very much longer.

Over the last few weeks this long-abandoned club has been taken over by a group of young event organisers for an ambitious, 99-day pop-up project called Counter Culture. The programme will deliver photographers and DJs, comedians and poets, art exhibitions and parties, a different lineup each night, spiriting this sprawling, downtrodden building straight into the 21st century. One of the four organisers, 23-year-old Lee Denny, meets me at the door, apologises for his moustache ("I'm not trying to look cool, I promise") and shows me around the venue he first discovered when he came to an underground party here.

Denny has some experience of pop-ups: five years ago, he started his own small music festival, LeeFest, in his back garden, and he still runs it each summer, albeit from a larger venue. He leads me into the smaller of the club's two main rooms, kitted out with old, over-stuffed sofas and a much more expertly stuffed fox head. The artist responsible for the fox only works with roadkill, says Denny, and he's particularly excited about a live taxidermy workshop she's going to be running.

We move on through a small changing room, where a pair of grubby grey y-fronts hangs from a high ledge, and out to the main stage. On the opening night, in late September, the club filled up with 980 people, "and musicians kept arriving," says Denny, "people who remembered the place, and had heard about what we were doing. There was Jazzie B from Soul II Soul, and Suggs from Madness. He said 'Have you got a trombone?' and then he got up on stage and was like," he holds one hand to his mouth and slides a fist deliberately through the air, "rum-pa-pum-pum-pum."

Counter Culture is just one of thousands of pop-up events that have opened in the UK and beyond over the last few years – ranging from the small to the large, the cool to the rubbish, the sublime to the ridiculous. There have been pop-up shops, restaurants and gardens; pop-up galleries– one in an abandoned Woolworths in Leytonstone – and cinemas – Tilda Swinton even carted one around the Scottish Highlands. There have been pop-up gigs in launderettes; restaurants in front rooms; films projected in disused petrol stations or on to hay bales in fields.

Those are the more guerrilla projects, the grassroots events, often put together on a wing, a prayer and a stiflingly small bank loan. But alongside these are the corporate-backed pop-ups, the temporary shops and bars and restaurants that appear with increasing regularity, often hosted by well-known venues.

The Double Club in London in 2008, a part-Congolese, part-western restaurant and bar backed by fashion label Prada, was particularly successful. A branch of Central Perk, the coffee shop from the TV series Friends, which opened in London's Soho for a fortnight last year, was used to promote a limited-edition box set of the series. In 2006, Nike opened a shop in New York for four days, selling a special edition basketball shoe at $250 a pair. Gap has used a school bus, kitted out with merchandise instead of seats, as a travelling pop-up shop in the US.

There have been pop-up projects that have opened for an hour, like Mary Portas's vintage clothes sale in 2008, and others so successful that they've eventually become a permanent fixture, such as Tom Dixon's Dock Kitchen restaurant in Portobello Dock in west London. But what unites these disparate projects is essentially a strong fascination with the temporary, with the here-today-and-gone-tomorrow, the idea of excitement, urgency and a dynamic interaction with urban (and it is usually urban) spaces. These are projects that stand in opposition to clone towns, to the idea of uniformity and unending drabness.

The debut of pop-up businesses is often traced back to 2004, when Rei Kawakubo of the cutting-edge fashion brand, Comme des Garçons, set up a temporary shop in a disused building in Berlin. Realistically though, while the "pop-up" description might be fairly new, the idea is as old as the hills. The current craze has echoes in everything from the restaurants traditionally run in people's homes in Cuba to the shop that artists Sarah Lucas and Tracey Emin opened in London for six months in 1993, where they made and sold mugs and T-shirts and ashtrays.

The artist Dan Thompson set up his first pop-up gallery with friends in a bakery in Worthing in 2001; he now runs the Empty Shops Network, which advises artists who hope to start projects in one of the country's many disused high-street stores. (It's estimated that 13% of all UK shops are currently empty – and that one in five of those may never be used again.) He says that his inspiration comes from the magical curiosity shops that have appeared for centuries in fiction, "which no one can ever quite find again. I love creating something that's gone so quickly that people say afterwards: 'Was that you? Did that happen?' I love that excitement that you can create in a town, that sense of – what's coming next?"

While these businesses have counter-cultural roots, there's no doubt they've become a corporate concern. As Ali Madanipour, professor of urban design at Newcastle University says, there are two key readings of pop-ups, which aren't mutually exclusive. One is that they can be "a positive way of making more intensive use of urban space," he says, "bringing life to parts of the city that are under-used – they can provide space for local activity, civil-society events, impromptu gatherings. But on the other hand, they can also be an aid to consumerism, in which brands create a stage setting, adding colour and texture to the general mall atmosphere that is the backdrop to many of our urban spaces. Pop-up businesses support shopping – they bring a festival atmosphere to shopping."

The exclusivity of pop-up events means those that are ticketed often sell out extremely quickly. Denny says he now finds it "impossible to get excited about a new place that's opening indefinitely – you think, 'Oh yes, I'll go to that at some point' and you end up there in 20 years. Whereas if it's temporary it's like: 'We've got to do it right now.'"

When pop-ups are hosted by established businesses, this exclusivity and popularity can lead to obvious rewards for both host and brand. Over the last few weeks, the London restaurant Meza has been hosting a MasterChef pop-up, with former contestants from the TV show cooking for diners at a cost of £49 for three courses. When I went there last week, the atmosphere was loud, buzzy, excitable – obviously good for the restaurant, and good publicity for MasterChef. It apparently sold out in 72 hours.

One of the attractions of pop-ups for businesses is that they can act as an informal, unacknowledged market research project. Last week the smoothie maker Innocent ran a pop-up event in London called the Five for Five cafe – offering a two-course meal designed to deliver five portions of fruit and veg for £5. Dan Germain, head of creative at Innocent, said that the event, held in a disused tramshed, was "a no-brainer. Put on a bit of a party for the people who buy the drinks, meet and hang out with them, and find out stuff you wouldn't discover in some weird research group . . . You get all these charts and graphs that say your customer is a certain age, that they live in a certain place, do a certain thing, and then you see the real people. We could just loiter in Sainsbury's by the fridges and watch the people who come and buy our drinks, but we'd probably get kicked out."

Like the MasterChef event, the Innocent cafe sold out quickly, and was cleverly run – the cavernous space was dressed with fairy lights, fruit trees and herbs on every table; there was friendly service, and good food. Any pop-up event this well thought out, prompting this much goodwill, is clearly an excellent piece of marketing.

Germain says a pop-up event is better value for money than running an advertising campaign. "You're getting a more intense return," he says. "Fewer people, yes, but you're hopefully forging relationships that will last a lifetime." Their pop-up event also enabled them to communicate their brand in an incredibly strong, concentrated way. "Everything we want to do was under that roof," he says. Their core message was literally: "up on the back wall, written in big letters: Eat your greens."

Stephen Zatland, a partner at management consultancy Accenture, says that pop-up businesses give retailers other benefits which might not be immediately obvious to the consumer. It's a chance, he says, "to try out a new store location, to see if the kind of people they want to attract will start flocking there before they invest in a permanent site. Manufacturers can try out new products, new services, deliver them direct to the customer, promote a new brand, or try and re-invigorate an older brand".

And they can carry out all this research and promotion for a relatively low price. Zatland says that compared to opening a permanent site, pop-ups are fairly inexpensive. The recession, with its surfeit of empty shops, has played a key role in this trend. "When a lot of Woolworths stores became available, for instance, retailers picked up on those and rented them for a short period to try out something new on the high street."

The pop-up trend has been so big, for so long, that there have been whispers that it must be about to fizzle and die. But Zatland suggests this is unlikely. "There's another interesting trend for a more permanent kind of feature," he says, "where there's a site for maybe eight different pop-up stores, and the content of that site will rotate, change, every eight weeks, or every three weeks. That will be good, I think, because it encourages customers to keep coming back to see what the new feature is."

When I ask Thompson about the corporate fashion for pop-ups, about the way they're being used to flog us more unnecessary stuff, I expect him to be disdainful. But it's quite the opposite. "I love it," he says, "I love the fact that such a daft idea, started by artists, has taken over. I went to a pop-up Gucci put on, and it was fantastic. It's like Quentin Crisp said – don't keep up with the Joneses, drag them down to your level. We've completely subverted all these great brands, who are now having to think differently, more creatively, and that has to be good for our town centres."

There's no doubt that pop-ups can aid regeneration and make a genuine difference. As Thompson points out, "if you live somewhere the size of Worthing or Coventry or Carlisle or Margate, and you lose a few shops, you really notice it. If that's your home town, and you're passionate about it, you'll fight to make it better."

Horton Jupiter (whose real name, he jokes, is "Mystic Rock") is less positive about some aspects of the pop-up phenomenon. He has been running a cafe called The Secret Ingredient from his front room in Newington Green, London, for over a year now, and says he prefers the term "home restaurant", because pop-up has "become something that people use as a marketing tool". He appreciates the temporary, impromptu nature of pop-ups, but projects like his, he suggests, are meant to be precisely an escape from capitalism, from the robot on the end of the phone, towards something more illicit, subversive, personal and warm.

For landlords whose properties have been empty for a while, these events are a great way to promote their building, bring people flooding back in, and perhaps get some free maintenance and decorating work done too. Thompson says he's never "paid anything more than a peppercorn rent – we cover business rates, we cover insurance, and in every shop we've been to we've left it in a better condition than we found it. We'll give it a lick of paint, a clean and tidy. We took a shop in Shoreham-by-Sea, initially for six months, but now for another six, and a place that had been derelict for 10 years has been completely refurbished – which has led to two other derelict shops nearby coming back into use as well."

Where artists go, corporations follow. And so does gentrification, as areas blossom, flourish and improve - and rents subsequently head skywards. Perhaps now, at a time of deep economic anxiety and trouble, we should just enjoy the most exciting of the pop-ups, those that bring life to depressed corners, flowers to abandoned skips, the flicker of film to the hollow beneath an underpass.

There is something slightly sinister about the marketing guile – and rampant consumerism – behind some of these projects, but many are straightforwardly brilliant, and there seems no shortage of people happy to get involved. "Every time I walk past an empty shop or building," says Denny, "I think: I've got to do something in there, I just have to! If I had time, every empty space that was remotely intriguing would be filled."