Loyalty Card Data - Perhaps the true cloud battleground.

In 2004 I worked on a project with a company called Rippledown Solutions, later their app became System42 then it got totally rebranded into Erudine. One of their demo pitches was loyalty card data and it's something I've been fascinated with ever since. To mine that amount of data takes some doing.

Anything with large datasets has the potential to be a cloud goldmine (if it's not already) including web usage data, video, photography and music. The main problem with loyalty card data is getting your hands on it... not the easiest thing in the world to do. At the end of the day it's a lot of numbers: date/time, store location, item, offer code... Tesco go deeper and create more data and a better idea of how to target their data.

In the UK there is one clear market leader who knows what they are doing with the data, and that's Tesco. That's more down to Dunnhumby who put a lot of backend consultancy together. Tecso mines the best part of 18 million customers data. From my dealings with Rippledown the reason, we found out on the grapevine, that you only got vouchers every three months was that's how long it took to mine the data.

This left the rivals in a complete state of non starter. The Nectar card scheme never really did anything and upon that Asda pretty much gave up on loyalty cards altogether. Nectar in the end got Peter Gleason to go from Dunnhumy to Nectar, he was one of the team who knew data analysis. Not a programmer that I'm aware of, just a smart marketer.

Consumer data is big money to the supermarkets as they can sell it on. And the faster they can process it means they can monetise it quicker. Sounds like a perfect elastic platform waiting to happen. There's a good chance it's already happening. One company that stopped selling their data though was Walmart, selling competitor information about suppliers is never a good thing. Wal-Mart operate their own data centre with over 460 Tetrabytes of data storage and boy do they know how to do stuff. CIO Linda Dillman could use the data to predict buyers strategy for hurricanes (PopTarts sales increase seven fold and the pre hurricane favourite was beer), predicting is much better than analysing after.

What doesn't come out in the wash is the speed that this data can be collated and processed. The Tesco problem was always sheer volume to process over a three month period, I'm not sure how Wal-Mart latency would be in all of that.

Instance replication is the big plus point here, getting servers to replicate during high loads of data processing. The processing of the New York Times archives (now a Cloud Computing must read) gives the sense of what's possible with instance replication. Sainsbury have gone to a system of offers at the point of sale, ie looking at the customer's basket and creating offers from there. It doesn't go in to any historical depth. A web service to a cloud backend (with all the previous data run during the night) could recall the new offers based on what they've already bought. The key here is the customer, it's easy to know out a voucher to reduce petrol/gas by 5p a litre but a whole other to push a new type of butter on the customer who's spent the last month buying margarine.

If there's one arena to keep an eye on it's this one.