The Need for Good Item Recommendation Systems

As discussed in a recent study group paper in which I participated, the online item recommendation systems currently employed are really crappy. A major problem stems from data sparsity because people's actual preferences are determined by what is known as the "long tail," which refers to the idea of rare events (where data is quite sparse) being the most important determining factor of something. (OK, what I wrote isn't a mathematical definition. See our discussion in the paper.) For example, you can't reasonably tell very much about which Wii games I might like based on the fact that I bought Super Smash Brother Brawl because so many people bought it. (You can certainly narrow things down to the fact that I might want games for the Wii as opposed to picking a system at random.) Instead, you would need to go to one of my preferences that is much less common to indicate what my preferences actually are. Then the trouble becomes how to do this effectively because the data sparsity in such regimes makes it hard to do such things effectively. It's quite mathematically challenging. Anyway, the reason I write this now is that because of the fact that I bought Super Smash Brothers Brawl, I have just gotten an e-mail from Amazon suggesting that I might also like to purchase Onechanbara: Bikini Zombie Slayers. Um, no thanks.

What happens in Montreal stays in Montreal

Tomorrow morning I am heading on a bus to the airport for my flight to the land of maple leaves, Celine Dion, and Tim Horton's. (Tim Horton's is apparently a big deal to many Canadians, but I must admit that I don't really get it. I've been to one and it was quite generic. Maybe some of the local Canadian folks want to educate me on this one?) In particular, I'm going to Montreal, which has various levels of association with all three (especially Celine Dion). Another big Montreal thing is smoked meat, which one can find ubiquitously in many restaurant menus there.

The reason for my travel is a fully-funded invite---that originally went to a more senior Oxonian, but he was busy and asked if I'd go---to a study group with industry. I went to one of these in Edinborough in April and it was fun but very exhausting. (I didn't end up walking around through Edinborough and seeing it properly, which is quite a shame given how cool a city it seems to be---and is, according to those who have truly experienced it. Hence, I will need to go back there and have a proper look around instead of just wandering the infinite corridors of Herior-Watt University.) At the Edinborough study group, I worked on a problem on item recommendation systems, which the industry presenters formulated as a probability problem but which our group recast as a bipartite network problem. (Funny, one wouldn't expect me to do that. :) ) The list of problems in Montreal includes one that is very TSP-like, and that's the one that seems the most interesting to me based on the short descriptions that have been published online. (The way it works is that on the first day, the industry folks present the problems---which are typically ambiguous and otherwise ill-formed---and the academics spend the rest of the week trying to get as far as they can with them, which for many problems can amount to just figuring out what the problem actually is.)

Anyway, I am very excited to have the chance to go to Montreal again. It's on the short short list of my favorite cities in the entire world (and, in this case, I am able to make that claim without any emotional bias due to the people or institutions in that city---well, except possibly for Youppi!), and I should have at least some time to go around and do cool things. (One big difference from Edinborough was that Heriot-Watt was away from the city center and in this case I should be able to just walk to where things are happening. If nothing else, the metro system in Montreal is really good, so I can haul ass over to St. Catherine's Street easily enough. There's an RPG store there that I might visit and I also want to pick up N+. Nope, I'm not going to be able to pick up N+ because it's been delayed another two weeks, so it's now slated to come out the day after I return to the UK. Bloody Hell. I was going to pick it up in Pasadena in January. I'll also be partaking in Montreal's many excellent cafes and restaurants. So I'm looking forward to going!) The last time I was there (March 2004), I tested Newton's laws of by running into a glass wall (they work) and had a butterfly give birth on my jacket while I was wearing it. (I have a picture to prove this. I think several of you have seen it before. There was also a Housequote related to this incident, but it was Jing Xu rather than me who uttered it.) The one time before that was in June 2003, and I can confirm that the weather in Montreal is much nicer during the summer than it is in March. :)

Oh, and in preparation for my upcoming exhausting study group with industry, I spent most of today (about 12 hours) revising a 40+ journal-page manuscript from top to bottom. I can overstate how painful this was! Now my collaborator has it and the hope is to submit it after his next pass. (It's conceivable that he'll ask me to go over specific parts again, but I hope he'll agree that we can submit it and post it on the arxiv. We'll then make more changes later when we deal with the referee reports and have had time to recover from the present ordeal.) One of my network science colleagues (and a fellow community-detector, no less) for whom I have a great deal of respect really liked it, which makes me feel very good about the paper.

What happens in Edinburgh stays in Edinburgh

In about an hour and a half, I will be boarding a train to Edinburgh to attend a study group with industry at Heriot-Watt University. There are direct trains from Oxford to Edinburgh, and my choice of the 1:36 train is so that I can have a route with 0 layovers. Then I can just chill in the train for a while and not have to pay attention at all if I don't want to.

This trip will include several firsts for me:

(1) It will be my first time in Edinburgh, which is where solitons were first observed. It's supposed to be a really nice city, and I bought new film and a new battery for my camera in preparation for it. (Note that I will finally get off my ass and buy a digital camera, which I've been intending to do for several years. I can be really lazy about things sometimes... I'm waiting until I'm in the States in May because it will cost me half as much if I buy it while I'm back home.)

(2) This will be my first time in Scotland. I am gradually visiting new countries. I'll also knock off Italy later this summer.

(3) This will be my first study group. On the first day, six industry groups will present their problems, and the attendees will spend the rest of the week seeing how far they can get on them. It should be very cool. I never got a chance to participate in the Mathematical Contest in Modeling, and this has that kind of flavor. Besides working on interesting problems, this has the chance to lead to data sets and longer-term projects and collaborations. It's an excellent opportunity and it gives me the chance to spend a few days away from home during the break (which in general is quite revitalizing). The problem that interests me the most based on the online descriptions is one involving recommendations for items with few reviews---essentially, it's difficult to deal with those kinds of tails because of small sample sizes, so the plan is to try to come up with clever ways of dealing with that. From my perspective, this is a networks problem and, in fact, I believe several of these issues have been studied in past papers that use, for example, data sets from Amazon's recommendation system. Plus I can see how the various Netflix prize solutions have dealt with similar issues.