A UCLA Math Undergrad's Data-Analysis Blog

UCLA undergraduate student Ritvik Kharkar, who is doing some research with me, is doing some spiffy data analysis on his blog. Go take a look at it!

It includes entries about community detection (in the Harry Potter universe), education issues, and UCLA salaries.

The Big, the Rich, and the Good (Data)

Nate Silver has written a fascinating article about the stellar success and rapid progress of baseball analytics versus the less-than-rapid progress of data analytics in other areas (e.g., economic and earthquake forecasts).

Putting the baseball angle aside (and of course I like that angle very much), one thing I really like is the very concise comments about "big data" versus what Silver calls "rich data" and why sports analytics have genuinely improved answers to many important-for-it questions, whereas other situations still struggle immensely to use their data for genuinely large increases in understanding.

Note that I have sometimes used the term "good data" before as a contrast to "big data", though importantly good data can be either big or small, and I think that Silver is thinking of Rich Data strictly as a subset of Big Data. See this blog entry of mine as well as additional blog entries referenced therein.

One could also, I suppose, ask whether these other systems are "more complex" than sports competitions, but I'm not sure (a) whether that's actually true and (b) how to quantify it in a way that goes beyond "Ooh, it's complex." Of course, we have measures of information content, but I expect there would be a lot of assumptions involved in crunching such numbers in these cases. Anyway, it's a thought I had, so I figured that I should at least bring it up.

How to Annoy Experts in Machine Learning

Fun fact: If you want to annoy a machine-learning expert, refer to what he/she studies as "data mining". Most of them hate that, and their reactions to it are often pretty damn funny. :P

"The Extraordinary SVD"

The final version of my article The Extraordinary SVD is now out.

Title: The Extraordinary SVD

Authors: Carla D. Martin and Mason A. Porter

Abstract: The singular value decomposition (SVD) is a popular matrix factorization that has been used widely in applications ever since an efficient algorithm for its computation was developed in the 1970s. In recent years, the SVD has become even more prominent due to a surge in applications and increased computational memory and speed. To illustrate the vitality of the SVD in data analysis, we highlight three of its lesser-known yet fascinating applications. The SVD can be used to characterize political positions of congressmen, measure the growth rate of crystals in igneous rock, and examine entanglement in quantum computation. We also discuss higher-dimensional generalizations of the SVD, which have become increasingly crucial with the newfound wealth of multidimensional data, and have launched new research initiatives in both theoretical and applied mathematics. With its bountiful theory and applications, the SVD is truly extraordinary.

Netflix movie recommendation prize

As discussed in this article from Wired Magazine, Netflix is offering a $1 million prize for improving its recommendation system by at least 10%. I have known about the prize---launched in late 2006---for a while, but my networks group has only recently started tinkering with that dataset. (By the way, a tip of the cap to group alum Eric Kelsic for letting me know about the article in Wired.) I don't particularly care about the prize (and I doubt we're even going to explicitly try for it at any point), but this is a fucking awesome dataset, and I do care a great deal about that!

The article's title is a bit misleading. If you read it (and it's an extremely interesting article), you'll find that the psychologist has a Masters degree in operations research, which provides extremely relevant mathematical tools, so the idea that he's not approaching this with mathematics is hogwash. I certainly appreciate his attempts to use psychological ideas to help drive the algorithms, and his comments are quite reminiscent of the things I have heard social scientists say about the mathematicians, physicists, and computer scientists who study social networks. Basically, they (correctly!) try to remind us that while we're focusing on crunching numbers, it helps considerably (and can be necessary---they would say that it's always necessary, but I don't quite agree with the statement when it's phrased with that strength) to remember that what we're actually dealing with people (or other relevant agents rather than numbers and to incorporate that as necessary in our algorithms.

The psychologist's daughter is apparently starting a mathematics degree at Oxford in the fall, so maybe I have a potential recruit for my research group?

Also, I very much appreciate the fact that a popular article like this one discusses singular value decompositions. I approve!