Usual disclaimer, I’m pushing this out earlier than I’d like (draft wise) because I’m too busy and shouldn’t be focussing on this…apologies for inaccuracies and poor writing :-). Last year I went to a talk “[I am therefore I buy – Measuring consumer preference in a digital age]1” That talk suggested (something like) products signal particular personalities, so people with certain personality traits buy/like those products to say something about themselves…I was somewhat sceptical about this for at least two reasons 1) the data is filthy, and 2) the notion is circular
In the last day or two there’s been a lot of coverage of a new story, with a headline along the lines of “Facebook Preferences Predict Personality Traits” (sciencemag), “Facebook users unwittingly revealing intimate secrets, study finds” (guardian)
I (cynically) assume this is mostly based on press-releases – gotta love
a bit of churnalism – based on this [PNAS article]2 First off, as
was the case when I went to the talk, I have issues with psychometrics
which influence my view, and I think some useful work is being done in
this case – it’s just that I wouldn’t want to generalise as far as has
been here. This is unsurprising – if we’re worried about employers and
governments collecting psychometric or demographic data on us using
implicit data, well I worry about using psychometric tests to inform
anything terribly useful (e.g. what does it mean to be 5 points more
‘extravert’ than someone else?). Also, in this case I think the
article raises some really important points regarding the ethical issues
of data collection using implicit trace on the web – there are
potential issues in raising awareness about privacy, ensuring
institutions (of whatever kind) cannot abuse such information, and
exploring the issues around ownership and use of data. They’re also
quite aware of some of these issues. However, I would raise a few
points on the study, including: * What the title should say is –
the ‘likes’ of people who are curious about how their personality is
displayed through facebook (and therefore added an app called
myPersonality) might tell us something about those people. Perhaps
facebook users are odd, but myPersonality app users are likely an odder
subset with particular characteristics including, I would guess, a
higher use of the site. (see below) * Conflating personality and
other traits is not helpful in any way, the paper refers to
‘attributes’ but quotations and news reporting (sigh, not the author’s
fault) are often just referring to personality. This is irritating
because it makes critique of specific parts of the study harder in some
ways, and because it implies personality and other features can be
lumped together which could lead to people leaning towards
personality/politics/sexuality being biological/psychological and
immutable/unlearnable * The ‘likes’ of people who declare may be
different from those who do not. What’s being claimed in some places
is that – from ‘likes’, we can assign particular attributes accurately.
Actually though the research uses profiles in which attributes are
declared and then checks their classifier against the declared
attributes. That’s a different issue – people who have already declared
they’re gay might be a different group (and like different things) to
those who have not done so. This criticism doesn’t apply to some of the
variables (e.g. parents staying together, personality, etc.) but other
criticisms do. To be fair, in the supplementary info they do say “An
important limitation of our sample is that some of the predicted
variables are from Facebook profile information. Individuals who declare
their political and religious views, relationship status, and sexual
orientation on their profile may be different from nondeclaring members
of those groups; they may associate with distinct Likes, which may lead
to an overestimate of prediction accuracies for these groups.” *
Classification success is based on culture and self-selected
samples. 88% success in classification of gay/not gay strikes me as
not great (in their sample, 4.3% males;2.4% females, I’d be interested
in the false positive/negatives) also a) this is self-selecting sample,
b) this tells us more about culture than it does about anything else
(e.g. ideas of masculinity – it’s interesting that the classifier was
only 93% accurate for gender, an example which also serves to reinforce
another point: there are not only two sexualities, nor two genders).
Another e.g. on the culture point “some of the likes were clearly
related to their predicted attribute, as in the case of the No H8
Campaign” – yeah, cuz wanting equality is like so gay. *
bisexuals classified as ‘heterosexual’ (never mind the fact the
facebook way of defining sexuality might be narrowing in any case):
Bisexuals ignored… “Sexual orientation was assigned using the
Facebook profile “Interested in” field; users interested only in others of
the same sex were labeled as homosexual (4.3% males; 2.4% females),
whereas those interested in users of the opposite gender were labeled as
heterosexual.” This is bad writing, the supplementary info makes it
clearer “Facebook profiles; users who listed being interested in only the
opposite gender were labeled as being heterosexual, whereas users who
listed only the same gender were labeled as being homosexual” *
Classifier success of 60% on what distribution? From a quick
search I can’t see stats pre-21, but it’s about 30% of first marriages
in the US (who knows about of facebook members generally). In this
study, n = 766; 56% stayed together and 60% of users were correctly
classified. As above my statistics (certainly off the top of my head)
aren’t good enough, but I wonder how good this is…randomly assigning how
successful could we be compared to this? Anyone any thoughts?…, I’d
like to see more detail and just how obvious some of the ‘likes’
leading to classification were…still, the potential for systematic
exclusion (whatever accuracy level) is scary. * (Note this
again ignores some nuance in how families are constituted, e.g. if I
have had a stepparent since I was a child, they might be considered my
‘parent’, self-reporting on this issue will be complicated). * You
might think you’re measuring x, but it’s actually a proxy for y (or
abc)…this isn’t necessarily a problem, but I think it’s probably
quite important to understand the stories. If we claim we can “detect”
some attribute, we might in fact be detecting some associated attribute
such as socioeconomic status which a) we might have higher success on b)
we might want to understand more (e.g. because it’s a societal problem),
and c) this is a reflection of culture, it isn’t immutable it is
dynamic, temporally and geographically located, etc. E.g. smoking status
and socioeconomic status are associated. Doesn’t mean I should charge
poor people higher insurance though… * I can’t believe how many times
I’ve said this in the last month…but: other countries have more than
2 parties; have more nuanced politics; have coalitions and minority
governments; have different sorts of debates. Again this is an issue of
culture, and of classification. * visual inspection for
ethnicity…misuse of this for discrimination is obviously
problematic, but it is also a) as the visual route shows, fairly obvious
(tho read on), and b) covered by existing legislation within the
US/elsewhere. However, the claim is problematic because 1) there are
more than 2 ethnicities, 2) visual inspection just makes me cringe
thinking e.g. of places deciding whether someone was black or not, 3) *
Implications for different institutions matter – the overarching
concern the paper raises is with how such data could, theoretically, be
used (or misused) by various institutions – including malevolent
governments. I’m concerned that a vague reference to misuse isn’t
terribly helpful though – what access would one need, what legislation
is currently in place, _what do we already know about personalised
search and similar customisation from trace, etc._ * if a
malevolent government wanted to categorise facebook users with a low
threshold for a ‘positive’ on some variable (i.e. they’re not too
worried about false positives), they might be able to do so using
facebook data – and that’s concerning. But I’m not sure what we do
about that other than democratisation, ensuring secure systems and data
privacy in so far as possible, etc. This is as true at home as it is
abroad – and steps by Western govts to seek access to personal data
should consider this. The issue here isn’t the capability to
categorise – that’s moot – it’s the deployment of such technology for
unethical purposes, that’s why accuracy doesn’t matter – dictators
rarely care much about false positives. * Facebook (Google, etc.) –
this is open to abuse, collaborative filtering stuff is an issue, and
the fact that governments may be able to access this data is
problematic, again see above * Casual users – are likely to interact
with people…they’ll find stuff out that way, not by either running
algorithms or even by assuming things from ‘likes’ (although they might
do that) – this is another problematic but separate area. As before, I
don’t have a problem with this sort of trace data research – hell it’s
what I do! But why on Earth would you try and push it in to the sort of
traditional psychometric research, with all of the problems that has? I
can’t see the value to marketers or individuals. So, trace is great
(and I’d love to read the paper “A Young Liberal Christian Woman Is
Searching The Web”) but I’m just not convinced we can make some of the
claims being made from this data…and I am convinced that some of the
claims are non-sequiturs, they’re important points, but they’re not
coming out of the data but out of other concerns (e.g. accuracy of
categorisation isn’t the concern, the ability to derive and assign to
categories by malevolent institutions is) 
Kosinski,
M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes
are predictable from digital records of human behavior Proceedings of the National Academy of
Sciences DOI: 10.1073/pnas.1218772110