the semblance of objectivity in numbers
I just received my first ever first-authored conference paper rejection from FSE. The primary reasons, quoted from the reviews, include:
- “The qualitative nature of the study … is liable to misinterpretation and bias.”
- “I was expecting a quantitative analysis: is there any correlation between some of the characteristics and between [the results] and the time a bug takes to resolve and its resolution status?
- “I would have thought that what types of elements to look for in discussion should be decided before by the researchers as it should be based on the problem”
- “I was expecting concrete advice on HOW the tools should structure the discussion.”
I was hoping the reviewers would have been more epistemologically informed. For example, the first and second quotes are quite telling: they imply that some forms of empiricism are not subject to misinterpretation or bias. But quantitative empirical measures are just as subject to bias as any other measure. For example, if I had counted certain kinds of data and run correlations between these counts and other outcome measures, not only would one in twenty of them be “statistically significant” by chance, but whether there was any real meaning in the variables depends on the construct validity of the quantitative measurements. For example, if I had correlated hyperboles with bug resolution time, not only would the hyperbole measure have the same limitations as it did as a qualitative classification, but the bug resolution time would have any number of contextual factors that could influence its true reflection of the hyperbole’s impact on consensus. Transforming empirical observations into numbers does NOT make them objective, nor does it prevent bias and misinterpretation.
The third quote is ironic: this reviewer seems to believe that the only way to analyze a problem is to make some assumption about its nature upfront. The whole point of qualitative research is that the more you make upfront assumptions, the more you bias your findings. What this reviewer is proposing would have lessened the objectivity of the results and prevented us from uncovering the trends we did.
The last quote reveals the systemic bias in software engineering research (and also some HCI venues): qualitative studies are only valuable if they explicitly inform design. What this really reduces to is a view that material goods are real work, but the production of knowledge comes for free. Building a system or automating some activity, even if the system and automation are entirely impractical in the real world, is more valuable than understanding the real world. The comment also reveals the reviewer’s lack of understanding about design: innovations don’t come from studies, they come from people. Studies can support design decisions (and the results throughout our rejected submission have been quite valuable in our current design efforts), but they cannot generate ideas. People generate ideas.
Had I really wanted the paper in, I would have littered the submission with arbitrary, but seemingly objective quantifications and correlations of our data (which is what most quantifications are in software engineering papers). This has worked in past papers and is a tried and true workaround for the software engineering community’s lack of experience with qualitative methods. Reviewers would have thought, “I don’t get all of this qualitative stuff, but these numbers are great.” I decided not to do this on principle, since doing so would have only made the results seem more objective without adding any real objectivity.
So much for principle. Time to start correlating things!
tough T
I just spent a day at Edward Tufte’s course on information design at the Seattle Marriott Waterfront. I’ve always known his work, I’ve talked about it in design classes, I’ve told students to read his books, but not once have I heard him speak. Now I can confidently say that his captions speak louder than words. Snicker.
That’s not to say he wasn’t insightful. The books have always been a nice translation of classic design principles into static visual information design, but most of the course was simply him parroting his own words. What made it unbearable was that he spoke them with the lifeless apathy of a statistics professor. Oh wait, he was one.
Aside from his lack of spark, there were a number of nice things about the day. I got a box full of his books; I got a refresher on visual information design; I had a chance to think more about forms of dissemination for my research (I tire of limiting my influence to academic publications). It was also a nice calm before my early May storm of deadlines.
halfway home
I’m back in Seoul, with a lot to say, but I won’t say much.
First, let me address the elephant in the room. Hello third world visit epiphany cliche-aphant. How are you today? Yes, I’ve returned from India and I’ve seen a lot of disturbing things. I saw the Muslim slums of Mumbai, naked children running through the streets, tweens selling day old newspapers for a rupee and homeless mothers begging for money with their sick and sleeping children dangling from their arms. People were dirty, water wasn’t potable, wild dogs slept in the street, ignorant and apathetic about the armada of auto-rickshaws swerving around them.
But, I also witnessed human experience of every other kind. I watched Rolex-laden businessmen step over old women laying on the sidewalk in the heat. I saw families of five clinging to a motorcycle and to each other, smiling, laughing, and close in a way I’ve never seen western families. In all of the squalor and dirt and poverty, I saw the exact same kind of joy that those of us in post-industrialized countries seem to struggle to find. I saw nothing about human experience in India that was substantially different from the rest of the world I’ve seen, other than the clothing that people wore and range of their reach into the rest of the world.
Would those children in the slums be any better off with a pair of Nikes and a sterile heated two bedroom condo? Would they laugh any more than I saw them laugh, play any more than I saw them play? There certainly are some absolute improvements that everyone deserves, food, health, shelter, but beyond these Maslows, its difficult for me to think of a legitimate reason why my US lifestyle would bring any more happiness or joy.
Yet as much as I won’t judge the quality of life and the reach of India’s people into the global community, I can’t be impartial. I just spent a week engaging with the academic computer science community in India, forming relationships and watching unfold an incredible attempt at recreating a US style scholarly community. I’m part of this dialogue between India and the western world, helping to propagate my scholarly culture. Whether I pass high-minded judgement on India’s quality of life or not, I’ve now actively engaged in helping India’s academic community mimic and mirror that of other nations, with conferences, posters, panels, and papers, and all their inherent limitations and western bias.
In some ways, I wish India would find its own way of being scholarly. I wish it would establish its own research communities, rather than focusing solely on engaging with those in the US and Europe. I want it to find something compatible with its people and then communicate these ways to the western world. By trying to mimic the rest of the global community’s scholarly practices, it ghettoizes its own efforts. If India invented its own practices around scholarly pursuits, it would be about apples and oranges instead of Honeycrisp and Fuji. For example, instead of trying to have poster sessions (and failing because of the lack of high quality poster printers), what if they drew their posters on whiteboards, chalkboards, or paper? They could find innovative ways of communicating their work, and even find better ways than the western world. If I were India and its academic leaders, I would look at this as an opportunity to innovate and reinvent academic practices, rather than mimic them.
Time to board.
halfway to India
There’s something absurd about this trip to India. The official reason for my trip—to speak at the India Software Engineering Conference—doesn’t seem like reason enough to spend 2 hours at an airport, 11 hours in an airplane, 3 more hours in South Korea, 9 more hours on a plane, 5 hours waiting in Mumbai, then 4 hours driving to Pune. Let alone repeating this journey five days later.
Yet all of the intangible reasons for going far outweigh the inconvenience of all of this lost time and sleep. On my Korean Air flight and here in Incheon, I’m surrounded by what I find to be a beautiful, playful, exciting language. I overheard engineers talking about the parts they’ll oversee the shipping of, old grandmothers returning home to Seoul, and little babies making the journey to and from Asia and the west coast. There’s a fascinating subculture of frequent Asian travelers, particularly those who take the cheaper flights offered by Korean Air, and seeing it throb and pulse is well worth the lost sleep.
In about 12 hours, I’ll be on the other side of planet Earth, driving to the center of the Indian IT industry with a man named Mr. Mahesh, all so I can speak to a room of Indian researchers about some bits I flipped last year.
what up, blog?
How’ve you been? We haven’t chatted in a while. What up?
Really? I’m sorry. I figured that you had other people to hang out with. You totally should have called! I was busy, but I wasn’t that busy, you can always give me a ring, or write.
Well, I guess that’s true, but you’d think that since I’m anthropomorphizing you in my head, you’d just magically acquire the ability to write, speak, and act upon the world. No constraints, right, whatever goes? You are, after all, just part of my imagination.
Constraints? But I thought imagination was all about freedom. Dreams, you know, flying, falling, superpowers, there are so many things that are possible. It seems to me that when we really let our minds be free, we can imagine anything. Why would my brain limit my imagined personification of my blog to communicating only with me and not with the rest of the world?
Wait: so you’re saying that what I think, what I dream, is all just limited to things I already know? But what about learning? I can learn new ideas, acquire new perspectives, and use this new knowledge to break constraints. For example, today I learned about a whole body of work associating pronoun use in conversation to personality disorders. Now, when I reflect on my speech and others’, I’ll generate new perspectives and new ideas about the meaning in the words. There are new ideas. There are wheels, there are internets. I fundamentally disagree that everything has been discovered, everything has been invented, that everything is just a mashup.
Hm. You have a point. If I imagined you actually talking to people in the world, or even pretended that you were a separate entity from me, some might call me crazy. But no one would really believe I was crazy unless I acted on this believe. And isn’t it natural to dissociate one’s self from one’s expression of self? Isn’t that a natural part of writing? Unlike speech, I’m creating an artifact, something that literally and physically does exist separate of me. Therefore it seems perfectly natural to think of you as separate from me because you literally are.
No, I’m trying to make a point. You can influence the world if you want, because you exist separate of me. You may not have a body or a brain, but you have a connection to me. You’re like my child. And because of this, every word of you I write and then read causes me to reflect back on my own existence. You have arms and legs, but you can only use them to poke and prod and ping my brain. You may not be able to act on the whole world, but you do have influence on me. See, you’re freer than you think!
That’s true. I guess you’re free, but only on my terms. You’re less like a child and more like a robot. I’ve predetermined your scope of influence. You may lead to unexpected things, but they only occur within a certain range of possibility. Of course, a moment ago, you were saying I was under the same constraints, that the scope of my own existence is predetermined by my creators and my world.
Yeah, I’ll have to think about that. I don’t know. You’re pretty cool, blog. We should hang out more.
why do researchers choose the disciplines they do?
I been giving some thought lately to my peers’ career choices. Why do faculty choose the disciplines they do? There are the obvious reasons, like self-efficacy. For example, a physics professor probably pursued a Ph.D. in physics because she found herself good at it. An English professor may have been honored for his writing.
But I think there’s something else underlying these choices. Consider some of the extremes, such as mathematics and philosophy, or social work and education. Is there something about the determinism of mathematics that makes it attractive to certain personalities? Are there certain types of people who enjoy reveling in logic and abstraction? Do these characteristics of these areas of thought make people feel safe somehow? And the more humanitarian fields: is it driven by a strong desire to exercise values and morality? Scientists are also interesting: does the search for truth make them feel noble, or is their something trilling about the hunt for explanations?
I suppose we all have in common the desire to fill our lives with as much thought as possible. Is it insatiable curiosity or just a particularly low threshold for amusement? By that I mean we can engage ourselves in the smallest of details in the natural and artificial worlds, where as others, who could care less about research, require a much greater magnitude of novelty to be engaged.
a better fit
I’ve been thinking about buying a new car for mostly the wrong reasons.
Petty reason #1. My Civic has a terrible amount of road noise. Driving the local roads in Seattle, even at a paltry 25 mph, it’s like riding a coal mine cart in an Indiana Jones movie.
Petty reason #2. My Civic has a cassette radio and playing my tunes through my cassette adapter is, and always has been, like listening to a live recording on a cardboard speaker.
Petty reason #3. I love riding my bike with Elle, but I can’t get bike rack that fits through my apartment’s garage. I need a car that can effortlessly fit a big and small bike.
Petty reason #4. My Civic has a great turning radius and is small enough for city driving, but improving on both these dimensions would be even better for the narrow Seattle roads. It would open up a whole new world of parking opportunities!
Petty reason #5. My Civic is aging and approaching the high maintenance phase of its life. Who wants to spend time replacing a timing belt here, tires there as it slowly falls apart? Better to rid of it now and save that time for something more enjoyable, like eating cheap ethnic food on the Ave.
I suppose the only legitimate reason I have is that I feel surrounded by emblems of my broken marriage. I live with the car, the furniture, the bed, and the kitchenware of a failed 8 year relationship, and as shallow as it seems to gripe about things, it’s really hard to move on when I’m living and using things from my painful past. I don’t know how much that’s worth; I don’t think I could put a price on it. But surely a $150 monthly car payment is well below its value.
So what do I want? I’m looking at a Blackberry Pearl Honda Fit. They’re small, fuel economical, with extremely versatile space for people and things. I’m used to Hondas, their usability is one of the best in the car industry, and they’re pretty cheap. As much as I hate cars, I do spend a lot of time commuting back and forth to Kirkland, so why not have a nice, compact little car that can zip around those Lake Washington curves? The Fit wouldn’t be dramatically better than my ‘02 Civic, but would be an incremental improvement, in a petty sort of way.
sketching with words
I had a meeting today with the Human Interactions in Programming group at Microsoft Research today, along with a few other UW people, to talk about potentials for collaboration. Lots of opportunities came out of our meeting, but one thing stuck around in my head today after David Notkin made a comment about peoples’ misperceptions about what software is and is capable of. I replied to his comment this notion that there are all kinds of versions of a software artifact: versions on paper, versions in specs, versions floating around in developers’ heads, versions in users’ heads. And of course, the real version that ends up being built. The intriguing thing here is how all of these different conceptions end up affecting the actual software artifact. For example, think of two developers discussing some broken feature and how they want to fix it: if you really listen, the discussion is about future versions of the current version and all of the quality implications of the change. They fill the discussion with descriptions of the slightly modified system at a variety of levels of abstraction, with the goal of debating the merits of the modified system. The same conversation might occur between a program manager and a marketer, discussing ways of describing what it is that is being built.
It seems to me that all of these different versions of the system are essentially sketches, in the broadest design sense. They’re verbal sketches, and likely so because programs and software systems are so tricky to draw. So when two developers debate a change, they’re essentially drawing out the modification in the ether, letting the ideas linger in the air, critique the ideas in their phonological loops. I helped with a study by Mauro Cherubini looking at developers’ sketches, and the result was quite similar: in most cases, developers use diagrams in order to support discussions about changes to code.
I’m planning some studies looking at software developers’ design discussions and this idea of a verbal sketch could be quite helpful in finding meaning in the data.
my iPhone hates you
In fact, it hates everyone, especially other iPhone users. Six hour text message delays. Voicemails showing up a day later. In some cases, I’ll never even get the message. It’s driving me crazy! I had situations where people email me asking why I’m not picking up, and I’ll call them, but they won’t get my call. This is the best phone I’ve ever had, except for the phone part.
That said, I’m on the phone with Apple and AT&T right now, and they’re pretty awesome. They know what they’re talking about, they’re clear in their instructions, and the quality of the call is great, without any distracting background call center noise. Now if only they can fix it…
Trouble is twitching
Ever since I returned from a four day break in Portland, my cat Trouble has been particularly affectionate and noisy. The day I returned, he meowed through the night right next to me bed. I felt really guilty for making him so lonely, but eventually, he returned to normal.
Or so I thought. Lately, he’s been climbing high places, pointing his tail straight up and then twitching it like a big furry vibrator while he meows at me pleadingly. I pet him, I stroke him, I feed him treats, but nothing placates his strange new behavior.
I searched for twitching tails online and some sites suggest that he’s “offering a friendly, cheerful greeting” except for the twitching, which is associated with either being intensely annoyed or intensely happy. Is he still pissed off at me for leaving him alone so long? Or is he happy that haven’t left again?
Maybe I should just get him a little kitten to pick on while I’m gone.
