Posts that are ene-ish
James Huff and I ran the second half of our “Developmental Theories and Engineering Thinking” class this afternoon. The topic: behaviorism and social learning. Being the pedagogically adventurous type, we decided to place the following restrictions upon ourselves:
- no powerpoint slides, no reading handouts
- no small group discussions and presentations
- all active learning, ALL THE TIME
This blog post chronicles our lesson plan for future reference. We split the class into two: Mel’s team (Farrah, Tosin, Canek, Les, Justin, Francesca, and John) and James’s team (Joi, Ruth, Dana, Nick, Farshid, Tasha, and Kelsey).
Part 1: behavorial conditioning with cards
James’s team goes off to another activity in a separate room – which we’ll reveal later. We’re left with Mel’s team in the classroom with several decks of cards, a buzzer, and a meditation bell kindly lent to us by Ruth Streveler. Mel’s team is given no verbal instruction; the buzzer and bell serve as feedback whether their actions are correct – but eventually they figure out they’re playing a game with hidden rules, namely Bartok (chosen for its similarity to the familiar game Uno as well as for its mechanism for adding new rules during gameplay).
Once they figure that out, Mel stops them and explains their challenge: they are to think of several new rules (the rule-adding phase of Bartok) and then try to teach the game – including the new rules – to James’s team when they return, without speaking to them or directly instructing them, implying that they need to show them the “correct” actions through example. The group is given some time to come up with the new rules and play a few rounds in order to get used to them; Mel gives the bell and buzzer to another member of the group, and play proceeds.
Part 2: enter social learning, stage left
James and his team return. They are told to pick a person on the other team to “shadow” and to see what they can learn. Mel’s team models the desired gameplay behavior for several minutes, then each person hands their cards to their “shadow” from James’s team and gameplay continues, with Mel’s team providing feedback (via the bell and buzzer) as to whether James’s team is getting the rules “right. After a period of gameplay by James’s team, the class gets back into one large circle for discussion of the card game — the rules are revealed, the strategies discussed, behaviorism debated.
Unknown to Mel’s group, James and his team have been spending their time outside the classroom talking about Erikson’s stages of psychosocial development, and they have their own game to bring in… but they’re not telling anyone. Each person has adopted an Erikson stage and is analyzing the behaviorism discussion in light of that specific stage; they are starting all their comments in the person of someone in their chosen Erikson stage, and their goal is to get their partner from Mel’s team to do the same. This is obviously a much more subtle and difficult task than playing a card game.
Again, we leave this open-ended; bell and buzzer may be used, verbal feedback (“no, that’s not right, you mean… <rephrase>”) may be given so long as explicit instructions aren’t revealed, James may indicate allowed and not-allowed responses and facilitate the modeling by following each comment from Mel’s group with a “correct” rephrasing from someone from his group, and so forth. The discussion will probably get a little stilted and sound odd, because… well, Erikson doesn’t exactly make for the most natural flow of conversation! How long will it take Mel’s group to figure out that something is going on?
Part 3: all is revealed
Finally, after everyone is thoroughly confused and/or frustrated (or amused), we’ll reveal everything we just wrote about, and see how the class responds. Some questions:
- What was going on during the solo phase of the card game? Behaviorism, obviously… but what sort of conditioning? How was it effective or ineffective? How would it have compared to direct instruction?
- How was social learning exhibited by both groups? How did the teams plan their social learning strategies? Did the social learning group (James’s) learn the card game’s rules faster than the behavioral conditioning ones? What did it feel like to be on the receiving end?
- James’s team knew they were supposed to pick up something via social learning (the card game); Mel’s group did not. Did this make a difference?
Readings
The class had to read several hundred pages of (already-assigned-by-the-professor) work on the topics; student presenters are usually expected to expand on this material. Our original readings from Dr. Evangelou:
- Crain, W. (2005). Theories of development: Concepts and Applications, NJ: Pearson. Chs. 8, 9, 12
- Jarvis, P. (2006). Towards a comprehensive theory of human learning. London: Routledge Ch. 8
- Bandura, A. (1969). Social learning theory of identificatory processes. In Handbook of Socialization Theory and Research. D.A. Goslin Ed. Ch. 3, p. 216-262.
- Bandura, A. (1989). Human agency in social cognitive theory. American Psychologict, 44(9), p. 1175-1184.
- Kelman, H.C. (2006). Interests, relationships, identities: Three central issues for individuals and groups in negotiating their social environment. Annual Review of Psychology, 57, p. 1-26.
In lieu of powerpoint slides and such for the background/expansion, we figured “hey, why reinvent the wheel?” and decided to contribute to the Wikipedia pages on our topics so they included all the information we wanted to give the class. Therefore, our “readings for further reference” follow. Note that these articles also have good links in them (to related topics, researchers, techniques, specific studies, etc) but we centered around these as the “core.”
Tuesday, February 21st, 2012 | ene | 1 Comment »
One of the best things about blogging over multiple years is that sometimes, you really do end up writing for your future self. Case in point: around my senior year of college, I had an epiphany that I should study “Engineering:Education” that came while I was…
…posting a two-page dense ramble of engineering education resources in response to Nikki’s innocent query to the metaf07 list for thoughts on ways to do a 2-credit independent study on pedagogy at Olin. “Mel, you covered 10x more material than any 2-credit independent study could cover in a semester,” said Marco. (April 30, 2007)
Flash forward almost 4 years, and I’m in the first year of my PhD in engineering education, hunting for good books to read deeply. I’ve been flat-out sick this weekend and have a next-revision of my lit review due today. So I go dig up that email, curious about what my younger self found so fascinating, and…
Warning: this email shows the tip of the iceberg of my obsession with engineering education. I can go on about this stuff for hours.
I think Andy’s suggestion is great. Your list looks awesome. One thing I’d make sure to do is look at what other schools are doing, as well as getting some outside readings on pedagogy/educational theory, since it’s helpful to have various frameworks for looking at these kinds of things. (imo, that’s one of the great things about Meta; you learn how to apply rigorous frameworks from different disciplines to a subject you’re interested in – in this case, Olin).
So get some readings on, for instance, different schools’ grading philosophies (Chris Morse has some readings on grading and evaluation – I’ve cc’d him here, hi Chris! and one book that totally spun my head around was “Punished By Reward” by Alfie Kohn, which Gill gave me Junior year.) Zhenya, Stolk, Somerville, Allen, Lynn, Raymond, Debbie, and Rob have thought a ton about this stuff over the years too. Talk to them! Our faculty came here because they’re really into the whole “teaching” thing as well as the whole “engineering” (or math or science or whatever) thing. They are your best resources EVER.
The IEEE Education Society and the ASEE (American Society of Engineering Education, which Sherra Kerns used to be president of) have some good journals and resources, as does the NSF (although you’ll have to dig a little harder for the NSF stuff, which is mostly “oh noes, students becoming less interested in technology, must find out WHY!”)
Another helpful thing may be to learn about research methods in the humanities and social sciences (esp. education) because their “way of thinking” and how they run their studies, form their theories/ideas, analyze data, what’s important to them, the terminology, etc. is very different from the way engineers talk. Arthur Applebee’s “Curriculum as Conversation” may help clarify some of this a little, but also read at least one research paper written by an Education PhD (maybe a thesis from the Harvard Grad School of Education? Nick Tatar’s predecessor in OSL, Zach First, attends HGSE now; ask Nick or Rod if they can get you in touch with him) and compare it – and its research methods – to an article in, say, the ASEE or IEEE pubs written by an engineer on education.
Caryn Park (Mark Chang’s wife) is doing her PhD dissertation in Education and has some fascinating stuff on multicultural education (mm, discussions on diversity in college admissions!) and also some very important stuff on imperialism that y’all should read (imperialism = “dude, we Know Better so we’re going to go Help Out all these Poor Unenlightened People and make them Happier.”)
OSL [student life] people – Rod and Nick et. al – can give you a completely different view of higher education – the bird’s eye view of why we have Trustees, the history of colleges (did you know we stole the idea of grad school from Germany, or the idea of a residential college – as in, dormitories – from England? Or that the first US colleges were made exclusively to train priests? Did you know that the lecture method of teaching comes from pre-Gutenberg days when monks copied down Bibles by listening to another monk read the Bible from a lectern? Replace “Bible” with “lecture notes” and you’ll get the idea.) They also have some very interesting perspectives on education in higher education… notice how NOBODY at Olin except for some folks in OSL actually have education degrees. Typically, faculty and administration at schools past the secondary level have received no training in education whatsoever! This has some very, very interesting consequences.
Read “Teaching Engineering” and “The Torch And The Firehose,” two classics on how engineering classrooms can be/should be/are run. (Rebecca may have interesting stories on “The Torch And The Firehose.”) It’s sort of like a manual for new engr profs. Also read “Understanding By Design” which is THE classic on curriculum design, and compare it to some of the classes you’ve been through.
Read a few of Piaget’s papers (he was the one who started the line of thought that, y’know, what students do is kind of important to their learning – previously everyone was operating under the assumption that kids were blank slates and that education was moving memorized facts from the prof’s brain to the students)’. I also like Papert’s stuff (he was the first person to propose that kids could learn about computers from — gasp! — playing with computers). Then again, theory might not be your cup of tea; that’s totally cool, the rest of the readings are definitely “more fun” than the ones in this paragraph. (note: I’m the kind of person who gets a kick out building up the notion of multiplication through through agonizingly painstakingly slow moves through abstract mathematics, this stuff is the education equivalent, you have been warned.)
Do an IEEE search for papers written on Olin by Olin (and non-Olin!) faculty – there is some interesting stuff on how the original curriculum was designed, a paper on competencies from when competencies were started, about Olin’s CS curriculum, etc. There’s also a book “Educating the engineer of 2020″ which has an entire section on Olin. In terms of the history of engineering education, try to find some stuff on MIT, which is where the idea of “engineering science” (teaching engineering as a science and a discovery process) got started. It’s also where the academic field of electrical engineering got started (after WWII, the government gave MIT’s research labs 6 months after their research ended to write down everything they’ve learned; the newly displaced researchers then proceeded to take these ready-made lecture notes to colleges across the country and set up EE programs there).
Also look at past Olin material – the ABET binders we’ve collected in the conference room by Ozgur’s [design professor Ozgur Eris] office are a good start. [now-retired director of institutional research] Ann Schaffner can point you towards all sorts of statistics. (Look at ABET’s accreditation criteria, btw; everyone goes on about how they’re “restricting us” but they’re like 18 pages long – ~2 pages on general engineering programs, and a half-page each for each degree, so only ~3 pages are actually relevant to Olin.) Look at ABET reviewer comments and external Expo evaluator comments if you can get them; look at exit surveys from the classes of ’06 (and soon ’07 – I feel old now) on how well they think their Olin education has prepared them.
Other good “general public” education books: “In Schools We Trust” by Deborah Meier, anything on multiple intelligences by Howard Gardner (“Frames of Mind” or “Multiple Intelligences” seem to be the classics – he was the one who came up with the ideas of
kinesthetic/visual/mathematical/etc. intelligence instead of just one measure of IQ; “Disciplined Mind” is also good), check out the TLL (Teaching & Learning Laboratory) at MIT [now in 2012, former TLL-er Sanjoy Mahajan is on the Olin faculty] and the HGSE (Harvard Grad School of Education), and look at the Futurepaths study [by Susan Silbey at MIT] (which is looking at Olin, among other places). For an interesting view on gender in engineering education, check out Smith (and talk to Zhenya). Also talk to Gill & Brian about the engineering certificate program at Wellesley.
On teaching: “What The Best College Teachers Do” is good, “Stuff You Don’t Learn In Engineering School” is a little hokey so don’t actually read past the table of contents but it’s fascinating to see what they include (ironically, their chapter titles match up reasonably well with stuff we cover in our competencies), and “Thinking Like An Engineer” is long and dense (so pick one chapter of it, if anything) but has some cool stuff on how one can think about ethics in engineering (and teaching students about such), and also on the Philosophy of Engineering (yes, it’s actually a field [which, in 2011, I learned drives me nuts to read]) which is the tacit stuff we absorb but don’t usually think about – for instance, why are we supposed to dislike bureaucracy and like action and hacking? Why do we value idiosyncratic individuals instead of folks who “fit in”? (the “great man” theory of science and technology) How are these values passed on, consciously and unconsciously, to students as they go through engineering school?
Also on Competencies, try to get the Big Conversations video featuring Woodie Flowers’ talk, and then after your minds have been sufficiently blown by his speech, go hunt down the research study he mentions in his talk. (And let me know if you find it. I’m still looking.)
If you want to get into the more new/subversive/radical educational philosophies check out Neil Postman (esp. “Teaching as a subversive activity”), “The Saber-Tooth Curriculum” (hilarious parody of the modern education system), “The Teenage Liberation Handbook” (on unschooling), anything by John Holt or John Taylor Gatto (also on unschooling – Holt is the founder of the movement, but Gardner has a semi-relevant book also called “The Unschooled Mind”).
Look for material on the Open Courseware movement and on the rising trend of adult education (University of Phoenix, etc.) because much of the “let’s involve the learner!” trends have been pushed forward by adult students who, after years of working as professionals, wanted to have control over their own educations. Chomsky (who is pretty much provocative no matter what he’s saying) has written a book called “Chomsky on MISeducation” but it’s not the most readable thing in the world. I’d do a chapter at most.
Also, look at how other fields educate their students – one thing I’ve heard over and over again from engineering faculty is how teaching is like acting, so I’m trying to learn how drama students learn theatre in college. How do musicians learn? Historians? English majors? Dancers, studio artists, athletes, skateboarders?
Phew. I’m undoubtedly missing stuff here, but that should give you a starting point… and again, I’m totally happy to talk to people about this stuff anytime. Aaaaanytime.
I am now terribly intimidated by my 20-year-old self. And… I think I have a few more books to put atop that lit review of mine now. Yes.
Monday, February 20th, 2012 | ene, olin | No Comments »
I’m at the airport on the way back from a lovely visit to Penn State’s Schuylkill campus (many thanks to Elinor and Tom Madigan for being such gracious hosts, with thought-provoking conversation and amazing food!) and gleefully eyeing the (gigantic!) moon pies sitting beside me.
One of the folks I met last night at dinner was David Beahm, who left a comment that I started responding to… and the response grew long enough that I decided “okay, it’s a post now.” So here goes.
You are also forcing me to rethink my approach to educational videos, which have been strictly screencasts with voice-over. How do you deal with those when you encounter them?
I turn the volume up really, really loud.
I’m halfway serious about that. I can hear some (bass), so I certainly watch and enjoy videos (especially music), but usually can’t understand words in them — I’d compare it to your experience watching an unsubtitled film in a language you don’t know. You get some things, sure — but you can’t assume you’ll actually extract and learn content from them, since you simply don’t know what you’re missing.
I’ve been blessed with some extremely good and patient friends, including a wonderful boyfriend, who are in my field. The “in my field” bit is important, since I’m an engineer and most people can’t grok the technical terminology even if they can hear. As I’ve gotten into more and more advanced material, it’s harder and harder to get support, so you can imagine the nightmares I’m having of my PhD defense in a few years.
Anyway, these wonderfully gracious people occasionally transcribe short videos for me (insanely time-consuming), or sit next to me and repeat the words back while the video is playing (so I can switch between lipreading them and seeing the video), or… similarly awkward arrangements — but they’re the best we can do, and it’s better than nothing. I am incredibly grateful for this — and it obviously doesn’t scale, nor is it a system I ought to rely upon. Because it’s so labor-intensive, this happens for perhaps one video every few months. Now, compare that to the frequency you probably watch web videos at… there are people who watch more web videos in a day than I’ve had made fully accessible to me in my entire lifetime.
Anyhow, returning to the original point: as far as educational videos go, I assume by default that I can’t use them. I find a book or website or something else on the same subject (guessing from the video title). There are platforms (such as dotsub and opencaptions) that let people transcribe web videos so folks like me can understand them, but the vast majority of videos are untranscribed. To get an idea of what the video world looks like for me, mute your computer audio, go to YouTube, and start trying to watch the clips you’d normally find interesting. Suddenly, the world looks much, much smaller.
This isn’t just a problem for people with hearing loss or auditory processing issues. Students with attention deficit, students who aren’t 100% fluent in English… captioning and transcribing has been shown to help them considerably as well. On a more pragmatic, publisher-facing note, no transcripts means the video is harder to find (because search engines don’t pick up keywords in the text).
Now, I think educational videos can be excellent and I’m not saying people shouldn’t do them — they are accessible to most, and if you can make more immersive, interactive multimedia — why not? (Heck, I make these videos myself sometimes.) Some people learn visually, some auditorily… the main thing, though, is to make sure that each student can get to the content in a format that’s accessible to them.
Let me repeat that, because it’s important. We need to make sure that each student can get to the content in a format that’s accessible to them. I’m used to having to ask for this myself, and wait for it (getting the content on a delay compared to my classmates in many cases), so when someone anticipates and accommodates me ahead of time, it’s glorious. (And sometimes students struggle silently and don’t ask. Sometimes I do this, too. I’m a very proud person…)
The easy solution for video is to post the voiceover script alongside it. It’s not perfect, but it’s better than nothing. If you want to consider visually impaired users as well, things like “selecting colors with high contrast for color-blind people” and “provide a text description of the visuals so screenreaders can read that out loud” become important. I totally realize that not everyone has time to do that all the time for every little video, which is why I’m also a fan of releasing your work under a Creative Commons license so that it’s possible for others to remix it into a more accessible format (translate into different languages, subtitle, remaster, etc).
That’s probably more than you wanted to know. I’ve dealt with this all my life, and can talk about it… pretty much forever.
Saturday, February 18th, 2012 | ene, teaching open source | 2 Comments »
All right, folks – it’s studyin’ time.
As many of you know, I’m deaf. Or more specifically, I’ve had a bilateral high-frequency severe-to-profound sensorineural hearing loss since age 2. Hearing aids have historically been awful at helping out with this sort of loss, so I had them as a child but never wore them (because they didn’t, y’know, help). I speak, lipread, and have mainstreamed my whole life because that’s how my (highly geeky) young life happened to turn out, but have tremendous respect for Deaf culture and would never impose technology or coping mechanisms on others that they didn’t want. I’m also an open source hacker, electrical and computer engineer, and all-around gigantic geek, and when I found out (very) recent technology developments in the hearing-aid department have been aimed directly at my type of loss, I immediately asked where I could sign up as a guinea pig.
And so next week I’m meeting with an audiologist and a hearing aids specialist who are also researchers at Purdue, and we are going to have a big geek-fest conversation about technology options, where they’re at, how they work, the state of the field, etc etc… and what might work best for me.
I want to preload my brain with as much useful information before this conversation as possible. As readers of this blog may have noticed, I’m trying my best to keep a record of what this sort of journey looks like as a patient and a curious hacker — it may be a useful story to have out there later on.
So. What should I know? What should I be familiar with? What questions should I ask? (What are you curious about?) Go!
A few things to keep in mind:
- I’m at a large research university with an excellent audiology department, live in the library and love librarians, and can probably get my hands on any papers and books (and websites, but that goes without saying) you point me to.
- I’m an engineer. Technical things don’t scare me – I love them. Advanced math, signal processing, geekery about chips and parallel processing and embedded packaging – yes. Throw them at me.
- I am studying education, and while I’m not an expert on developmental psychology or cognition or neuroscience or any of that stuff, I’ve had more exposure to it than your average layperson, and am actively pursuing learning more. I am fully aware that part of using hearing augmentation tech includes reprogramming your brain to make sense of the new inputs it’s getting (“aural rehabilitation” is the term) and want to learn about that too.
- However, my biology/physiology/medical-literature/audiology background is not as great; I took a single intro-to-Bio class as an undergrad, and my knowledge of the way the auditory system operates is exactly equivalent to the contents of Wikipedia articles on it. I am a researcher, so medical research papers don’t look wholly unfamiliar – but they are very much not in my domain (engineering education).
- This is an intense side interest. My graduate studies and work take up the bulk of my days; I love tackling tough things with intensity, but I am time-and-resource limited in what I can do with this. So “read this entire journal series, start attending all these conferences, buy this $50,000 development kit, and prepare to do research in this field for the next 50 years” is unrealistic, but I am happy to plunge into specific difficult papers, talk with individuals, experiment with technical platforms that don’t require a lot of time or money (grad student, remember!) startup cost, and so forth.
Thank you!
Wednesday, February 15th, 2012 | ene, fedora, olin, teaching open source | 7 Comments »
Jason Priem from UNC Chapel Hill’s library/info department gave a talk on altmetrics, total impact, and decoupled journals this morning at Purdue. Actually, he’s giving it as I type this — welcome to the talk liveblog.
One common theme throughout the presentation is that scholarly communication has always been limited by the best-available technology. Writing individual letters by hand made sense when we didn’t have the printing press – at least it let us get our ideas out to others. Printing centralized journals made sense when we didn’t have the internet – we could reach far more people this way, and that was worth the constraints of large-scale centralization and all its accompanying restrictions of limited print space and needing to wait many months or even years to hear about scientific progress. But now that we do have the internet, we no longer need industrial-scale replication in order to reach a wide audience – and so the next publishing revolution will promote a diversity of outputs.
Research has several different types (or stages) of output; the web changes them all.
- Data – raw output. Logs. Pictures. Probe data. Satellite dumps.
- Analysis – what you do to the data. As computerized tools make their way deeper into the research world, it becomes easier to track the actions you took – the commands you told your statistics package to add, the codes you assigned to this portion of the interview – and suddenly it becomes possible, even easy (technologically, if not legally/culturally) to release this to the world as well.
- Stories – once we’ve analyzed the data and learned something from it, we tell the tales of our journeys in knowledge, often via journal articles.
- Conversations – how we talk with colleagues about them. The web has been transforming how we have conversations – witness the proliferation of reference managers (zotero, mendeley), blogs (wordpress, blogger), social bookmarking, and social networking software. Some of these conversations are, and could be – gasp – scholarly.
Jason discussed the example of using Twitter for research. He no longer reads the table of contents of many journals – he simply follows a hundred or so other researchers and reads the papers they tweet about. More and more researchers are beginning to use Twitter as a scholarly tool — and cooler yet, there’s no significant differences in adoption or usage between disciplines or stage of career. (See “Prevalence and use of Twitter among scholars” here.) This means English grad students and tenured chemistry faculty were equally likely to use Twitter intensively for research. I can’t help but wonder if this sort of thing might lead to interdisciplinary conversations that may not previously have taken hold.
(A better version of the graph can be viewed here.)
There are several different types of scholarly tweets. “Primary” tweet-citations (“citwations”) link directly to a scholarly paper, but far more common are secondary citations — tweets referring to blog posts that in turn refer to scholarly papers. This is due in part to questions of access; researchers are hesitant to link to papers behind a paywall their colleagues (and followers) may not have access to, but they know that everyone can read a blog post.
This isn’t an entirely new idea — it’s just that we’ve recently acquired the tools to actually do it well. The Science Citation Index was created by Garfield in 1961. The big idea was to use crowdsourcing (remember, this was 1961, long before “crowdsourcing” was a buzzword) to fill the role formerly held by individual expert judges. Instead of asking one person how “good” an article was, why not just see what researchers use — what data do they mine and cite? The more folks use it, the better it’s likely to be. In other words, Garfield invented Google PageRank decades before the internet took hold.
So why does Google PageRank make a lot more sense to us than “scholarly citation indexes?” Well, a citation index has limitations. It only deals with academic people using scholarly articles as resources for a single use – writing other scholarly articles. And of course, there are a lot more people than researchers. And even researchers do a lot more with articles than write other articles about them. There’s this entire universe of usage we aren’t capturing or counting.
For instance, I’m an engineering education researcher(-in-training); my work is designed to be read by – and impact the work of – people who do not publish scholarly articles. Engineers. Hackers. As a grad student who hasn’t yet been primary author on a peer-reviewed journal paper, my citation impact is zero – but my impact is arguably nonzero. My work on teaching open source has reached dozens of faculty, touched hundreds of students, been read by thousands of hackers, been blogged, tweeted and dented, found its way into classes being taught, textbooks being written, conference talks…
When I stand up several years from now and present my portfolio and defend my dissertation, I want this work to count. When I look for a faculty position (if I do), I want to go to a school that values the sort of impact I care about — and I want a way to show them what that is and how they can evaluate me on it. That’s the idea behind Altmetrics, and I’m sold. I think I’ll pitch Shannon on letting me do my end-of-term project on publicly instrumenting my scholarly life to gather and display altmetrics and getting all my old research open-access, so I’ll be clear to “do it right” going forward.
Altmetrics impact is mostly orthogonal to traditional citation impact, so if you care about it, it’s important to make sure you gather it because it’ll be invisible in your portfolio otherwise. There’s a great slide showing the correlation of one type of citation to the other (the image below, grabbed from Jason’s slides). You can see that html and pdf downloads correlate with everything, that social media has its own cluster of correlation in the lower right, and that traditional scholarly citations have their cluster in the upper left. But the picture is clear; relying on scholarly citations alone misses a giant portion of the real impact that your work is having.
You can also look at large bundles of altmetrics data and see the different “types” of articles — there are, of course, a few (3%) that become popular with all walks of people, scholarly and nonscholarly, as well as many articles (a bit over 50%) that aren’t popular with anyone at all. But there are articles that do well with traditional citation metrics but aren’t “popular” on the internet – canonical papers, methods papers, things researchers cite as foundational work. And there are articles people share on scholarly networks like Mendeley but don’t cite in their own papers – what do we make of those? (There are more of these than we may think. 80% of the articles in PLoS are in at least one researcher’s Mendeley library.) How about articles with a high Mendeley ranking but no Facebook posts — are those the ones of interest to a specialized population but nobody else?
As for gaming the system — yes, every system can be gamed, including our existing citation system. But the more data you have, the easier it is to spot gaming by eye or by automation — it’s a classic instance of “more eyeballs make bugs shallow,” – the eyeballs need sufficient data on the “bugs” to make them “shallow.”
All right. We’ve talked about a different sort of citation and impact tracking system, but we’re still applying those trackers to conventional outputs – papers in scholarly journals. What if we applied web tools to those as well, and started publishing with the best tools the 21st century had to offer, instead of the best tools the 17th century had?
Let’s talk feature set. Anything that tries to supplement or replace journals will have to have feature parity on four fronts: certification (“yes, this research is high-quality”), dissemination (getting research out there), archiving (keeping it findable in perpetuity), and registration (are you who you say you are?).
Of these four, existing simple web tools do a far better job of everything except certification. (Heck, mailing lists do a better job.) Certification is still vital, though; it’s why we have peer review. We want high-quality research, and high-quality anything requires careful scrutiny, feedback, and filtering.
Jason’s point was that this filtering doesn’t need to be done in a centralized place by a single group of people. And he launched into a description that looked surprisingly similar to the discussions on content stamping we were having at OLPC some years back.
And it’s already being done. “My twitter feed is like a private journal,” Jason said. “The people who care about my work – the… you know, four people who care about my work – they follow me on twitter, and they’ll read everything I write, and that’s peer review.” More large-scale examples are F1000 and ArXiV, and there are others.
Like any good hacker, Jason had a call-to-action in his talk. Want to try things out for yourself? Check out Total-impact, a web tool that lets people put in collections of (scholarly or nonscholarly) materials and get a wide range of citation stats on them. I wonder how well the opensource.com/education articles play in the semi-scholarly spaces… probably not that well (since that’s not the intended audience), but we could be surprised.
Tuesday, February 14th, 2012 | ene, olin, teaching open source | 2 Comments »
As our adventures in radically transparent engineering education research continue, we find ourselves staring at IRB and going “well, all right, how is that going to work?”
Well, here’s how it’s going to work.
- Linda gets individual permissions to make transcripts & identities public (and keeps track of the emails granting permission).
- Some participants may request certain edits before their transcripts are released. We do this editing.
- Transcripts are posted online under a creative commons license. We may want to post permission emails (stripped of email addresses) alongside the data as the closest thing we have to “signed informed consent forms.”
- The Purdue group visits Purdue’s IRB to get these interviews cleared as a public use dataset. Specifically, we’re in category 6b on page 3 of that document.
Or at least that’s the plan as of now. If we need to adjust, we’ll adjust. But if this all works, we’ll have an open data set that can be used by any future Purdue researchers for any future research without any further IRB approval – it basically places it in the same league as, say, publicly available census data.
Of course, this doesn’t solve the problem for other institutions. But Purdue has a pretty good reputation as a research university, so if we post the notice that Purdue’s IRB is ok with it, it makes it easier for other schools to go “well, if Purdue says it’s ok, we guess it’s ok here too.”
Wednesday, February 8th, 2012 | ene, olin, teaching open source | 1 Comment »
It’s been interesting to hear everyone’s reflections as we continue our journey towards radical realtime transparency. I’ve asked Robin for permission to share some of her thoughts on that and on the progress of our little research project. Here they are in full, emphasis mine.
I should be clear about my bias. I feel the change idea is so undertheorized and that we only look at the people in that process as objects (they do or don’t adopt something) – but don’t look at them as learners. Those theories we do throw around (e.g., get administrative support, don’t do this until you’re tenured) seem to be a very limited view that creates barriers to change – and yet there are so many examples of people doing this stuff regardless of all the barriers…which suggests “there must be something else going on”.
The “change knowledge” idea gives me a set of lenses to explore “the other bits” – that perhaps there is a lot of knowledge out there but that we don’t give it full recognition perhaps because it runs counter to other ideas or because we have a limited view on “change” (e.g., it doesn’t count as a change if you don’t fully adopt someone else’s idea yet a change happened). So… my bias… the developmental piece of change – and the transformative theory work seems to be a useful way to think about this.
As I was writing that last email I was thinking “how to get this in the public space” and out of “email space” so – yes – let’s move it into the public space…do you think a blog is the way to go? what is something that captures our history (like a journal) but keeps the timeline/conversational bits (like a threaded conversation)? Oh – and it has to be something that is low motivation threshhold – in other words, super easy to naturally do.
And… people need to feel comfortable with it – or at least feel comfortable with being on the periphery until they feel comfortable joining in. It may be that we need to define “public” – e.g., it is in a public space but unless someone is doing a specific search to find us, [they] won’t find us – so it is sort of protected in a “we don’t know about you” place.
A thought that bubbled up in my mind (for a conversation down the road) – is if we think of this as a model – what would it mean if we invited teachers into something like this (or mentors?) who typically don’t have access to data about people talking about their experiences. Would this be a new model for linking research and practice – a form of participatory research in which educators would help researchers see the important themes through their eyes and would gain a better understanding of how this kind of research can help them?
So yeah, this is what we sound like in each other’s inboxes, long before anything gets formatted as a shiny journal paper – and this is the sort of conversations we’re hoping to expose and make available to others. We will someday have an open mailing list… but first we need to get that data public, because “we have discussions regarding non-public data” is one of the biggest reasons it’s a closed list right now.
Wednesday, February 8th, 2012 | ene, olin, teaching open source | No Comments »
Debbie Chachra was brave enough to volunteer as the first faculty subject for my “let’s get everyone’s stuff open access! project on Olin’s institutional repository!” Thank you, Debbie. I finally got all her journal publications from her time at Olin into the repository; it took about an hour per publication. (Now that I understand how to work the software better, it’s down to 15min/publication. Still labor-intensive, though.)
This means all the works are listed online – but not everything is open access and available online. I’ve been doing some copyright checking, and things fall into 3 categories:
Category 1: The publisher already makes the full text of your paper available freely online (i.e. “they are already open access, so let’s just link to them”). We’re done, no action needed.
Category 2: The publisher doesn’t have your paper freely accessible online, but is ok with you putting it up there yourself. Specifically, these publishers allow us to upload postprints (the edited document after reviewer comments – what it looked like before they formatted it with all the journal branding) without asking for further permission. I asked Debbie to send me postprint pdfs for these; once I have them, I’ll upload them, and that’ll be done too.
Category 3: Some publishers don’t grant you anything at all, so need to specifically request permission to make anything available to anyone. So I’ve asked Debbie to decide what version she wants to try to open up (preprint, postprint, etc) and send that version to me so I can prepare a letter to the publisher asking for permissions just for that specific document. As soon as they say ‘yes,’ we’ll be able to post that too; it’s just that the road is a bit longer.
Some things I’m learning as I go through this process:
- I want to set myself up for open access and download/citation metrics now, as a grad student, before I even have journal publications – I want my publications list to be completely up-to-date and fully instrumented for “impact measurements” at any given moment.
- I am now proactive about copyright assignment, open licensing, etc. for everything scholarly I do. I was pretty proactive before, but this… goes to a different level.
- I am looking at the few scholarly things I’ve done in the past – conference papers, panels, presentations – and trying to get retroactive permission for that stuff to be posted, while everything’s still fresh in my mind.
It’s like scholarly housekeeping. I tell myself it’ll make a giant difference when I go up to defend my dissertation someday, years down the road – and when (and if) I apply for postdocs and faculty positions, and when… the list goes on and on. I gotta keep this somewhere, and it’s best to start when your career is young.
Wednesday, February 8th, 2012 | ene, teaching open source | No Comments »
Today was a busy day. Over lunch, Purdue Librarian Mike Witt regaled me with historic tales of the early days of Purdue’s institutional repository (IR). He should know: he started and spearheaded it. He’s also the CS librarian and a huge open access advocate, which explains another reason why we were sitting across from each other in the dormitory dining hall, chattering busily between mouthfuls of strawberry cheesecake. (Purdue visitors: Earhart Hall has an incredibly impressive repast. I would go again. Recommended!)
“The point of an institutional repository’s collection policy is so that you can say no.”
It’s not about what you will collect, it’s about what you won’t collect – how you don’t want to use your limited resources. Lunch with Mike was a joy; his casual comments (because he’s deeply embedded in his field) expanded to a wealth of information when I tracked them down. For instance, the Panton Principles (rationale for open data) – how did I not see this before?
I learned a lot about how someone thinks about an institutional repository. (I know it’s one view out of many possible ones, but still – getting a rich slice of that one view was extremely useful.) Content in Purdue’s IR is organized the same way its departments are organized; interdisciplinary research usually comes out of Discovery Park and goes in its corner of the repository. There are common actions that an IR’s workflow should handle well: consuming (downloading someone else’s papers), submitting (sending in your own), accepting (what it sounds like), and batching (creating collections of papers). Make sure you check and know all four.
Mike confirmed some other things I had been thinking about, like how catering to faculty self-interest (“ah, I can track publication downloads and use this for my tenure case!”) actually works well. Mandating something that should ostensibly be in faculty’s self-interest (“only publications listed in the IR count for your tenure evaluation!”) is… hard, and I’m not sure if I agree with that goal (I still live in a dream world where people do good things without being required to do them, apparently) but it’s got an undeniable impact on compliance rates.
On the way back, he introduced me to the work of Jason Priem, who looks at altmetrics – can we more accurately measure scholarly impact if we look at social media instead of merely journal citations? – and whose open access koans made me grin. Apparently Jason is speaking at Purdue in 2 weeks – I’m totally going. And apparently Jason is getting his PhD at the same place as Bryan Behrenshausen, last summer’s opensource.com intern (and the man behind the POSSE profiles from last summer – we’re currently having a debate about the future of POSSE, by the way).
My brain is swimming — and I haven’t even started my German translation homework yet. It’s been a good, good day. Extremely good. A day full of The Learn.
Thursday, February 2nd, 2012 | ene, teaching open source | 2 Comments »
“Quick, Joi — I need a codename so I can blog about our project. What’s your favorite baby animal?”
And so begins the public chronicle of Project Puppy. I must apologize for the obscurity; we’re still waiting to get consent from our interviewees to be 100% transparent about the whole project, but the short version: Project Puppy is radically transparent engineering education research. What would it mean to run an engineering education project like an open source one – if you assumed an abundance rather than a scarcity mentality, if you shared rather than hoarded your data, if you welcomed surprises and “uninvited” contributions instead of carefully curating access? What would happen?
At least that’s the lens I’m bringing to it – my other collaborators all come from different backgrounds and perspectives. Linda Vanasupa from Cal Poly started the whole story; some time ago, Linda recorded 8 fascinating interviews, but didn’t have time to go through and analyze them. So she poked Robin Adams at Purdue, who brought up the idea to a few grad students: Joi-Lynn Mondisa (doing mentoring research), Junaid Siddiqui (doing transformation research), Dana Dennick (using conceptual change frameworks), myself (radical transparency and the open source way). And off we ran.
We’ve poked around the data a bit, but today was the first conversation we had about what it might mean to make this an “open research project.” What if we asked our interviewees – there are only 8, after all – for permission to release their transcripts under a Creative Commons license (Linda’s idea) and then did our coding process in public so that people could see what it looked like to do engineering education research?
Sure, there are concerns to deal with (for instance, Junaid brought up the question of what happens if someone does a writeup on “our” data that is a distorted misrepresentation – how would we deal with that?) but this opens up some interesting possibilities. For instance, not only would we have (we hope) some papers, but we’d get out a manageable-sized dataset for coding practice in a qualitative methods research course for anyone who wanted it. We’d be working with questions of open access – which Dana, with her library science background, intellectually understands but doesn’t practice, and which I, with my open source background, practice but may not be fully conscious of. We’d be looking at an interesting model of research mentorship and resource allocation; Robin already mentioned that it was refreshing for her to not be “the boss” of the project, and to not have the pressures of funding breathing down our necks!
In short, it’s an adventure. Now, Project Puppy isn’t about radical transparency per se. It’s a “normal” engineering education research project (on a topic I can’t yet reveal, but Linda’s working on that) we’re trying to conduct in a radically transparent way. My job, in part, is to model this radical transparency for the rest of the group – and so here comes this blog post. More stories to come!
Wednesday, February 1st, 2012 | ene, teaching open source | 6 Comments »
