Well, here’s how it’s going to work.

1. Linda gets individual permissions to make transcripts & identities public (and keeps track of the emails granting permission).
2. Some participants may request certain edits before their transcripts are released. We do this editing.
3. Transcripts are posted online under a creative commons license. We may want to post permission emails (stripped of email addresses) alongside the data as the closest thing we have to “signed informed consent forms.”
4. The Purdue group visits Purdue’s IRB to get these interviews cleared as a public use dataset. Specifically, we’re in category 6b on page 3 of that document.

Or at least that’s the plan as of now. If we need to adjust, we’ll adjust. But if this all works, we’ll have an open data set that can be used by any future Purdue researchers for any future research without any further IRB approval – it basically places it in the same league as, say, publicly available census data.

Of course, this doesn’t solve the problem for other institutions. But Purdue has a pretty good reputation as a research university, so if we post the notice that Purdue’s IRB is ok with it, it makes it easier for other schools to go “well, if Purdue says it’s ok, we guess it’s ok here too.”

I should be clear about my bias. I feel the change idea is so undertheorized and that we only look at the people in that process as objects (they do or don’t adopt something) – but don’t look at them as learners. Those theories we do throw around (e.g., get administrative support, don’t do this until you’re tenured) seem to be a very limited view that creates barriers to change – and yet there are so many examples of people doing this stuff regardless of all the barriers…which suggests “there must be something else going on”.

The “change knowledge” idea gives me a set of lenses to explore “the other bits” – that perhaps there is a lot of knowledge out there but that we don’t give it full recognition perhaps because it runs counter to other ideas or because we have a limited view on “change” (e.g., it doesn’t count as a change if you don’t fully adopt someone else’s idea yet a change happened). So… my bias… the developmental piece of change – and the transformative theory work seems to be a useful way to think about this.

As I was writing that last email I was thinking “how to get this in the public space” and out of “email space” so – yes – let’s move it into the public space…do you think a blog is the way to go? what is something that captures our history (like a journal) but keeps the timeline/conversational bits (like a threaded conversation)? Oh – and it has to be something that is low motivation threshhold – in other words, super easy to naturally do.

And… people need to feel comfortable with it – or at least feel comfortable with being on the periphery until they feel comfortable joining in. It may be that we need to define “public” – e.g., it is in a public space but unless someone is doing a specific search to find us, [they] won’t find us – so it is sort of protected in a “we don’t know about you” place.

A thought that bubbled up in my mind (for a conversation down the road) – is if we think of this as a model – what would it mean if we invited teachers into something like this (or mentors?) who typically don’t have access to data about people talking about their experiences. Would this be a new model for linking research and practice – a form of participatory research in which educators would help researchers see the important themes through their eyes and would gain a better understanding of how this kind of research can help them?

So yeah, this is what we sound like in each other’s inboxes, long before anything gets formatted as a shiny journal paper – and this is the sort of conversations we’re hoping to expose and make available to others. We will someday have an open mailing list… but first we need to get that data public, because “we have discussions regarding non-public data” is one of the biggest reasons it’s a closed list right now.

Both are software programs made to assist professors in the tedious task of team selection, which is a complicated and highly subjective process that needs human input and oversight, but nevertheless relies on lots of tedious data-pushing best automated away so the humans can use their brains to, y’know, think. Very similar. Except one was made as a quick project by two faculty members at a tiny (300-undergrad) engineering college, and the other is a massive research project done by an entire team at a university 200 times the size – Purdue graduates more engineering students every year than Olin has ever enrolled in its entire existence to date.

So the SCOPE-o-Matic is, in effect, mini-CATME. Or rather, CATME is SCOPE-o-Matic on steroids. Both let you select and sort by a variety of hard and soft criteria; not just technical binaries like “do you code yes/no,” but things like “how comfortable are you with writing?” Both try to get the tedious stuff out of the way and narrow down the options so the final call can be made by human beings, with all the lovely squishy judgement abilities we have in our heads but not in our computers.

Similarities end there. The SCOPE-o-Matic’s data entry is done by teachers; CATME has students enter all their data. This makes sense. When you’re processing thousands of first-years, they know themselves way better than you do, and besides, do you really want to do that data entry? But when you’ve worked with 80 seniors for the past 3 years, you may have noticed things about them that they don’t know (or don’t want to admit) quite yet; the collaborative judgements of multiple mentors are likely to end up with a more accurate triangulation than the self-perception of a 21-year-old.

Then there are extra features. CATME asks students to put their schedules in, and tells teachers (when picking teams) and students (when receiving them) how much and when their mutual free times are, which is nice. It can also be used to survey students on team dynamics throughout and after the course, a feature SCOPE-o-Matic doesn’t have. Then again, Olin classes are tiny, and it’s humanly possible for a professor to sit down for “marriage counseling”with each of the 6-10 teams in his class in the span of a week. At Purdue? You’d never have time for anything else.

Different strokes for different institutions. It’s neat to see how the same idea adapts and springs up independently in multiple schools.

[0] Yes, Mark Chang actually called it marriage counseling. As I recall, my Computer Architecture team was a dysfunctional polygamous triad where all three members were in open relationships (had other team projects going on that semester) and had serious problems with the division of child-rearing (completing our labs) when people intermittently went deadbeat and refused to pay child support. The three of us remain good friends to this day, but we learned some hard lessons about team dynamics that semester: good friends don’t automatically make good teammates.

It may lead to your work being picked up by a new audience. “One of the things we’ve found is that new audiences are constantly revealing themselves to us,” the repository manager of Cal Poly said in 2009, “and that’s been the most surprising piece.” (From Jean-Gabriel Bankier and Courtney Smith’s paper on Repository Collection Policies [pdf].)

It’s also something that can benefit your students, who can also often place their work (essays, dissertations, presentations, etc) in the repositories of their colleges and universities as well. If those repositories are open access, the students have an instant public portfolio for their scholarly work, which can lead to fascinating conversations with future teachers and employers. In other words, it’s the academic equivalent to our usual argument for getting students involved in open source projects; open source gives you a portfolio of outputs of practice, whereas open access gives you a portfolio of outputs of scholarship.

Yes, grad schools can usually access published papers already, but that’s a tiny fraction of a student’s work; how many undergraduates have published papers? Even those who do will likely have a lot more good work that is not published anywhere (for instance, an undergraduate thesis), and plenty of students will want to show their portfolios to prospective industry employers, who mostly need open access to see any of this at all.

(This is, by the way, one way some faculty get into open access and institutional repositories themselves — they nudge their students to put things into their school’s repository, then go “wait, I could do that too.”)

And this isn’t just about big research schools. Jonathan Miller, the library director for Rollins College, writes about OA for liberal arts colleges.

Colleagues are surprised because they assume OA is an issue for researchers and the large universities that employ the majority of them. I argue that OA is not just the concern of research universities. In fact, it might be even more relevant for smaller colleges than for larger schools.

Rollins is a largely undergraduate, teaching intensive school with a liberal arts curriculum. This means that, at least in one sense, we need broad not deep access to information. We are net information consumers, rather than net producers. The subscription model of collecting a relatively small number of periodical titles “just in case,” doesn’t make much business sense for a school like us. What we really need is “just in time” access to a broad array of information resources, none of which will be used particularly heavily… the librarians are the faculty and students’ guides and partners in a larger, richer, but more complicated  information environment.

If you care about teaching, you should care about open access. If you’re curious and wondering where to learn more, here are the best resources that I’ve found.

Anyone who writes and publishes (researchers, authors, students, and those who want to work with them) should read this little document on Author’s Rights [pdf] by Charles Bailey that gives a clear how-to and why-to introduction on getting started with open access. It covers reasons for making things open access, explains copyright and publisher agreements, shows you how to quickly find (and respond to) the policies of the publishers you use, explains creative commons licences and open access journals… I found it to be an easy-to-read guide (10 pages not counting cover page and bibliography) that I’ll use as a quick-reference in the future.

For anyone who wants to give a quick talk on open access, use or remix Isaac Gilman’s sharp little (public-domain!) slidedeck on “open access in 15 minutes or less.”

Finally, if you’re an open access advocate at your institution and looking for some empirical data to take to people, print out this handy crib sheet for conversations on open access with both authors and administrators. It contains a list of selling points for each audience, plus (I love this) links to actual data and examples to back up as many points as possible.

It doesn’t quite cover everything, though, so you’ll also want to download all 17 pages of The Open Access Citation Advantage by Alma Swan, which is an annotated bibliography covering (as of 2010) the empirical research that’s been done on the effects of OA on, quite simply, “the things researchers get evaluated on.” It’s essentially a big table of papers and their findings, so you can (for instance) search for studies specific to your field, sort by sample size, check their analytical approaches, and so on. They consistently find that, yes, Virginia, there is an OA citation advantage. (Of the 31 studies covered, 27 demonstrated a clear OA advantage, and 4 demonstrated either no advantage of a disadvantage — if anyone takes a look at those 4, I’d love to hear what you find!)

Bells started to go off in my head. It sounds like they’re saying that the academic institutions that have the most to gain from open access are small ones, less research-based ones… teaching institutions. The same types of schools we keep on seeing in Teaching Open Source. The same professors that show up summer after summer at POSSE. These places aren’t heavily populated by the “traditional” tech research powerhouses like MIT or Carnegie Mellon; we get teaching faculty. They’re little liberal arts schools, they’re satellite campuses, they’re places where professors spend more time in classrooms than in labs.

And yet these same faculty do have to publish something, or they willperish. Could open access – or something about it, habits like it – have a turbo-boost effect on these professors and make it easier for them to meet and surpass what they’re expected to do for research? (The other two things faculty are traditionally evaluated on are “teaching” – self explanatory – and “service,” which means committees, advising, and other administrative duties.) It looks like there’s an article on exactly that - Open Access and Liberal Arts Colleges: Looking Beyond Research Institutions. I’ll be reading it next week – I’m curious what it’ll say.

Other bits and pieces I found useful:

The literature has proposed three major theories to explain why open access increases the impact of scholarship.

The Open Access Postulate theorizes that because open access articles are more easily accessed, they are read more often. Convenient access alone, according to this argument, increases the likelihood of citation.

The Early Access Postulate suggests that articles benefit from their quicker “start out of the gate” over competing articles on the same topic, and therefore the citation rate is higher for articles that are posted early in the publication process.

The third offered explanation is the Self-selection Bias Postulate, which argues that authors self-select to publish their best articles online thus increasing their citation rate, assuming that these are also the “better” articles in their respective subject areas.

from “Citation Advantage of Open Access Legal Scholarship“

The article swiftly goes on to disagree with each of them. My summaries:

• The Open Access Postulate doesn’t hold true for legal scholarship; legal researchers have access to all this material already without open access, since their libraries have subscriptions. So it’s not that these are information-impoverished law faculty who desperately seek access to any legal papers whatsoever. (That’s the conclusion of the authors – I don’t quite buy this, because access and really easy access are two different things; the water fountain may be right outside my office door, but my hydration rate increased dramatically when I started putting a water bottle on my desk.)
• The Early Access Postulate also doesn’t hold true, since the data used in this study comes from both before and after the “we can archive preprints of our work online!” revolution in law, and doesn’t show any correlation of citation rate to whether articles were put online preprint or much later.
• The Self-selection Bias Postulate looks like it may also not hold true; as more papers get placed online by default, it seems to have no effect on the increased citation rate.

Finally, when I was asking about the spread of OA through various disciplines (did Law pick up on OA faster than, say, English? Much later than Chemistry? Why? What factors influence the speed of OA adoption in a community?) Amy and Dana responded by saying “hrm, perhaps you could parse some arXiv data to find that out.” I scratched my head. “What’s arXiv?”

Turns out it’s an archive for electronic preprints of scientific papers in the fields of mathematics, physics, astronomy, computer science, quantitative biology, statistics, and quantitative finance which can be accessed online.” (Thank you, Wikipedia). It’s big, it’s historic (for the internet – it’s 20 years old), it’s awesome and impressive – and I immediately burst out laughing. “That’s  a more open, non-web-2.0 version of Scribd!” (Note that I use Scribd as an example mostly because it’s what people will know – it’s definitely not as openly accessible as I want it to be.)

If you think about it, though – they both host pdfs so they’re easier to find and access, they deliberately don’t count as “publishers” so it’s really about the access… sure, arXiv provides more scholarly-flavored metadata fields, but you could pretty much use an existing service like Scribd to build an arXiv of your own. (Although the point of arXiv is that there weren’t existing services at the time – it predates Scribd.)

So it’s a lot of fun, seeing what’s out there. Now I’m off to the humanities library to see if someone can help me figure out good search terms for answering the question “are there any information-management habits common to successful scholars?” because… I’m swimming in so much learning and information now that my coping strategies are straining, and I desperately need new ones.

Hup!

By open access, we mean its immediate, free availability on the public internet, permitting any users to read, download, copy, distribute, print, search or link to the full text of these articles, crawl them for indexing, pass  them as data to software or use them for any other lawful purpose.

–from Citation Advantage of Open Access Legal Scholarship

I spent it (unexpectedly) engrossed in Citation Advantage of Open Access Legal Scholarship (by James M. Donovan and Carol A. Watson), which took articles from an 18-year span from each of 3 law journals published by the University of Georgia and checked for correlations between citations of an article and its open access status (having “open access” status here was defined by “we could find the full text by using Google” – regardless of having the full text out there was technically legal or not).

Short version: yes, there was a nontrivial impact; OA papers got cited 58% more than non-OA ones. No surprise here; we’ve found this before. Two new things caught my attention, though.

First, how long does the “OA impact” last? 17 years. Donovan and Watson found the impact on citations trailed off over time, petering out at the 17 year mark. Why 17? No idea. (Would they have found different things if they’d looked at more than 18 years’ worth of data? I don’t know.) I wonder what you could do with a knowledge of that 17-year timeout – is that 17 years after publication, or 17 years after it first gets placed online? If it’s the latter, could individual researchers use this as a strategy to revitalize interest in work they did over two decades ago?

Second, OA had no (significant) impact on nonscholarly use of the material. The paper was written about legal scholarship, so what this means in practice is that while OA increases the use of an individual article by legal academics, it did not increase its use by courts. I wonder if similar things hold true in engineering; does OA make research more likely to be used by practicing engineers building real products, or does it only have an impact on engineering researchers? (Yes, I realize most practicing engineers hardly ever read research papers.) If it doesn’t, what strategies would facilitate the transfer of engineering research discoveries into the “real world” of “actual products” and the things that “working engineers” know?

Fascinating.

Next week I’ll finish the reading portion – far more efficiently, because I plan on coming in with printed copies of pre-triaged papers.  Boy, am I glad we budgeted some start-up time; this morning’s biggest accomplishment was getting a basic Zotero workflow up and running (not the software so much as the habits I developed in order to use it effectively). I only managed to read one paper, mostly because I spent far too much time taking overly detailed notes on it; in the future, I’ll reserve detailed note-taking for really important papers and write short summaries for the rest. (Next week I’ll arrive with a printed stack of papers already triaged by importance and really try to blast through them.)

I also found the first reading for the open* reading group Seb Benthall and I are going to do this term: selections from The Access Principle, which (as its subtitle says) makes a “Case for Open Access to Research and Scholarship” – and an empirical one, too.

First, I think it’s important to note that “ABET” and “accreditation” don’t mean the same thing. Most colleges are accredited by a few different bodies. For instance, Olin is located in the northeastern US, so in addition to ABET, it is accredited by NEASC, a regional body that grants it “real college” status (eligible for federal financial aid, able to grant student visas, etc – for a student’s “why should I care” perspective on NEASC and similar bodies in other regions, see Kim McCraw’s old Frankly Speaking article). And ABET, while known for its accreditation process, is actually a federation of 29 engineering-related societies like ASME, ASEE, IEEE, and others – so it does a variety of other things. So ABET is just one of many groups that gives colleges some sort of rubber stamp, and it does things other than accreditation.

However, when most folks say “ABET,” they are talking about either the process of getting the +1 from ABET, or the status of having that +1. This rubber stamp is applied to an entire engineering degree program, not to individual engineers – it’s a way to certify a course of study as something that “produces real engineers.” Now, there are some famous engineering programs (for instance, Stanford) that are not ABET accredited – so clearly it’s not the only way to be considered a “real engineering program,” just like getting a CS or Software Engineering degree is not the only way you can get a programming job. But it is generally accepted as a screen of quality – you don’t have a made-up degree from a sham school, you are probably a reasonably competent novice professional.

There are some clear advantages to ABET accreditation. Among other things, it allows a school’s graduates to take the FE and PE exams that will make them eligible for the title of “Professional Engineer.” (This is for the USA; other countries have their own variants and requirements for licensed engineers.) This may or may not be helpful to an engineering graduate. Of course, if you’re using engineering as a jumping-off platform to another field such as medicine, business, law, (or circus performance, acting, and world travel, as some Oliners have done) this title doesn’t matter to you or your future teachers and employers. It does matter to most civil engineers and to some mechanical engineers, largely ones doing work for the government; the exam and certification was originally designed for engineers and surveyors, if that’s any indication. But by and large, you can work as an engineer without that certification without much trouble – some subfields (like my own, electrical and computer engineering) disregard it completely.

Accreditation also makes it easier for a program’s classes to count for transfer credits, because other schools go “ah, yes, that class was from a real engineering school.” This mostly matters for transfer students and those studying abroad. As far as Olin students at the time of our accreditation 5 years ago were concerned, those were the big salient benefits; we’d prove we could (and weren’t unable to or simply being contrary), we’d get the ability to be certified Professional Engineers (which few of us have actually gone for), and credits would be easier to transfer (though it’s not impossible to transfer them without accreditation – it’s just sometimes harder). Do these reasons still hold? Are there new ones now?

There are definitely benefits to getting accredited; there’s also a cost. The ABET accreditation process is 18 months long, which is about par for the course for academic accreditation processes in general; this isn’t a trivial thing. You can read about the process if you’re curious; basically, programs are evaluated by a jury of their peers, which in this case means professors from other engineering schools.

Far more important and illuminating is the criteria for accreditation. If you do nothing else before talking about ABET, read this. It is only 24 pages long (and the first 3 are cover pages, so really 21) and describes what ABET accreditation actually means and what it requires a school to do. (For Olin, which has 3 degree programs, the relevant pages were the introduction on p. 4-7 and the bits on Electrical and Computer Engineering on p. 11, General Engineering on p. 11, and Mechanical Engineering on p. 16.) It’s important to actually look at the document, because without it people tend to make assumptions. For instance, did you know…

• ABET does not require letter grades; it just says (p5) “student performance must be evaluated.” How? By whom? How often? ABET leaves these as open questions.
• Schools do not force (for instance) MechEs to take “outdated classes like thermodynamics” (as some students have grumbled) “because of ABET.” Right on page 6: “The curriculum requirements… do not prescribe specific courses.”
• Engineering faculty don’t need to do research, have PhDs, or even have academic backgrounds at all. Also on page 6.

Finally, the recent history of ABET accreditation is fascinating. It hasn’t “always been this way” – the ABET requirements and process we know today were actually adopted (after furious debate) in 1997. Search for “EC 2000″ (it stand for “Engineering Criteria”) if you want to learn more; IJEE published a study of the impact of the changes (pdf) EC 2000 really started getting implemented in 2001 (coincident with Olin’s Partner Year) and represented a big change in how engineering programs were evaluated. It used to be that ABET looked at “teaching inputs,” which loosely translates to “students must take classes ABC.” With the advent of EC 2000, ABET looked instead at “learning outcomes,” which – again, loosely translated – means “hey, since the point of students taking classes ABC was to learn XYZ, why not just require them to demonstrate that they’ve learned XYZ, and let them accomplish that any way they want?”

So that’s some preface. As for what I personally think… around the time of Olin’s first ABET accreditation, professor Gill Pratt (now on extended sabbatical at DARPA) wrote a Frankly Speaking article on ABET, and one sentence sums up my sentiments exactly.

It is not accreditation we should worry about, but the aversion to risk that worrying about accreditation generates. — Gill Pratt

That’s why it’s so important to be informed; when you don’t know precisely what’s required, you tend to err on the side of safety – which tends to make you exceedingly boring. After all, it’s safe to do what everyone else does; they’ve got to allow that, right?

Sure. But there’s so much more you can do – think of the requirements as a jazz fake sheet to riff from. If accreditation could inspire schools to push the envelope instead of staying inside it, if the process could challenge the boundaries of what we think of as engineering and of engineering education — that would be a marvel. It would also take a lot of extremely brave people at all levels of the equation — students, faculty, administrators, and members of the evaluation team — who’d be willing to be brave, perhaps at high cost to themselves, for an exceedingly long time. And it might fail. It might fail.

Let us always keep ourselves in places of enough abundance to be able to take risks like that.

From the introduction:

This is a preliminary and extremely incomplete literature review surveying the current academic scholarship on open source and education. It can be summarized in three words: there’s not much. I’ll begin… by discussing what I am not looking at… I am not looking at the use of open source software in educational contexts… open source as an IT solution. Nor am I looking at open source with a focus on content licensing… open source as an information access solution.

Instead, I am concerned with an analysis of what sorts of practices and processes for learning are exhibited in open source communities themselves and how these practices might be made transferable back to the classroom. In other words, I see open source as a way of operating learning communities: radically cross-functional, collaboratively constructed realtime transparency.

Aaaaand here’s the whole dang thing, below. Why push now? Well, after a certain point, I decided to stop agonizing over it and release early and often because hey, maybe what I need are some other eyeballs to kick me out of the mental ruts I’ve gotten into. Also, I finished writing it and turned it in 20 minutes ago (which may explain why the quality of the writing degrades considerably in the last 3 paragraphs). It’s been a rough week. Month. Semester, really.

Literature review: the open source way and education (alpha)

Essentially, I spent my entire first semester of grad school finding that nobody’s really done research on my topic of choice (engineering education happening in open source communities) before – which is both awesome (because that means I get to do it) and depressing (because WHY AM I ALL BY MYSELF OUT HERE I DON’T KNOW WHAT I’M DOING). And now I’m standing here unsure of what to do next; I want to read some of these more deeply, talk about them with people, have folks tell me where I’m wrong and why, get into debates over a bunch of these ideas, throw them off the wall, bounce them off other people’s heads…

And so I fling this out to you, my friends. Thoughts? I apologize that this is not in comic book form – it was a little harder to get away with that for a lit review… but I’ll keep trying.

I’m tremendously grateful to everyone who took a chance on me and helped me grow; if OLPC was where I took my first stumbles into the big, wide world of open source and learned to walk, Red Hat was where I learned to run, and to run alongside the best. The chance to grow both Fedora Marketing and FUDCon into mature, community-run entities, the chance to lead POSSE from an idea to a reality that’s touched dozens of professors and hundreds of students around the world, the opportunity to play a crucial role in the early stages of Teaching Open Source, the chance to learn from everyone around me – these are experiences I couldn’t have had anywhere else, and things I could not have done without my kickass teammates, mentors, and friends. The list is far too long to include here, but suffice it to say it’s been the people who’ve made the journey as amazing as it’s been, and I have no plans to say goodbye.

I’m still going to be in many of the same spaces I was in before (especially Fedora, TOS, and opensource.com), so in practical terms, what this means is that I’m swapping a .com email address for an .edu one so I can devote more time to graduate school and other personal projects along the same general theme of Awesomeness and Learning. And I’m still doing research on the open source way — what are the elements of the methodology, and how can they be made systematically and measurably transferable and reproducible? It looks like answering that question (or whatever it turns into) will be a large part of the next few years of my life, and I’m looking forward to it and will definitely keep you all updated.

It’s a big wide world out there. Time for the next adventure.