Posts that are olin-ish
Some of my Olin buddies (Sebastian Dziallas, Colin Zwiebel, Andy Pethan, and DJ Gallagher) are putting together their first Fedora event, a FAD focused on Etherpad deployment. Predictably, it’s called the Etherpad FAD. In preparation for this, Colin asked some questions about Fedora Infrastructure that I thought other newcomers might have, so I’m posting my responses here in the hopes that people can (1) correct me if I’m wrong, and (2) transfer this information somewhere else more useful (wiki?) if I’m right.
By the way, if you’re interested in Etherpad development or deployment and would like to participate in the event, get in touch with Colin Zwiebel and he’ll get you started. Packagers, js/scala/java developers, infrastructure folks, experienced Etherpad developers and deployers along with new folks who want to learn… we need all sorts of people! It’s in the Boston area, and some travel funding is likely to be available, or you can participate remotely (I’ll be pitching in remotely from Cape Town, South Africa). Again, get in touch with Colin and he’ll get you started.
Now for Colin’s questions…
How do things normally go up on Fedora Infastructure?
#fedora-admin. That’s why I was trying to point you there. :) Really, just catch me on IRC sometime and we’ll get your questions answered there in realtime.
Do you need someone to maintain the new installation?
Probably. :)
If so, what qualifications does that person need? How can we become/find that person?
How Fedora Infrastructure works in a nutshell: if you want something (say, Etherpad) deployed in production, it has to first move through publictest (“you’ve got root on this random box, experiment and break things and configure until you think you’ve got it right”) and staging (“now that you think you know what you’re doing, write us out detailed instructions on exactly how to replicate your setup, and we’ll see if your instructions can be automated”). Once it’s verified that you’ve got things in a state where they can be automatically and stably deployed, then you go into production, which is the “hurrah! it’s launched!” state that you’re looking for.
So the first step is getting access to publictest machines so you can play around. For this, you’ll want to get formally started with the Infrastructure team, as they are the ones who can grant access. http://fedoraproject.org/wiki/Infrastructure/GettingStarted is their getting-started page; you want to get sponsored, so you’ll want to read http://fedoraproject.org/wiki/Infrastructure/GettingSponsored, and the FIG (Fedora Infrastructure Group) you want is sysadmin-test, http://fedoraproject.org/wiki/Infrastructure/FIGs#sysadmin-test.
Once you get access to the sysadmin-test group, you should have root privileges on all of Fedora’s publictest machines; an admin in the #fedora-admin channel can tell you more about that. The next step after that is filling out an RFR (Request For Resources) as described in https://fedoraproject.org/wiki/Infrastructure/RFR and you’ll soon have root access to whatever sort of environment you need to set up things.
I think that’s it, but I’m going to blog this introduction to Planet Fedora to make sure I’m not steering you wrong, and also because the text may be useful for others getting started with the Infra team.
Tuesday, August 31st, 2010 | fedora, olin, teaching open source | 3 Comments »
I ran across this on Planet Fedora and thought some Oliners (particularly ECEs such as myself) might appreciate the documentation of a random-number-generating USB key device called the Entropy Key. I thought it was a nifty “oooo, that’s how it works” writeup.
The Entropy Key uses P-N semiconductor junctions reverse biased with a high enough voltage to bring them near to, but not beyond, breakdown in order to generate noise. In other words, it has a pair of devices that are wired up in such a way that as a high potential is applied across them, where electrons do not normally flow in this direction and would be blocked, the high voltage compresses the semiconduction gap sufficiently that the occasional stray electron will quantum tunnel through the P-N junction. (This is sometimes referred to as avalanche noise.) When this happens is unpredictable, and the occurrence of these events is what the Entropy Key measures.
These noise generators are then coupled to a 72MHz ARM Cortex-M3 CPU on the device. This processor samples the generators at a high frequency, forming a stream of random bytes. These streams of bytes are then analyzed using Ueli Maurer’s universal test for random bit generators whereby the amount of entropy in the streams is estimated rather conservatively. The streams are also exclusive-ORed together and that stream’s entropy is estimated in the same manner. If the raw streams appear to have severely reduced entropy then it indicates a fault in that generator, if the third stream has low entropy then it indicates that the generators have correlated and are not independently gathering entropy. Any of those three states are considered a failure mode and will result in the eKey locking itself out of the host, returning only an error code instead of generating entropy packets.
The two raw streams are then processed further in a de-biasing process invented by John von Neumann. Their entropy is estimated after the de-biasing process has been performed. Again, if the estimated entropy in the streams is seen to vary too wildly at this stage, the Entropy Key will lock itself out. The processed streams are then mixed into a pool made with a secure hashing function. Once at least 50% more (estimated) entropy has been mixed into the pool than it could possibly hold it is finalised and another pool initialised. Once enough pools have been processed to fill 20000 bits, the totality is subjected to the tests stipulated in FIPS 140-2. These tests produce a PASS/FAIL indicator for the block. On its own, this is not useful, since a perfectly random block could quite plausibly fail the tests. The Entropy Key therefore keeps running statistics on the FIPS 140-2 tests and will lock itself out if the ratio of failed blocks to passed blocks rises above a conservative estimate of the statistical likelihood of failure.
Once the block has been analysed, regardless of its PASS/FAIL indication, it is chopped up into 32 byte packets and these are handed off to the protocol handler in the device. Through this process therefore, each 256 bit block of data handed to the host was formed from somewhere in the region of between 3000 and 5000 bits read from the generators.
This has been your geeky moment of the day. (Or if you’re lucky, one of many. ;-) Carry on.
Wednesday, August 4th, 2010 | olin | 1 Comment »
I found this mailing list conversation snippet to be very insightful, and wanted to share it.
Scott: “Open to critique” isn’t quite the same as “responsive to critique”. From an outside perspective, it seems that frequently SugarLabs is just not listening to people who offer contrary opinions. This is better than flaming them, but maybe not as good as it could be. For an end-of-year report, I’d like to see instances enumerated where SugarLabs actually internalized some outside critique and responded in a positive way — some concrete change made to the UI, or Sugar, or to process. That would be more convincing that simply stating, “we are now open to critique”.
Bernie: We’re definitely intimidating to non-technical people. At least, this is what I sensed at the Realness Summit. OLE also seems to be doing a better job at connecting with educators. I’m not completely sure what corrective actions should be. We might need to do some work on the wiki, maybe add web forums, which non-geeks tend to prefer…
Scott: I suspect that the answer to this problem does not involve installing additional software.
Later in the day, Jeff and I were having this conversation on #teachingopensource.
Jeff: Is IRC really a barrier to entry? maybe I have simply been using it too long, but it seems immediately recognizable to me. I think one barrier might be the attitudes that crop up. Even with emoticons, sometimes it’s hard to discern intent. Hard enough in email, but sometimes devastating in real time.
Mel: Actually, yes. I had a really, really hard time figuring out IRC. First, figuring out that it existed and I had to use it. Then how to get it, how to set the software up. Then what the heck networks and channels and whatnot were – and why channels? my IM paradigm was “you have a buddy list and you ping people individually.” So “chatrooms are the default!” wasn’t hard to understand once I realized it, but it took a while to realize because I wasn’t looking for it.
And then “oh man, who are all these people? I am nervous about pinging them, will they yell at me?” And then all the /commands I had to remember. It was so bewildering and terrifying and new and it was being used as a way to present new information to me at the same time, sort of like… taking your first calculus class in… Mandarin, if you’ve also just started studying Mandarin as a foreign language. You can’t concentrate much on the calculus because you’re going “zomg it’s in CHINESE.”
It’s hard to remember how hard things can be, especially when you’re surrounded by a community of people who are the ones who self-selected and made it past that hardness. By definition, if you’ve gotten into FOSS, the current participation mechanisms worked for you… so why fix them?
Because we want others to join us.
Tuesday, July 13th, 2010 | fedora, olin, olpc, sugar, teaching open source | 9 Comments »
I’m at a curriculum development workshop at Olin working on the design of POSSE. It is… well, let’s just say I think I understand what POSSE participants feel like (this is also a week-long faculty workshop; I’m the only participant who’s not a professor) – it’s an experience that’s making me see a world (in this case, curriculum design at universities) in an entirely new light, and finally starting to gain the ability to learn at the level I’d like to learn at… I’m nowhere near there, but I think this week is bootstrapping me up to the point where I can learn much of the rest by doing (a lot, for a long time – now I need a lot more experience). This workshop is giving me cultural context. Whoa.
Awesomeness of the day (one moment of many): running into Sanjoy Mahajan, an MIT professor who was my advisor for my humanities capstone on open content engineering textbooks. Sanjoy is also a FOSS geek, and he’s a visiting professor at Olin this coming year. Back in the day, we talked about software for writing textbooks. Turns out he kept working on it, and now one of his students has made an open source textbook writing software called nb – the code is actually under an MIT license, I’m told, but there’s no (easily findable) public repo of it yet, etc… we’ll fix that soon.
A Guided Tour of NB from Sacha Zyto on Vimeo.
Basically, it’s heatmapped commenting on textbooks in pdf format, inspired by the comment workflow on the GPL v3 draft back when it was a draft a few years ago. This way you can see at a glance where folks have commented, and how much, so you know what areas of the text you need to work on. See Sanjoy’s book, Street Fighting Mathematics, for an example of an annotated text.
How would this work in practice?
- source code: TeX
- compiler: LaTeX
- submit a comment/patch: write an annotation on the book using nb
- accept/push a patch: revise the upstream TeX, “recompile” the pdf in LaTeX
- forking: get the TeX source and build your own
- etc.
I don’t know how well it would work, but I’ll be poking the maintainer over the weekend, trying to get an instance of nb up, and then throwing the latest instance of the TOS Textbook up on it to see how that goes, just as an experimental “let us try this tool!” branch. Hrm. When… do I have free time? That is the hard part. If this works out, it would be amazing to have students in classes (Heidi and Tim, I’m looking at you) comment on the textbook as they’re using it.
Wednesday, June 30th, 2010 | fedora, olin, teaching open source | 1 Comment »
Had a little fun today while talking with Diana about her research. She was explaining how it’s hard for her to figure out who’s an active Fedora contributor since there are so many ways and means and places (git, wiki, lists, etc) to contribute and everyone contributes in a different place (someone may maintain a lot of packages but never blog, another person may blog and never touch the wiki, etc), so I pointed out that just about everything in Fedoraland was a website with FAS authentication, so “fire up twill, scrape ‘em all down, do some text processing, and you’ll have a per-user portfolio you can analyze to get an activity count.”
8 minutes of coding and 29 minutes of documenting later, a quick and dirty solution prototype is up in the form of FAS scraper. It takes a list of FAS usernames and makes little portfolios for each user using his/her recent activity from a variety of services (so far, just wiki recent changes and packages-maintained). This isn’t meant to prototype the architecture of the code (this code basically has no architecture, it’s 11 lines long), it’s meant to be a rough demo of desired functionality. Think about making the user-portfolios themselves more query-able and you’ll have a notion of how this could be extended – it would be neat to run queries like:
- How many people who blogged on Planet at least twice a month for the last 6 months are also frequent wiki editors?
- Show me all the users who maintain at least 10 packages and are also members of the syadmin-test FAS group?
- For each user with recent wiki edits to the F13 Talking Points page on the wiki, give me all their emails to the advisory-board list in the past month.
I’m sure you can think of better ones. For students (and professors), I actually think this might make a neat variant of a problem set I was once given in college.
To run it, you’ll need the python and python-twill packages (that’s what they’re called in Fedora, dunno about other distros), so this is what I think most people reading this will want to do, for easy copy-pasting into a shell script or a terminal.
sudo yum install python python-twill
mkdir fasscraper
cd fasscraper
wget http://mchua.fedorapeople.org/FAS_scraper/FAS_scraper.py
python FAS_scraper.py
You’ll get a directory full of output that looks like this. They look reasonably pretty on account of they’re straight scrapes of the html pages. This is the sort of thing I pull up to show students when I speak about what a “FOSS portfolio” looks like, so I might just use it to quickly autogenerate “portfolio pages” for the folks I’m introducing them to on IRC. And yes, I realize some of these services are likely to have better APIs to interface with than “scrape the webpage, then consider parsing the html in later versions.” I do not know what they are or where they are.
I am, lines-of-code-per-unit-time-wise, one of the slowest programmers I know, because my docs-per-line-of-code ratio is ridiculous. It’s an old habit that comes from writing APIs usable by mechanical engineers. Fighting the temptation to rewrite. Will probably cave at some point and do a more proper version than the kludge that’s up now, but… if someone takes it off my hands before then, I’ll rest easier and not stay up all night fiddling with this.
In terms of moving this forward, what actually needs to happen is for this to be re-architected into a good general-purpose python library for getting data from FAS-authenticated services. Do things like “instead of manually defining the list of FAS usernames in the code, grab the list of usernames from the actual FAS system.”
Any takers? The first thing to do, methinks, is to get this baby under version control.
Monday, March 1st, 2010 | fedora, olin, teaching open source | 5 Comments »
I am so proud to be a part of my alma mater right now. Really, really proud.
A couple hours ago – a few minutes past 10pm here in Boston – I threw out an email to the Olin alumni mailing list that started like this:
Olin’s got about 22 hours to write and submit a draft proposal for our
Grand Challenge Scholars Program (GCSP) to ARB – the GCSP being
something we’re pioneering for the National Academy of Engineering (NAE)
in our usual role of guinea-pig; other schools are going to be
implementing GCSPs, so this is an attempt to translate Olin’s existing
culture and curriculum into this format so it’s portable to them.
We need feedback and help. Big time. If you imagine bleary-eyed,
sleepless, incredibly hosed Olin students holed up in a team room just
before the Candidates’ Weekend crunch, you’ll have roughly the right
picture.
And lo, edits began to flow all over the wiki, everything from small typo corrections to structure additions to comments to tough questions and their answers. My various IM screens lit up; we flooded into an IRC channel and kept debating. “I’m about to sleep but this is awesome and I’m going to look at this when I wake up tomorrow morning” messages started to hit my inbox. (I mean, I did send it out at 10pm, and apparently some of my friends sleep now…)
There is an 8pm tomorrow evening deadline and hence a 6pm feedback-on-current-version cutoff – the feedback will continue to go and improve the current draft, but a couple students will be freezing a snapshot of whatever version/feedback is there at 6pm and branching it off to polish for improvement and submission to ARB (Academic Recommendations Board – the group at Olin that decides the curriculum).
It is now 2am; most folks online have now gone to sleep (a lot of Olin alumni stick around the Boston area – although we did get David Nelson popping in from Japan, which was ubercool). I got asked if logs would be sent to the list, so of course now I’m going and sending that out before I can pass out (which, at this point, I very, very, very much want to do, even if it’s early).
Oliners know how to work together to change things quickly. You just have to give them the chance to care. The cultural similarities between Olin and the open source projects I love and work in every day are striking – and now I really think we’re starting to see the two converge. FOR GREAT JUSTICE.
Wheeeeeee!
Friday, February 12th, 2010 | olin | No Comments »
Dear students involved in Fedora, Sugar Labs, OLPC, and/or any other open source project:
Please take a minute to explain to people with fancy titles why you are awesome.
That is all. I hope to see some of you in Boston in April spreadin’ the good news.
Thursday, February 11th, 2010 | fedora, olin, olpc, sugar, teaching open source | No Comments »
While reading Matt Jadud’s blog via Planet TOS, I came across the blog of Cory. Cory is one of the students working with Matt on Operation: Stick Figure Army. His recent post “how to cope with the design phase?” was about how he (as a blind hacker) goes about a process that most engineers rely on highly visual tools for (UML, sketching on whiteboards, etc). I can’t comment on his blog without a login I’m not sure of how to get, so I’m writing a longer blog entry as a response instead. (For those who don’t already know, I’m a deaf* hacker.)
First of all, I like the way Cory and his professor handled the question on how to assess his understanding of UML diagrams (a visual convention for describing program structure and a required topic in a class Cory is taking). He has to demonstrate understanding of the concepts; it’s just that the input and output methods for that understanding are different.
…even though I may not be drawing diagrams, that doesn’t mean that I’m not responsible for knowing how each diagram is used and how to describe one.
Reading Cory’s description of how he describes UML diagrams in text reminded me of the time in elementary school where my music class was going through the instruments of the orchestra; we were listening to sound clips from different instruments and had to write about each one. Since I can’t hear high frequencies, my reports went something like this: “The tuba sounds like this, the bassoon sounds like that, the piccolo has a fascinating history and an intricate key mechanism that I will now diagram…”
I don’t believe that a fundamental property of the software development cycle is that it is visual. I think we make it that way because most people think it is more convenient…
I agree. And I don’t believe that a fundamental property of high-bandwidth conversation is that it’s auditory, either. I know many people who, at the present moment, find phone conversations to be the easiest way for them to communicate with others long-distance. But that’s different from saying phone conversations are the most effective way of doing so, depending on your goals (for instance, phone conversations currently – usually – don’t get logged for posterity, let alone logged in a way that can be automatically translated). Similarly, there are undoubtedly highly effective non-visual ways of doing design. As someone who’s highly visual myself, I don’t know what they are, but I would love to learn. (One of the reasons I enjoyed reading Cory’s posts is that my hearing forces me to rely so heavily on visual input that I often forget to run thought experiments suspending the assumption that I can.)
I’d actually like to learn more about the design practice of looking at edge-case users (not sure if there’s a better term for this). Maybe posts like Cory’s can shed some insight into the advantages of non-visual design systems, or the disadvantages of visual design systems, in a way that makes both of them better for everyone (not just the visually impaired). I look a lot at the benefits of alternatives to auditory-by-default systems because I have to, and sometimes the adjustments I make end up being useful to other people.
Matt pointed out in his response to Cory’s post that the majority of the software development world does not use visual input either.
…the overwhelming majority of our communication and collaboration regarding software developing is written/verbal, not visual. That is, we’re not shipping pictures back-and-forth 24/7—we’re chatting on mailing lists, IRC, and blogs to get things done.
However, I do wonder whether the dominance of text-based communications in software development will continue as tools like inkboard (collaborative Inkscape) continue to be developed, or if an (initially secondary – possibly only within a subculture at first) alternative, more graphical/auditory discourse will start happening. The parallel for me is podcasting and vlogging. They haven’t replaced forums and mailing lists in general online discussions yet, but they are definitely a presence that I grow increasingly more disadvantaged for having to ignore.
Well, mostly ignore. Strictly speaking, I do have the advantage that I can catch some audio, and that I have friends who’ll sometimes take the time to write a video summary for me, or sit next to me while a podcast is running and re-mouth the words so I can lipread them, but for most practical purposes, that’s like saying that publishing documentation in Tamil should be entirely sufficient for English speakers because of the presence of Google Translate. It takes a lot of extra conscious effort, the availability of specific tools and helpers, a lot of extra time, and much is still lost in translation, so it’s usually not worth the investment to even try.
I find it fascinating to see how other people adjust and hack inclusion into a world that often doesn’t assume them in its default case. At least with open source I get to hack on things – and with things – that give me the freedom to shape them into what I need (yay visual system beeps!) but the burden’s still on me to do the shaping and the constant reminding of others that I need accessibility to the things they’d like me to contribute to (for instance, project meetings by phone virtually guarantee my silence). At least the burden here comes with the tools I need in order to assume it. (Mostly. We could do better, but that’s a longer post.) And I’m glad projects like Sugar try to make themselves more-hackable-by-default.
*re: “deaf” – I’m trying to get used to being able to use this word as well, though I can hear some sounds (my hearing loss is classified as “severe”) and grew up mainstreamed in the hearing world (with lots of hacks). It’s a cultural adjustment that I’m consciously learning (with tremendous latency and deep discomfort) to make.
Thursday, November 19th, 2009 | fedora, olin, sugar, teaching open source | 1 Comment »
The perfect is the enemy of the good. Instead of entertaining grandiose thoughts of posting a transcription of my talk with the slides nicely set inline, I should just start by putting out the slides themselves…
The Invisible Traceback: blockers that make potential contributors drop out (and how to fix them) – This is the talk I gave at the Ontario Linux Fest. The slides make a lot more sense with the accompanying narration, but questions are welcome, and I can always come back in and fill in the remainder of the transcript at Some Point Later.
Abstract: Unix Philosophy #12, Rule of Repair: “When you must fail, fail noisily and as soon as possible.” This applies to both code and culture; when someone gets stuck and hollers for help, they are helping their community find and fix a participation process bug. However, the new contributor on-ramp pipeline is particularly tricky to debug; potential participants often struggle in silence, giving you no indication of their presence, let alone why they were unable to begin working with your project community. We’ll go over some common blockers that quietly prevent students (and other new contributors) from beginning to participate in open source, and how to fix them no matter who you are.
Tuesday, November 10th, 2009 | fedora, olin, sugar, teaching open source | No Comments »
For those interested in open source in higher education…
The Teaching Open Source Summit (TOSS) and the Free Software and Open Source Symposium (FSOSS) it was a pre-conference to have finished, and notes are up from both the full day of TOSS and the TOS session at FSOSS, replete with plenty of whiteboard pictures. Thanks to everyone who helped with notes, from IRC backchannel transcription to whiteboard scribing to photographing to transcribing notes to editing the wiki page, and everything else that’s going to be helpful for posterity.
Here’s one of our whiteboards – the link to the notes has many more. This one is the “what is needed?” project voting board.
Now it’s off to the mailing list to see what happens next.
Friday, October 30th, 2009 | fedora, olin, sugar, teaching open source | 2 Comments »
