So Much Wit

If you need to find huge sets of data to work with, you could do worse than the Stack Exchange data dumps. Stack Exchange is a net­work of question/​answer sites that nerds of all stripes can use to get their ques­tions answered. All of the con­tent on the net­work is user-​​generated, and it is all licensed under a very per­mis­sive Cre­ative Com­mons license. Every cou­ple of months, the Stack Exchange guys dump their entire data­base onto the net for any­one who wants to use it. Giga­bytes and giga­bytes (com­pressed!) of inter­con­nected, struc­tured data.

Nerdgasm.

Any­way, I though I’d ingest some of that data and see what it looked like. I picked a small site, because the pop­u­lar sites on the net­work (such as Stack Over­flow) gen­er­ate way too much data to deal with quickly. Their Apple site is pretty small, but with over five and a half thou­sand users, that’s still a bit much for super-​​casual data min­ing. For­tu­nately, each site has a com­pan­ion meta-​​site, where peo­ple can ask and answer ques­tions about the site itself (in other words, if you have ques­tions about your Mac, ask them at the Apple site; if you have ques­tions about the Apple site itself, ask them at the meta-​​Apple site). Far fewer use these meta sites than use their non-​​meta coun­ter­parts, and the meta-​​Apple site only has about eight hun­dred users — a much more man­age­able dataset.

Any­way, long story short, I asked a few of ques­tions. For one thing, how many ques­tions and answers have been asked to date? Four hun­dred and forty four. And there are only about seven hun­dred users who have inter­acted with the site in any mean­ing­ful way. So, a bit under two users per post. Not too bad, I guess. But I assumed that there were some users who are really active on the site, mod­er­a­tors of a sort. Prob­a­bly these users gen­er­ate a lot of con­tent, while other users basi­cally cre­ated their accounts to ask some spe­cific ques­tion, and, hav­ing got­ten their answer (or not), never came back. Map­ping users to the posts and com­ments they’ve gen­er­ated yielded a pretty good visual sense of this. And, inci­den­tally, it’s kind of a pretty pic­ture — really, the only rea­son I wanted to write this whole thing was to post it:

At the cen­ter of each cir­cle is a user, and all the blue lines point to ques­tions, answers or com­ments that user has gen­er­ated. It becomes pretty clear that on a site with eight hun­dred users, about a half-​​dozen of them account for a dis­pro­por­tion­ately large chunk of the gen­er­ated content.

Which isn’t sur­pris­ing, and prob­a­bly isn’t even that inter­est­ing, and I’m sorry you read this far just to learn that.

The John­nies in the audi­ence might remem­ber an assign­ment from early in the first semes­ter of Fresh­man Geom­e­try: visu­al­ize the depen­den­cies within the propo­si­tions of Book I of Euclid’s Ele­ments of Geom­e­try. The 48 the­o­rems and con­struc­tions that make up the first book’s propo­si­tions are related through a com­plex set of cita­tions and ref­er­ences: while some stand on their own (such as Propo­si­tion 1: To con­struct an equi­lat­eral tri­an­gle on a given finite straight line), oth­ers use ear­lier propo­si­tions as the premise for their solu­tions. Visu­al­iz­ing these depen­den­cies can be use­ful in under­stand­ing the struc­ture of the work and the the­matic threads that hold each vol­ume of the trea­tise together.

As it turns out, visu­al­iz­ing these rela­tion­ships is not com­pletely straight­for­ward. At least, not if you want the result to be at all use­ful. There are a lot of dependencies.

I don’t remem­ber what my solu­tion looked like, but I don’t think it was very good. Some of my friends’ efforts were bet­ter; and some peo­ple on cam­pus got excep­tion­ally cre­ative with the assign­ment. One woman, whom I don’t remem­ber at all, made a sculp­ture! It con­sisted of 48 wooden blocks with the propo­si­tion num­bers burned into their faces. Each block was con­nected to its depen­dents via a series of wires, and the most frequently-​​cited propo­si­tions were low­est in the sculp­ture, form­ing the base on top of which every­thing else bal­anced. It was extremely cool, but still pretty visu­ally noisy, and not very use­ful as a learn­ing aid (in my opin­ion, which I shared at the time and was accused of being jeal­ous of the cre­ativ­ity the stu­dent had dis­played, which I was, but that was beside the point).

In a cou­ple of weeks I’m going to go to work full-​​time for a seman­tics research com­pany. Their flag­ship prod­uct is a soft­ware tool that helps ana­lysts and researchers gather, orga­nize and visu­al­ize com­plex sets of struc­tured data. Some peo­ple use the soft­ware to track ter­ror­ists. Oth­ers use it to mon­i­tor money laun­der­ing schemes. In an effort to bet­ter learn to use the soft­ware, I thought I’d see what it could do with Euclid.

The first step was to get the data. I relied on an online ver­sion of The Ele­ments to get a list of the depen­den­cies within Book I, since my copy of the book is in a box some­where and I couldn’t be arsed to get up and look for it. Once I had the data (which I actu­ally gath­ered for the first five books, since it was all there), I needed to for­mat it in a way that would be easy to import into the visu­al­iza­tion tool. CSV files work really well for this, so I made two: one that lists what propo­si­tions belong to what book, and one that lists what propo­si­tions depend on one another. For the sake of com­plete­ness, you can find the scripts I used to gen­er­ate the files here, though I assure you it’s really quite boring.

The sec­ond step was to actu­ally import the data into the visu­al­iza­tion tool. This was the inter­est­ing part for me, since the whole point for me was to get prac­tice with the tool. But, you don’t have the tool, and can’t afford it any­way (nei­ther can I), so we’ll just skip this part and move onto the pretty images.

The visu­al­iza­tion tool has a few dif­fer­ent orga­ni­za­tional pre­sets, and as I have been play­ing around with dif­fer­ent data sets, I’ve been able to see how the dif­fer­ent presents lend them­selves well to dif­fer­ent kinds of data. The first pre­set I tried is the hier­ar­chi­cal view, which most closely resem­bles the visualization-​​as-​​sculpture I men­tioned above (you can click on all these images to get the full-​​sized versions).

Neato. The propo­si­tions with the most direct and indi­rect depen­dents are at the bot­tom (I.1 and I.4 being notable among these); I.48, on the other hand, is way up at the top.

The visu­al­iza­tion tool can pro­duce other, some­what more com­pact visu­al­iza­tions. This is the star view:

Here, the propo­si­tions with the most depen­dents are at the cen­ter of the map, while those with the most depen­den­cies are on the outer edge.

Yet another view con­fig­ures the data in a big cir­cle. I recall that sev­eral stu­dents at St. John’s took this approach, though with­out the touch of putting the most depended-​​upon propo­si­tions in the center.

Finally, for the sake of com­plete­ness, here’s an orthog­o­nal ver­sion of the star view. I don’t like it as much as the oth­ers, but it might be a lit­tle eas­ier to trace the lines of depen­dency in this con­fig­u­ra­tion, as they’re all hor­i­zon­tal and vertical.

And there you have it. Once I had the data in place, I was able to do all of this with the push of a but­ton. The moral of the story? I like my job!