August 2006 – Brain Lint

Ode to unit tests and course-grained objects

I recently became a believer in the value of unit testing. I sort of understood the benefits of the test-first approach academically, but always felt that what I was writing would just be too difficult to unit test (UI stuff etc). Before this, ed I think the last thing I unit tested in a formal, framework using way was this directed acyclic graph thing I knocked out for workplace. This project dropped the prospect of taking in a pile of data and at the end spitting out some numbers, which would be easy to test, but that’s not what got me there.

The former code that did similar functionality (ok, really the entire project) is a tangled mess of collections of fine-grained value objects that mostly map to rows in tables. So a given section of code may be juggling three arrays and repeatedly iterating (where did i put that one?) to find objects that relate to objects in each of the other arrays. This is obviously inefficient and very hard to read (especially when the objects passed around have fields that are overloaded, their values depending on the computation phase as they are passed around in a disgustingly procedural manner), so I proposed taming the colections with more intelligent, course-grained objects that would serve as indexes of sorts. Now, instead of looking through the array (or several) to find the sales for item X at store Y the code could just ask an object “get me the data for item X at store Y”. Even better if there’s some kind of normalization or reduction of the data, it can be done as the object is filled with the value objects. So this presents a perfect unit testing specimen, because I can fake up some value objects, toss them in and then test what comes out, with and without manipulation.

None of that is rocket science, but it does allow two things: significant complexity can be hidden and thoroughly tested without hooking it up to an app server (fixing something in a JUnit harness is much faster than redeploying to the app server), and secondly that the code to actually do the final calculations can be astonishingly easy to write, read and integrate into the application. Indeed, in this case, the code slipped into place with only one bug.

From the outside, it seems like a lot of extra work to mock out supporting objects and write tests, but having done it it I feel like it saved me several days of total development time. The end product is better too. I’m not sure I can get behind actually writing tests first, but writing tests (in parallel in my case) definately helps to crystalize my thinking about what my objects actually need to do.

Shifty serving sizes

General Mills (secretly? sneakily?) altered the serving size of the multi bran chex I eat for breakfast. Instead of 190 calories per serving now its 160 calories. It just seems disengenuous that any food should be made to appear better for you by altering its serving size. Then again, does anyone actually eat just one serving of cereal? More mysterious is that the requisite 1/2 cup of skim milk factored in didn’t change. Do people suddenly like a higher milk to cereal ratio?

I see the benefit (to not eating too much, not so much the environment) of smaller individually packed serving sizes because there’s a certain mental barrier to opening another bag, but not the benefit of arbitrarily reducing the serving size.

30 Days

I really like the Morgan Spurlock created show 30 Days. If you’re not familiar, Morgan Spurlock is the guy who made Super Size Me, and he’s extended that type of stunt documentary to create the series. People spend 30 days in someone else’s shoes to better understand other’s lives or the other side of a polarizing issue The first season included Spurlock and his fiance spent 30 days living on minimum wage, a christian living with a muslim family, and a homophobic man living with a gay man. This season has had an atheist living with a mega-church-attending christian family, an anti-immigration minuteman living with a family of illegal immigrants, and the finale will have Spurlock going to jail for 30 days.

The iTunes music store has both seasons of the show avaiable for sale. The powerful illegal-immigration episode and a making-of the jail episode are available for free.

Time for a new phone.

My trusty Sony Ericcson T637 is in the process of shuffling off this mortal coil as you can probably tell from the picture at right. I’m looking around for a new phone but not seeing anything I really like. Cingular seems to be overrun by Motorola phones, and I’ve never liked their interface design. Their cell phone selection seems to be influenced by some echo of Henry Ford. You can have any phone you like as long as its a Razr. In four colors!

I want something that’s not really a smart phone, but rather a “smarter” phone. I don’t want a whole computer to carry around, but it would be nice if it had a basic HTML browser so I could use it with Backpack or some other online organizer. Perhaps some basic calendaring and organizer capabilities. It has to play nice with isync. Music playing and picture taking aren’t priorities because I already have those covered. Hopefully it’s not huge either.

I tend to overconstrain problems so that they can’t be solved, so maybe I’m already there with the above needs.

The Nokia Nseries looks like it could be a winner for me, but I don’t know if or when cingular would offer one of their phones, plus only the top of the line appears to support GSM850. They do offer the 6682 which appears to meet my needs, but they’re not doing me any favors; at $250 its only 50 bucks cheaper than getting it directly from Nokia, unlocked.

I like sony ericcson as well but I can’t decipher their current phone line and cingular only offers one of their phones right now, the Z525a, which doesn’t meet my needs. The k790 and w810
look like they could meet my requirements as well, but they’re not on Cingular yet either, although they are rumored to be showing up soon.

I guess there’s always ebay, but that’s complicated slightly because I have an AT&T; sim card rather than Cingular, so all the Cingular locked phones there will probably not work.

Suggestions anyone?

“Unimaginable” overused?

Lately in the news I’ve been struck by an overuse of the word “unimaginable”. Lets take a look at some examples. The first time I noticed this trend was in coverage surrounded the foiled not-even-close-to-execution plot to take down planes with liquid bombs. Here’s a quote from Deputy Commissioner Paul Stephenson of the Metropolitan Police:

… It is a very, very serious plot… Put simply, this was a plot to commit mass murder on an unimaginable scale.”

Next I saw an article in the metro (I can’t find the original quote) talking about how the returning student population was stressing the road system on top of the problems with the big dig tunnels, that one flat tire or accident could cause “unimaginable gridlock”. Searching for this article in google found this quote from a fiction excerpt instead:

The camera switched to an aerial view recorded earlier showing a scene of unimaginable gridlock stretching to the horizon in all directions.

Which is ironic, because given an aerial view, one doesn’t even have to use imagination! Another example of this use is about the evacuation preceeding hurricane Rita:

The evacuation triggers unimaginable gridlock, which in turn sparks anger, exhaustion and confusion

My final example comes from the public editor’s column in todays New York Times. The article is on the accuracy of polling data, and this quote is at the end of a paragraph on false precision.

The survey would have to interview unimaginably many thousands for that zero point eight to be useful.

I wonder whether these seemingly incorrect uses of the word unimaginable are a failure of vocabulary or of imagination?

Let’s revisit those quotes:

“mass murder on an unimaginable scale”: I can certainly imagine mass murder. In fact I don’t have too, because I could read about Hitler or Stalin. Blowing up ten airliners would kill three thousand people give or take. The death toll at the world trade center was 2,752.

“unimaginable gridlock”: I’m pretty sure I can imagine lots of cars waiting to go someplace, slowly.

“unimaginably many thousands”: I can imagine lots of thousands. How’s a million thousands. In fact in a survey, one could actually calculate (imagine that!) how many thousands it would take to have precision down to the tenth of a percentage point. Got math?

Time for some imagination.

US Centers of Population on a Google Map

I found a document (pdf) on the US Census Bureau site with data on the mean and median population centers of the United States over time. Unfortunately they didn’t stick it on a map so one can get a real sense of the slow south west migration. Fortunately this provided me with a good excuse to learn the google maps API and put the points on a map.

I’ve put the maps up here. I’m not sure I expected the median and mean to be so similar (though unfortunately there’s a hundred years less data for median). The mean ends up further west than the median because the distance of the west coast amplifies the weight of its population.

I was actually looking around the US Census site looking for some interesting data to build a Treemap with, but didn’t find anything yet with a two level hierarchy that didn’t end up at the county level. Got any ideas of something that could be interesting in treemap form?

An interactive line-chart without plugins

I wrote a quick and dirty line chart that is interactive without using flash or any other plugin – it uses Microsoft’s Vector Markup Language (VML) along with javascript. Although I feel dirty for doing something Microsoft specific, anabolics I’m planning to build it into a demo for work (where the apps are alarmingly short of data visualization), and the web-applications are IE-only due to activex plugins.

The example is here.

The chart could be improved in many ways, such as a fill under the line, some markers to indicate the original position of the chart as it is edited, larger click targets for the vertexes, some animations on mouseover. But since I’m not sure any of that will ever happen, I’m putting it up in case someone else can learn from it.

With Canvas and SVG support making it into the the other browsers, VML in IE, and a graphics api (like dojo.gfx/dojo.2d) that will hide the differences, it looks like there could be quite a bit of client-size rendering coming to the web in the not too distant future. (google maps already uses vml as a fall back from transparent pngs for drawing the route in IE)

Incidentally I did most of the work using Aptana which is a quite capable Javascript editor.

Does everyone use Java still?

I was talking to a recruiter the other day, and mentioned that I’d like an environment where people don’t use Java as the default implementation language. I feel that a company full of “Java programmers” are more likely to be less intelligent and proficient programmers and designers compared to a developers at a company that makes use of Python or Ruby or any other scripting languages (as well as more mainstream tech) would be more likely full of intelligent, intellectually curious people, because those people took the time to go and learn those new languages on their own.

While computer languages really are just a way to tell a machine what to do, making the “what to do” the most important piece of the design, I think learning several different languages makes one look at problems from different angles and come up with more elegant (and perhaps robust?) solutions. Perhaps the same way learning eskimo with its twenty something words for different types of snow would change how one would look at the weather?

Anyways having split my time recently between javascript and java, I’m frequently annoyed by the lack of first class functions and closures in Java. On the other hand Java provides a lot more API, so you win some and you lose some.

To wrap up the recruiter story, she said she doesn’t think she ever hears about jobs with python or ruby etc. There’s got to be some people out there using this stuff, but how to find them?

Click data as heatmaps

A colleague pointed out this open source project that allows users to visualize the mouse movements of users as a heatmap – the hotter the area, the more the mouse has been used there. Its a neat idea, a well executed visualization and a great that the code is shared, but I wonder about the utility of the resulting data.

Heatmaps are usually used in this context with eye-tracking data – that is it shows where the users look on the page, the movement patterns between sections, and how long they spend there. This data is useful for understanding if the layout makes sense, to understand where to place things so the user will see them.

I don’t think that cursor position is a good proxy for attention while using any application on a computer. I don’t constantly mouse over things on a page, paticularly when reading some amount of content. Maybe its the case that I do it unconsciously enough that there is some real meaning to the data, or maybe there are groups of users who do this all the time?

My colleague pointed out one good use for this – to identify elements of the page that are misleadingly affording interaction – are people clicking on stuff that isn’t clickable? Otherwise I fear people will read too much into the heat maps, and would be better off with just a click stream around the page. I wonder if even a clickstream provides solid enough data upon which
to draw conclusions with any degree of certainty.

Theres a commercial offering of a similar capability called Clicktale. They provide video simulations of the user’s mouse interactions with the page – from the limited information they have, it doesn’t look like they have visualization tools, and who has time to watch all that video?

Down with Monarch Computers

Don’t shop at Monarch Computers. If you have to make a return, you’ll regret it. I wanted a refund, instead I got a run around.

I found monarch computers browsing through Pricewatch for memory for my new macbook. It must have been late because it turns out I accidentally bought desktop RAM instead of laptop RAM. This was June 14. As soon as I opened the box I realized my mistake, so I contacted them and got an RMA number and had the memory on its way back in a day or two. They would only give me store credit, less the 15% gouging fee for restocking, so I waited my time until they processed my return. And waited. Called to ask why I hadn’t got a store credit notification yet even though the package was already there for days. I get told the “returns department” is taking care of it – but seriously how many people could actually be working for this place?

Seven days after the return gets there I finally get an email saying my store credit is ready. I instantly place an order for the correct memory so I can use my store credit. Only it turns out that they don’t apply the credit to the order:

keep in mind that orders processed on the website will be completely billed to your credit card or pay pal account, and in turn, the store credit will be refunded back to your account within 10-15 days of the order being processed.

Or do they? I get two emails that say this instead

Your RMA credit is currently being applied to your order. Any credit available to you will be refunded back to you.

Seems to be some internal confusion about their policies.

The order was placed on 6/30 and finally shipped 7/6 or so (because of the holiday). Add on the 10-15 days and my money is a little over two weeks overdue.

I start calling. They don’t even answer the phone half the time. Last monday (7/30) they told me that the return credit was to be processed that day by the “credit department”. Today (8/7) I call, again, looking for my refund and am told again it was sent to the “credit department” and should have been taken care of. I ask to speak with this mysterious “credit department” and the guy (Mark?) tells me he has no way of providing me that information (wtf?). So I tell Mark that unless I hear from someone by the end of the day I’m going to have to start a chargeback through my credit card – he takes my number but I didn’t hold my breath on hearing from them.

When I got home, I started my first ever chargeback.

Its disappointing really – I usually prefer to deal with smaller companies I find through pricewatch (especially if they’re on the east coast so shipping is faster and cheaper) but this is the first time I’ve had a problem.

Steer clear of monarch computers.