Skip to main content

On Automated vs. Human Testing

A lot of people are pointing to Joel Spolsky's Talk at Yale: Part 1 of 3

and it's a great read. If you read it further, I guess closer to the end of Part 1, he makes some very valuable comments on automated testing and the need for human intervention, using Vista as an example:

Speaking of the old approach to testing - "they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one."

And because of the reliance on automated scripts - "And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. "

"nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP."

Now that sounds a little harsh but it's also very true. No one will argue (ok, I'm sure some will) that UAC is a better way for managing security, or that putting all the but the reasons that Joel mentions above are precisely the reasons why Chris went off on his rants about the Vista UI last year. Now Ed did counter with some great points but many did agree and the result (6 months after the release of Vista) was still the same.

I'm all for automated testing and test suites, etc - but let's remember that software is only useful if people can use it. And the people who write it aren't always the ones who find the bugs. (great post by Goran Zidar on that)

I often think of this when I wonder why we haven't done more implementing new VFP 9 technology in various products (including Foxfire!). After all, many of the features are completely awesome to put in. But the user experience has to count for something. And while putting a variety of code into the Comments field is one way and while sure, you can build a dialog with a bunch of checkboxes on it - I don't know if you come away making an interface easier to use. Now, most of my readers here are developers - so we want the cool features and the UI on these dialogs certainly works the way most of us would expect it. But I'm taking the leap here from a development IDE to an end user product. Just as a user doesn't always get all the different format options that can be dropped into the Picture box, nor do I think they will get all of the other features. So the user experience has to be considered priority and that is not something that can happen properly with automated testing.

I'm doing some work with outsourced resources from Russian but I've also seen the same results with local contractors. Early on, I have learned that if I want a dialog to say something, I have to spell it out completely - as in word for word, spelling out each dialog. If I don't, I know others will come back and clobber me for it.

Likewise with good user interfaces, it's something everyone needs to struggle with. What is the easiest way for users to discover a feature? (some might argue that users shouldn't have to discover a feature but in this case, I equate discover to inductive user interface). Now some don't get IUI and use obviously bad examples to describe its shortcomings. But that's not the point.

It's all about consistency and while I'm sure there are some automated tests that ensure buttons are lined up perfectly and the right fonts are used, they still don't take the "human impact" into consideration. I don't think any automated test would have shown that Apple's UI decisions would be as impactful as they have been in the software world.

Those that are looking for automated tests to do everything should look into good usability tests. Morae's a useful tool here but sometimes, it's simply enough to set up a Snag It  or a GoToMeeting and just record what people are going through. 

Blogged with Flock

Update: Argh! And just as I posted this from Flock, I was presented with another UI inconsistency. WHO was it that decided that any open-source or non-MS tool should reverse the order of the buttons in a dialog so that the most commonly used option was on the furthest RIGHT instead of the Cancel?


Tamar E. Granor said…
Andrew - I'd argue that programmers/developers shouldn't be designing UI at all. We know too much, we like computers too much, and we don't see the world the way most of our users do. UI should be designed by designers.
Andrew MacNeill said…
But how realistic is that?

While it would be nice for a programming team to be made up of:

a) a designer
b) a programmer/developer
c) a documenter

That could add a fairly big overhead to a project.

Sounds like a good discussion for a FoxShow - you up for it?

Popular posts from this blog

Well, that explains CodePlex...

In a move that will be sure to anger open source (or rather anti-paid software, anti-Microsoft open source)  zealots, Microsoft is planning to buy GitHub . A year ago, I mused about why Microsoft would shut down CodePlex and how the world needs competing source code repositories to be strong. I'm not the only one per this Slashdot article  : "...   people have warned about GitHub becoming as large as it did as problematic because it concentrates too much of the power to make or break the open source world in a single entity, moreso because there were valid questions about GitHubs financial viability...." - Jacques Mattheij I will be interested in seeing this play out - whether developers jump ship or not. Have all the efforts Microsoft has made in pushing towards open source be seen as genuine or will all the zealots jump ship or maybe even attack? Microsoft's comment about why they shut down CodePlex referred to how spammers were using CodePlex. Well, GitHub

Attending Southwest Fox 2019 could change your life - Find out how

Southwest Fox is coming up in October and as I do every year, I spoke with the organizers Rick , Doug and Tamar on the FoxShow. Deadlines for Southwest Fox: Super-saver price (before July 1): $695 Early-bird price (before August 1): $770 Regular price (August 1 and later): $820 This year, I took a different approach with separate shows for each organizer but the main message is still the same : July 1st is their Go/No-Go date. Conferences don't talk about this very often. I don't think developers really question if Apple will hold their WWDC in June or Microsoft will hold their Build conference - but that's because those conferences are vendor-led. Southwest Fox is a community-driven conference - it's not driven by a company with an agenda. Listen to the interviews and you can hear how important each of the organizers feel the live connection between speakers and among attendees.

Virtual FoxFest - A New Way to Conference

If you haven't been keeping up with the news around the Fox community, the Southwest Fox conference has gone digital now showing up as  Virtual FoxFest .  At $49, it's a steal and a great way to learn some new ideas and get inspired. While the reasoning for this change is fairly obvious with the year of COVID - for me, this is something that has been a long time coming. I appreciate many people's needs for a physical conference but the world is very large and it's difficult to get people from around the world into a single physical location. I recently attended a single-track conference via YouTube (a Quasar conference). YouTube's Live stream provided a very handy way to watch, rewind and communicate with people online. While Tamar, Doug and Rick are still making decisions related to the streaming platform, there are lots of great options available. I'm really looking forward to it. The FoxPro community has also really felt its international roots