Skip to main content

On Automated vs. Human Testing

A lot of people are pointing to Joel Spolsky's Talk at Yale: Part 1 of 3

and it's a great read. If you read it further, I guess closer to the end of Part 1, he makes some very valuable comments on automated testing and the need for human intervention, using Vista as an example:

Speaking of the old approach to testing - "they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one."

And because of the reliance on automated scripts - "And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. "

"nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP."

Now that sounds a little harsh but it's also very true. No one will argue (ok, I'm sure some will) that UAC is a better way for managing security, or that putting all the but the reasons that Joel mentions above are precisely the reasons why Chris went off on his rants about the Vista UI last year. Now Ed did counter with some great points but many did agree and the result (6 months after the release of Vista) was still the same.

I'm all for automated testing and test suites, etc - but let's remember that software is only useful if people can use it. And the people who write it aren't always the ones who find the bugs. (great post by Goran Zidar on that)

I often think of this when I wonder why we haven't done more implementing new VFP 9 technology in various products (including Foxfire!). After all, many of the features are completely awesome to put in. But the user experience has to count for something. And while putting a variety of code into the Comments field is one way and while sure, you can build a dialog with a bunch of checkboxes on it - I don't know if you come away making an interface easier to use. Now, most of my readers here are developers - so we want the cool features and the UI on these dialogs certainly works the way most of us would expect it. But I'm taking the leap here from a development IDE to an end user product. Just as a user doesn't always get all the different format options that can be dropped into the Picture box, nor do I think they will get all of the other features. So the user experience has to be considered priority and that is not something that can happen properly with automated testing.

I'm doing some work with outsourced resources from Russian but I've also seen the same results with local contractors. Early on, I have learned that if I want a dialog to say something, I have to spell it out completely - as in word for word, spelling out each dialog. If I don't, I know others will come back and clobber me for it.

Likewise with good user interfaces, it's something everyone needs to struggle with. What is the easiest way for users to discover a feature? (some might argue that users shouldn't have to discover a feature but in this case, I equate discover to inductive user interface). Now some don't get IUI and use obviously bad examples to describe its shortcomings. But that's not the point.

It's all about consistency and while I'm sure there are some automated tests that ensure buttons are lined up perfectly and the right fonts are used, they still don't take the "human impact" into consideration. I don't think any automated test would have shown that Apple's UI decisions would be as impactful as they have been in the software world.

Those that are looking for automated tests to do everything should look into good usability tests. Morae's a useful tool here but sometimes, it's simply enough to set up a Snag It  or a GoToMeeting and just record what people are going through. 

Blogged with Flock

Update: Argh! And just as I posted this from Flock, I was presented with another UI inconsistency. WHO was it that decided that any open-source or non-MS tool should reverse the order of the buttons in a dialog so that the most commonly used option was on the furthest RIGHT instead of the Cancel?


Tamar E. Granor said…
Andrew - I'd argue that programmers/developers shouldn't be designing UI at all. We know too much, we like computers too much, and we don't see the world the way most of our users do. UI should be designed by designers.
Andrew MacNeill said…
But how realistic is that?

While it would be nice for a programming team to be made up of:

a) a designer
b) a programmer/developer
c) a documenter

That could add a fairly big overhead to a project.

Sounds like a good discussion for a FoxShow - you up for it?

Popular posts from this blog

Programmers vs. Developers vs. Architects

I received an email this morning from Brandon Savage's newsletter. Brandon's a PHP guru (works at Mozilla) but his newsletter and books have some great overall perspectives for developers of all languages. However, this last one (What's the difference between developers and architects?) kind of rubs me the wrong way. Either that, or I've just missed the natural inflation of job descriptions. (maybe, it's like the change in terminology between Garbage man and Waste Engineer or Secretary and Office Administrator)

So maybe it's just me - but I think there's still a big difference between Programmer, Developer and then of course, architect. The key thing here is that every role has a different perspective and every one of those perspectives has value. The original MSF create roles like Product Manager, Program Manager, Developer, Tester, etc - so every concept may pigeon hole people into different roles. But the statements Brandon makes are often distinctions I…

Security in Windows 10

 discusses some Windows 10 privacy settings and their implications.

"Finally, we will access, disclose and preserve personal data, including your content (such as the content of your emails, other private communications or files in private folders), when we have a good faith belief that doing so is necessary." "In other words, Microsoft won't treat your local data with any more privacy than it treats your data on its servers and may upload your local data to its servers arbitrarily"
I did a quick install on a VM choosing the Express settings. When I fully deploy this on a real workstation, I will likely choose to wade through all of the individual pages, as David recommends.

Of course, losing one's privacy is nothing new - it's happening all over the place (despite Santa Ana's police force's lawsu…


I'm not TRYING to be "fanboy-flame bait" but what I saw yesterday was a typical "Do it this way, now do it this way and then we'll go back to this way" all over again.... a move similar to what Microsoft does to developers on an ongoing basis.

Remember the first iPhone? Smooth and curved, at least as far as it could be back then. I still pull out my 3G and can see the curves on it.

Then the 4 came out and "boxy" was all the rage. Everything should be "tight with corners"

Now iPhone 6.... smooth and curvy is back. Granted I don't have the actual device yet, but that's the message.

Guess that means the iPhone 8 will be back to boxy.

And honestly, Apple Watch is not worth "one more thing" --- especially when everyone knows it's going to be shown. "One more thing" would be something no one saw coming.  The device itself ? Very interesting and yes, definitely lots of potential but "one more thing" wor…