Skip to main content

On Automated vs. Human Testing

A lot of people are pointing to Joel Spolsky's Talk at Yale: Part 1 of 3

and it's a great read. If you read it further, I guess closer to the end of Part 1, he makes some very valuable comments on automated testing and the need for human intervention, using Vista as an example:

Speaking of the old approach to testing - "they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one."

And because of the reliance on automated scripts - "And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. "

"nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP."

Now that sounds a little harsh but it's also very true. No one will argue (ok, I'm sure some will) that UAC is a better way for managing security, or that putting all the but the reasons that Joel mentions above are precisely the reasons why Chris went off on his rants about the Vista UI last year. Now Ed did counter with some great points but many did agree and the result (6 months after the release of Vista) was still the same.

I'm all for automated testing and test suites, etc - but let's remember that software is only useful if people can use it. And the people who write it aren't always the ones who find the bugs. (great post by Goran Zidar on that)

I often think of this when I wonder why we haven't done more implementing new VFP 9 technology in various products (including Foxfire!). After all, many of the features are completely awesome to put in. But the user experience has to count for something. And while putting a variety of code into the Comments field is one way and while sure, you can build a dialog with a bunch of checkboxes on it - I don't know if you come away making an interface easier to use. Now, most of my readers here are developers - so we want the cool features and the UI on these dialogs certainly works the way most of us would expect it. But I'm taking the leap here from a development IDE to an end user product. Just as a user doesn't always get all the different format options that can be dropped into the Picture box, nor do I think they will get all of the other features. So the user experience has to be considered priority and that is not something that can happen properly with automated testing.

I'm doing some work with outsourced resources from Russian but I've also seen the same results with local contractors. Early on, I have learned that if I want a dialog to say something, I have to spell it out completely - as in word for word, spelling out each dialog. If I don't, I know others will come back and clobber me for it.

Likewise with good user interfaces, it's something everyone needs to struggle with. What is the easiest way for users to discover a feature? (some might argue that users shouldn't have to discover a feature but in this case, I equate discover to inductive user interface). Now some don't get IUI and use obviously bad examples to describe its shortcomings. But that's not the point.

It's all about consistency and while I'm sure there are some automated tests that ensure buttons are lined up perfectly and the right fonts are used, they still don't take the "human impact" into consideration. I don't think any automated test would have shown that Apple's UI decisions would be as impactful as they have been in the software world.

Those that are looking for automated tests to do everything should look into good usability tests. Morae's a useful tool here but sometimes, it's simply enough to set up a Snag It  or a GoToMeeting and just record what people are going through. 

Blogged with Flock



Update: Argh! And just as I posted this from Flock, I was presented with another UI inconsistency. WHO was it that decided that any open-source or non-MS tool should reverse the order of the buttons in a dialog so that the most commonly used option was on the furthest RIGHT instead of the Cancel?

Comments

Tamar E. Granor said…
Andrew - I'd argue that programmers/developers shouldn't be designing UI at all. We know too much, we like computers too much, and we don't see the world the way most of our users do. UI should be designed by designers.
Andrew MacNeill said…
But how realistic is that?

While it would be nice for a programming team to be made up of:

a) a designer
b) a programmer/developer
c) a documenter

That could add a fairly big overhead to a project.

Sounds like a good discussion for a FoxShow - you up for it?

Popular posts from this blog

FoxInCloud Stats

FoxInCloud sent this link a while back about their statistics regarding visits to their site:

http://foxincloud.com/blog/2017/12/27/VFP-community-lessons-from-foxincloud-site.html



What's interesting here is the breakdown of people. Yes, I think it's understandable that the Fox community is getting older.

Another factor is the growth of the mobile and web environments taking over development. These environments really do push people towards the newer non-SQL or free SQL/hosted environments but more towards hosted storage options like Amazon and Google. A tool like FoxInCloud that helps MOVE existing applications to the cloud inherently competes with those environments.

But FoxInCloud also allows developers to extend their application further by giving them a starting point using Javascript and the basic CSS (such as Bootstrap). If you're not rebuilding your application from scratch, it's certainly a great step forward.

Well, that explains CodePlex...

In a move that will be sure to anger open source (or rather anti-paid software, anti-Microsoft open source)  zealots, Microsoft is planning to buy GitHub.

A year ago, I mused about why Microsoft would shut down CodePlex and how the world needs competing source code repositories to be strong. I'm not the only one per this Slashdot article :
"...people have warned about GitHub becoming as large as it did as problematic because it concentrates too much of the power to make or break the open source world in a single entity, moreso because there were valid questions about GitHubs financial viability...." - Jacques Mattheij

I will be interested in seeing this play out - whether developers jump ship or not. Have all the efforts Microsoft has made in pushing towards open source be seen as genuine or will all the zealots jump ship or maybe even attack?

Microsoft's comment about why they shut down CodePlex referred to how spammers were using CodePlex. Well, GitHub has its own …

The World of Updates Today

I just received an update for Office 365. It certainly includes some cool features - including starting in one environment and picking it up in another environment. In recent years, I've certainly enjoined the use of Continuity on a Mac and in fact, I feel spoiled being able to start a message in one environment (even Google) and then finish it off on another.  This has become some pervasive when we were reviewing our most recent backlog at a client site, a similar feature was added to the current workload.

But with web applications, the trend is to reduce the amount of software on a client machine. I used to have automatic backup for all of my machines (thanks Carbonite!) but these days, many of my machines don't need anything beyond the core OS and some basic applications. Certainly that's the feeling with Chromebooks and even the lightweight aspect of many iOS apps. The functionality is mostly in the cloud.

When you upgrade your system, you expect it to a big update. So…