Thursday, September 15, 2005

Ring my friend I said you’d call Doctor Robert

So, after a very slow period of blogging, mostly due to hectic home AND work life, I got my StatCounter email regarding hits to the site.

Much to my surprise I saw a spike in activity, and I was able to track it back to this blog entry by one Robert Collins. So in pure InstaPundit fashion:

Welcome Advogato readers!

Anyway, back to Robert's comments on my "rant" about testing frameworks and my struggle with JUnit.

I think the overall point I was trying to make was a bit different than what Robert conveyed. And perhaps that's because I didn't explain it very well. Myself, and many others, work in places where JUnit is considered the "Standard" testing tool...not just for unit tests, but for all tests.

It kind of flows like this. Folks start using JUnit. JUnit gets integrated to builds. Pretty reports come out of it. Managers like pretty reports. Manager talks to other manager, shows reports. Other manager says "We need to use JUnit for all tests and I want pretty reports".

If you haven't seen that in your IT shop, you're pretty lucky.

My point was more along the lines of, stop trying to force JUnit to do things it doesn't. The key for using something like TestNG is that it produces "pretty reports" as well. You can make it do things that JUnit doesn't allow you to do without overriding the entire framework.

Part of the issue is that these tests are not unit tests. They're integration tests or limit tests to test valid input from a client. Unforutnately I can't dive deeper into specifics due to my own personal restrictions on actual stuff going on at work. However, let's just say that the way we are reverse engineering the input stream back into its core data components presents challenges that most folks are not likely to encounter in their daily development life unless they are doing systems integration type work.

Probably the best example would be if you were writing a program to screen scrape a 3270 session and you wanted to be sure you captured all of the data into the appropriate objects properly. You've got 100,000 permutations of the same screen to test. You think you've got 99.99% correct, but that still leaves 10 problem instances. How do you know if you make your adjustment to those 10, that you don't create 10 new problems.

That's the sort of think we're talking about testing. You have to automate it, JUnit is a good way to do that...but it is not the right tool for this job. TestNG was because it was developed in a way that made it very simple.

Hopefully that example explains a little more about the problems that exist and why I went with a testing framework rather than a unit testing framework.

No comments:

ShareThis