Re: [IMP-dev] Improving the test framework

On May 29, 2008, at 9:50 AM, Ben Webb wrote:

But the nightly builds do
not complete if any of the tests fail, regardless of whether they're run
in separate processes, and then people queue up at my desk the next day
to complain about New Feature X not being usable yet in the nightlies.

Ahhh.

That said, it is annoying to have to turn off a test when we discover a bug which is not going to be fixed immediately. It should just stay as a failed test as a reminder of non-working functionality (after all, it is still a bug). If it weren't for the memory corruption issue, we could presumably just tell Python.unittest how many tests we expect to have fail and have it modify the return state accordingly. At the very least, such bugs should be stuck somewhere on the wiki (or the doxygen todo list) rather than just working around the failure, but I would much rather have a test failure.

The Python unittest tests are not for testing C++ code, but the Python
interface. If you want to write tests for something which is not exposed
to Python, it makes more sense to write C++ tests rather than exposing
it just so you can test it. I seem to recall that Boost has a unittest

framework, which would probably be the best thing to use here.

So where do I put C++ tests when I have written them? I currently just delete them when I am done. As a note, I mostly test in C++ until I get rid of the obvious bugs and then change it to python since it is hard to debug across swig.

I don't think we really want to have two separate test frameworks (or rather, I don't think you really want to have two separate test frameworks :-)

Of course, if the functionality
*is* exposed to Python, you'll need a Python testcase, at a minimum to
make sure the SWIG wrapping is working correctly, so you may as well use
this Python testcase to also test the underlying C++ code unless you
want to have a lot of overlap in the testcases.