Anyway, my team has been suffering from lack of on target testing for a while now. Most of the existing testing is more of a hardware exercise test at best. The verification of functionalities and integration test are left for the last minute surprise. There were some standard approaches such as building drives and applications into the binary image and download to the target device, relying on the start up scripts to do its job. Again, most of these tests are more of automated hardware exercise testing rather than simulation. I guess one of the subtly here is that to simulate run time device performance in a realistic working environment is much involved when various components with timing constraints, system load difference, hardware limitation as well as failure mode recovery are really non-deterministic by nature. Some would argue the best we can do really is to do our deterministic part and leave nature to do its part. Well, we all know nature likes to surprise us every single time, just to show our ignorance I guess, for fun :-). We thought about adopting some off-the-shelf testing framework to write our test programs checking the components while they are running on the target device and integration test programs will simulate as many as possible scenarios we can think of, of course this would only be possible when the device running modes are well captured and defined. These simulation tests are not trivial to develop nor easy to maintain. The best part is, when something gets changed, or tests failing to be updated accordingly with the production code, things start to fall apart and no one has a clue what the heck is going on other than running around to setup JTAG debugger or gdbserver and start to get into our ancestor beloved register and assembly world.
My idea is simple, in fact, the engine I wrote on the train is only 200 lines of code. To define a light structure where each time when a new component is dropped to target, some standard test scripts or test programs will be put under certain location. They will be automatically picked up when a high level unit test mode command is issued. For the traditional integration test, we can run multiple scripts simultaneously, following the pattern the application demands. And finally, the best part is Panic test, where a high level command will be something like this: run_as_wish. This mode will certainly cause quite a lot of failures but I guess it is better to know your bottom line earlier rather than keep looking for it all the time.
The engine is written in Lua. So we will need to get Lua onto the target first. It defines three modes for invoking existing test scripts/programs in a protect mode. If you want to know how to kick off multiple scripts in parallel, this post worth reading, and of course luathread. The beauty of Lua is that you can easily integrate it into all sorts of different languages, even .Net, via certain interface. Since Lua has been used widely as embedded script engine in games, you should not be surprised too much. In my engine, again, simple and easy, use a global script to kick off the whole process and iterate through all those scripts needed to be concurrently running, in the scenarios you defined. Obvious question would be, why do not we just write multiple C programs? Okay, first of all, since Lua scripts does not have tight dependencies to the low level OS APIs, they are much easier to port. Secondly, with RemDebug, you can actually remote debug Lua test scripts from PC. Thirdly, when your tests gets bigger and heavier to maintain, it is unavoidable that your production logic sneaks into your test code, if they are written in the same language, that is even more convenient for us lazy people to do. This way, it force you to look at your production code independently and poke it in a protected environment. By doing this, you do not have to rebuild your image every time, which is really a time killing task!
No comments:
Post a Comment