On target automation test engine

While I was on the train the other day to see a crappy doctor for my eyes (I am sorry but he has certainly proved that the visit does not worth my fifty pounds!), I thought just to see whether I could implement a small idea with one eye open, just you know, the other one was in pain...

Anyway, my team has been suffering from lack of on target testing for a while now. Most of the existing testing is more of a hardware exercise test at best. The verification of functionalities and integration test are left for the last minute surprise. There were some standard approaches such as building drives and applications into the binary image and download to the target device, relying on the start up scripts to do its job. Again, most of these tests are more of automated hardware exercise testing rather than simulation. I guess one of the subtly here is that to simulate run time device performance in a realistic working environment is much involved when various components with timing constraints, system load difference, hardware limitation as well as failure mode recovery are really non-deterministic by nature. Some would argue the best we can do really is to do our deterministic part and leave nature to do its part. Well, we all know nature likes to surprise us every single time, just to show our ignorance I guess, for fun :-). We thought about adopting some off-the-shelf testing framework to write our test programs checking the components while they are running on the target device and integration test programs will simulate as many as possible scenarios we can think of, of course this would only be possible when the device running modes are well captured and defined. These simulation tests are not trivial to develop nor easy to maintain. The best part is, when something gets changed, or tests failing to be updated accordingly with the production code, things start to fall apart and no one has a clue what the heck is going on other than running around to setup JTAG debugger or gdbserver and start to get into our ancestor beloved register and assembly world.

My idea is simple, in fact, the engine I wrote on the train is only 200 lines of code. To define a light structure where each time when a new component is dropped to target, some standard test scripts or test programs will be put under certain location. They will be automatically picked up when a high level unit test mode command is issued. For the traditional integration test, we can run multiple scripts simultaneously, following the pattern the application demands. And finally, the best part is Panic test, where a high level command will be something like this: run_as_wish. This mode will certainly cause quite a lot of failures but I guess it is better to know your bottom line earlier rather than keep looking for it all the time.

The engine is written in Lua. So we will need to get Lua onto the target first. It defines three modes for invoking existing test scripts/programs in a protect mode. If you want to know how to kick off multiple scripts in parallel, this post worth reading, and of course luathread. The beauty of Lua is that you can easily integrate it into all sorts of different languages, even .Net, via certain interface. Since Lua has been used widely as embedded script engine in games, you should not be surprised too much. In my engine, again, simple and easy, use a global script to kick off the whole process and iterate through all those scripts needed to be concurrently running, in the scenarios you defined. Obvious question would be, why do not we just write multiple C programs? Okay, first of all, since Lua scripts does not have tight dependencies to the low level OS APIs, they are much easier to port. Secondly, with RemDebug, you can actually remote debug Lua test scripts from PC. Thirdly, when your tests gets bigger and heavier to maintain, it is unavoidable that your production logic sneaks into your test code, if they are written in the same language, that is even more convenient for us lazy people to do. This way, it force you to look at your production code independently and poke it in a protected environment. By doing this, you do not have to rebuild your image every time, which is really a time killing task!

Lua in .Net

ALthough I have mentioned before that there are a few options to enable scripting in .Net projects, I was thinking write a interface library to wrap up the unmanaged C style Lua library into a CLR compliant module. It is actually pretty straightforward process. Some nice people have already written ref class LuaDLL to solve the marshalling between unmanaged functions to unmanaged APIs. Essentially, you could use LuaDLL directly within your .Net project. To simplify things, what I was looking for is to wrap Lua into an object oriented way, as some OO nuts like to brand it. What LuaDLL does not do is to offer an easy way allowing Lua script accessing CLR type objects.

Without too much efforts, to pack all the fancy Lua interpreter, table, function and user data into classes, and some counter-effective strings to be executed by Lua runtime to set the meta data. Here we are, a nice interface class - LuaInterface, which defeats my original plan to be the first integrator ... And, it was done long time before my consciousness.

Anyway, to use this interface could not be easier. You can access CLR objects from Lua scripts which essentially will be string format to be executable by Lua runtime and manipulated from CLR objects vice versa. What I like about this is to hook up event handler and use delegate just as easily. On the contrary, I could not locate any good summary on this subject thus I listed in this post.

• Define delegate in C# to use Lua functions:
  

// Lua interpreter
private Lua lua;
// function pointers to functions in Lua
public delegate double PlusDelegate(double a, double b);

public Script()
{
lua = new Lua();
// define variables in Lua
lua["num"] = 2;
lua["str"] = "a string";
// define functions in Lua

lua.DoString(@"
function plus(x, y)
return x + y
end
");

// use functions defined in Lua
PlusDelegate plus = lua.GetFunction(typeof(PlusDelegate), "plus") as PlusDelegate;

if (IsDefinedInLua(add))
{
double res = plus(10, 2);
rtbOutput.Text = (String.Format("result: {0}", res));
}
}

private bool IsDefinedInLua(Delegate delegateOfFunction)
{
if (delegateOfFunction.Target is LuaDelegate)
return (delegateOfFunction.Target as LuaDelegate).function != null;

return false;
}

• Define 'delegate' in Lua to use C# functions:
  

public void CallCSharp(string s)
{
rtbOutput.Text += ;
}
lua.RegisterFunction("callCSharp", this, this.GetType().GetMethod("CallCSharp"));

Just to be aware that the function has to be public to be registered into Lua. You could also use Lua function by instance as:

// use a Lua function by instance
LuaFunction luaFunction = lua.GetFunction("minus");
double ret = (double) luaFunction.Call(10,2).GetValue(0);

• Implement event defined in C# from Lua, a.k.a bind events to scripted handlers.
  

public delegate void RaiseAttentionEventHandler(object sender, EventArgs e);
public event RaiseAttentionEventHandler RaiseAttention;

lua.DoString(@"
function clickMe (sender)
sender.Text = 'It worked'
end
");

lua.DoString(@"
newPanel = ScriptPanel('aNewPanel')
newPanel.RaiseAttention:Add(clickMe)
");
if (this.RaiseAttention != null)
RaiseAttention(sender, null);

• Implement event defined in Lua from C# is a little twisted logic. I could not think of a good use case right now but for the sake of completeness sake, I did a little trial and error. Before moving on, you might want to refresh a little about the event support in Lua here. Another good event module implementation using C library could be found here. The easiest way I found is to register C# functions into Lua as global function and use them to handle the events in Lua. 
  

private void btLua_Click(object sender, EventArgs e)
{
// Implement event defined in Lua from C# and trigger it in Lua
LuaEventHandler handler = new LuaEventHandler();
lua.RegisterFunction("eventInLua", this, this.GetType().GetMethod("EventInLua"));
handler.handler = lua.GetFunction("eventInLua");
handler.handleEvent(sender, e);
}

public void EventInLua(object sender, EventArgs e)
{
rtbOutput.Text = ("Trigger from Lua ");
return;
}