Open source target motion control - Part I

I have always wanted to be a pioneer to introduce an open source ARM based dual core processor into machinery control. Most of the machine control platform right now are either based on third party generic design motion controller (i.e. Baldor) or in-house microcontroller and DSP based for very specific control application. 

Basically, we need a host enviroment, a target-simulation enviroment and a real target device to do the work. In simplest term, it roughly go like this: develop and then port your programs from host to the target-simulator, and then download the image to the target device.

• Host, in this case I use Ubuntu
• Target-simulator (or a cross-compiling platform). Install Scratchbox on Ubuntu Hardy
• Target device, the latest dual core automotive control SoC device DIOPSIS from Atmel

wget http://repository.maemo.org/stable/3.1/maemo-scratchbox-install_3.1.sh 

chmod +x maemo-scratchbox-install_3.1.sh 

sudo ./maemo-scratchbox-install_3.1.sh -d

/scratchbox/sbin/sbox_adduser USER yes

/scratchbox/login

[1] For the inconsistency detected by ld.so: rtld.c: 1192: dl_main: Assertion ‘(void *) ph->p_vaddr == _rtld_local._dl_sysinfo_dso’ failed! problem, it’s necessary to disable “vdso”:

echo 0 | sudo tee /proc/sys/vm/vdso_enabled

[2] For problems when building packages (mmap: permission denied when building), it’s necessary to reduce the mmap minimum address:

echo 4096 | sudo tee /proc/sys/vm/mmap_min_addr

[3] To make these changes permanent across reboots, one can add these settings to /etc/sysctl.conf:

vm.vdso_enabled = 0

vm.mmap_min_addr = 4096

Then run

sudo sysctl -p

Unfortunately, I am running scratchbox in Ubuntu Hardy, which in turn is within a virtual box on Windows XP SP2... Anyway, the fact is you will not be able to access network within scratchbox after the installation. Somehow this is not a problem if you install on Debian distributions (this shows how much scratchbox people love Debian!)

What you need to do is to modify the following two conf files:

• /scratchbox/etc/nsswitch.conf (change nameserver to the correct IP address in your subnetwork)
  
• /scratchbox/etc/resolv.conf (change the line "hosts: xxxx" to "hosts: files dns")
  

One caveat is that you will not be able to change these two files IN scratchbox, even you do chmod 666 magic. You have to logoff scratchbox and change these two files outside. (Display hidden port and programs which are using them, netstat -nap)

Now you can use apt or wget to install packages onto your target simulation linux device. Here is a quite template list for /etc/apt/sources.list. Mine are

deb http://ftp.us.debian.org/debian/ stable main contrib non-free

deb-src http://ftp.us.debian.org/debian/ stable main

deb http://security.debian.org/ stable/updates main contrib non-free

After the enviroment is ready, one of my first step was to get my favorite messaging middleware onto ARM + Linux combination - zero messaging queue.  Malo has reported that he has successfully done the porting. Somehow I found there are many more quite involved problems. I will find some other time to list them all.

Be aware that if you are trying to build latest version of autotools or GNU M4 for marco scipting you will find them in " /usr/local/bin", rather than the "/scratchbox/tools/bin". So make sure in your sh script file you are referring to the correct location to use the right version.

Some useful links:

Boost

http://www.internettablettalk.com/forums/showthread.php?t=8241

http://linux.zipitwireless.com/wiki/Scratchbox

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dai0221a/ar01s02s01.html

Change the size of existing image

Publish and subscribe - real time system

I seems to get the feeling that this is not the first time I wrote something about this pattern. Since I could not control my temptation, as always, I decided to pick it up again. It does make perfect sense, to me at least, that there are generally two roles everything plays, producer or consumer. In another word, every entity in the system is either producing something or consuming, or both. 

In a system where the scale is by nature big, there is little hope that you can manage and control the behavior via tight-bound relationships among the entities you are able to indentify. Under this circumstance, what has bigger chance to survive is a loose and distributed system - we are expecting each individual cell is 'good' enough to be self-sufficient, have its own purpose of life ;-).

Topic-based messaging publish and subscribe structure is quite useful and flexible to meet this requirement - a word I am a little reluctant to use right now. There are some good articles descrbing how to implement this concept in WCF and raw socket. Compared with zeromq, I am still yet to try which one offers better flexibility. I think I the easiest simulation is to come up with a set up producing tons of publishers and consumer and see how the time latency and throughput would hold up. When it comes to real time (or near real time) communication, whether leaving the publish service running in each individual entity or centralising the service publishing capabilities in server (entity itself will only issue publish or subscription requests to the centralised service host), where redundancy could be applied is still a bit question mark for me. If each entity hosts the service publishing itself, that means itself will need to maintain all subscribers' information locally, as well as the callbacks for all the published service on the network this entity is interested, or even maybe those publishers' information as well. There will a burst of multi-casting packages whenever new entity joins or leaves the party. While in the centralised server structure, most of the communication will be uni-cast.

Down another level, to use TCP might not be a good idea for the de-centralised structure as the overhead of maintaining the connection, and also the fact TCP packets demanding more load on the network might present a bit of problem for the overall throughput when the number of entity increases significantly. In the real time control scenario, some of the data are timing and quality critical, sometimes 'best-effort' delivery just is not good enough. Field bus is really for this purpose. I am thinking maybe a different priority or type of delivery-assured channel could be available when a service is to be published by certain entity, with default higher QoS level. (I wonder there is an easy way for one multi-casting receiver to determine whether the other receiver has consumed the same packet or not?)

Legislative Control System

I always get the feeling wanting to punch a bag when getting something I am really proud up and running. This is one of them. After many weeks hard work, we have managed to get this whole new system to a decent, demonstrable state, in a pretty much 20% time investment sort of way (well, at least for me). Great job everyone!

This is what I have always believed, do what is opposite to everyone else does. Otherwise we are just gonna be another someone else. There was no what so called by-the-book project management, we have confidence and trust on each individual and believe in their commitment and skills to deliver their parts. This is one of the situation when everything was well-considered up front, and they are plugged together, they just worked! When I looked back to those intense nerve-breaking argument and challenges everyone brought onto the table, they suddenly all make sense. I am so proud what we have achieved so far and really looking forward this to be recognised sooner by our customers and competitors, well, maybe in different ways.

Of course this does not mean that we are just a bunch of ad hoc headless chicken. Quite oppositely, I realised that once the best practice becomes habit, it is hard to not to make it work. We have some great team players who are willing to keep their head up, taking up all the shitty, bread-and-butter daily supporting work to allow us being able to tackle with problems. They are real heroes. I could not remember how many times I have shouted to get people's mind around and brought all of us to focus, they took it in the constructive way I would never be able to do myself. Thank you very much for putting up with me and the credit should be with you.

Here is what we have done.
We have initially scribbled down all the desirables end users would like to be able to feel about, of which they are very descriptive. Everyone is not sure where we are going to start as it did not make sense by any book of knowledge as where the system architecture is going to be. To manage this stage of fuzzy feelings and ideas are quite a tricky thing as people would naturally have doubts and suggested whether it would make more sense by starting to do something for real, i.e., writing some function spec. Spec is important, no doubt about that. But without sound and very objective outside views, I would not dare to think what would come out of months and even years of work without sense of customer perspective. So I would suggest, knowing your customer, knowing who you are serving is paramount. To achieve so, is to sit tight and being critical about every single assumption or limitation in the system-to-be.

Next stage is what we are confident and comfortable to a reasonable degree where we want to achieve. We scan through the very basic functionalities user are expected to be see within first 5 minutes when they are exposed to the system-to-be. Believe it or not, the first 5 minutes are extremely important when you are evaluating a new system. Sexy UI, plug and play, easy and almost transparent self configuration of every piece of system, dynamic update of system status, well captured of all the thinkable failure scenarios and their fallback positions, these are what some experts would refer to as user experience.

Then we start, eventually, working on the design. We literately locked ourselves in a 'dark' room for a good couple of months. In very human unfriendly languages, we have abstract the core system into pieces and pinned down the fundamental data support structure, component definition, logic break down of the system layers, messaging and communication protocol, generic external device control interface. Let's just say at that point, when we walked out of that room, we do not know how to speak English (maybe I have never been able to do anyway). We do not know whether we needed coffee or just another sandwich. What an enjoyable painful period!

Coding could not be easier. I know people kept saying not everyone could be great programmer unless you have mastered a lot of cunning skills in the language you use or the subtlety the platform you are working on could provide. But at the end of day, what really seems to matter is how you drive, how you picture and to find the piece in the puzzle fit your purpose. We did not choose to let the language or OS API to lead us. We did the opposite, yet again. Coding could not be easier, as long as you know what you want. In fact, I got annoyed every time when I tried to pair program on one of team member's computer as he kept setting the keyboard setting to some bizarre configuration and I could never be able to find my icon and not mention all the familiar shortcuts which have become essentials to me. The code was far from being great as some of them are not that straight forward to understand. In fact, I started to think whether easy-to-understand is really a fair criteria. The dynamic performance of the code requires us to put a lot of dummy code behind us. Being a real time control system, we just could not afford to have ten lines of code to manage the stream overloading or fancy dictionary. What actually came out is a nice flowing picture how every cell of the system passing information and behave itself when they are supposed to be working together.

Test, is not a nice to have, it is a nature of this business. There is no more to say about this. We are by far from being testing expert, nor do we have great automated test suite. But we test every component in our mind and by means of poking the component from every possible angle dynamically. We have had a nice clock mechanism in placed where it will initiate some random sequence of system behaviour and update with no assumption of what preconditions are. Test the code itself seems to be too nice a fantasy to have, but we certainly tasted a slice of it. With a graceful failure capture engine in place, we could know exactly where the failure happen and thanks to the master clock, we were allowed to replay exactly the sequence which made the failure happen. The key point is, we have something always running from almost the first week of the development. Of course many of those failures come from our 'dummy' internal users! Many thanks to you and we are grateful to your support along the way, doubts, challenges, rants :).

This is certainly not the end, we all know there are lots hard work in front of us. But this day is to be remembered, and this day we are so proud of what we have achieved.

Use standard COM component in unmanaged code

This subject itself is a bit messy, I know. Sorry, but as I am sitting in between a bunch of 'well-designed' unmanaged C++ code and a nice, shiny C# code, and a third party COM component supplied with absolutely no documentation, here I am. Two hours later, I plodded through a path.

First thing, if you are not familiar with how to write or use one of the Microsoft's oldest 'open' component-oriented design - COM, you might want to start from a good tutorial. Believe it or not, there are countless man months of many great engineers' professional lives have been ruthlessly swallowed by Microsoft.

Ok, I had my rants. Here is the trick, and I have not found any help or information in dealing with such problem via Google, so my two pennies.

You could use Visual Studio 2003/2005/2008 etc to work on your own unmanaged code if you are indeed a solid C++ engineer and have belief in the pure logic world. Problem comes when you try to use COM component within your unmanaged code (not via Interop which is to serve marshalling between managed and unmanaged boundary). 

First of all, you use magic #import "xxx.tlb" to import type library into your unmanaged world, and choose your fancy namespace, such as:

#import "MyCOM15.tlb"

using namespace MyCOM15;

Then you initialise your code to be "COM aware" by calling 

CoInitialize(NULL);

in the beginning of your code snippets. In the meantime, put CoInitialize(NULL); before you are done. (Apology if this reads like teaching grandma sucking eggs.) After this, you retrieve the famous Dispatch interface and create instance of choice based on uuid:

CLARITYCOMLib::IMyField10Ptr fieldPtr = NULL;

fieldPtr.CreateInstance(__uuidof(MyField10));

Now you think you can relax... so did I.

You can not compile!!! You will see error like this 

error C3861: 'TEXT': identifier not found C:\Program Files\Microsoft Visual Studio 8\VC\include\comdef.h 255

"TEXT" is a macro for different character set setting. You might think to make sure all your project have Unicode character setting would make this go away. Well, apparently we are wrong again. Long story short, the way to fix this is to change "TEXT" into "(_TCHAR*)&". For example, the original line in comdef.h:

_COM_PRINTF_S_1(m_pszMsg, 32, TEXT("IDispatch error #%d"), wCode);

will become

_COM_PRINTF_S_1(m_pszMsg, 32, (_TCHAR*)&("IDispatch error #%d"), wCode);

Also, include "tchar.h" in your header files.

At last, make sure your unmanaged code project does not have /clr:pure or /clr:safe turned on. These two types of common runtime support are not compatible with what we are trying to achieve here.

Fox networked!

You could not get better morning wake up calls like this. Our product manager just announced that Fox business network has passed its two weeks period with 39 robotic heads online in one system, in full redundancy mode. Within these, 6 of them are remotely located in Chicago, LA and Washington. There are 6 control stations simultaneously available for control any of the robotic devices in this system. I am sure news will come out soon but I could not resist it. 

Kudos, well done team! Here we are, lead the competition again, to network the real time robotic control on much larger scale and distributed environment.

SNMP

Agent++ is a pretty good platform independent implementation of SNMP. I felt dizzy as well when first look at OID like this: .1.3.6.1.4.1.6827.50.19.3.29.0.

However, SNMP is an extremely powerful management protocol in most of the networking devices you can see. In fact, more and more other systems have now embedded SNMP capability in (by running SNMP Agent), such as the one I was working on today - Evertz broadcast video combiner. Sun also has a decent MIB toolset within Java Dynamic Management Kit. 

Other than those, if you are unlucky as I did, you'll have to start from MS SNMP library and plod through the MIB trees, for which I found this post quite helpful.

Btw, this project will pop up a heap corruption problem if you are trying to build the source in .Net environment. The problem is the way SNMP_realloc is used  to create SnmpVarBind. Here is the change of the code which works (.Net 2.0, 3.5):

int SNMP::action (string oid, int actionCode, SnmpVarBind * args) 

{

long int errorIndex = 0;

long int errorStatus = 0;

char * cOid = (char *)oid.c_str ();

if (SnmpMgrStrToOid (cOid, &args-> name)) 

{

SnmpVarBindList snmpVarList;

snmpVarList.list = NULL;

snmpVarList.len = 0;

snmpVarList.list = (SnmpVarBind *)SNMP_malloc(sizeof(SnmpVarBind)); 

snmpVarList.len++;

// Assigning OID to variable bindings list

SnmpUtilOidCpy(&snmpVarList.list[0].name,&args->name);

SNMP_CopyVarBind(&snmpVarList.list[0], args);

try 

{

SnmpMgrRequest(mgrSession,

actionCode,// SNMP_PDU_SET,

&snmpVarList,

 &errorIndex, 

&errorStatus);

} 

catch ( ... ) {

PostError("Snmp Request Failed");

}

// free the list, to avoid memory leak

SnmpUtilVarBindFree(snmpVarList.list);

SnmpUtilVarBindListFree(&snmpVarList);

}

return errorStatus;

}

Google app engine and all the jazz - Part I

It should not be too difficult to set up the development environment for google application engine. Since at the moment GAE only supports python as its compilation platform, and Eclipse is argubaly the most popular open source IDE with its nice plugin architecture, I decided to set out for the fun. After a beer and an hour effort, I found out Google has a post about the pretty much same setup :-).

First of all, to get Pydev installed in your Eclipse and you might also want to avoid problems in receiving console input while you are working under windows os by override the input() and raw_input() with the following functions:

def win_raw_input(prompt):

    return raw_input(prompt).replace("\r", "")

def win_input(prompt):

    return eval(win_raw_input(prompt))

Also should you want to be able to use console in Eclipse the same way as the interactive shell in python IDLE, here is the link to do so by setting up external tools. Now you can use Eclipse to do your python coding.

I am not going to repeat what has been described by Google's post. There are couple of other things though. Sometimes you will see this error by following the last step by adding user-defined starter for GAE development server using "${project_loc}/src".

This is caused by the fact that the project you are working get its focus snitched by Eclipse IDE.  I got annoyed after a few instances and decided to replace ${project_loc} with ${workspace_loc}\my_project_name\. Just to be aware that your yaml file should be located under your project root folder to allow dev_appserver to pick it up.

The other way is to use external tools to set up a configuration, i.e. "run Python for GAE". In the main tab of the program configuration, add the following parameters:

"C:/Program Files/Google/google_appengine/dev_appserver.py" 

--port=8080

"${workspace_loc}\HelloGAE\"

--port=8080

Now I am all set to rock and roll. 

Crisis management

We are in a pretty deep hole right now. Jun didn't spend too much time to convince me that the worse thing to the current economic situation is our children, and children's children will have to spend considerably long time to pay for the economic debt we are creating right now. Increasing public spendings, lower the saving rate to encourage personal spendings and investment are all trying to stimulate the spending power, consequently create needs for working with sufficient cash flow. All these spendings will need to be balanced or paid back one way or another from either tax policy or decrease of benefits which our offspring will have to face with. This is what I could comprehend so far.

What I would like to talk about here is more of a miniature within a business organisation, which I call it crisis management to distinguish with terms like risk management, budget management etc.

We are all witnessing what is the unavoidable effect on individual business under current circumstance - decreasing budget, shredding jobs, cutting head counts, shrinking investment. All other things being equal, I could not easily find a logical outlet with such reactions. I also do not see how these reaction would help a business and positively produce contributions to global economy either. All the immediate reactions could only destroy consumer's confidence and power to spend. It is more like the start of the snowball. We can only pray for that at certain point in the future, the slope starts to slow down and things will pick up slowly.

Instead of reacting, there are at least certain things we could do. Economy climate change always has a hint, a signal of some sort. Honestly, as naive and dummy as I am in finance, the first time when I picked up the news that greedy mortgage lenders sell mortgages to customers without any appropriate evaluation, there was a voice in the back of my mind that something was not quite right. As an organisation, there are many signals, measurements could be used to trigger the alarm system. Management decisions should really at least be rehearsed before the crisis really hit us. One thing we should have learnt from the bank and logistics is that to simulate all different scenarios before hand is not a trivial job, which should really be the focus of our strategical management, if that really happens.

Personally, it is just not good enough to tell the people working for me that I do not know where we are heading towards, nor do I have better ideas or directions of what we should be doing other than simply react to the bigger environment. Let's face it, reaction, as one of the fundamental biological behaviour, is rather rudimentary. It can hardly be considered as intelligence. Rather than only being honest with employees, a good leader should do his homework - knowing what options we have, what are the fallback positions, and this is the time when the rehearsed strategy could be put in effects. A good leader also should realise that crisis could also be good opportunity to redefine the market, revisit your product portfolio, restructure the team, basically, you can create a whole new game, given that you are well-prepared enough and tough enough to get through the winter. You may not be able to keep the company going no matter how many heads to be cut off. That alone will not help you to push the ball uphill. We need to bear in mind is that the goal is not to save cost, which is only means, arguably may not be the best one though. Rather than destroy the morale completely, it might give you different perspective by fight with all backs against the river.

So, whenever I have free time slots, I keep reminding myself of the importance of home work - always be prepared before the rain comes. All those boring figures, charts, predictions, numerous versions of plans of attacks (no doubt most of them will go to the bin), they are worthwhile.