Data logging...

Logging data from "Things", I've run computer operated planes, rockets, cars, helicopters. All of these share a similar problem. How do you log data?  I like to log ALL raw sensor readings unmodified.  This ends up in a steam of multi rate data, ie I see raw IMU readings at 200Hz, Compass readings at 20Hz  GPS at 20Hz, and various things like mode changes and responses at varying rates from 200Hz to to once at power up.

Some things I've learned:

• I don't trust a file system. If your thing crashes the data you are really interested in is the last few bits of data before impact. I've lost too much data to trust a file system to be robust when the recording media is de-powered and spattered all over the landscape
• You can't squeeze everything you want to record in a telemetry stream.
• I keep reinventing this same wheel.
• In the course of a project the format of what you are recording evolves. Keeping the code for the data display and the data recorder in sync is difficult and often I end up unable to read earlier logs.

My current approach is as follows:

 Record everything to flash directly, it makes no difference if its a SD card or dedicated flash chip I use the same format.  I fully erase the flash (all the sectors on SD) then on power up I scan the flash page or sector by sector looking for the first blank page/sector.  I then start recording new data in that sector....

My data recording has a big circular buffer that records the data in stream format....

All the recorded data is in blocks with the format:

START ID ... (Next START)

Where START is a specific byte and ID is a numeric value representing the kind of data...
If the value START appears in the data is is escaped...The packet ends with  the next start....

I then have a background process that takes this circular buffer and writes it to flash sectors where there is a full sector worth of data.


Now I might record something like my GPS reading...


typedef struct
{
 double lattitude; //Radians
 double longitude;//Radians
 double ht;  //m
 unsigned long msec;
 unsigned short week;
 double ECEF_X;//m
 double ECEF_Y;//m
 double ECEF_Z;//m
 float hspeed;//m/sec
 float head; //radians
 float vv; //m/sec
 unsigned char flag1;
 unsigned char flag2;
 unsigned char nsat;
 WORD seq;
}CombinedTrimGps;


So that would be stored as

START GPS_ID the contents of the structure.....

Now if in the past if I changed the record so I added say a list of  SNR reading to the record I'd have to redo the code that decodes the GPS data from the data logs....

In an ideal world C++ would have introspection and it would be able to put the format of the structure into the data log the first time its used.....  then when the data reader sees the data it immediately knows what format the data is in....

Since C++ does not have introspection... I wrote some code that comes very close...


void LogRecord(CombinedTrimGps & item)
{
  LogStart(LOG_TYPE_BD960 ,"GPS");
  LogElement(lattitude,"lattitude " );
  LogElement(longitude,"longitude " );
  LogElement(ht       ,"ht        " );
  LogElement(msec     ,"msec      " );
  LogElement(week     ,"week      " );
  LogElement(ECEF_X   ,"ECEF_X    " );
  LogElement(ECEF_Y   ,"ECEF_Y    " );
  LogElement(ECEF_Z   ,"ECEF_Z    " );
  LogElement(hspeed   ,"hspeed    " );
  LogElement(head     ,"head      " );
  LogElement(vv       ,"vv        " );
  LogElement(flag1    ,"flag1     " );
  LogElement(flag2    ,"flag2     " );
  LogElement(nsat     ,"nsat      " );
  LogEnd();
}

Using Macros this expands into...



void LogRecord(CombinedTrimGps & item)
{
static bool bIntroed;
BYTE tid=LOG_TYPE_BD960;
if(!bIntroed)
{
bIntroed=true;
StartItemIntro(tid,sizeof(item),"GPS");
ShowElement(tid,&item,&item.lattitude,item.lattitude,"lattitude " ) ;
.
.//All the elements
.
}
LogRawRecord(tid,(const unsigned char *)&item,sizeof(item));
}

I have a different ShowElement  overloaded function for each type I might want to log....


Thus the first time this data type is logged it puts the format into the data log record....
This is something I've wanted since I first did the rocket stuff 6 years ago.

I've now got a self describing log format that is complete and efficient....
I can also use the exact same format for Telemetry....

On my rockets I had an optional parameter that went to the logging function that selected if the data went to the LOG, to telemetry or to both locations....

Last night I got the whole chain to work and the picture below is derived from this chain of events...
IE I drove the AVC car around my driveway -> recorded log -> process log generating records -> put in excel -> Then made a graph of locations....

Now I have to get the data replay tool I wrote a few years ago to read this format and my data logging tools will be complete!