If your not a pilot this is probably not meaningful for you.
My FAA medical lapsed in 2007 and and I have not flown since. Long time followers of my blog will remember that 3 or so years ago I had a post about Sleep Apnea and getting a CPAP machine. This significantly complicates getting a flight physical. Before I figured out the sleep apnea there were some other minor medical issues due to the poor sleep that I dealt with that also complicate the flight physical. The FAA flight physical does not have a recency or last x years stipulation, the form says "have you ever".... and in today's realm of pervasive computer databases your going to get busted if you are not 100% honest about EVERYTHING!. Every yes on the form must be documented and dealt with.
The FAA Medical process as it is currently construed has a really nasty catch 22, One can fly light sport aircraft (2 seats, day vfr, gross less than 1320 lbs) with a drivers license instead of a medical, as long as you have never been denied a medical. So going to the FAA to get a 3rd class medical has risk, if you try and fail then you loose the LSA option, If you never try and never fail you keep the LSA option.
So the only way to really combat this is to gather all the necessary paperwork, all the tests reports, documentation records etc... and go individually do equivalents of all the FAA medical tests, vision, etc...so you know what the answers will be before you actually start the process.
How do you know what tests and documents will be required? You need expert advice, and while there are large companies charging thousands of $ for this help I found them to be unresponsive and bureaucratic.
(Yes I paid the initial FEE to a large aviation medical consulting co and got ZERO value for the large $) and then I found Dr Bruce Chien participating on the Pilots of America medical forum.
I contacted him directly and I could not be more pleased with the whole process. He is a AME that specializes in difficult medicals. He worked with me to identify possible issues, then gather and document these in a way that would be acceptable to the FAA. Finishing this process required that I fly to his office in Il, in the end even including airfare, hotel, rental car etc.... it was less than half the cost of going through one of the big aviation medical consulting firms.
So I now have a brand new 3rd class medical, and I'm really looking forward to flying again....
Tuesday, October 29, 2013
Monday, August 05, 2013
Space and futures....
It looks like Armadillo is now dormant or dead.
Based on some hints I've received, I've suspected that this was coming for awhile...
The suborbital business never made any sense to me, its hard and other than artificially created markets (cruiser program) and thrill seekers I see no fundamental business case to be had at a price that is remotely appropriate to the level of effort.
Its been 10 years since space ship one and I see no one close to having a profitable business
launching suborbital passengers or payloads.
I've stared longingly at this business, I'd really like to do something here, but no matter how hard I look it has never made sense as a business...
There is remote possibility that someone might come up with a nanosat launch business that makes sense.(I've even written a somewhat optimistic business plan toward that end), but once you add it the difficulties of testing and regulatory compliance the whole thing is really really hard. For the time being it looks like the only real progress to be made will be done by large well funded organizations like SpaceX and possibly Blue Origin, but even Blue is subject to the whims of Bezeo's attention, he may find it more fun to play with his new newspaper...(He jsut spend 250M of his own $ on the Washington post)
I'd like to see someone try to build an expendable launcher where the primary aim is automated manufacturing and really low cost. I think that even if Spacex gets its 1st stage re-usability right it would be possible to beat them on cost, they currently have close to 3K employees, at reasonable aerospace wages at 33 flights a year that is 10M a launch just for wages..... Too bad Beal failed, I think they were really on the right track for low cost space access...
Dam stubborn gravity well....
Based on some hints I've received, I've suspected that this was coming for awhile...
The suborbital business never made any sense to me, its hard and other than artificially created markets (cruiser program) and thrill seekers I see no fundamental business case to be had at a price that is remotely appropriate to the level of effort.
Its been 10 years since space ship one and I see no one close to having a profitable business
launching suborbital passengers or payloads.
I've stared longingly at this business, I'd really like to do something here, but no matter how hard I look it has never made sense as a business...
There is remote possibility that someone might come up with a nanosat launch business that makes sense.(I've even written a somewhat optimistic business plan toward that end), but once you add it the difficulties of testing and regulatory compliance the whole thing is really really hard. For the time being it looks like the only real progress to be made will be done by large well funded organizations like SpaceX and possibly Blue Origin, but even Blue is subject to the whims of Bezeo's attention, he may find it more fun to play with his new newspaper...(He jsut spend 250M of his own $ on the Washington post)
I'd like to see someone try to build an expendable launcher where the primary aim is automated manufacturing and really low cost. I think that even if Spacex gets its 1st stage re-usability right it would be possible to beat them on cost, they currently have close to 3K employees, at reasonable aerospace wages at 33 flights a year that is 10M a launch just for wages..... Too bad Beal failed, I think they were really on the right track for low cost space access...
Dam stubborn gravity well....
Monday, June 17, 2013
LENR I don't think I'm crazy...
The AVC is over its time to talk a little bit about my LENR project.
24 years ago or so Pons and Fleischmann made a big splash with claims of Cold Fusion, ie using simple electrochemical cells generating excess heat beyond what could be explained by chemical energy.
This caused quite a controversy and after a few years the furor died down and it seemed like Cold Fusion was largely discredited. At the time my Dad was a commodities trader and I bought Pd futures and made a few hundred dollars by riding it up and most of the way back down.
Every once in awhile since then I'd hear of some experiment or another and then it would go dark. About 6 mo's ago a non-technical someone asked me to review some stuff in this field for technical validity as they were contemplating an investment. I expected to report back that t was a fraud or junk science. What I discovered surprised me.
You may believe me or not but this is what I now believe...
The Pons and Fleischmann effect is real. (here after called PFE)
To this day we still do not understand what is going on, replicating the PFE requires Pd of specific structure, and unknown contaminates. A number of cases have been shown where a specific Pd rod that works in one lab will work in a different lab with different apparatus. Just plain unscreend Pd works about 5% of the time. Pre screening the Pd for high D-Pd loading ratios without Pd swelling can reduce this ratio to about 33%, ie if a Pd rod will absorb > 0.95 D for each Pd atom without swelling and cracking the odds of it demonstrating the PFE is 30% or better.
Whatever is happening it does not generate neutrons expected of traditional hot fusion., and is probably not traditional fusion. There have been a large number of theories presented, none of them presently fit all of the facts, so we effectively have no theory. Over 60K peer reviewed papers have been written about High temp superconductivity and we have no theory there either, so if science was functioning properly this lack of theory should not be a total block to progress.
I think one of the pathological failures was calling it Cold fusion, because it's probably not fusion in the traditional sense. Much more likely to be some path following the weak force rather than the strong force...Also there is now a growing body of evidence that whatever is going on can happen in regular water and ni, so some of the early "dead" experiments that were used to prove an instrumentation error were later found to be lower grade PFE reactions on their own.
Another body of evidence also shows that the Pd needs to have some "Stuff" plated on it to work really well, P+F were a tiny bit sloppy about contamination, so when it came time to replicate the more meticulous the lab the less chance they had of replication. Its also been shown that most of the successful replications from the time of early P+F all came from the same batch of Pd rod all contaminated with a few ppm of Rhodium and other trace elements.
Some of the experiments show that it takes 100hrs+ to load a Pd rod enough for it to turn on... again it does not help replication....
My Friend Dave W, says its sort of like the story: I've been told that if I put a bent piece of shiny metal and small bit of wire on the end of a string and dip it in water I can pull up free food. I've been doing in my lab with sterilized string and distilled water for three weeks and still no food....
We clearly don't know what we are missing...
An ideal energy source that runs off of water is too good to be true and it attracted a bunch of scammers eager to make a buck. These scammers did not do the real scientists any favors. I'm still not convinced that the most notorious of the lot Rossi is not a scammer/huckster.
I am convinced that the Navy SPAWAR lab, SRI, briillion, Ed Storms, Toyota and others have done very careful science, addressing every issue brought up in opposition to their results and have proven to me beyond a doubt that there is a PFE and there is science here we don't understand.
So all the scientists with a desire to be accepted by other scientists have worked very hard doing careful Calorimetry running experiments lasting 100's of hours proving that the result can't be chemical.
A properly done long term calorimitry experiment is the gold standard of proving its not a chemical effect....
If your experiment takes 100hrs to complete then its really hard to do fast cyclical testing.
Each sample you try is a major commitment of time and care.
My idea is this, accept the PFE is real. Try lots and lots of samples and screen for it.
Don't wiat 100hrs to toss a sample that is not working. Toward that end I've built a tiny little stainless reaction chamber that I can heat and pressurize with H or D and stimulate the sample with RF or electrical pulses ... (the system is 50 ohms up to and back out of the sample chamber.) I've got an fast reading 15msec IR pyrometer with a tiny spot size that I can use to measure the instantaneous temperature of the sample, so if some stimulation causes the sample to "go" I don't have to wait hours to know what I just did.
(The IR pyrometer looks through a sapphire window rated for the temps and pressures I expect)
I want to try dozens of samples a day, try 1000's of possible stimulation....
I also have 4 wire restive measurement of the sample and a large pancake style GM tube to detect any radiation form the experiment.
The wright bothers did not build a plane first, first they built a wind tunnel.
I'm building a LENR wind tunnel, the samples will be tiny and useless, but if they generate excess power above the chambers elevated temperature I'll know quickly and hope to be able to optimize to maximize the effect. in the next day or two I'll organize my pictures of the reactor and test equipment and post it all here...
I'm on my second revision of the reactor, the first revision was a bit too clunky to assemble and disassemble, the current version seems much easier.
In this same theme I'm going to the ICCF-18 in July...
24 years ago or so Pons and Fleischmann made a big splash with claims of Cold Fusion, ie using simple electrochemical cells generating excess heat beyond what could be explained by chemical energy.
This caused quite a controversy and after a few years the furor died down and it seemed like Cold Fusion was largely discredited. At the time my Dad was a commodities trader and I bought Pd futures and made a few hundred dollars by riding it up and most of the way back down.
You may believe me or not but this is what I now believe...
The Pons and Fleischmann effect is real. (here after called PFE)
To this day we still do not understand what is going on, replicating the PFE requires Pd of specific structure, and unknown contaminates. A number of cases have been shown where a specific Pd rod that works in one lab will work in a different lab with different apparatus. Just plain unscreend Pd works about 5% of the time. Pre screening the Pd for high D-Pd loading ratios without Pd swelling can reduce this ratio to about 33%, ie if a Pd rod will absorb > 0.95 D for each Pd atom without swelling and cracking the odds of it demonstrating the PFE is 30% or better.
Whatever is happening it does not generate neutrons expected of traditional hot fusion., and is probably not traditional fusion. There have been a large number of theories presented, none of them presently fit all of the facts, so we effectively have no theory. Over 60K peer reviewed papers have been written about High temp superconductivity and we have no theory there either, so if science was functioning properly this lack of theory should not be a total block to progress.
I think one of the pathological failures was calling it Cold fusion, because it's probably not fusion in the traditional sense. Much more likely to be some path following the weak force rather than the strong force...Also there is now a growing body of evidence that whatever is going on can happen in regular water and ni, so some of the early "dead" experiments that were used to prove an instrumentation error were later found to be lower grade PFE reactions on their own.
Another body of evidence also shows that the Pd needs to have some "Stuff" plated on it to work really well, P+F were a tiny bit sloppy about contamination, so when it came time to replicate the more meticulous the lab the less chance they had of replication. Its also been shown that most of the successful replications from the time of early P+F all came from the same batch of Pd rod all contaminated with a few ppm of Rhodium and other trace elements.
Some of the experiments show that it takes 100hrs+ to load a Pd rod enough for it to turn on... again it does not help replication....
My Friend Dave W, says its sort of like the story: I've been told that if I put a bent piece of shiny metal and small bit of wire on the end of a string and dip it in water I can pull up free food. I've been doing in my lab with sterilized string and distilled water for three weeks and still no food....
We clearly don't know what we are missing...
An ideal energy source that runs off of water is too good to be true and it attracted a bunch of scammers eager to make a buck. These scammers did not do the real scientists any favors. I'm still not convinced that the most notorious of the lot Rossi is not a scammer/huckster.
I am convinced that the Navy SPAWAR lab, SRI, briillion, Ed Storms, Toyota and others have done very careful science, addressing every issue brought up in opposition to their results and have proven to me beyond a doubt that there is a PFE and there is science here we don't understand.
So all the scientists with a desire to be accepted by other scientists have worked very hard doing careful Calorimetry running experiments lasting 100's of hours proving that the result can't be chemical.
A properly done long term calorimitry experiment is the gold standard of proving its not a chemical effect....
If your experiment takes 100hrs to complete then its really hard to do fast cyclical testing.
Each sample you try is a major commitment of time and care.
My idea is this, accept the PFE is real. Try lots and lots of samples and screen for it.
Don't wiat 100hrs to toss a sample that is not working. Toward that end I've built a tiny little stainless reaction chamber that I can heat and pressurize with H or D and stimulate the sample with RF or electrical pulses ... (the system is 50 ohms up to and back out of the sample chamber.) I've got an fast reading 15msec IR pyrometer with a tiny spot size that I can use to measure the instantaneous temperature of the sample, so if some stimulation causes the sample to "go" I don't have to wait hours to know what I just did.
(The IR pyrometer looks through a sapphire window rated for the temps and pressures I expect)
I want to try dozens of samples a day, try 1000's of possible stimulation....
I also have 4 wire restive measurement of the sample and a large pancake style GM tube to detect any radiation form the experiment.
A picture inside the reactor the depression holds an alumina plate and the ceramic feed through on both sides are 50 ohms up to the wall, and with 0.016 thick wire the plate is the right thickness for 50 ohms.
Both sides of the reactor with the sapphire sample window visible.
The reactor inside its heater jacket with thermocouples attached, the PID controller holds a constant temperature. Design operating point is 300 PSI and 600C with SF of 3.0. You can see the IR pyrometer lying on its side next to the reactor, I still need to fabricate an adjustable support to make it peer through the window.
I'm building a LENR wind tunnel, the samples will be tiny and useless, but if they generate excess power above the chambers elevated temperature I'll know quickly and hope to be able to optimize to maximize the effect. in the next day or two I'll organize my pictures of the reactor and test equipment and post it all here...
I'm on my second revision of the reactor, the first revision was a bit too clunky to assemble and disassemble, the current version seems much easier.
In this same theme I'm going to the ICCF-18 in July...
Saturday, June 15, 2013
AVC Part 2
The world does not want me to write this post....
The car makes it through the hoop but misses the ramp... slightly to the left.... I'm faster than Savage solder, but they hit the ramp... after the first heat the score is Savage Solder 440 NetBurner (me) 412.. so I was 22 seconds faster... but missed the 50pt hoop bonus.
I hurry back and look at the small car code.... I see where I commented out too much and that my {} in the state machine did not line up as expected, id turned off the IMU/gyro processing. It is with high hopes I go back to run heat two of the Cheapshot small car.... This time the car starts right up, turns hard just like its big brother and flips over on the start ramp.... I'm now zero for 2 on this vehicle.
Time to adjust the course to try and hit the ramp and run the big car again....I move the way points over about 3 ft to hit the ramp and we run heat two.... It lurches off the start gets going solidly.... just three feet before the hoop it swerves out of the way just missing the hoop.... when it gets to the ramp it goes way wide and actually rubs on the curb..... I moved the way points the wrong way.... it scrapes along the curb recovers and finishes the race..... I'm now solidly in first place.
(After heat 2 Photo by Ben Brockert)
I'd worked on it for about an hour and forgot to open a new tab when navigating away to find a link and blogger lost it.....
On the day of the event I arrive at the venue at around 7:45, Unloaded my stuff and setup on the provided worktables under the tent. They open the course for testing at 8am and I slowly walk the doping class car around the course gathering way point data. For the first turn I decide to go waaay long so as to try and avoid some of the traffic. In the videos you can see the car goes way past the first turn this is intentional.
My first heat is with Cheapshot the little car. Its heat two and a rush to be ready. I strip out the RC code and setup so the system arms on reset..... I test this in the pits by pressing the go button and it seems to work.
I sit at the line its time to go I hit the button and nothing happens..... I then hit rest and wait out the 20 seconds of gyro zero and other things then the car finally takes off just after the first place car finishes.....
The car goes out about 10 feet and turns in a large never ending never varying circle round and round...
Heat 2 round 0 Zero....
So I have 6 heats or so to get the big car ready.... I revert the code to what it was for testign last night. Rather than hurrying to strip otu the RC arming code I just comment out the one line of processing received chars from the DSM2 receiver and add a serial port command to arm the car....
I press the button and the car turns hard right smacking into other cars... then it straightens out and runs the course solidly. (Here is the heat one video..)
Video by Ben Brockert
The car makes it through the hoop but misses the ramp... slightly to the left.... I'm faster than Savage solder, but they hit the ramp... after the first heat the score is Savage Solder 440 NetBurner (me) 412.. so I was 22 seconds faster... but missed the 50pt hoop bonus.
I hurry back and look at the small car code.... I see where I commented out too much and that my {} in the state machine did not line up as expected, id turned off the IMU/gyro processing. It is with high hopes I go back to run heat two of the Cheapshot small car.... This time the car starts right up, turns hard just like its big brother and flips over on the start ramp.... I'm now zero for 2 on this vehicle.
Time to adjust the course to try and hit the ramp and run the big car again....I move the way points over about 3 ft to hit the ramp and we run heat two.... It lurches off the start gets going solidly.... just three feet before the hoop it swerves out of the way just missing the hoop.... when it gets to the ramp it goes way wide and actually rubs on the curb..... I moved the way points the wrong way.... it scrapes along the curb recovers and finishes the race..... I'm now solidly in first place.
They now take a break and let everyone practice for an hour before the next heats. Realistically the small car has no chance of even placing so I focus on the getting the big car to hit the ramp. I go out on the course and carefully measure the center of the ramp.....
I do a quick review of the code for the small car and realize that steering code has a static variable in it to measure the rate of turn... between loops and since most of my testing had the course starting to the north this would not be apparent.. I change the code so it ignores this for the first pass through navigation and prepare to try again.... This time the small car ends up in the fence... I'm zero for 3 very frustrating..
I'd planned on going faster than I was going. In testing I ran the straightaways at 3X the speed and 50% faster on the corners, but for the last run I had such a lead that if I just finished I would win. No incentive to go faster... It was still fast enough to jump the finish line:
So I left the car just as it was and the third heat looked just like the first two....it went straight at the hoop and the ramp and dodged away just missing at the last possible minute.
It did not matter I was not fastest, I was most consistent I'd won.
I think I had 1131 points and the 2nd place car (the team that won last year 0x27) had 571 points.
That is a pretty solid win. If Id just focused on the one car I could have fixed a few bugs and gotten another 250 pts...(2 hoops and 3 ramps).
So what did I learn? Again I relearned the don't try and do too much... one entry, not three. I Was lucky that my starting out weirdness did not cause a crash and if the 2nd/3rd place cars had been consistent I would have taken third.
The way I tested I drove the car around an area laying out a track... then placed way points on the map. Since the map faced north all of my testing had me starting to the north.... something about my starting code worked on a heading of 0... but not at a heading of ~270 I thought I was testing like I'd race... alas that was not the case.
The BD960 trimble GPS and Omnistar service was flawless the entire time I used it. Sitting in the tent at the race it was reporting an estimated error of between 5 and 15cm. Without this level of accuracy winning the race would have required a lot more work. Its pretty amazing when you realize that its measuring better than 20cm at 20hz.
The GPS track was flawless. better than 20cm accuracy at 20Hz, so why did I miss the hoop twice and the ramp every time? After my midnight suspension repair the car pulled to the right. The code took care of this with an integral steering term in the PID that turned heading into steer direction. As all the turns were in the same direction and I did not want windup I zeroed the I term at each waypoint switch.... so since I put waypoints right before the hoop and the ramp.... right before the hoop and the ramp the I term was zeroed and the car swerved to the right until the error built up again.... I should have checked the heading change and not zeroed the I term for changes less than some value. I should have figured this out in testing,....
In my testing I'd set the hoop in the track and made sure the car was consistent. I never tested hitting a spot the was pre-configured, only in being consistent run after run... (You can see the in the video from my previous post) The track was very consistant, but it wasn't wher eI wanted it to be.
Last year I handicaped myself by not using the good GPS. This year I wanted to prove a point that the lowest cost NetBurner device was powerful, so I gave myself a deliberate handicap by choosing our lowest powered CPU. In testing I recorded telemetry and got data from the devices. During the actual race I had no on board data recording. If I had I would have easily seen both the the start up and ramp missing problems in the data in a way that I could fix them... regardless the car won with an SBL2E cpu, a network attached CPU for less than $15.
I've now won with a car so next year I'm going to finally enter a flying machine. Id prefer to enter a plane, but if the rules don't change they heavily favor a VTVL vehicle. So I'll probably pull my autonomous helicopter out of mothballs and use a new modern CPU. For fun here is the best piece of video my autonomous helicopter ever took:
I do a quick review of the code for the small car and realize that steering code has a static variable in it to measure the rate of turn... between loops and since most of my testing had the course starting to the north this would not be apparent.. I change the code so it ignores this for the first pass through navigation and prepare to try again.... This time the small car ends up in the fence... I'm zero for 3 very frustrating..
I'd planned on going faster than I was going. In testing I ran the straightaways at 3X the speed and 50% faster on the corners, but for the last run I had such a lead that if I just finished I would win. No incentive to go faster... It was still fast enough to jump the finish line:
(picture by maushammer)
It did not matter I was not fastest, I was most consistent I'd won.
I think I had 1131 points and the 2nd place car (the team that won last year 0x27) had 571 points.
That is a pretty solid win. If Id just focused on the one car I could have fixed a few bugs and gotten another 250 pts...(2 hoops and 3 ramps).
So what did I learn? Again I relearned the don't try and do too much... one entry, not three. I Was lucky that my starting out weirdness did not cause a crash and if the 2nd/3rd place cars had been consistent I would have taken third.
The way I tested I drove the car around an area laying out a track... then placed way points on the map. Since the map faced north all of my testing had me starting to the north.... something about my starting code worked on a heading of 0... but not at a heading of ~270 I thought I was testing like I'd race... alas that was not the case.
The BD960 trimble GPS and Omnistar service was flawless the entire time I used it. Sitting in the tent at the race it was reporting an estimated error of between 5 and 15cm. Without this level of accuracy winning the race would have required a lot more work. Its pretty amazing when you realize that its measuring better than 20cm at 20hz.
The GPS track was flawless. better than 20cm accuracy at 20Hz, so why did I miss the hoop twice and the ramp every time? After my midnight suspension repair the car pulled to the right. The code took care of this with an integral steering term in the PID that turned heading into steer direction. As all the turns were in the same direction and I did not want windup I zeroed the I term at each waypoint switch.... so since I put waypoints right before the hoop and the ramp.... right before the hoop and the ramp the I term was zeroed and the car swerved to the right until the error built up again.... I should have checked the heading change and not zeroed the I term for changes less than some value. I should have figured this out in testing,....
In my testing I'd set the hoop in the track and made sure the car was consistent. I never tested hitting a spot the was pre-configured, only in being consistent run after run... (You can see the in the video from my previous post) The track was very consistant, but it wasn't wher eI wanted it to be.
Last year I handicaped myself by not using the good GPS. This year I wanted to prove a point that the lowest cost NetBurner device was powerful, so I gave myself a deliberate handicap by choosing our lowest powered CPU. In testing I recorded telemetry and got data from the devices. During the actual race I had no on board data recording. If I had I would have easily seen both the the start up and ramp missing problems in the data in a way that I could fix them... regardless the car won with an SBL2E cpu, a network attached CPU for less than $15.
I've now won with a car so next year I'm going to finally enter a flying machine. Id prefer to enter a plane, but if the rules don't change they heavily favor a VTVL vehicle. So I'll probably pull my autonomous helicopter out of mothballs and use a new modern CPU. For fun here is the best piece of video my autonomous helicopter ever took:
Lastly I get requests for the code on these projects.... since I write code for a living for NetBurner I'm hesitant to post my hobby code because its not up to the standard of code that I would release in my professional capacity. So as long as you realize that this is hobby code written in a hurry while suffering from sleep deprivation you can have at it..
.
.
Here is the code for the car.. as a a few days before the AVC...
and
here is the QT code for taking logs and placing waypoints... again a few days before the AVC....
I just returned from the SparkFun 2013 AVC. It was another interesting experience.
This is my second year competing and I'd hoped I'd learned something. (See 2012 report).
At one level this year was a success I won the doping class rover with over 1100 pts, 2nd place was at 571 pts. I set out to win and I did, beyond that it was much more painful than I expected it to be.
The format of this years contest differed from last year and my LASER range finder approach would have been problematic. In any case the Trimble BD960 L2 Omnistar GPS I had from the lunar lander contest was the perfect instrument to make this contest seem easy....
Last year I ran a red cat racing 1/5 scale buggy and so did last years winner, team 0x27.
This year I decided to bring three vehicles to the event:
Insanity is doing the same thing over and over and expecting a different outcome. Last year I tried to do both a plane and car and failed from having too much on my plate this year I again tried to do too much... I got much farther with the plane than last year, it was very close to ready. I could have successfully done the minimum necessary to complete the task. The ball drop, and autoland were not ready and hence it could not win, or even place.
So lets back up a couple of months....I'm a pilot from way back. (my Dad ran an Alaskan bush airline and I've been flying since I was a teenager.) And I love airplanes, my heart was reallyu into building an airplne for the contest and toward that end I developed a custom autopliot board using the same sensors as the 3D robotics APM and a NetBurner MOD5213 for the CPU. I put together a nice clean installation in a park zone Extra 300 . One pof the stretch goals was to develop some hardware and software that might eventually become a product.
Picture of the Autopilot in the canopy of the test plane.
I spent about three months working on the code and hardware to make all of this clean and tight in the end I ran out of time. Along the way I tried to use a Q2 Robotics controller I got from kickstarter and getting that to work correctly ate a whole month and was finally abandoned after I re-cpu-ed it and ran out of time.
I'd hoed to use the controller for both the plane and the cars, in developing an autonomous vehicle its really useful to be able to get real time feedback and to have lots and lots of mode switches. I also liked the four pots allowing one to tune 4 different PID tuning parameters in real time.
Here is a picture of my final version:
In preparing the cars I started by bolting the high end BD960 GPS to last years car and getting it all working.
I also chose a new chassis, a Traxss XO-1 the fastest production RC car on the planet. (I was playing to win)
About this time I decided that since 99% of the work was in the software and since I wanted to run both a leess than $350 class car and a doping class car I chose the same CPU for both. TheCheapShot got the Netburner SBL2E Break out board for the CPU. Since this CPU uses a different branch of the NetBurner RTOS environment it meant that the car code was an almost 100% complete re-write. I used the same CPU in a different package for the eventual Doping class car. This CPU is limited and is made for really low cost network attached device so it presented some challenges:
Here are pictures of the three
Last years CPU had SDCARD and 64Mb of RAM so logging data was not an issue.
This years CPU did not have either a SD Card or enough RAM to log all data.
So I started using a SparkFun data logger to record data, but it soon became apparent that at 115K baud it just did not keep up. So I added an Xbee to radio telemetry to the computer and stay under the $350 cost target. I eventually ended up with two schemes one using the Xbee on the CheapShot car and one using a
a WIFI router on the Doping car.... as the tiny car was not big enough to carry the router.
So I developed two systems of telemetry encoding and recording... ugly...
On last years CPU I had a really nice system for setting and modifying parameters... without recompiling the code... I ran out of time to port this system to the new CPU so everything was compiled in as code constants.... I should have taken the time to fix this it would have been a win in time and effort, but hindsight is 20/20. In the heat of the process the pain never got bad enough to make me fix this.
I developed a weird trap bug about a week before the contest... I spent three or four days wringing custom diagnostic tools to find my bug only to find that it was an interrupt priority problem I had two interrupts with exactly the same priority and level and this causes a known bug in the CPU interrupt controller.....
So about two weeks before the contest I had the doping class car hardware finished and it was driving around in my driveway...
At this point it looks pretty good...
It was still missing:
This is my second year competing and I'd hoped I'd learned something. (See 2012 report).
At one level this year was a success I won the doping class rover with over 1100 pts, 2nd place was at 571 pts. I set out to win and I did, beyond that it was much more painful than I expected it to be.
The format of this years contest differed from last year and my LASER range finder approach would have been problematic. In any case the Trimble BD960 L2 Omnistar GPS I had from the lunar lander contest was the perfect instrument to make this contest seem easy....
Last year I ran a red cat racing 1/5 scale buggy and so did last years winner, team 0x27.
This year I decided to bring three vehicles to the event:
- A full on doping class vehicle done as well as I could with almost no restrictions.
- A low cost <$350.00 entry.
- A plane.
Insanity is doing the same thing over and over and expecting a different outcome. Last year I tried to do both a plane and car and failed from having too much on my plate this year I again tried to do too much... I got much farther with the plane than last year, it was very close to ready. I could have successfully done the minimum necessary to complete the task. The ball drop, and autoland were not ready and hence it could not win, or even place.
So lets back up a couple of months....I'm a pilot from way back. (my Dad ran an Alaskan bush airline and I've been flying since I was a teenager.) And I love airplanes, my heart was reallyu into building an airplne for the contest and toward that end I developed a custom autopliot board using the same sensors as the 3D robotics APM and a NetBurner MOD5213 for the CPU. I put together a nice clean installation in a park zone Extra 300 . One pof the stretch goals was to develop some hardware and software that might eventually become a product.
I spent about three months working on the code and hardware to make all of this clean and tight in the end I ran out of time. Along the way I tried to use a Q2 Robotics controller I got from kickstarter and getting that to work correctly ate a whole month and was finally abandoned after I re-cpu-ed it and ran out of time.
I'd hoed to use the controller for both the plane and the cars, in developing an autonomous vehicle its really useful to be able to get real time feedback and to have lots and lots of mode switches. I also liked the four pots allowing one to tune 4 different PID tuning parameters in real time.
Here is a picture of my final version:
In preparing the cars I started by bolting the high end BD960 GPS to last years car and getting it all working.
I also chose a new chassis, a Traxss XO-1 the fastest production RC car on the planet. (I was playing to win)
About this time I decided that since 99% of the work was in the software and since I wanted to run both a leess than $350 class car and a doping class car I chose the same CPU for both. TheCheapShot got the Netburner SBL2E Break out board for the CPU. Since this CPU uses a different branch of the NetBurner RTOS environment it meant that the car code was an almost 100% complete re-write. I used the same CPU in a different package for the eventual Doping class car. This CPU is limited and is made for really low cost network attached device so it presented some challenges:
Here are pictures of the three
The original low cost car.
The Low cost car as run....
The doping class car as it was run
Last years CPU had SDCARD and 64Mb of RAM so logging data was not an issue.
This years CPU did not have either a SD Card or enough RAM to log all data.
So I started using a SparkFun data logger to record data, but it soon became apparent that at 115K baud it just did not keep up. So I added an Xbee to radio telemetry to the computer and stay under the $350 cost target. I eventually ended up with two schemes one using the Xbee on the CheapShot car and one using a
a WIFI router on the Doping car.... as the tiny car was not big enough to carry the router.
So I developed two systems of telemetry encoding and recording... ugly...
On last years CPU I had a really nice system for setting and modifying parameters... without recompiling the code... I ran out of time to port this system to the new CPU so everything was compiled in as code constants.... I should have taken the time to fix this it would have been a win in time and effort, but hindsight is 20/20. In the heat of the process the pain never got bad enough to make me fix this.
I developed a weird trap bug about a week before the contest... I spent three or four days wringing custom diagnostic tools to find my bug only to find that it was an interrupt priority problem I had two interrupts with exactly the same priority and level and this causes a known bug in the CPU interrupt controller.....
So about two weeks before the contest I had the doping class car hardware finished and it was driving around in my driveway...
At this point it looks pretty good...
It was still missing:
- Start on a push button.
- Way to record and enter Waypoints other than by hand.
- IT was still dependent on the RC transmitter for throttle control.
At this point I started trying to run the code to the the less than $350 car.
I did not do much S/W development on this car as the low cost GPS was not working well enough to reliably stay in the driveway. I figured that I'd tune and tweak the settings for this car when I got to testing in a larger more open venue.
I did not do much S/W development on this car as the low cost GPS was not working well enough to reliably stay in the driveway. I figured that I'd tune and tweak the settings for this car when I got to testing in a larger more open venue.
Its now about a week before the contest and I go back to working on the plane.
(I also ran a 1/2 Marathon on Sunday the 2nd that I had signed up for before the AVC was on the schedule.)
I continued to have the plane will be ready fantasy until Wed morning when the canopy came off in flight and I totaled the primary air frame...
In the evenings when it was too dark to fly I started working on a mapping app to help me enter way points.
This was my first real QT app and while buggy and incomplete it worked reasonably well...
I left San Diego at noon on June 5th, headed for Boulder. After 1100 miles we arrived in boulder on the afternoon of the 6th. We went to the boulder reservoir and started driving around the parking lot making sure things were working well. At this point I had a way to manually enter waypoints overlayed on a telemetry track..and to run the car at fixed throttle without touching the RC transmitter....
As the parking lot started to fill up at 5pm it got too crowded to really test the cars... we went on to find our hotel.... I then spent the evening moving the electronics from the tiny fragile car to the less tiny less fragile car for the $350 class. After that was don I went to the parking lot across from the hotel to do some tuning and testing..... The GPS precision of the tiny car was so much poorer that the good gps tht it did nto drive very well, I needed to use the magnetic heading and turn down the response to the GPS errors.
So for the 350 car on Thursday I
- Moved the electronics
- Re-calibrated the magnetometer
- Spend several hours tweaking the steering and tracking gain.
At this point I have both cars driving around the parking lot with the RC receiver still hooked up and taking abort commands...
So back at the hotel ... I do the following...
The contest rules say no communication with the car, so I need to bring a car with no telemetry and no RC receiver to the event. With no telemetry how do I record the position of all the waypoints ?
so I add a system to record specific points into memory using the "Gear" switch on the RC transmitter.
I then go across the parking lot and test this.... The doping class car works just fine, the Low cost car with the pooer GPS and lower GPS rate drives like a drunken sailor.... with an hour or so of loop gain tweaking it pretty much drives the course as programmed.
The one thing I have left to do is to program acceleration and deceleration into the Doping class car... so I take a bunch of data on how hard it accelerates/decelerates. What speeds it can corner without slipping, how much brake is necessary to decelerate well etc.... given all this data I sit in the parking lot across the street and try to implement this....its all pretty much done when a security guard asks me to leave.
I tell hip its an empty parking lot nd I only need about 10 more minutes. ?he says as he makes his rounds he will be back in 5 min... I have till then...
So I hurry things up and go out to test the car.. in a hurry I put the new code in the car and rushing to get the car tested I drop the RC transmitter on the tailgate of my truck.... while holding the car. The car goes to 100% throttle... (Over 100mph with this car) so I'm holding a car screaming like a banshee with significant gyro forces trying to tear it out of my hands. I reach down and slam the throttle stick to full brake.
This is more than the speed control can take an the car, batteries and speed control errupt in flames.
Its now 10:15 pm the day before the race and the car just caught fire.
Back to the hotel room to strip the speed control off of last years car In the spares there is a similar speed control ..I dissemble the car and install that.
His speed control is wired to a single battery connector and my ole one was wired for dual.. as my list of things to do included moving the battery for better balance I do that at this time too...
The battery voltage changed from 6S to 4S and the speed control changed from Mamba Monster to Mamba Monster II. I download the castle creations programming software to the lap top and try to reprogram the speed control into linear mode like the last one.. alas it does not support this mode.
So all the acceleration deceleration mapping data I took is now invalid.
Its midnight I'm out in an empty theater parking lot redoing the acceleration/deceleration curves.
I get this done and test the code.. a few tweaks later the car is running pretty good. Its insanely fast and I'm really happy with the results. I decide to do one last run takes off really fast, makes the first corner ... turns left accelerates between corners at the second corner it does not slow down.... it doesn't' turn it just goes straight and disappears from view as it leaves the area of lit parking lot at full speed....
Its now 1:30 am and I'm wandering a huge dark parking lot searching for my car... after about 15 minutes I find it, it hit the curb at speed and the left front shock mount snapped off the bolt holding it on and is dangling loose. The error is mine.. I let the battery that runs the GPS and CPU get too low and the GPS stopped reporting. I go back to the hotel and dig through my spares.. I brought spares of all the suspension components... except the bolts that hold them together..... I head out to the all night walmart hoping they have bolts.... their bolt selection is really weak and and back at the hotel at 3am I end up stealing screws from other parts of the car to fix the broken parts. Its 3:30 am I'm getting up at 6:30 to go to the event... I think the car is ready... I still have to strip out the code that listens to the RC receiver... I'll do that in the morning. Much better than last year I get 3 hours of sleep. (As compared to 0)......
To be continued I'll cover the day of the race in my next post.
The one thing I have left to do is to program acceleration and deceleration into the Doping class car... so I take a bunch of data on how hard it accelerates/decelerates. What speeds it can corner without slipping, how much brake is necessary to decelerate well etc.... given all this data I sit in the parking lot across the street and try to implement this....its all pretty much done when a security guard asks me to leave.
I tell hip its an empty parking lot nd I only need about 10 more minutes. ?he says as he makes his rounds he will be back in 5 min... I have till then...
So I hurry things up and go out to test the car.. in a hurry I put the new code in the car and rushing to get the car tested I drop the RC transmitter on the tailgate of my truck.... while holding the car. The car goes to 100% throttle... (Over 100mph with this car) so I'm holding a car screaming like a banshee with significant gyro forces trying to tear it out of my hands. I reach down and slam the throttle stick to full brake.
This is more than the speed control can take an the car, batteries and speed control errupt in flames.
Its now 10:15 pm the day before the race and the car just caught fire.
Back to the hotel room to strip the speed control off of last years car In the spares there is a similar speed control ..I dissemble the car and install that.
His speed control is wired to a single battery connector and my ole one was wired for dual.. as my list of things to do included moving the battery for better balance I do that at this time too...
The battery voltage changed from 6S to 4S and the speed control changed from Mamba Monster to Mamba Monster II. I download the castle creations programming software to the lap top and try to reprogram the speed control into linear mode like the last one.. alas it does not support this mode.
So all the acceleration deceleration mapping data I took is now invalid.
Its midnight I'm out in an empty theater parking lot redoing the acceleration/deceleration curves.
I get this done and test the code.. a few tweaks later the car is running pretty good. Its insanely fast and I'm really happy with the results. I decide to do one last run takes off really fast, makes the first corner ... turns left accelerates between corners at the second corner it does not slow down.... it doesn't' turn it just goes straight and disappears from view as it leaves the area of lit parking lot at full speed....
Its now 1:30 am and I'm wandering a huge dark parking lot searching for my car... after about 15 minutes I find it, it hit the curb at speed and the left front shock mount snapped off the bolt holding it on and is dangling loose. The error is mine.. I let the battery that runs the GPS and CPU get too low and the GPS stopped reporting. I go back to the hotel and dig through my spares.. I brought spares of all the suspension components... except the bolts that hold them together..... I head out to the all night walmart hoping they have bolts.... their bolt selection is really weak and and back at the hotel at 3am I end up stealing screws from other parts of the car to fix the broken parts. Its 3:30 am I'm getting up at 6:30 to go to the event... I think the car is ready... I still have to strip out the code that listens to the RC receiver... I'll do that in the morning. Much better than last year I get 3 hours of sleep. (As compared to 0)......
To be continued I'll cover the day of the race in my next post.
Monday, May 06, 2013
Cute coding trick...
In building cars and planes and rockets, I collect a lot of data log files for all of the projects....
I backup all the source files using source control on every build..., but I presently had no way to connect a data file to the exact source that was used to make it... I'm sure this has been done a thousand times... before but I reinvented the wheel last night...
I wrote a simple header file...
call it FileList.h
//A Class to make a static linked list of ALL files included in the project....
class FileList
{
private:
static FileList * pHead;
FileList * m_pNext;
const char * m_pName;
public:
//The constructor will add the name to the global static linked list...
FileList(const char * name)
{m_pNext=pHead;
pHead=this,
m_pName=name;
};
static void ShowList();
};
//A Static opbject that does the real work...
//__BASE_FILE__ is the name of the file being compiled rather thatn this header file
static FileList FL(__DATE__ "," __TIME__ "," __BASE_FILE__);
I backup all the source files using source control on every build..., but I presently had no way to connect a data file to the exact source that was used to make it... I'm sure this has been done a thousand times... before but I reinvented the wheel last night...
I wrote a simple header file...
call it FileList.h
//A Class to make a static linked list of ALL files included in the project....
class FileList
{
private:
static FileList * pHead;
FileList * m_pNext;
const char * m_pName;
public:
//The constructor will add the name to the global static linked list...
FileList(const char * name)
{m_pNext=pHead;
pHead=this,
m_pName=name;
};
static void ShowList();
};
//A Static opbject that does the real work...
//__BASE_FILE__ is the name of the file being compiled rather thatn this header file
static FileList FL(__DATE__ "," __TIME__ "," __BASE_FILE__);
If you #include "filelist.h"
in every single C++ file in your project... this will make a linked list of
all the file names, dates and times..... a list I can prepend to the data log on startup...
so now the whole code set and its date time are included in the data log on every start...
Maybe this is common knowledge, but the __BASE__FILE__ macro was a new one to me...
Monday, April 29, 2013
Data logging...
Logging data from "Things", I've run computer operated planes, rockets, cars, helicopters. All of these share a similar problem. How do you log data? I like to log ALL raw sensor readings unmodified. This ends up in a steam of multi rate data, ie I see raw IMU readings at 200Hz, Compass readings at 20Hz GPS at 20Hz, and various things like mode changes and responses at varying rates from 200Hz to to once at power up.
Some things I've learned:
Where START is a specific byte and ID is a numeric value representing the kind of data...
If the value START appears in the data is is escaped...The packet ends with the next start....
I then have a background process that takes this circular buffer and writes it to flash sectors where there is a full sector worth of data.
Now I might record something like my GPS reading...
typedef struct
{
double lattitude; //Radians
double longitude;//Radians
double ht; //m
unsigned long msec;
unsigned short week;
double ECEF_X;//m
double ECEF_Y;//m
double ECEF_Z;//m
float hspeed;//m/sec
float head; //radians
float vv; //m/sec
unsigned char flag1;
unsigned char flag2;
unsigned char nsat;
WORD seq;
}CombinedTrimGps;
So that would be stored as
START GPS_ID the contents of the structure.....
Now if in the past if I changed the record so I added say a list of SNR reading to the record I'd have to redo the code that decodes the GPS data from the data logs....
In an ideal world C++ would have introspection and it would be able to put the format of the structure into the data log the first time its used..... then when the data reader sees the data it immediately knows what format the data is in....
Since C++ does not have introspection... I wrote some code that comes very close...
void LogRecord(CombinedTrimGps & item)
{
LogStart(LOG_TYPE_BD960 ,"GPS");
LogElement(lattitude,"lattitude " );
LogElement(longitude,"longitude " );
LogElement(ht ,"ht " );
LogElement(msec ,"msec " );
LogElement(week ,"week " );
LogElement(ECEF_X ,"ECEF_X " );
LogElement(ECEF_Y ,"ECEF_Y " );
LogElement(ECEF_Z ,"ECEF_Z " );
LogElement(hspeed ,"hspeed " );
LogElement(head ,"head " );
LogElement(vv ,"vv " );
LogElement(flag1 ,"flag1 " );
LogElement(flag2 ,"flag2 " );
LogElement(nsat ,"nsat " );
LogEnd();
}
Using Macros this expands into...
void LogRecord(CombinedTrimGps & item)
{
static bool bIntroed;
BYTE tid=LOG_TYPE_BD960;
if(!bIntroed)
{
bIntroed=true;
StartItemIntro(tid,sizeof(item),"GPS");
ShowElement(tid,&item,&item.lattitude,item.lattitude,"lattitude " ) ;
.
.//All the elements
.
}
LogRawRecord(tid,(const unsigned char *)&item,sizeof(item));
}
I have a different ShowElement overloaded function for each type I might want to log....
Thus the first time this data type is logged it puts the format into the data log record....
This is something I've wanted since I first did the rocket stuff 6 years ago.
I've now got a self describing log format that is complete and efficient....
I can also use the exact same format for Telemetry....
On my rockets I had an optional parameter that went to the logging function that selected if the data went to the LOG, to telemetry or to both locations....
Last night I got the whole chain to work and the picture below is derived from this chain of events...
IE I drove the AVC car around my driveway -> recorded log -> process log generating records -> put in excel -> Then made a graph of locations....
Some things I've learned:
- I don't trust a file system. If your thing crashes the data you are really interested in is the last few bits of data before impact. I've lost too much data to trust a file system to be robust when the recording media is de-powered and spattered all over the landscape
- You can't squeeze everything you want to record in a telemetry stream.
- I keep reinventing this same wheel.
- In the course of a project the format of what you are recording evolves. Keeping the code for the data display and the data recorder in sync is difficult and often I end up unable to read earlier logs.
My current approach is as follows:
Record everything to flash directly, it makes no difference if its a SD card or dedicated flash chip I use the same format. I fully erase the flash (all the sectors on SD) then on power up I scan the flash page or sector by sector looking for the first blank page/sector. I then start recording new data in that sector....
My data recording has a big circular buffer that records the data in stream format....
All the recorded data is in blocks with the format:
START ID ... (Next START)
Where START is a specific byte and ID is a numeric value representing the kind of data...
If the value START appears in the data is is escaped...The packet ends with the next start....
I then have a background process that takes this circular buffer and writes it to flash sectors where there is a full sector worth of data.
Now I might record something like my GPS reading...
typedef struct
{
double lattitude; //Radians
double longitude;//Radians
double ht; //m
unsigned long msec;
unsigned short week;
double ECEF_X;//m
double ECEF_Y;//m
double ECEF_Z;//m
float hspeed;//m/sec
float head; //radians
float vv; //m/sec
unsigned char flag1;
unsigned char flag2;
unsigned char nsat;
WORD seq;
}CombinedTrimGps;
So that would be stored as
START GPS_ID the contents of the structure.....
Now if in the past if I changed the record so I added say a list of SNR reading to the record I'd have to redo the code that decodes the GPS data from the data logs....
In an ideal world C++ would have introspection and it would be able to put the format of the structure into the data log the first time its used..... then when the data reader sees the data it immediately knows what format the data is in....
Since C++ does not have introspection... I wrote some code that comes very close...
void LogRecord(CombinedTrimGps & item)
{
LogStart(LOG_TYPE_BD960 ,"GPS");
LogElement(lattitude,"lattitude " );
LogElement(longitude,"longitude " );
LogElement(ht ,"ht " );
LogElement(msec ,"msec " );
LogElement(week ,"week " );
LogElement(ECEF_X ,"ECEF_X " );
LogElement(ECEF_Y ,"ECEF_Y " );
LogElement(ECEF_Z ,"ECEF_Z " );
LogElement(hspeed ,"hspeed " );
LogElement(head ,"head " );
LogElement(vv ,"vv " );
LogElement(flag1 ,"flag1 " );
LogElement(flag2 ,"flag2 " );
LogElement(nsat ,"nsat " );
LogEnd();
}
Using Macros this expands into...
void LogRecord(CombinedTrimGps & item)
{
static bool bIntroed;
BYTE tid=LOG_TYPE_BD960;
if(!bIntroed)
{
bIntroed=true;
StartItemIntro(tid,sizeof(item),"GPS");
ShowElement(tid,&item,&item.lattitude,item.lattitude,"lattitude " ) ;
.
.//All the elements
.
}
LogRawRecord(tid,(const unsigned char *)&item,sizeof(item));
}
I have a different ShowElement overloaded function for each type I might want to log....
Thus the first time this data type is logged it puts the format into the data log record....
This is something I've wanted since I first did the rocket stuff 6 years ago.
I've now got a self describing log format that is complete and efficient....
I can also use the exact same format for Telemetry....
On my rockets I had an optional parameter that went to the logging function that selected if the data went to the LOG, to telemetry or to both locations....
Last night I got the whole chain to work and the picture below is derived from this chain of events...
IE I drove the AVC car around my driveway -> recorded log -> process log generating records -> put in excel -> Then made a graph of locations....
Now I have to get the data replay tool I wrote a few years ago to read this format and my data logging tools will be complete!
Friday, April 19, 2013
Finding a Firmware bug...
I spent the last few days hunting down a bug.
It was interesting enough that I thought I'd write a short blog post....
First some background...
In all of the NetBurner products we try very hard to make sure address space 0 is unmapped,
ie any attempt to reference a NULL pointer should cause an error.
The particular processor I was bug hunting on boots at address zero so in most normal scenarios address 0 is mapped to some kind of memory.
In our case this is not so...after the boot monitor boots, it turns off the hardware memory mapping to address 0...and relocates all valid memory up higher in the memory map...
(We are not using an MMU)
The bug we were hunting was an interrupt latency bug, 99.99% of the time the part has an interrupt latency of 0.9 usec or so... every once in a while it would have a longer latency, randomly varying from 1 to 520usec. 520usec is WAAAAAY too long.....
So to instrument this I setup one of the on chip timers to make a pulse and reset the counter at a fixed rate...
I hooked this timer output up to the non maskable interrupt and in the interrupt the first thing I did was read the timer counter value... this is an all on chip direct measurement of the interrupt latency...
And soon found that it varied randomly from 0.9 to 520usec.
Step 1 complete we can replicate the problem.....
After this I tried a whole bunch of things to hunt this down, looked at code, moved stacks and vector tables to/from different classes of memory... nothing changes the result....
If I make the whole code set small enough so that it all fits in the instruction cache it goes away....
As a random idea I changed the bus timeout monitor from very long to something shorter...
and the latency improved... this was the clue I needed to hunt down the bug....
Here is the bug:
We use a lot of C code in the system of the form:
It was interesting enough that I thought I'd write a short blog post....
First some background...
In all of the NetBurner products we try very hard to make sure address space 0 is unmapped,
ie any attempt to reference a NULL pointer should cause an error.
The particular processor I was bug hunting on boots at address zero so in most normal scenarios address 0 is mapped to some kind of memory.
In our case this is not so...after the boot monitor boots, it turns off the hardware memory mapping to address 0...and relocates all valid memory up higher in the memory map...
(We are not using an MMU)
The bug we were hunting was an interrupt latency bug, 99.99% of the time the part has an interrupt latency of 0.9 usec or so... every once in a while it would have a longer latency, randomly varying from 1 to 520usec. 520usec is WAAAAAY too long.....
So to instrument this I setup one of the on chip timers to make a pulse and reset the counter at a fixed rate...
I hooked this timer output up to the non maskable interrupt and in the interrupt the first thing I did was read the timer counter value... this is an all on chip direct measurement of the interrupt latency...
And soon found that it varied randomly from 0.9 to 520usec.
Step 1 complete we can replicate the problem.....
After this I tried a whole bunch of things to hunt this down, looked at code, moved stacks and vector tables to/from different classes of memory... nothing changes the result....
If I make the whole code set small enough so that it all fits in the instruction cache it goes away....
As a random idea I changed the bus timeout monitor from very long to something shorter...
and the latency improved... this was the clue I needed to hunt down the bug....
Here is the bug:
We use a lot of C code in the system of the form:
if ( pDHCPTick )
{
pDHCPTick();
}
This code checks a function pointer to see if its null and calls the function if it isn't...
This chunk of code is inside one of the system tasks and is used to service the DHCP client if its active...
This compiles to assembly code something like:
moveal pDHCPTick,%a0
testl %a0
beqw skip
jsr @%a0
skip:
So if pDHCPTick is null the jsr @%a0 will never be executed....
But the processor I'm having the issue with has aggressive pipelining....
So it tries to load the @%a0 even if it does not call it.
This causes an instruction fetch of address 0....
If address 0 is not in the instruction cache then this forces a cache miss and a cache line read from physical address 0.....
Physical address 0 is un-mapped so the bus timeout monitor goes off terminating the transaction 520usec later...
If the interrupt has the misfortune of trying to go off while we are hung in the bus time out we get long latency....
So the problem was solved by setting up the system so there is a valid bus ack at address 0....
Makes it so we don't catch NULL pointer accesses... but fixes the interrupt latency...
We are working on a better solution.. but I thought the problem crossed enough different hardware/software domains where it was interesting.
Wednesday, April 03, 2013
Teaching a computer to fly...
For this years spark fun AVC I'm planning to bring both a Car and a Plane. The Car is primarily last years car upgraded with a lower center of gravity and a much much better GPS. The plane is a different story...
As part of the Lunar Lander Challenge I built an autonomous helicopter to test out my code. Right now I have a vectored thrust rocket I'm flying as well, all in all these two efforts have the actuators and response of the vehicle fairly well coupled and tight. a rocket or helicopter is a brute force thing that beats the environment into submission. An airplane is a gentle elegant creature. I 've been a pilot since I was a teenager, and when I look at most of the open or openly discussed UAV autopilots out in the world it strikes me that a lot of the people working on them don't understand this distinction.
I have a copy of x-plane 10 with I'm using as a simulator for the autopilot development.
I've set this up as a hardware in the loop simulator and I've been slowly teaching the NetBurner to fly.
Much like my Dad taught me to fly a long time ago.
- Manage your airspeed,
- Coordinate your turns,
- Hold a heading,
- Hold an altitude,
- Make smooth level turns...
- Turn to a heading...
Now learn to do all of the above while climbing, descending, accelerating, decelerating, you have a precision tool with the grace of a light saber, feel the grace. If the engine dies it should still be in control when glides gently to earth.....
An airplane can be flown with a single axis rate gyro, anyone that has passed a partial panel IFR checkride understands this. I have all the rate gyro integrating quanterion attitude estimaters that everyone else has.... alas most of this is not needed you can do everything you need to do to control the airplane with airspeed , rate of turn, magnetic compass and altitude.
The interactions can be subtle... does the throttle control speed? does it control altitude? what variable does it control? How about energy..... I found this paper to be especially good on this point.
The basic bottom level control laws are simple....
Inputs:
Desired Airspeed
Desired Heading
Desired Altitude (or rate of climb)
Sensors:
Pitot AirSpeed (IAS)
Magnetic heading
Rate of turn.
Altitude (either baro or GPS driven)
The controls are simple...
The elevator is controlled with Airspeed and TargetAirspeed
The rudder does nothing more than keep the "Ball" centered in flight or the aircraft on the runway on takeoff.
The Aileron is controlled by rate of turn, magnetic heading and target heading
The throttle is controlled with the current energy state ie airspeed ^2 and altitude vs their target values.
(I've also found it useful to have a bit of feed forward from turn rate to throttle to add a little power in the turns)
In any case I'm having fun writing this....
Subscribe to:
Posts (Atom)