Friday, September 05, 2008

Software Woes

A long time ago( a year ago) we had a software design.

We created a code structure for this software that matched this design, and then it branched.

One branch turned into vehicle control software optimized to run static rocket tests and gather data.

The other branch learned to fly the helicopter.

Over time the two branches grew and mutated until they were not really recognizable as twins.

The vehicle branch controls all the rocket actuators and  logs data while running static tests.

The helicopter branch hovers the helicopter in place and successfully moves it to a selected 3D waypoint.

The vehicle branch gains such things as differential GPS and a flash file system to record data.

The helicopter branch gains things like status LED’s and remotely programable modes selectable via the RC transmitter.

About a Month ago as the physical vehicle reaches completion we started working to merge what is common in the two branches in preperation for hovering the vehicle.

We split out the parts that are defferent between the two vehicles into vehicle specific modules and merged the parts that are common.

This merge took about two weeks and the merged software passes all the ground tests we can throw at it.

The merged software almost flies the helicopter, it is just a tiny bit unstable. The helicopter holds position sort of and if its not disturbed almost flies, disturb it and it goes into ever increasing wobbles….

So we added the ability to tweak the control system gains in flight and flew the helicopter, we reverted the GPS to WAAS mode and flew the helicopter, we played with gains and flew the helicopter, we stuck the tail roter in the dirt, we fixed the tail roter, we flew the helicopter, we stuck the tail roter in the dirt we replaced the tail roter gear box, we adjusted gains , we flew the helicopter, we adjusted gains, we stuck the tail roter in the dirt we replaced the tail roter belt, we flew the helicopter, we had a dumb thumbs moment while trying to check the blade tracking in manual mode, and we replaced the gear, the training poles, the main roter blades, the tail boom, the landing gear, and the tailroter ie rebuilt the helicopter, and we adjusted the gains, and our hair turned grayer, and we swore at the helicopter.

We went back to the source control, checked out the last version of the Helicopter before the attempted merge, we reverted the the GPS to WAAS mode, and we flew the helicopter. It flies perfect…..arghhhhhhhh!

So we know its not the GPS, not the IMU, not the servos, not the helicopter mechanics, or any thing else it’s the software and as far as I can tell the control laws and delays are identical, just rearranged into different files.


Anonymous said...

Unitialized variable?

john g said...

Sounds like it *could* be a timing issue to me. Mixing the new code in could have introduced delays that were not there before. Have you checked that the loop is running at the same rate as before? No new lags in the actuator commands? I have seen network lag in simulation throw off a controller in a similar fashion, making it marginally stable or mildly unstable. Then again you said you tried new gains and that didn't work, so that goes a bit against this theory.

gravityloss said...

Can you "fork" the problem, ie replace only half the code and try again? And if the problem persists replace only a quarter etc to narrow down where it is...

Also, if you can save all data from a helicopter flight you should be able to run simulations offline. If the inputs are exactly the same, the outputs should be too. Then you can run sw tests in seconds and no loss of hardware. I realize doing offline runs for feedback software is of limited use (you can only replicate exact results) but still...
I don't know if it is feasible to save all data though.

Anonymous said...

sounds like timing lag.

can you make the code run a simulator?

feed the code into XPlane and try and debug
the code

Paul Breed said...

It looks like it is related to heading. We assume that the vehicle is pointing true north. We measure magnetic north and adjust for local magnetic variation. Given pointing true north then an east west error can be corrected by rolling left/right. If we are not pointing true north then the errors cross couple. A roll left does not just change the Longitude, it also changes the Latitude requiring a pitch corrections which also couples to the e-w etc..... We flew 14 versions of the software today and it comes down to the new software not holding the proper heading.
I did a bit of work on this and
I just flew off two batteries and the "new" branch worked well.
I had the helicopter flying squares 10 meters on a side going from waypoint to waypoint. Now to get the helicopter doing the equivalent of a 14.9 second tethered rocket flight, up 1M, hover level, down.