Sunday, January 13, 2008

(Update) A Day of testing.

(Update I found the receiver problem, read to end)
I drove out to Mojave Friday night and spent all day Saturday at FAR. There was a TV program shooting pictures of large high power rockets and that was kind of cool to watch. It took them all day to do some background video and two flights. The flights went perfectly. I had often contemplated inviting some TV to our tests, but after watching them all day I don't think I could handle the time impact, its much more intrusive than I would have expected.

In any case I set up shop in the propellant mixing shed and did some flight testing. My previous helicopter testing I'd writte some code, drive somewhere to test it, go home to review the data and do it again. Saturday I had the laptop in a reasonably comfortable dim space 10 steps from where I could fly the helicopter. (Dim is necessary because you can't really read a laptop in bright sun). I made a lot more progress than I would have the traditional way. It was really nice to be able to change code do a 30 second flight review data and then change code again all inside 10 minutes. It was not perfect, the smallest screen I usually do development on is a nice 24" LCD. Doing development on the laptop fees cramped, but the quick cycle time is a big win.

Flight #1I verified that the CCPM code on the helicopter worked correctly in flight. I also verified that the heading hold code developed on the lazy susan works properly in flight.

Flight #2 I fixed the heading hold gain and removed it from the adjustable gain lever in preparation for doing altitude hold. I also did some hard vertical climbs and descents to gather some altitude data. On review the GPS altitude data was really choppy. I added an integration of the GPS velocity to the variables calculated and modified the data review code to allow me to zero this and display it. (The display code is a lot more complicated than the flight code and hence harder top work with on the Laptop)

Flight #3 Integrated GPS velocity was much better, still not perfect, the data had a downward bias. Sat down and looked carefully at the GPS data, GPS altitude was jumping in 4 m steps, very strange. I had not yet looked at the GPS position data, it looks reasonable, but Id not carefully examined it. So I paced off a 20M square oriented to true norht in the dirt and marked the corners.

Hand carry #1 I then carefully hand carried the helicopter over the 20 M square. First at waist height, then with the helicopter held as high as I could over my head. On review I got zero data. So I swap out the sd card in the data logger and do a 30 second run .I got data this time. A note on the data collection process. The Laptop can display all the data in real time, but the useful mode is to command the telmetry suitcase to shut down. Take the SD card out of the telemetry suitcase, put it into the reader, copy it into the laptop data directory, rename it to something meaningful, and write a summary in the lab notebook of what the flight test was while you still remember, then review the data on the laptop. I need to streamline this process.

Hand Carry #2
I carrnyone watching would think I'd gone crazy and was out waltzing with my helicopter. On review thy the helicopter around the 20 m square spinning 360 degrees at each corner (so I can see the heading change on the data to know where the corners were.) Ae GPS position data just isn't right. The altitude data is also doing 4M jumps. Only the GPS data is suspect, everything else is fine. The GPS data goes from the GPS to the onboard computer in binary form then its converted to be computer endian and packaged into another structure and set out telemetry. So I add another telemetry frame type to ship the raw unconverted GPS binary data to down the telemetry stream without touching it at all. I then modify the data display program to have duplicate fields for all the GPS data. (The PC is the same endian as the GPS no conversion necessary).

Hand Carry #3 Same path around the 20M square spinning at each corner.The two data sets do not AGREE!!!!! ARGHHHH. In looking at the endian conversion on the PC telemetry side I'd used HTONL host to network long it works as expected because the PC does not store data in the expected network byte order. On the Netburner side the natural storage is network byte order so HTONL is a no op on the platform. SO I'd written my own byte swap. It had typo/ error, where the 3rd byte of the float got the 4th byte duplicated. On the 8 byte doubles the 3rd got the 4th and the 7th got the 8th. So I fix this.

Hand Carry #4 The Data sets match, and the altitude is sensitive enough and not jumpy. It even detects me setting the helicopter on the ground , on the work table and holding it over my head! Cool.

Flight #4 Lift off and the helicopter is shaking/ vibrating really badly. Land immediately. One tail rotor blade had split, probably hit the sand on the flight #3 landing. Change tail rotor blades.

Flight etc... Helicopter is still vibrating, main blade tracking is really off. I do a short flight and try to adjust the tracking. I have some people trying to help me spot which blade is high the red or blue tip. The tracking just seems random, either I'm making to big adjustment or something else is wrong. Still vibrating badly. On the 6th or 7th hop helicopter goes from low hover down to land abruptly, not damaging hard, but more abrupt, and uncommanded by me.

Helicopter is sitting on the ground upright and running and I have no control. Thats easy I've set up so the motor controller is not run by my computer system it's run directly by the RC receiver so in the worst case I can shut it down without relying on software. no dice, the system does not shutdown???? I continue to fiddle with the transmitter, I shut it off. (RC receiver failsafe should shut it down... it does not) Flip the transmitter on and off. Walk around the helicopter trying fiddeling with the transmitter. It shuts down. Quickly go over and disconnect the main power battery. Back on the work bench table I test the receiver battery pack and it seems very low. All the parts of the rotor head seem ok, so I suspect that the rotor blades are failing in some way. I remove the rotor baldes and run the helicopter on the bench. Very smooth and no vibration. I take my spare blades and using a sharpie paint a black stripe on one tip. I install a fresh receiver battery and the new blades. All the controls work, I check all the cable bundels for a loose connector or other problem. All looks good.

Flight N+1 I set the helicopter down on the ground and command the motor to start. I immediately loose control. The Motor is running the helicopter has negative collective commanded so its firmly on the ground but I have no control. I start fiddeling with the transmitter on-off nothing the helicopter will not shut down. After about 5 minutes of this I crawl up to the front of the helicopter reach in and disconnect the power battery.

Is it the transmitter or the receiver?
I take out the small Trex 450 and fly it, it's perfect. The problem is not the transmitter.
Start packing up. I put everything away tables back in my container. Clean up pack etc.
I pack up everything but the little Trex. So I fly that some just for fun. I did my first helicopter rolls in real life!. I've done them on the simulator a lot, but this was the first time in real life. I pack up. Its 4:45 pm I arrive back home at 9:30 PM (Traffic in LA was bad) I unpack the car. All in all a great day.

I suspect that my receiver problem is related to this.

I apologize to all that this has become the unreasonable helicopter blog. But we should be back to rockets in the next 4 to 6 weeks.

I did some more testing today outside the garage. The on/off stick on problem persists even
when I changed the receiver. It goes away if I power the receiver separately.It seems that there is a loose or flaky power problem with the system. I'm going to put the receiver on a separate power source. I also rebuilt the bearings in the blade grips and replaced the plastic blade grips with metal and now the rotor tracks perfectly. In any case I now have a spare receiver and I've done the rework recommended in the link to my earlier suspected problem.


gravityloss said...

It's good that you can test with the helicopter... sorts out a lot of kinks with cheaper flights and cheaper crashes!

Then when you have your rocket in working order it's just plug and fly. Well, that's the idea. :)

Anonymous said...

Just out of interest, what are you writing the control code with ? C++ ?
Being an embedded developer myself, thats interesting to me.
I have run into similar issues in control code, often enough that i have started applying stringent unit testing at least to the basic chunks of code that i do.

I enjoy reading these posts, thanks for sharing. One of the most interesting sites ever i have read was from a guy who did code for MERs on vxWorks :)

Paul Breed said...

I write in C++, alas I don't use some of the bigger C++ parts such as the standard template library etc...

Its more of C with some extensions.
I've done a lot of heavy duty C++, but anything that does allocation (malloc/new) invisibly has no place in a real time embedded system.(IMHO)

The display code/PC console is written in MFC on windows.

Anonymous said...

same here, with a few caveats. RTTI and exceptions has been a big no-no for embedded. however, i have found STL and templates to be of some use, if you manage your own allocators and carefully keep an eye on template proliferation.
At some point quite some time ago we did a lot of reading and research on best practices and recommendations etc on do's and don'ts on embedded C++, in the end using what one is familiar with and gets the needed results worked out best.
Joint Strike Fighter coding standards document was an interesting read, nonetheless :)

Paul Breed said...

Didn't the F-22 have a problem when it crossed the dateline?

Paul Breed said...

Try again at the post...

Paul Breed said...

Arghhhh the comment system truncates.

Now try that.