Paul’s AVC postmortem….
In the end it did not go as well as I would have liked. On 6 official attempts I only completed one
lap.
I thought I was pretty close to ready; The car was following
complex arced paths well…
This picture is from testing Wednesday The black and pink
arcs are on top of one another.
The big change that enabled this precision was to add 200
msec of position prediction to the turn algorithm. IE star turning 200 msec
early (200mse seems to be the command to actual steer servo is in new position
latency)
As you can see the difference between the black actual path and "pinl desired path is insignificant.
In addition, with nice straight building walls the laser scanner was correctly finding the walls and correcting both the horizontal offset and accumulated heading errors against the walls.
When I left for Denver on Thursday I had three software bits unfinished….
I’d done testing and analysis on all three, but they were incomplete.
1)Obstacle avoidance.
In addition, with nice straight building walls the laser scanner was correctly finding the walls and correcting both the horizontal offset and accumulated heading errors against the walls.
When I left for Denver on Thursday I had three software bits unfinished….
I’d done testing and analysis on all three, but they were incomplete.
1)Obstacle avoidance.
2)Stop at the stop sign/pedestrian.
3)Hoop Location.
So Early Friday morning I wrote my barrel avoidance code and got to start testing on the
actual course Thursday…. By 3PM I had my barrel code working the car was going
through the barrel section mostly avoiding the barrels 5/6 times or so….The car
was a bit too large and the barrels were a bit tight so I really needed to go
slow and turn sharply. To make this happen I turned the steering gain way up
for any time the car was in barrel
avoidance mode.
I then took some data to figure out how to detect the cross walk the pedestrian walked on and thus stop at the stop sign. In testing this chunk of code it worked one time in four or five and I worked on this code until 7:30 pm where I discovered my error…. I was using a static variable to indicate that I’d already done the stop and to not do it again…So the code worked after a clean reset, but not a test restart…
Once I fixed that I was reliably stopping at the
pedestrian..
I put every thing together and did a full run where
everything worked. Then the course
closed at 8pm and I was done for the day. I had one issue remaining. The start
line was too high it high centered the car, so I had to start behind the start
line and get a running start and jump over the start line.
This basically added 50 to 70” to the path as the wheels spun. I did a hack and added in 70” to the path in code. I’ll test this in the 2 hours of testing Saturday morning…
This basically added 50 to 70” to the path as the wheels spun. I did a hack and added in 70” to the path in code. I’ll test this in the 2 hours of testing Saturday morning…
Saturday Race Day #1,
Its cold in Denver and I’m there when the course opens for practice and rather than use my 70” hack I manually record a new track path starting 24” behind the start and use that recording to layout a new path.I do a test run and the car crashes into the hay bales straight ahead…. And breacks the 3D printed bracket that held the LIDAR and electronics… I brought spres of every 3D printed object on the car (6 parts) except the one bracket, because that was not on car #2, that was metal on car #1.
So with one hour and 15 till race start I’ve got to get the 2nd car ready, it was 100% complete except I had not wired the start switch so I put both cars in the back of the retal SUV and drive back to the hotel… (about ¼ mi) When I get there I discover the back gate of the SUV had not closed and neither car was in the car. I went back and found both cars lying in the road, otherwise undamaged.
I brought them back to the room and wired in the start switch on car #2 (I’d planned to do that Saturday afternoon after day one was done. Rules were I ran one car on Sat, and a different one on Sunday.
I now have 15 min till practice is over and 30 min till my
heat starts and car #2 had not yet been
turned on in Co. I run the test and it
impacts into the same hay bale that broke the previous car. And I remember that my extra 70” hack
was still in place, I remove it and get in one last run it gets tied up in the
barrels, I’m not sure why this is not working….
Saturday Heat 1:
I’d planned to spend the morning fine tuning the path to center it up on the course and perfect it. Alas I spent the morning soldering switches and removing my 70” hack.
I’d planned to spend the morning fine tuning the path to center it up on the course and perfect it. Alas I spent the morning soldering switches and removing my 70” hack.
Run one it gets a tiny bit hung up in barrels, but otherwise
does the first half of the course, its supposed to go just inside the ramp,
alas the lack of track tuning it hits the ramp/jump and goes off the edge of
the ram, causing it to turn a bit and impact the hay bales on the other side.
End of heat 1.
Saturday Testing between heat 1 and heat 2.
The car is now acting unreliable. IT runs sometimes, and
other times it just wanders off course in random ways. A couple of times when I try to gain manual
control with the RC receiver it won’t steer….
In trouble shooting this power cycling always seems to fix this… I’m wondering if I have a hardware problem. Out of 3 or so test runs one works, one is random and one gets hung up in the barrels…
The main CPU board on the car is a NANO54415, on this car was an old dusty first rev prototype board that I had used on many projects So I figure there is no harm in swapping the board.
So I swap CPU boards for the spare in my electronics bag. I reprogram it with the car code, program in the course and I’m out of testing time. Time for heat 2….
In trouble shooting this power cycling always seems to fix this… I’m wondering if I have a hardware problem. Out of 3 or so test runs one works, one is random and one gets hung up in the barrels…
The main CPU board on the car is a NANO54415, on this car was an old dusty first rev prototype board that I had used on many projects So I figure there is no harm in swapping the board.
So I swap CPU boards for the spare in my electronics bag. I reprogram it with the car code, program in the course and I’m out of testing time. Time for heat 2….
Saturday Heat 2:
I press the go button it immediately turns right into the wall and dies.The system saves a bunch of configurable parameters in flash, one of these is use the magnetic compass, or just the bare gyro for navigation. I’d found the bare gyro was more repeatable than the compass, but use compass defaulted to on…. So car turned right into the wall due to the compass…
Practice between heat 2 and 3 on Saturday…
This went reasonably well, I tuned the path a little bit.
This went reasonably well, I tuned the path a little bit.
Saturday Heat 3:
The car ran pretty much perfectly, it dodged the barrels well, missed the ramp, caught the hoop and almost stopped at the stop sign. It was supposed to stop within 12” of the line, it stopped about 18” short, so while it stopped and missed the pedestrian it it did not get the stop bonus.
This was the VERY first car to complete the entire cours at this years AVC!
One other car in a later heat also finished the course so only two cars finished on Saturday.
So I spend the afternnon bumming some aluminum angle from one of the people working oon the manned AVC car hack saw off a chunk, drill and tap add some strategic tye wraps and repair the broken bracket that died earlier this morning on Car #1.
In general the build and wireing quality of car #1 was better than Car #2 (I built #2 first) so I was feeling pretty good about Sunday and went out to dinner with my Wife and my Sister in laws. (One Sister in law flew out with us, and the other sister in law lives near Denver.).
I code up both these changes The course opens for testing at 8am, so I put a report screen together on the LIDAR avoidance and wander around the hallway of the hotel carrying the car and looking at how it sees and reports obstacles…it works reasonable well…. I take it out side and try it agains a wall in the parking lot, it works perfectly… even when command to steer into the wall at a 30 deg angle it will skim along just avoiding the wall. Its perfect…
So I take a good manual course survey and set up to fine tune the route…
It starts to act really unreliable… runs some times, doesn’t run others…
Sunday heat one:
It makes a bad decision on how to go around the very first barrel and gets wedged between the
barrel and hay bale. In past times this would not have been an issue as it would push the
barrel aside a little bit and continue. This run there is so much loose hay on the course that the car just spins its wheels on the loose hay.
So while other heats are running on the course I go out in the big empty parking lot and just run…
The car ran pretty much perfectly, it dodged the barrels well, missed the ramp, caught the hoop and almost stopped at the stop sign. It was supposed to stop within 12” of the line, it stopped about 18” short, so while it stopped and missed the pedestrian it it did not get the stop bonus.
This was the VERY first car to complete the entire cours at this years AVC!
One other car in a later heat also finished the course so only two cars finished on Saturday.
So I spend the afternnon bumming some aluminum angle from one of the people working oon the manned AVC car hack saw off a chunk, drill and tap add some strategic tye wraps and repair the broken bracket that died earlier this morning on Car #1.
In general the build and wireing quality of car #1 was better than Car #2 (I built #2 first) so I was feeling pretty good about Sunday and went out to dinner with my Wife and my Sister in laws. (One Sister in law flew out with us, and the other sister in law lives near Denver.).
Sunday Morning..
I woke up really early Sunday (4am Denver time) with two things in my mind to accomplish, improve the barrel avoidance to give the car some more clearance so it does nto drag the sides, and add general purpose collision avoidance, so that it it some how looses its navigation position it will steer away from obstacles..I code up both these changes The course opens for testing at 8am, so I put a report screen together on the LIDAR avoidance and wander around the hallway of the hotel carrying the car and looking at how it sees and reports obstacles…it works reasonable well…. I take it out side and try it agains a wall in the parking lot, it works perfectly… even when command to steer into the wall at a 30 deg angle it will skim along just avoiding the wall. Its perfect…
Practice Sunday morning…
On the first run the new barrel avoidance S/W is perfect. It doesn’t touch a barrel.So I take a good manual course survey and set up to fine tune the route…
It starts to act really unreliable… runs some times, doesn’t run others…
Sunday heat one:
It makes a bad decision on how to go around the very first barrel and gets wedged between the
barrel and hay bale. In past times this would not have been an issue as it would push the
barrel aside a little bit and continue. This run there is so much loose hay on the course that the car just spins its wheels on the loose hay.
So while other heats are running on the course I go out in the big empty parking lot and just run…
Something is very wrong its gotten really unreliable…
It stops steering in
manual mode even… so I unplug and re-plug the Steering servo and it comes back
to life… for about 30 seconds I do this about three times, and
it comes back to life each time…
Then dies… On the last plug in I plug in the connector off by one pin and put 5V on a 3.3V pin and the CPU power supply dies…
I rush back to my work table reach into my spare bag pull out a new DEV board…
Oh no the boards on the Car’s have header pins soldered in for the connector, the spare had just holes, it won’t work…
So I disassemble the CPU dev board from Car #2 and put it on car #1. I take my last spare CPU board from the useless dev board spare plug it in remount it . I also remove the steering servo from one car to the other… I’ve got both the steering servo and the dev board swapped and I have 15Min of testing before the heats start…
So lets take a break and talk about why I do this… I think its important that a company eat its own dog food. I’m doing all of this development on a new NetBurner branch 3.0 it will be our new multiplatform release, ColdFire, Arm etc… I’m using this new code to find bug and issues and improve the quality of our new release….this now bites me hard.
We changed the way that IP configuration and setup works to be all web based with no special tools. 95% of our testing has been on DHCP based LANS. With a direct cable connection un-configured the system is supposed to use AutoIP to set up configuration. It has not been tested recently, and it no longer works…. I can’t talk to the board to download paths, or set parameters, because it has no IP address….. so I start hacking the system code to force it to have a static IP address…
Then dies… On the last plug in I plug in the connector off by one pin and put 5V on a 3.3V pin and the CPU power supply dies…
I rush back to my work table reach into my spare bag pull out a new DEV board…
Oh no the boards on the Car’s have header pins soldered in for the connector, the spare had just holes, it won’t work…
So I disassemble the CPU dev board from Car #2 and put it on car #1. I take my last spare CPU board from the useless dev board spare plug it in remount it . I also remove the steering servo from one car to the other… I’ve got both the steering servo and the dev board swapped and I have 15Min of testing before the heats start…
So lets take a break and talk about why I do this… I think its important that a company eat its own dog food. I’m doing all of this development on a new NetBurner branch 3.0 it will be our new multiplatform release, ColdFire, Arm etc… I’m using this new code to find bug and issues and improve the quality of our new release….this now bites me hard.
We changed the way that IP configuration and setup works to be all web based with no special tools. 95% of our testing has been on DHCP based LANS. With a direct cable connection un-configured the system is supposed to use AutoIP to set up configuration. It has not been tested recently, and it no longer works…. I can’t talk to the board to download paths, or set parameters, because it has no IP address….. so I start hacking the system code to force it to have a static IP address…
(In hind site I could have used the local IPV6 configured
address and solved my problem, but I did not think about that at the time.)
This takes me about 15 minutes to hack in a solution and get the board to boot
with a static IP… (This bug will be fixed before release) Now they are calling my heat I’m out of time…
I program in the path, the config settings and run to the race track. The car
has not even run 1 inch since I swapped the steer servo and the brains… I’m
hoping the servo center is close enough that
the steer PID loop can correct….
The car goes through the barrels and does not touch anything, it just misses the hoop by about 3 inches, but it detects the pedestrian, and does an absolutely perfect stop at the stoip sign, wiatsfor the path to be clear and continues…
It drives all the way around the course to the finish line and stops 3” too
short.
It had been programmed to go about 3 ft past the finish line, alas with all the hay on the course it got 39” of wheel slip and was 3” short. If it had gone 4” farther it would have been the hi scoring run of the entire event. This was just crushingly demoralizing…
Between heat 2 and heat 3.
I extend the stop zone by 8 feet and do a practice run, its perfect…
I turn the speed up 50% and run it again, its perfect….
I park the car and spend the next hour packing up all my stuff into the SUV and getting ready to go home. I don’t touch the car….
With everything packed in the SUV, the only thing left to do is run the last heat and go home. Having not finished the first two heats I’m no longer in the running to place, I’m half tempted to head out before running the last heat. I stay.
Sunday Heat 3.Sunday Heat 2:
The run is almost perfect…The car goes through the barrels and does not touch anything, it just misses the hoop by about 3 inches, but it detects the pedestrian, and does an absolutely perfect stop at the stoip sign, wiatsfor the path to be clear and continues…
It had been programmed to go about 3 ft past the finish line, alas with all the hay on the course it got 39” of wheel slip and was 3” short. If it had gone 4” farther it would have been the hi scoring run of the entire event. This was just crushingly demoralizing…
Between heat 2 and heat 3.
I extend the stop zone by 8 feet and do a practice run, its perfect…
I turn the speed up 50% and run it again, its perfect….
I park the car and spend the next hour packing up all my stuff into the SUV and getting ready to go home. I don’t touch the car….
With everything packed in the SUV, the only thing left to do is run the last heat and go home. Having not finished the first two heats I’m no longer in the running to place, I’m half tempted to head out before running the last heat. I stay.
I press the go button the car accelerates hard about 30 ft and then just stops.
When I walk up the speed controller if flashing a red fault light, either it got to hot (its cool to the touch) or I ran the car battery down to much… (more likely) my AVC is done.
Final thoughts:
1)The car was too big for how tight the barrels were. The barrels were much tighter this AVC than previous AVC;s. I’d chose the chassis to be able to go insanely fast, I needed rugged more than speed, the lack of ground clearance and wide stance were both issues. The Slick tires just don’t work well in straw. The car would be superior on smooth dry pavement, not so much on the AVC course.
2)When I turned the steer servo gain way up for the barrels I was driving the steer servo too hard and it was overheating and shutting down. A normal RC driver does not continuously drive the servo stop to stop hard, when it barrel following mode I was doing just that and the servo over temped and died.
Adding in the hyper sensitive obstacle avoidance only made this problem worse… as the car might change its steering direction mind significantly at 50Hz.
3)I was not really ready, I should have had my spares more ready, had a spare for ALL 3D printed parts, had the wireing done on Car #2. Have tested the barrel mode enough to show the steering servo flaw etc….
Conclusion:
Projects with a hard unmoving deadline are painful.
For general info:
