Paul’s AVC postmortem….

In the end it did not go as well as I would have liked.  On 6 official attempts I only completed one lap.

I thought I was pretty close to ready; The car was following complex arced paths well…

This picture is from testing Wednesday The black and pink arcs are on top of one another.

The big change that enabled this precision was to add 200 msec of position prediction to the turn algorithm. IE star turning 200 msec early (200mse seems to be the command to actual steer servo is in new position latency)

As you can see the difference between the black actual path and "pinl desired path is insignificant.

In addition, with nice straight building walls the laser scanner was correctly finding the walls and correcting both the horizontal offset and accumulated heading errors against the walls.

When I left for Denver on Thursday I had three software bits unfinished….
I’d done testing and analysis on all three, but they were incomplete.
1)Obstacle avoidance.

2)Stop at the stop sign/pedestrian.

3)Hoop Location.

So Early Friday morning I wrote my  barrel  avoidance code and got to start testing on the actual course Thursday…. By 3PM I had my barrel code working the car was going through the barrel section mostly avoiding the barrels 5/6 times or so….The car was a bit too large and the barrels were a bit tight so I really needed to go slow and turn sharply. To make this happen I turned the steering gain way up for  any time the car was in barrel avoidance mode.

Here is a picture and video of the zigzag path through the barrels..


I then took some data to figure out how to detect the cross walk the pedestrian walked on and thus stop at the stop sign. In testing this chunk of code it worked one time in four or five and I worked on this code until 7:30 pm where I discovered my error…. I was using a static variable to indicate that I’d already done  the stop and to not do it again…So the code worked after a clean reset, but not a test restart…

Once I fixed that I was reliably stopping at the pedestrian..

I put every thing together and did a full run where everything worked.  Then the course closed at 8pm and I was done for the day. I had one issue remaining. The start line was too high it high centered the car, so I had to start behind the start line and get a running start and jump over the start line.
This basically added 50 to 70” to the path as the wheels spun.  I did a hack and added in 70” to the path in code.  I’ll test this in the 2 hours of testing Saturday morning…

Saturday Race Day #1, 

Its cold in Denver and I’m there when the course opens for practice and rather than use my 70” hack I manually record a new track path starting 24” behind the start and use that recording  to layout a new path.

I do a test run and the car crashes into the hay bales straight ahead…. And breacks the 3D printed bracket that held the LIDAR and electronics…  I brought spres of every 3D printed object on the car (6 parts) except the one bracket, because that was not on car #2, that was metal on car #1.
So with one hour and 15 till race start  I’ve got to get the 2^nd car ready, it was 100% complete except I had not wired the start switch  so I put both cars in the back of the retal SUV and drive back to the hotel… (about ¼ mi) When I get there I discover the back gate of the SUV had not closed and neither car was in the car.  I went back and found both cars lying in the road, otherwise undamaged.

I brought them back to the room  and wired in the start switch on car #2 (I’d planned to do that Saturday afternoon after day one was done. Rules were I ran one car on Sat, and a different one on Sunday.

I now have 15 min till practice is over and 30 min till my heat starts and  car #2 had not yet been turned on in Co. I run  the test and it impacts into the same hay bale that broke the previous  car. And I remember that my extra 70” hack was still in place, I remove it and get in one last run it gets tied up in the barrels, I’m not sure why this is not working….

Saturday Heat 1:
I’d planned to spend the morning fine tuning the path to center it up on the course and perfect it. Alas I spent the morning soldering switches and removing my 70” hack.  

Run one it gets a tiny bit hung up in barrels, but otherwise does the first half of the course, its supposed to go just inside the ramp, alas the lack of track tuning it hits the ramp/jump and goes off the edge of the ram, causing it to turn a bit and impact the hay bales on the other side. End of heat 1.

Saturday Testing between heat 1 and heat 2.

The car is now acting unreliable. IT runs sometimes, and other times it just wanders off course in random ways.  A couple of times when I try to gain manual control with the RC receiver it won’t steer….
In trouble shooting this power cycling always seems to fix this… I’m wondering if I have a hardware problem.  Out of 3 or so test runs one works, one is random and one gets hung up in the barrels…

The main CPU board on the car is a NANO54415, on this car was an old dusty first rev prototype board that I had used on many projects  So I figure there is no harm in swapping the board.
So I swap CPU boards for the spare in my electronics bag. I reprogram it with the car code, program in the course and  I’m out of testing time. Time for heat 2….

Saturday Heat 2:

I press the go button it immediately turns right into the wall and dies.

The system saves a bunch of configurable parameters in flash, one of these is use the magnetic compass, or just the bare gyro for navigation.  I’d found the bare gyro was more repeatable than the compass, but use compass defaulted to on…. So car turned right into the wall due to the compass…

Practice between heat 2 and 3 on Saturday…
This went reasonably well, I tuned the path a little bit.

Saturday Heat 3:
The car ran pretty much perfectly, it dodged the barrels well, missed the ramp, caught the hoop and almost stopped at the stop sign.  It was supposed to stop within 12” of the line, it stopped about 18” short, so while it stopped and missed the pedestrian it  it did not get the stop bonus.

This was the VERY first car to complete the entire cours at this years AVC!
One other car in a later heat also finished the course so only two cars finished on Saturday.

So I spend the afternnon bumming some aluminum angle from one of the people working oon the manned AVC car hack saw off a chunk, drill and tap add some strategic tye wraps and repair the broken bracket that died earlier this morning on Car #1.
In general the build and wireing quality of car #1 was better than Car #2 (I built #2 first) so I was feeling pretty good about Sunday and went out to dinner with my Wife and my Sister in laws. (One Sister in law flew out with us, and the other sister in law lives near Denver.).

Sunday Morning..

I woke up really early  Sunday (4am Denver time) with two things in my mind to accomplish, improve the barrel avoidance to give the car some more clearance so it does nto drag the sides, and add general purpose collision avoidance, so that it it some how looses its navigation position it will steer away from obstacles..

I code up both these changes   The course opens for testing at 8am, so I put a report screen together on the LIDAR avoidance and wander around the hallway of the hotel carrying the car and looking at how it sees and reports obstacles…it works reasonable well…. I take it out side and try it agains a wall in the parking lot, it works perfectly… even when command to steer into the wall at a 30 deg angle it will skim along just avoiding the wall. Its perfect…

Practice  Sunday morning…

On the first  run the  new barrel avoidance S/W is perfect. It doesn’t touch a barrel.
So I take a good manual course survey and set up to fine tune the route…
It starts to act really unreliable… runs some times, doesn’t run others…

Sunday heat one:
It makes a bad decision on how to go around the very first barrel and gets wedged between the
barrel and hay bale. In past times this would not have been an issue as it would push the
barrel aside a little bit and continue.   This run there is so much loose hay on the course that the car just spins its wheels on the loose hay.

So while other heats are running on the course I go out in the big empty parking lot and just run…

Something is very wrong its gotten really unreliable…

It stops  steering in manual mode even… so I unplug and re-plug the Steering servo and it comes back to life… for  about  30 seconds I do this about three times, and it comes back to life each time…
Then dies… On the last plug in I plug in the connector off by one pin and put 5V on a 3.3V pin and the  CPU  power supply dies…

I rush back to my work table reach into my spare bag pull out a new DEV board…
Oh no the boards on the Car’s have header pins soldered in for the connector, the spare had just holes, it won’t work…
So I disassemble the CPU dev board from Car #2 and put it on car #1.  I take my last spare CPU board from the useless dev board spare plug it in remount it . I also remove the steering servo from one car to the other… I’ve got both the steering servo and the dev board swapped and I have 15Min of testing before the heats start…

So lets take a break and talk about why I do this… I think its important that a company eat its own dog food. I’m doing all of this development on a new NetBurner branch  3.0 it will be our new multiplatform release, ColdFire, Arm etc… I’m using this new code to find bug and issues and improve the quality of our new release….this now bites me hard.

We changed the way that IP configuration and setup works to be all web based with no special tools. 95% of our testing has been on DHCP based LANS.  With a direct cable connection un-configured the system is supposed to use AutoIP to set up configuration. It has not been tested recently, and it no longer works…. I can’t talk to the board to download paths, or set parameters, because it has no IP address….. so I start hacking the system code to  force it to have a static IP address…

(In hind site I could have used the local IPV6 configured address and solved my problem, but I did not think about that at the time.) This takes me about 15 minutes to hack in a solution and get the board to boot with a static IP… (This bug will be fixed before release)  Now they are calling my heat I’m out of time… I program in the path, the config settings and run to the race track. The car has not even run 1 inch since I swapped the steer servo and the brains… I’m hoping the servo center is close enough that  the steer PID loop can correct….

Sunday Heat 2:

The run is almost perfect…
The car goes through the barrels and does not touch anything, it just misses the hoop by about 3 inches, but it detects the pedestrian, and does an absolutely perfect stop at the stoip sign, wiatsfor the path to be clear and continues…

It drives all the way around the course to the finish line and stops 3” too short.


It had been programmed to go about 3 ft past the finish line, alas with all the hay on the course it got 39” of wheel slip and was 3” short. If it had gone 4” farther it would have been the hi scoring run of the entire event.  This was just crushingly demoralizing…

Between heat 2 and heat 3. 
I extend the stop zone by 8 feet and do a practice run, its perfect…
I turn the speed up 50% and run it again, its perfect….
I park the car and spend the next hour packing up all my stuff into the SUV and getting ready to go home. I don’t touch the car….

With everything packed in the SUV, the only thing left to do is run the last heat and go home. Having not finished the first two heats I’m no longer in the running to place, I’m half tempted to head out before running the last heat. I stay.

Sunday Heat 3.
I press the go button the car accelerates hard about 30 ft and then just stops.
When I walk up the speed controller if flashing a red fault light, either it got to hot (its cool to the touch) or I ran the car battery down to much… (more likely) my AVC is done.

Final thoughts:
1)The car was too big for how tight the barrels were. The barrels were much tighter this AVC than previous AVC;s. I’d chose the chassis to be able to go insanely fast, I needed rugged more than speed, the lack of ground clearance and wide stance were both issues.  The Slick tires just don’t work well in straw. The car would be superior on smooth dry pavement, not so much on the AVC course.

2)When I turned the steer servo gain way up for the barrels I was driving the steer servo too hard and it was overheating and shutting down. A normal RC driver does not continuously drive the servo stop to stop hard, when it barrel following mode I was doing just that and the servo over temped and died.
Adding in the hyper sensitive obstacle avoidance only made this problem worse… as the car might change its steering direction mind significantly at 50Hz.

3)I was not really ready, I should  have had my spares more ready, had a spare for ALL 3D printed parts, had the wireing done on Car #2. Have tested the barrel mode enough to show the steering servo flaw etc….

Conclusion: 

Projects with a hard unmoving deadline are painful.

For general info:

How to evaluate a Launcher Startup.

This post started as a 7 tweet tweet book . Several people asked me to put it together in a way that can be permanently linked. This is the result.

I'm assuming there is a valid business case. The business concept with  cost estimates and identified customers etc... needs to work before any of this matters.  I assume that any business evaluation will include the business fundamentals. I personally believe that a low cost dedicated nanosat launcher makes business sense and I have personally worked on  such a business plan. I see a lot of launcher startups being funded so I must assume that others agree the business case might work.

What I see looking out at the world is a complete failure in technical evaluation of the various launcher startups.  I'm trying to provide a rough guide for technical evaluation of a launcher startup.

To state the obvious, the first goal of any launcher  company will be to get payloads into orbit,

that is the hard part of the business. Added services, responsiveness, cool features etc... are all irrelevant until you have a vehicle getting to orbit.

So what does it take to get to orbit and how does that differ from getting to space?

The official line of space is 100Km or 328K ft. In the scale of the earth a tiny distance.  Suborbital vehicles, like Space Ship 1, and Blue Origin's New Shepard are suborbital. They go to space, but they do not go into orbit. This may be useful as a tourist vehicle, and for limited scientific experiments, but its not really relevant if you are trying to build an orbital vehicle.

The potential energy necessary to get to space (straight up) is (approximately)

Pe=m*g*h

where m is mass, g is acceleration due to gravity and h is height. So for one kg  

1*9.81*100,000m = 981,000 nm of energy.

The minimum speed to be in orbit is around 7500 m/sec

Ke=0.5*m*v^2   

so for the same 1 Kg in orbit we add an additional

0.5*7500*7500=28,125,000N-m of energy.

So the total is ~ 29,106,000 NM or 29 times the energy of a suborbital vehicle.

So if a space company says we have gotten to space and we will be in orbit real soon now; realize it's like an airline company offering you a ride from San Francisco to New York City, yet the farthest their plane has ever flown is from SFO to Sacramento.

So what does it take to actually get to orbit:

1. Mass Fraction (Hard)
2. Motor performance (Hard)
3. Guidance (Medium Hard and getting easier)
4. Regulatory approval (Medium Hard)

Mass Fraction.

Mass fraction is the ratio of propellant mass to empty weight. For example a 2L coke bottle has a mass fraction of about 40.  I.e., the bottle full of coke weights 40 times what it weighs empty.

This is a good mental model for a rocket, the mass ration of the upgraded Falcon 9 is estimated to be 25:1.

Any orbital vehicle using chemical propulsion will look like a big tank.

So when evaluating a potential launcher company you need to ask what is the Mass Fraction of hardware you have on hand and can show me.  When evaluating this don't accept paper numbers, insist on knowing what the numbers are for the hardware on hand.

Motor Performance.

Rocket motors (assuming they work at all) have one major and two minor performance parameters.

• ISP. It's usually stated in seconds. IE a motor with an ISP of 300 will make 300lb sec of thrust for one lb of propellant.Orbital vehicles have had rocket motors varying in ISP from 220 (shuttle big solids) to 450 (high stress staged combustion Lox/H2 SSME) Lox RP1 motors are in the 300 to 330 range.
• Thrust to Weight: What is the ratio of motor weight to motor thrust. If the rocket motor is accounted for in the Mass Fraction of the overall vehicle, this is sort of irrelevant. Its also a bit hard to score as the value varies depending on where you draw the line between the rocket motor and the rest of the vehicle. The Main propellant valves for instance, do they belong to the tank or the Motor?
• Burn Duration: The rocket motor need to last long enough to get you to orbit. This is not really relevant to most liquids, but for small solids, it's really hard to increase this number due to the physics of solid propellant burning and insulation requirements.

I've marked this part of the four fundamentals as Hard.  I'd actually score the motor as Medium hard, but getting all the instrumentation and test together to optimize and really know what your motor is really doing pushes this into the hard category.  

Guidance

The rocket needs to go where you want it to go. You'll also have to convince the regulatory agencies that you really do have control of where it's going so it won't be a hazard. As computer simulations get better and better getting this part right gets easier. It's really easy to have the simulation right, and have errors in the vehicle where the actuator direction is backwards or the simulation inertia scale is wrong lbs/kg anyone?

So, does the rocket company have a plan to verify and reality check the hardware with the simulation without destroying the rocket?  I test flew my hovering rockets under a forklift so I could discover guidance errors without destroying the vehicle. The difference between this video and this video is the sign of a single term in the guidance equations.

Regulatory

If you're in the U.S. you are going to have to get FAA permission to launch your vehicle.

Where you launch from and how you mitigate risk to the general public are a big part of this process.

It's not really that hard, it just must be planned for in the development of the vehicle. It's not something you can just "Stick OK" a completed vehicle if you have given zero thought to things like ranges, range safety and EC (expected casualty)  The FAA is going to care about what the rocket parts can hit if things do not go to plan. That's why I've always given very little credence to inland spaceports like the one in NM or Odessa as a place for orbital launches. If you launch from NM you're going to stage somewhere over the densely populated parts of Tx. Short of a planet wide emergency,  like in Seveneves, it's never going to be allowed.

Putting it all together with the rocket equation

The rocket equation   

Delta V= ln(MI/MF)*ISP*g

MI initial stage mass, ie the weight of everything....including upper stages and propellant

MF final stage mass ie the weight of everything minus all the burned propellant.

ISP the Motor ISP number. (Not really a fixed number improves with altitude)

g 9.81 m/sec acceleration due to gravity.....

Calculate this Delta V for each stage in the vehicle.

The Mass of the fully fueled stages above the one you are calculating Dv for must be included in both the initial and final weights.

Once you have calculated ALL of the stage Dv and added them up, if the number is > 9000m/sec then you have a chance at getting to orbit. This number will vary somewhat with the vehicle acceleration profile and size. I personally like this paper "How Small can a launch vehicle be" for evaluating concepts as a first order check.

If the number is not yet to 9000m/sec and the plan to fix mass fraction and ISP to get to 9000m/sec is not the number one issue for EVERY technical person working on the vehicle, I would be doubtful they will succeed.

The Team

Building a new Rocket is an exercise in creativity. There is not a formula that given inputs generates a rocket. Unless the engineering team shows the ability to design, build, evaluate, and repeat in a creative way, nothing else is going to matter. Building hardware is different than building software, rockets are hardware. When you talk to the CTO/Chief engineer/Chief Wizard without notes can he tell you what his mass fractions, ISP and current achieved weight vs planned weight numbers are?

If he can't then who in his organization can? If no one can, then they are not going to succeed.

Walk through the shop and engineering, do you see a scale?  Does every engineer building a piece part know what his weight budget is and if he's going to make the weight budget?

Before XCOR failed, I personally thought about investing in their company several times, every time I got a tour of their facility, I was turned off by the complete lack of weight control. The half finished linx had many parts that look like they belonged on a truck, not a space craft. The complete lack of detailed weight control on all the minor parts spelled failure to me and I never invested for just this reason.

Eventually the team will need to learn operations, and as it grows it will need HR, middle management and all the other parts of a company. But unless it gets the creative engineering part right up front it's going to run out of $$ long before any of these things matter.

To quote from my tweet storm:

• Pretty Paint does not matter.
• Fancy Building does not matter.
• Previous Dinospace experience does not matter.
• Cool animations of a paper rocket do not matter.
• Until they have hardware that can do 9000m/sec DV, operational issues don't matter.

Why doesn't previous dinospace experience matter? Because other than SpaceX, all of the other launch providers had their creative how do we do the base design done in the 60's and 70's. The creative steely eyed missile men that made that fabulous leap to the moon and beyond are no longer part of the current dinospace environment. It's a very different thing to manage and evolve an existing system than it is to create a new one from scratch.

Please feel free to comment on errors or omissions or areas that need clarification in this post. I will try to correct, enhance and improve over the coming few weeks.