Monday, November 02, 2009

After Action Report.

We destroyed silver and badly damaged blue. The root cause was not enough time and budget to do the necessary testing. Primary lesson only work on one vehicle at a time.

In Early September we flew multiple long flights, as long as 106 seconds. We then parked the vehicle. On Thursday I merged the navigation code I'd flown on the helicopter with the hover code on blue. I picked the wrong code branch and it did not have the wiggle correction in it, so it wasted too much thrust wiggling and did not have sufficent thrust to lift off. After sitting with what must have been an in inadequate purge the cat pack was badly corroded. On Saturday I changed the cat pack and added the gain scheduling to correct the wiggles. I was worried based on Friday that it might not get off the ground so I loaded what I calculated to be the minimum necessary propellant+ 2% , it flew for 85 Seconds then crashed on the 2nd pad. See: You Tube from on board and Video looking out. It looks like my code slicing and dicing also missed the roll or yaw control fix I put in the blue ball as yaw seemed largely uncontrolled. Based on the takeoff acceleration we could have put a lot more propellant into the vehicle and done the full 90 seconds.

I was really happy with the way silver turned out. It was very clean , light and capable. I just could not get io hover stably. It looked like a control loop integral wind up issue. From a sportsman ship stand point both Ben and Ian of Masten asked suggested that was the problem. I had turned off the Integral Gain on position hold as one of my first steps, alas the strange mix of code from three sources had a cut and paste error so there was still an integral term inplay even after I thought I'd turned it off. I'm 90 % sure that with 2 extra hours to slow down and review I'd have fixed it. We came very close: Silver ball hover.

The eventual fault that killed it was a bit subtle. On the blue ball when I hit the command abort it shuts the motor off cleanly it just closes the main valve. On the silver we purge on shutdown, and we purge hard, this causes a shut down thrust transient as residual propellants are forced through the motor. Couple that with the choice to center the motor and kill active attitude control when aborting and one gets wild gyrations on abort. Yes can see Ben's video of its final flight.

On a personal level I'm really glad I got the one 94% successful flight out of the blue ball. It at least shows we were close. It also shows what a small team can accomplish. On the flip side we walk away from the contest with nothing tangible to show for the effort. This project has been a really big part of my life for the last year and it leaves a big hole. At the end it became an unhealthy all consuming obsession as the clock wound down and we ran out of time and budget. From a financial stand point it also means no playing with rockets for at least 6 months to a year as I try to rebuild the rocket fund.

Only time will tell if I eventually see the overall project as a positive thing. It does not seem that way this morning.


Anonymous said...

I know what it's like to work on a project for years, and see it come apart at the last minute, but those projects have almost always led me to something much better on the next try. You guys have done an incredible job for such a small team, with limited time, manpower, and resources! You've got a lot to show for you work, and should be proud of what you've accomplished! How many people can say they've built as many hovering rockets in their spare time?

Anonymous said...


I am reading your blog since a long time and I learned a lot. I am truly grateful for that.

I hope you will continue to build VTVL rockets after some well deserved rest! The size of your blue and silver ball impressed me. However, why not building it a little smaller now since it does not matter anymore how long they fly?

All the best.

Paul Breed said...

The size of blue and silver were set by the 25Kg payload requirement for the LLC. Without the 25Kg requirement they would be a lot smaller.

Anonymous said...

I am really impressed with what you have accomplished.


Jesse said...

I know it's probably hard to take a 'moral victory' away from this, but try and step back and see what you and your son accomplished!

No one else has come this close with such a small team. Certainly prospective customers can see that with time and money, Unreasonable Rocket could do some really amazing things.

Anyway - congrats on the successes, you've done incredible work.


William said...

Paul A and Paul T -- you have both been an incredible personal inspiration to me and to the rest of my colleagues at the X PRIZE Foundation. I know you must wish the weekend went a bit better than it did, but I hope it's some small consolation to know that your efforts are absolutely pushing out the bleeding edge of what is capable, and inspiring people literally around the planet along the way. You all rock!

C. Scott Ananian said...

Hear, hear to the congratulations! I've been following Unreasonable on line for years, like others, and you put on a great show for us at the end. There were ~600 people watching the webcast at any given time on Friday, Saturday, and Sunday -- it probably wasn't apparent at FAR, but you had a huge virtual crowd there cheering you on.

The days after a project is over are always a downer, but I am certain that in hindsight you'll value the opportunity. There are people like me who've been vicariously living through your blog *wishing* we could be the ones putting in those last night and long hours on a project you (and we!) loved. Thank you for taking the time to blog and share it all with us.

ian said...


I am so sorry about the accident with the Silver Ball and that I could not help at all in the rush. I have profound admiration for you and your son, and only hope that you recover and get back at it.

Ian (MSS)

Will Baird said...


It *IS* a net positive thing. However, it is imperative you do *NOT* stop where you are. You have a very positive and important angle to contribute to RLV development.

I *COMPLETELY* (with a blink tag, no less) understand your frustration. Better than most. We're soldiering on, as should you.

Anonymous said...

from this spectator's perspective you achieved the same that armadillo and masten achieved, except you rushed untested software and it crashed the hardware. big deal. every big company does that.

sometimes it is a mistake and sometimes it is a calculated risk that pays off.

life is a game of chance.. like poker and rocket launches. you didn't need to rush at 110% and risk both rockets. so maybe the most important thing to learn is about how much risk you should be taking in a given situation.

if you wanted to invest your only two big lump sums in a chinese gold mine stock I'd say no, don't risk so much, because if it goes bust you'll lose it all and won't know where to go from there.

but if it happened I'd just say go back and work harder and smarter than ever before.

Peter Lund said...

I hope you will do a write up of the whole project (or let others write it).

Think of "The Soul of a New Machine" -- there could be another Pullitzer in there for someone :)

(Or a phd or two on rocketry or project management!)

Thank you for a most wonderful show so far.

PS: Take a look at Mercurial. It might help with the source code problems. It works under all of Linux/OS X/Windows, is fast and efficient, has a great manual AND is easy to use.

Joe Stanton said...

I consider myself extremely honored to have been able to help in some small way over the past 10 days. You have excellent crisis management skills and critical path decision making ability. I learned a lot from watching you in action. There is a lot more that Unreasonable Rocket Men can do for the suborbital and near earth aerospace industry.

"There is no try, there is only do or do not." - Yoda.

You DID and you will DO again.

heroineworshipper said...

The legacy is the customer rockets, the rocket fuel factory, & the construction for FAR. Driving 8 hours every week to do testing may seem like a killer compared to Masten who can just walk out & fly, but we spend 10 hours/week driving to our day job. Realistically, Carmack should be the only winner because he invented the standard rocket design from scratch.

John Carmack said...

We have had a couple opportunities present themselves at Armadillo that could have resulted in a major expansion in our efforts (don't anyone get excited -- nothing looks imminent at this point). I have firm beliefs about what it is going to take to really change things in space access, and "Put more men on the job!" isn't high on my list, but at some point you do need more hands on deck. My first thought has always been "I want the Breeds." Not that I wouldn't shamelessly poach Masten and XCOR, but you have been at the top of the list. I hope that I get to make the pitch to you someday.

It may be to early to look at the last year dispassionately, but the lessons are worth revisiting: As you say, the primary issue was an over commit on resources with two different vehicles. I have worried about that with you all year, and I tried to convince you of it. If you had made a second identical blue ball vehicle and rotated them through flight tests, the tragically heroic efforts with the new engine and control system wouldn't have been taken.

A second order effect was the low margin on the vehicle. Building at a somewhat larger scale and higher engine thrust would have given enough margin to absorb some of the hardships. The price of peroxide influenced the scale, but with a proper accounting of everything, it should have been a smaller factor.

Tying in with both of the above was the relatively limited testing before the event. It takes a lot more flight operations than it seems like it should to have anything like confidence that you will be able to make a pair of flights on the clock.

After we tragically wrecked the initial Super Mod with the cartwheel and tether break, I'm surprised everyone didn't take that to heart and double the strength of all things tether related. It is a shame the Silver Ball had to hit the ground.

I still can't make myself satisfied with the way things turned out. You should be repairing the silver ball to compete for level 2 second place in 2010.

Ben Brockert said...

Paul, I know you guys are down now, but I can't see your work as anything as a positive thing.

Considering all the friendships you've made at FAR, business relationships you've made with suppliers, and lots of publicity for you guys and your company, you'll come out of this on solid ground.

Two years ago, would you have predicted that you'd be regularly firing a 10-30 k lbf rocket engine?

You built three really impressive vehicles in a very short time, and got experience with every aspect of rocketry. When you get your feet back under you in six months, I look forward to seeing the altitude shots with the composite rocket.

David said...

Paul - I can understand your deep disappointment and I hope that hole that your feeling does in the future gets filled with further rocket development.

Watching the blue ball bounce along through the air was truly amazing. Each wobble, each tilt causing me to bend, move and gasp while hoping it stays aloft. The landing although hard made me jump out of my seat and yell "YES!!!!". I have followed for years and I had not realised until then how much emotional commitment/connection I had with your efforts and project. I get up every day and check your blog and wish I could be there and help - inspirational. As little as it will help at the moment, I (and likely many others) understand where you are at the moment. I hope you can continue development.

I would like NASA to see how positive this prise has been and provide more options - maybe even start a second round for Level 2 to encourage the remaining teams to compete. I truly hope you find the motivation to keep working and moving forward.

Your an inspiration to so many. That legacy may be the most important one.

Anonymous said...

Congrats to you guys for fighting it out. I was impressed by blue's flight and your personal tech abilities. As a father/son team you guys made a real show! Keep it up and show those big corporations how real work is done.

Joe Stanton said...

Would John Carmack care to expand on that statement about competing for 2nd prize in Level 2 in 2010? Is Armadillo withdrawing their qualifying flights?

George Katz said...

What an amazing adventure! You guys are providing inspiration for so many people. It is absolutely amazing what you guys have achieved so far. Thanks for sharing all the ups and downs along the way over the years.

It was great watching Blue fly live on the net, the whole family enjoyed the show. My six year old was jumping up and down and cheering when Blue took off. :)

Well done guys!

Paul Breed said...

I think John means Masten should not have won this year as they did it on the 5th down ;-)

Joe Stanton said...

I agree with that sentiment entirely also.

But John could really make this interesting by doing so. A) He doesn't really *need* the money or glory, B) it would be a prime example of forcing the world to adapt to him, and therefore John would become Unreasonable.

C. Scott Ananian said...

Might be worth using the "recharge the rocket fund" time to clean up your source code and change management mechanisms. IIRC in a previous post you mentioned that this was one of those things that kept getting put off because of the time pressure to get hardware flying. A previous poster mentioned Mercurial, that might work. I personally use git these days, which I find is really powerful *especiallhy* if you've got two different branches you're working on and want to keep tabs on how they're different. I'm sure there's all sorts of other code janitor work you've been saving up as well.

Doesn't sound as much fun as building rockets. ;-)

Anyway, congratulations again. Among your other successes, keep in mind that you really wowed everyone watching with your quick cycle time and small crew size. I think that was a significant engineering accomplishment.

noel.wade said...

As I said before - I hope you take some down-time, enjoy the family and other pursuits for a couple of weeks. Then gather your wits and see what technology you can license or what components you can sell. It may not be the same scale of money as winning an NGLLC level, but its definitely something positive to show!

And who knows - if MSS or Armadillo or some other new company is successful and growing, you've certainly put yourself on the map as someone who could be a valuable addition to their team (as JC has noted)!

I know that in the bitterness of the aftermath of this weekend that a lot of these comments are going to have little impact on your emotional state; but I hope that in the future you can look back at these comments and feel some pride and a sense of accomplishment - even if you didn't take the prize.

amphlett said...

Following your progress has been an inspiration to myself and my two sons. I hope you'll appreciate at some point the positive impact you've made in so many other peoples lives with what you've both done so Unreasonably :)

Also, as someone who's been following the blog from early on, it's very clear to me how much you've learned and progressed over the duration.

I think you knew a few weeks before this that you were stepping out into the bounds of the unlikely. The thing I've enjoyed more than anything about your work is your disciplined and realistic approach. No-one can blame you for succumbing to that tantalising bright shiny hope and trying for the goal despite what you know. The experience has to have been worth it surely.

Congratulations for a bloody fantastic attempt, and for sharing so much of the experience.

Anonymous said...


This week is why Single Malt scotch
was invented. Spend a week enjoying a nice MacAllen, and then spend a week working out a strategy.

Anonymous said...

Y'all are an inspiration.

Simon Arthur said...

You've clearly built a positive reputation for yourselves that's 100 km high. Congratulations.

You've got lots of options for your next project, as you have the knowledge, tools, and a giant pile of hardware. Make something even more impressive.

rjbrassell said...

Hold your heads high!

I too have been a follower of your blog since..... well.. forever. I felt like I was right along for the ride.

All the best,
Bob Brassell

Anonymous said...

You gentlemen are awesome! You have achieved remarkable results on a schedule-limited and resource-limited project, and a competition you entered years later than two teams that, in the end, won the event on the day of the contest.

Keep up the good work of being Unreasonable. Someone's got to do it.

Anonymous said...

Sometimes after a loss, it is best to just sit with the person and say nothing. Your presence is enough to show your support for the person while they grieve. Since we cannot sit with you, we support you with our words and thoughts. If I may just mention what I take from this after following the blog for so long - you owning up to your mistakes in a humorous way, that time you spent with your dad in the shop answering the same questions over and over, you mentioning the blessing of working side-by-side with your son for the last three years, the determination that young Paul has had especially during these final days which I am sure came through a great mentor, and you taking the time to thank the "extended" team with your wife at the top of the list. It may not have gone off as planned, but you sure seemed to be doing the right thing through it all! Thanks for the ride, and you can bet we will all be waiting to jump back on if you decide to continue:) Jerry in Tucson
P.S. This may seem a strange message to put in a public blog, but when you follow it for so long and the guy on the other end has opened his hopes, dreams, and trials to you, you cannot help but respond back in a personal way.

Michael said...

Paul, Paul, and volunteers to Unreasonable Rocket:

Y'all came within a whisker of pulling it off. Carry yourselves with pride. Y'all operated on the shoe-stringiest of budgets, and still completed a flight profile. How many other groups have EVER done that?

Rest up, lick your wounds, and get back after it as soon as you can. The industry needs three capable players, or it isn't really an industry.

Best regards,

Mike G

Laurence said...


To save cluttering up your comments page with a long post, I've put my comments on a blog posting here:

I've pondered for a while as to whether to post it or not - after all it's mostly full of my own woes. In the end I decided it was important that you realise how many people your vision and ethos has reached and inspired during your journey.

In short - please hold your head up high - it's nothing short of staggering what you and your team have achieved.

Laurence, Dorset, England