Sep 11 2021

Elon Musk on Design, Development, and Manufacturing

Recently, I went down a SpaceX rabbit hole where Elon Musk shared his thoughts on design, development, and manufacturing. Here are some choice insights from Elon as he talks about the SpaceX Starship, a fully re-useable rocket.

  • If a design is taking too long, the design is wrong and therefore the design needs to be modified to accelerate progress.
  • One of the most fundamental errors we’ve made to advance development is to stick to a design even when it is very complicate and to not strive to delete parts and processes.
  • Everyone is chief engineer, everyone must understand how all the systems work so that you don’t have subsystem optimization.
  • The product errors reflect the organizational errors. Whatever departments you have, that will be where your interfaces will be. Instead of questioning the constraints, one department will design to the constraints that the other department has given them without calling into question and saying those constraints are wrong.
  • You should actual take the approach that the constraints you are given are some degree wrong, the counterpoint would be that they’re perfect.
  • One of the biggest traps for smart engineers is optimizing a thing that shouldn’t exist.
  • We often try to answer the questions we are given, instead of questioning the premise of the question itself.
  • We can produce boosters and ships way easier than we can make the launch site.
  • It’s 10 to 100 times more effort to design the manufacturing system than the engine (a first of its kind).
  • The amount that goes into the design rounds into zero, relative to the amount of effort that goes into the manufacturing system.
  • All designs are wrong, it’s just a matter of how wrong. Everyone is wrong, no matter who you are, everyone is wrong some of the time.

In large part from his experience at Telsa, Elon is now using this Five Step Process at SpaceX.

Five Step Process

  • Make your requirements less dumb, your requirements are definitely dumb. It doesn’t matter who gave them to you. It’s particularly dangerous if a smart person gave you the requirements because you might not question them enough.
  • Try very hard to delete a part or process. If you are not occasionally adding things back in, you are not deleting enough.
  • Simplify or optimize. The reason this is the third step is that it is very common for a smart engineer to optimize a thing that should not exist.
  • Accelerate cycle time, you are moving too slowly but don’t go faster than until you’ve work on the other things first. If you are digging your grave don’t dig faster, stop digging your grave.
  • The final step is automate.

About the above process, Elon states the following.

  • I’ve personally made the mistake of going backwards on all five steps, multiple times.

Mar 10 2020

Standard Log Format

There isn’t a standard log format that spans all applications. There is a Common Log Format used by web servers, and there are logging frameworks that helps in the mechanics of producing logs. Unfortunately when it comes to composing the actual message that will be written out to log files the developer is left to his own devices.

Here are some best practices that I’ve found useful when adding logs to an application I’ve working on.

  • Each log event should be written to a single line.
  • Each log event should start with an event timestamp, preferably in UTC.
  • Include the log level in the log line.
  • Add a short description about the event that is being logged.
  • Included all pertinent data about the event in name=”value” pairs.
  • Quote all data values.
  • Use and log a transaction id as requests move through your applications.
  • Be consistent about field names used across applications.
  • Add a prefix to the field name to add context, for example for input or output data.
  • Mask out passwords, credit cards, and any other Personal Customer Information (PCI) data.
  • The log file name should include the application identifier and hostname.
  • Log files should be rotated out based on time and/or size.

Feb 4 2020

Software Development Checklist

During the summer, our team usually hires an intern. We find that the interns we hire are more than capable with the technical requirements but usually need some help on process. In order to help new team members adapt our process we’ve developed a number of useful checklists of some best practices. Here are some useful checklist to go through when working on new features.

Ticket Checklist

  • Add clear and concise title and description
  • Add reproducible steps
    • Use realistic sample data
  • Add specific acceptance criteria
  • Add related tickets

Code Checklist

  • Create your own feature branch
  • Check code into your branch often
  • Merge from master often
  • Run test suites on your own branch
  • Add and update test cases
  • And and update wiki documentation
  • Add comments to ticket

New Project Checklist

  • Source
    • Git Repository
    • Dependency Management
      • Maven
    • readme.md
  • Wiki documentation
    • Design Diagram
    • API Documentation
    • Common Configuration
    • AWS Permissions
    • Technical Specifications
      • Ports
      • Hardware
    • Sample log output
    • Sample Splunk queries
  • Splunk friendly logs
    • Splunk Dashboards
    • Splunk Alerts
  • Test Cases
    • JUnit
  • Release Management
    • Bamboo Plan
    • Ansible Playbook
    • Package
      • Docker Image
      • RPM Package

Oct 18 2019

Favorite How I Built This Episodes

I’ve been listening to podcasts before every comedian had one. Back in 2007 I listed my favorite technology podcasts. One of my favorite podcast now is How I Built This with Guy Raz. Guy talks with established founders and entrepreneurs on how they got their business off the ground. The following are my favorite episodes of How I Built This because of the perseverance, determination, and humility described by the guest to launch their successful business.


Jun 7 2012

The Cost of Doing it Wrong

As the saying goes, if a thing is worth doing, it’s worth doing well. It has been proven that doing a thing right the first time is most efficient and cost effective. But what is the cost of doing something wrong? In terms of money, I’ve seen cases where something is done incorrectly and then had to redone at a loss of $100k. In terms of time, you can model it in the following thought experiment.

If a task takes x amount of time to complete and it is done wrong the first time, how long does it take to fix it?  More than 4x. It takes to x amount of time to complete the task the first time, probably at least x amount of time to find out it was done wrong, another x amount to figure out what the first guy did, and finally x amount to fix it correctly.

We all can agree that time is money and if a task is done incorrectly it may cost at a minimum as much as four times as much as it would cost if done correctly the first time around.


May 28 2012

The 80 Percent

I always give 110% to any task I set out to complete. If you ever give anything less than 100% then you get the following logic…

It’s okay if we do 80%, but the 80% needs to be done 100%… We can’t deliver 80% of the 80% we actually attempt of the 100% we are committed too.