Ardupilot Postmortem
A big thanks to those who helped from the Ardupilot Forum
Last week, I took a few extra days off from work to give myself a nice long break over Thanksgiving week.
I spent a lot of time finishing an RC plane we had some replacement parts for. This one was going to be autonomous. I spent a few days soldering the flight controller and electronics, gluing the plane together, measuring and trimming everything perfectly. It was perfectly balanced, perfectly weighted. We finally took it out for its maiden flight today, Sunday November 27. Sadly, this story ends in pieces.
This post is a postmortem of the incident to make sense of what happened. I was flying using Ardupilot and had an SD card in recording the data. Although I can't be certain, I'm pretty sure we can use this data to root cause the issues.
Plane Hardware
- Plane: Bixler 3
- Flight Controller: Holybro Kakute F7
- GPS, Compass: Beitian BN-880
- Ardupilot: Arduplane V4.3.1
Tools & Files
- UAV Logviewer
- Ardupilot LOG FILE
Theories & Investigation
The flight controller has a BMP280 barometer to measure the altitude quickly, and a ICM20689 IMU for acceleration and gyro. There is also a slower-updating GPS, so between these instruments, I expect fairly accurate altitude and position readings. The plane behaved erratically twice in the short two-minute flight. After combing through the available data, I suspect two failures during this flight. What follows is a timeline of the events of the flight and notes on each incident.
Events in Flight
1. Bad Altitude Readings Pre-Launch
The first noteworthy item is that the altitude readings from the barometer and accelerometer are bogus. I seem to be getting more realistic readings from the GPS. Going forward, any screenshots will be from the GPS altitude.
The altitude readings are super interesting here. You can tell even before I launch the plane, the GPS altitude readings are between -13m and 0m. Not sure what this altitude is (above sea level? Above ground level?? Either way, this flight took place very nearly at sea level.)
More concerning, the barometer altitude is between -10 and -70m in the ~1 minute before launch on the ground. The calculated AHRS (Attitude Heading Reference System) altitude at somewhat similar, -9 and -80 🫤
ROOT CAUSE
As mentioned, the Ardupilot software uses a combination of an accelerometer, GPS, and barometer to calculate the current altitude. In this instance, the hardware was not faulty, but badly installed.
On this flight controller, the barometer is on the bottom of the board. It has a little tiny hole in it to allow pressure sensing. In my 🤡'y install, this is covered by hot glue and velcro.
While this didn't cause a crash, it shows that I didn't ground test the altitude, and any attempts at keeping altitude in autopilot mode may have been dangerous.
2. Launch
The maiden launch itself in manual mode was nominal. You can hear me saying "Good luck little plane". It needed more than luck it seems!
3. Fly-By-Wire Test
At ~1 minute 22 seconds into the flight, I turned the flight mode from manual to Fly by wire A, which should have had the effect of assisting the plane in straight line. Looking at the radio stick input, I did not change the heading, but the autopilot sent this into a steep dive at this time. A near miss where we avoided a crash here.
Reversed Elevator
12:24 is interesting here. If we look at the time we engage the autopilot, we can see the pitch heading down into the ground, before our roll even begins to change. It ends up to a point where it's almost completely nose down before we save it by switching to manual flight mode.
Root Cause
What's interesting is that the pitch continues to fall, and the AETR elevator gets more and more positive. Is it possible that we're trying to correct the nose down and instead making it worse? Here I did NOT check how the autopilot responds on the ground, and I instead tested in manual mode, and adjusted the Servo Output functions to work. I should have instead checked this in combination with the Radio Calibration reverse settings
4. Crash (elevator failure?)
This event makes me believe that I had an elevator failure during the flight. In a panic I "checked" my autopilot settings, and accidentally turned it on.
Events:
- Nominal flight approaching turn
- Banks left to enter turn
- Starts pitching down, control toggles are pulling back on the elevator, manual flight mode still engaged
- Plane is not responsive and is still pitching down
- After loosing much altitude, in a panic, RTL is engaged
- Plane starts pitching nose down further once RTL is engaged (reversed elevator theory?) causing the epic crash with almost full speed, full nose down
Upon inspection, the suspected failed elevator servo is the one shown in black here. When moving the control horns without power, I noticed that it slipped. It's possible it could have slipped in flight causing the loss of the ability to pitch up. It's also possible that the 915mhz SIK radio I was flying with for telemetry interfered with the servo (they do twitch then they're close, but I had thought I had moved this far enough away). I guess we'll never truly know!
Lessons
- Don't fly with so much throttle. Mistakes happen a lot faster than it could otherwise happen
- Test the FBWA modes on the ground. Ensure that the elevator, ailerons and rudder is behaving as expected in relationship to pitch, roll movements.
- Make sure you setup BOTH the reverse setting on the controller input, and servo output.
- Be sure to glue in everything! I'm not convinced a servo could have come loose during this flight.