iXRD
A couple of days ago, iXRD launched which is one of the things I was working on before joining Fuel Labs for about 2 years. It feels weird to have a thing that you contributed in outer space, that is for sure. Here I am discussing some of the things we encountered during the development since some lessons were learned during these 2 years.
Sharjah-sat (which hosts iXRD) launching onboard SpaceX's Transporter 6 mission
What is [i]XRD
First of all, what is iXRD? HEALAB (high energy astrophysics lab at Sabanci University) designed an x-ray detector to be launched with a cube-sat which is deployed in 2017. Unfortunately due to the cube-sat, it is shipped with not receiving any signals from ground control the mission was a failure. After that development of iXRD started to be launched with Sharjah-sat. iXRD is a custom-designed detection circuit that hosts a fancy ASIC and a detection crystal to detect high-energy particles. Why this is needed you might ask. Why not build a huge telescope in the ground? Since most of the stuff that is exciting to observe is filtered by the atmosphere of our earth, we need to observe them from space.
Sharjah-sat and us; probably doing something that we shouldn't be doing
The software
I worked on the firmware of the detector targeting a microcontroller from MSP 430 family. The idea was that our onboard controller would communicate with the detection ASIC; initialize it and read the data from ADC which reads the output of the ASIC, process the data, save it to an SD card and finally serve the data to the onboard computer of the whole satellite.
Evolution of my ideas regarding embedded software during the development:
- If the hardware is not %100 reliable, embedded software is a pain in the ass
- File system is most of the time overkill. We were using FAT32 but had so many problems that we ended up removing it completely
- We started implementing KISS protocol for all the communication which ended up taking more than it brings
- C, CCS any stack targeted for novice-embedded developers that you get out of the box sucks let's all switch to Rust all together
Hardware reliability
TL;DR: Writing and using a check-up software that checks all possible points of the hardware before trying a new code for bug fixes might be worth the extra effort since it is pretty hard to narrow bugs to software only.
iXRD had two main components, a motherboard, and a daughter board. Since the ASIC we were using was incredibly sensitive to noise, we tried to put as much space as possible between the noisy components (for example the clock of the processors in the satellite ended up adding a considerable amount of noise to the system such that we could clearly see some noise at the running frequency of the different processors in the satellite in our tests) and the detection circuitry. Since we did not have proper documentation for the ASIC we were working with, we had to reverse engineer the communication between the software provided with the ASIC and the ASIC itself to understand the required bit patterns we had to send over to the ASIC for various operations. Not surprisingly that requires lots of trial and error. Since we had such a modular structure of the hardware and lots of other reasons too, it was nearly impossible to make sure that we were doing controlled experiments. There were simply too many things we either could not check easily or cannot control at all.
File system
TL; DR: File systems are probably unnecessarily complex for your embedded needs. At least that was the case for us. We ended up writing raw blocks in a cyclic buffer to the SD card with some minimal header. Worked a lot faster!
iXRD had old software that has been created for the XRD that launched in 2017. That used some file system abstraction with SD cards. Sometimes, randomly, the initialization step was taking forever. We ended up removing the filesystem completely. Interfaced with the SD cards directly and write the blocks in a cyclic-buffer manner. We added a small header to each block (like 7 bytes long) and to find a specific block we basically did a binary search as the block headers kept increasing. Initialization was instant, and the speed was great. Since the detection had a specific timing requirement, we had to write the data before the new data comes. To comply with that we were doing some caching and holding the data in the memory until the detection is over. Writing to the SD card without the filesystem was simple and fast enough that we could remove the extra caching. Removing it was important as it freed up some considerable amount of RAM usage. It is important to note that we were using an official FAT driver that is provided by TI, maybe a complete rewrite would help with speed as well but never had the chance to try that.
KISS (AX25) PROTOCOL
It is cool and simple but implement it with care. We had lots of small issues and inherent bugs with our implementation that were really hard to debug that at some point made us consider removing it completely.
C -> Rust
Let me start with a confession, at the beginning of the project, I was a junior in college with minimal exposure to embedded systems, that's for sure, and especially because of that, we should have started with Rust. Rust eliminates lots of weird stuff I wrote especially at the beginning which was allowed by C. As a newish C developer, I had the power to make such critical mistakes. Writing C that works may seem like easier than Rust but writing good C is considerably harder than writing good Rust. So if you are doing some embedded stuff, consider Rust, and if you decide not to go with Rust, try to have strong reasons for that. My hope for 2023 is that I will have some chance to work with embedded Rust in cool side-projects.