Understanding the Oak code stack

[OakTestUser]

Hello all,

I've got my Oaks today, it did an update (the led blinks 3 times now, 0.5sec pause, this is correct?) and was able to connect it to the particle.io cloud with the config app. I didn't fiddle more with it as I'm trying to understand it more beforehand so I'm reading up. And I've got a lot of questions that keep stacking up and were I can't find a good answer for, I hope to find some answers here.

First, I know the Oak has an ESP8266 module on it. I have some esp modules myself, but beside a blink program, I didn't do to much with them. Now with the Oak I hope to change that but I'd like to understand it first.

So I'd like to start with some basic questions and then some more technical questions.

• The Oaks is programmable with the Arduino IDE, digistump made a complete "Oak" toolchain on it that is downloadable with the boards manager. You compile a sketch and program the Oak over usb (usb to serial converter is inside I assume). Same thing with particle web IDE but you program it OTA. (Or does Arduino IDE also program OTA and not via usb?)
• To be able to program your Oak over usb/OTA, the module has what is called firmware running that will write your sketch to a specific flash memory to be executed endlessly afterwards. Moreover, the Oak is able to update it's own firmware OTA. (This is basically analogous to the Arduino boatloader on Atmel mcu's)
  
• This complete firmware "package" holds all the Oak/Particle programming stuff, and contains all the libraries that you can use inside your sketch. This makes the sketch small and makes it quickly programmable. But there is perhaps also other stuff I'm missing in this firmware "package"?
  
• If you brick your Oak you need to really reflash it with this firmware "package". You pull Pin2 to GND and reboot it, Oak is now in boot mode and you flash the firmware binaries with ESPTOOL or a similar tool. This is different from "programming" your Oak over usb/OTA like I asked previously as you do not need to have the Oak in boot mode. Firmware inside handles "reprogramming" the sketches you make for it.

Ok now the more technical questions.

• The Oak or esp modules for that matter has flash memory. When you compile binaries or find binaries for it, it will give you several .bin files (I tried to setup a complete esp toolchain with the espressif SDK but always gave up on it, hence these questions). The .bin files are often "blank.bin" "boot.v1.4.bin" "esp_init_data_default.bin" or maybe "irom0text.bin" or if you want your AT commands on the ESP you often flash only one .bin file. In case of the Oak your flash "oaksetup_restore.bin" to 0x0081000 "oakupdate_restore.bin" to 0x101000 etc. Can anyone give me a clear explanation why these different .bin are there and why they sometimes have to flashed to a specific starting address or why you sometimes only have one .bin file. How do you all know this, where/what bin files go where.
• All development on esp modules (and thus Oak) use the espressif SDK or do I got it totally wrong here? A project like the Oak or bringing the esp8266 to the Arduino IDE (and thus making sure "digitalWrite()" etc arduino stuff works for the esp) where do you start from? Totally from scratch or an SDK or libraries provided from espressif? I've troubles understanding this part of the story.
  
• In clear langauge, what is SPIFFS exactly? Why is it used or why is it in the esp modules and is it really needed? Secondly, what is "elf", I see this pop up sometimes in the source code?
  
• Lastly, what is the "OakSystem" on github used for? To build a config rom? I also apparently do not understand this part.

I'm really trying to understand all parts of the story here, I got really interested in electronics and embedded programming but I just can't get my head around the ESP8266/Oak.

Thanks

[werner]

Hi

I'm just a newby myself but will attempt to answer your questions as I understand things at this point.  Please don't take this as gospel as I might be wrong about some of this:

The Oaks is programmable with the Arduino IDE, digistump made a complete "Oak" toolchain on it that is downloadable with the boards manager. You compile a sketch and program the Oak over usb (usb to serial converter is inside I assume). Same thing with particle web IDE but you program it OTA. (Or does Arduino IDE also program OTA and not via usb?)

  You compile a sketch and program the Oak over usb (usb to serial converter is inside I assume)

No, USB connector is just there for power.  Currently when you compile in Arduino IDE, your binary gets uploaded to Particle which will then send it via OTA to the Oak.   Programming via serial cable only supported via Python scripts and not recommened at this stage.   (You will need to connect FTDI USB for that anyway)

To be able to program your Oak over usb/OTA, the module has what is called firmware running that will write your sketch to a specific flash memory to be executed endlessly afterwards. Moreover, the Oak is able to update it's own firmware OTA. (This is basically analogous to the Arduino boatloader on Atmel mcu's)

Yes, that about it.

This complete firmware "package" holds all the Oak/Particle programming stuff, and contains all the libraries that you can use inside your sketch. This makes the sketch small and makes it quickly programmable. But there is perhaps also other stuff I'm missing in this firmware "package"?

Just like with regular Arduino you have libraries.  The compiler and linker decides what to include in your binary when you hit the program button.

If you brick your Oak you need to really reflash it with this firmware "package". You pull Pin2 to GND and reboot it, Oak is now in boot mode and you flash the firmware binaries with ESPTOOL or a similar tool. This is different from "programming" your Oak over usb/OTA like I asked previously as you do not need to have the Oak in boot mode. Firmware inside handles "reprogramming" the sketches you make for it.

Yes, except it is still just a regular porgram like any other that gets loaded via the bootloader.   The purpose of thise program is just to set up the Wifi and connect you originally to the Particle cloud.  As soon as you download your own app, it overwrites this initial program.

Lastly, what is the "OakSystem" on github used for? To build a config rom? I also apparently do not understand this part.

As I understand it, it is just a regular program like any other you write yourself as well.  It helps to configure the WiFi access and initial connection to the Particle cloud.  Once you load you own app, it replaces the original OakSystem.   But of course because your own app is built using the same libraries as the OakSystem it still has the same fundamentals on the device

[OakTestUser]

Thanks werner for the reply!

So usb connector is just power! Yes ok got it! Arduino is also only OTA and via a python script and compiled binaries you can flash the Oak offline but you need a usb-serial converter. Familiar with this as I flashed esp8266 countless times. I think if you take the NodeMCU flasher and load up your own binaries you can just flash it this way as well? Furthermore the bootloader thing I get now as well.

Just like with regular Arduino you have libraries.  The compiler and linker decides what to include in your binary when you hit the program button.

How does this compiler with linker works? How and when should you use it? Am I correct a linker checks the necessary or used functions/library includes and add them to the compiling binaries making a total package of your code + needed library/sdk code. Anything uncessary is not used or linked in this case and thus not added inside the binary?

Yes, except it is still just a regular porgram like any other that gets loaded via the bootloader.   The purpose of thise program is just to set up the Wifi and connect you originally to the Particle cloud.  As soon as you download your own app, it overwrites this initial program.

What exactly do mean "just a regular program" that "gets loaded via the bootloader" if your Oak is bricked and you need to completely reflash I think you need to flash everything include the bootloader with it?

[PeterF]

Hey OakTestUser,

I'm just starting to get my head around the Oak also, as I haven't had time to follow it's progress or delve into its depths... so is quite possible I've gotten a few things wrong

I think as far as the reference to 'just a regular program' may be a bit misleading, as the firmware is able to update both itself, and the user program. Erik posted this in a recent comment about the Oak in regard to another discussion, which might help aid in filling in some of the blanks:

An Oak has 3 rom slots, one holds your Config rom, one holds your User rom (what you upload), the other is empty. When you upload a new User rom or a new Config rom (due to an update), it goes to the empty slot and once confirmed complete and good the pointer to that rom is changed to point to that slot.

The memory addresses which you are specifying are the addresses that the Oak's MCU is expecting to find those ROM programs. It is different to other ESP based boards as Erik has divided up the memory into those slots. As you can see from where you were mentioning "oaksetup_restore.bin" and "oakupdate_restore.bin", you are filling these two slots with those two default programs - one which set up the oak, and the other being the system updater. The memory addresses are probably only relevant to the bootloader, as it should be the only code that is operating at that low a level. I think there are a few other slots we'll find about as the documentation is updated, which hold the unique device id's for the Particle Cloud (at 0x100000 and 0x201000).

Regarding serial programming, it is indeed possible. You change your upload settings (under Tools -> Upload) to"Serial (Expert Use Only - Requires Python)". However, I believe you then lose OTA programming until you restore the stock firmware. However, it would upload significantly faster, and is a possible workaround to using an Oak until the setup process becomes more reliable.

ELF stands for Executable and Linking Format, and apart from be able to store fuse values and EEPROM data when used for the Arduino boards, I believe it also contains the the final interpreted version of your code, as well as the assembly version. It's main use is for being able to run your code through a debugger or simulator if one exists.

Pete

[digistump]

As you can see from where you were mentioning "oaksetup_restore.bin" and "oakupdate_restore.bin", you are filling these two slots with those two default programs

I guess this gets a bit more complex than the 3 slots - we combine the updater and the setup roms into a single file and load them together into one of the three slots. The bootloader actually supports dividing each of the 3 into quarters, currently we only divide the system slot into halves though and user slots are left whole - probably best to dive into the bootloader code if you really want to understand it on that level as it is all abstracted away for the user experience (bootloader code: github.com/digistump/OakBoot)

[OakTestUser]

Thanks for the reply's

Ok so first things first, with bricked it is meant that the Oak/ESP8266 will not function at all. You need to reflash everything, bootloader firmware, everything else + your own code. Or with bricked that the user code is bad and causes runtime errors -> programming (bugfree) new user code unbricks it in this case.

Concerning the ELF files. These files are basically the same as binary files but with ELF files you can flash it at any memory place. The linker makes sure that all locations to data or functions are being correctly offset from a base memory address whereas a bin file has this hardcoded inside.

Concerning flash memory. You can divide it up in any way you want it as far as I see and make any slot system with it as you want? As Digistump replied, digistump chose to divided the Oak into 3 "slots" in these slots you flash binaries into specific memory addresses. As digistump said, the Oak bootloader is able to divide these slots into quarters but the bootloader handles this and we do not need to be concerned about it. However the bootloader currently divides a "system slot" in halves, this slot has the bootloader, Oak setup code, OTA code in it etc if I understand correctly and leaves the "user slot" whole. In this user slot is our code being placed by the bootloader.

So you flash the bootloader on the Oak or any ESP8266-12. With bootloader you can program the Oak with user code or update completely or parts of the bootloader/Oak system stuff binaries? To flash the bootloader or restore the oak, flash "0x1000 blank.bin 0x2000 oaksetup_restore.bin 0x0081000 oakupdate_restore.bin 0x101000 blank.bin 0x102000 blank.bin 0x202000 blank.bin" which I got from the github. These bins include the bootloader, Oak libraries, OTA update, Particle integration.

Concequently -> Flash bootloader & system binaries on the Oak. Oak will be able to update itself OTA, update user code OTA, your user code is able to use system binaries which include the particle integration etc.

An other thing I'm wondering. Is there a specific start address the Oak/ESP8266 start executing code from or are you able to somehow "set" the entry point of the bootloader. The bootloader then gives execution to user code.

Best regards