How to build OakSystem

[thinbits]

I posted a comment on the GitHub bounty page (https://github.com/digistump/OakCore/issues/54) asking about how to build OakSystem to debug AP connection issues, and was asked to start a thread here to collect all the information in one place.

Here is what I've tried:

I've grabbed the source for OakSystem, oak_fallback (from bounty comments), and oak_update (from bounty comments). I'm using Arduino IDE 1.6.5 and installed the 2.0.0-rc1 folder from oak_fallback into ~/Library/Arduino15/Hardware (Mac OS).

My Arduino settings are: Oak by Digistump (Pin 1 Safe Mode - Default), Serial (Expert Use Only), 80 MHz, and the port is set to my USB -> Serial adaptor.

I can now get OakSystem.ino to compile, but I'm unclear on how to get it installed onto my Oak. I've tried both uploading from the IDE and exporting the compiled binary and using esptool, but in both cases, when the upload is complete and I re-boot the Oak the LED flashes 3 times, pauses, and repeats.

The esptool command I tried was:

Code: [Select]

esptool.py --baud 115200 --port /dev/tty.usbserial-A50482AV write_flash -fs 32m 0x1000 blank.bin 0x2000 OakSystem.cpp.oak1.bin 0x0081000 OakSystem.cpp.oak1.bin 0x101000 blank.bin 0x102000 blank.bin 0x202000 blank.bin (total crapshoot, cobbled together from bounty comments)

What is the correct way to get a new OakSystem installed for debugging?

[PeterF]

So I take it from what you said about the blink pattern it is doing this one? Indicating that it has updated the firmware?

I may be talking complete nonsense, but it may be worth doing an OakRestore first, and then re-loading your own compiled firmware the same you have done it with esptool. I suppose you could check if your own custom version of oakSystem was being loaded would be at line 584, change the

Code: [Select]

String version = versionResponse(); to

Code: [Select]

String version = "42"; or some other custom number. Then when you are using the config web page, or when you interrogate http://192.168.0.1/system-version, you would get your own custom number instead of 0 (or 1-5, depending on firmware update installed). You could then change it back to the original code once you know your code is being installed.

Pete

[jdeon]

There's pretty good instructions for building OakSystem on a lot of the bountied issues on GitHub - https://github.com/digistump/OakCore/issues/28

There are a few steps involved, including modifying the hardware platform files.  There's an example serial flash command you can modify for your use as well.

It sounds like you got OakSystem to build already, but one issue I ran into is that OakSystem.ino and oakrestore.ino won't build on Arduino 1.6.6 - they changed something with requiring function prototypes.  If you run You'll have to use 1.6.5.  You've got the compile/upload settings correct, so there's no problem there either.

From what I understand, there's space on the Oak for two builds - when shipped from factory, they are the "oaksetup_restore.bin" and "oakupdate_restore.bin" from this page - https://github.com/digistump/OakRestore 

Downloading code from the Arduino IDE via serial or from the Particle Cloud overwrites...  something.  I'm not sure if it's one of the oaksetup/oakrestore images, or if there's another slot for user images.  That part's not very clear to me.

Your esptool.py command writes your new OakSystem build to both "known" slots, which is probably overkill.  You probably want to write "oakupdate_restore.bin" to the second slot, like:

Code: [Select]

python esptool.py --baud 115200 --port YOUR_COM_PORT write_flash -fs 32m 0x1000 blank.bin 0x2000 OakSystem.cpp.oak1.bin 0x0081000 oakupdate_restore.bin 0x101000 blank.bin 0x102000 blank.bin 0x202000 blank.bin
All that said though, it seems like you're doing it right.  The "three blinks" means that you're running the latest firmware.  Is the ACORN-XXXX wifi network visible from another wifi device?  You should be able to run through the same config stps on the wiki - http://digistump.com/wiki/oak/tutorials/arduino and get your Oak connected to Particle at this point.

Right now, there's code available for building:

• Factory version of "oakupdate" - https://github.com/digistump/OakCore/issues/54#issuecomment-191158318 in oakupdate, requires that you also have the factory version of the "oakfallback" hardware from here https://github.com/digistump/OakCore/issues/54#issuecomment-191492393
• Latest version of "oaksetup" - https://github.com/digistump/OakSystem, which depends on the "oak" hardware package from here https://github.com/jldeon/OakCore

When you're building OakSystem.ino, you're building the latest version of the "oaksetup" file.  This is part of what gets downloaded when you connect your Oak to the internet and run the setup app for the firs time.  I believe the "firmware_v1.bin" file that is downloaded by the Oak contains the latest OakSystem + oakupdate_restore.bin.

If what you're trying to do is make a small change to the factory oaksetup, I don't believe the code is available for that right this second.

If you feel like you need to change the hardware platform code (ie, OakCore or OakFallback), I have had success in building those under Windows and can probably point you in somewhat the right direction.  Don't know if that's of interest, however.

[thinbits]

The specific problem I'm trying to track down is a r = -1 result from the configure-ap endpoint on the Oak, which is the source of my problem which I documented here http://digistump.com/board/index.php/topic,2005.0.html. From reading the forums, other people have encountered this issue too.

After some testing, it has something to do with processing of the network name ("Dave & Judy's" in my case), specifically, if I remove the '&' it no longer returns -1. The code in configure_ap_from_JSON (in particle_core.cpp) looks correct, so I was going to add more debug output to isolate the problem.

If I restore with the stock OakRestore command, I see the Acorn-XXXX network in my Airport menu. If I then restore with your suggested command (0x2000 OakSystem.cpp.oak1.bin 0x0081000 oakupdate_restore.bin), I no longer see the Acorn network but I get the rapid 3-flash repeating pattern on the LED after a bit.

Seems odd that I would get the blink pattern indicating a firmware download from the net after a fresh restore of the device.

I've run out of time to tinker with this tonight. I'll try to dig deep during the week.

[PeterF]

jdeon, there's another slot for user images, so downloading via Serial or via the Cloud shouldn't be overwriting oakSetup or oakUpdate if I understand the memory model used for the Oak properly. Going back to an earlier comment by Erik

An Oak has 3 rom slots, one holds your Config rom, one holds your User rom (what you upload), the other is empty. When you upload a new User rom or a new Config rom (due to an update), it goes to the empty slot and once confirmed complete and good the pointer to that rom is changed to point to that slot.

and he elaborated on the first slot a bit further

I guess this gets a bit more complex than the 3 slots - we combine the updater and the setup roms into a single file and load them together into one of the three slots. The bootloader actually supports dividing each of the 3 into quarters, currently we only divide the system slot into halves though and user slots are left whole

So my take on that is that the Oak's memory has been divided into three parts, with the first being the system rom, the second being the user program, and the third being empty. When you upload a new program, it goes into the empty slot, and once it is successfully downloaded, becomes the program slot, with the other (which contained the old user program) becoming the empty one (by changing which slot the pointer points to). The system slot has been divided into to halves - consisting of the oakSetup and oakUpdate images, and are uploaded as two files as it makes it easier to maintain them separately, and are expected to start at specific addresses in memory due to their two-file nature.

Need to shove this on the wiki somewhere, as it helps get a better understanding of how the Oak has been put together.

Pete

[jdeon]

The specific problem I'm trying to track down is a r = -1 result from the configure-ap endpoint on the Oak, which is the source of my problem which I documented here http://digistump.com/board/index.php/topic,2005.0.html. From reading the forums, other people have encountered this issue too.

After some testing, it has something to do with processing of the network name ("Dave & Judy's" in my case), specifically, if I remove the '&' it no longer returns -1. The code in configure_ap_from_JSON (in particle_core.cpp) looks correct, so I was going to add more debug output to isolate the problem.

That makes sense.  I've run into this sort of thing before, I got a r=-1 response when trying to send commands down that had multibyte or weird extra characters.  Have you looked at the raw command being sent from the browser in something like Chrome DevTools or Firebug?  That might help as well.

If I restore with the stock OakRestore command, I see the Acorn-XXXX network in my Airport menu. If I then restore with your suggested command (0x2000 OakSystem.cpp.oak1.bin 0x0081000 oakupdate_restore.bin), I no longer see the Acorn network but I get the rapid 3-flash repeating pattern on the LED after a bit.

You should see the ACORN network when you've got the "3 flash" pattern going.  That's the second half of the setup phase - once the firmware downloads, then the setup app has you reconnect to the Acorn network, confirm the firmware update, and then configure against the Particle cloud.

Have you tried power cycling the Oak?  You might be running into the hidden network bug - https://github.com/digistump/OakCore/issues/28

Seems odd that I would get the blink pattern indicating a firmware download from the net after a fresh restore of the device.

If you're downloading your custom-built OakSetup, then you're actually building the latest version, since we don't have the code for the factory OakSetup image.  The OakSetup.ino you get from https://github.com/digistump/OakSystem is the version that is downloaded from the internet in the first part of the setup app's process.  (I attempted to explain this in my previous post, but probably failed to make it clear )

[jdeon]

jdeon, there's another slot for user images, so downloading via Serial or via the Cloud shouldn't be overwriting oakSetup or oakUpdate if I understand the memory model used for the Oak properly. Going back to an earlier comment by Erik

An Oak has 3 rom slots, one holds your Config rom, one holds your User rom (what you upload), the other is empty. When you upload a new User rom or a new Config rom (due to an update), it goes to the empty slot and once confirmed complete and good the pointer to that rom is changed to point to that slot.

and he elaborated on the first slot a bit further

I guess this gets a bit more complex than the 3 slots - we combine the updater and the setup roms into a single file and load them together into one of the three slots. The bootloader actually supports dividing each of the 3 into quarters, currently we only divide the system slot into halves though and user slots are left whole

So my take on that is that the Oak's memory has been divided into three parts, with the first being the system rom, the second being the user program, and the third being empty. When you upload a new program, it goes into the empty slot, and once it is successfully downloaded, becomes the program slot, with the other (which contained the old user program) becoming the empty one (by changing which slot the pointer points to). The system slot has been divided into to halves - consisting of the oakSetup and oakUpdate images, and are uploaded as two files as it makes it easier to maintain them separately, and are expected to start at specific addresses in memory due to their two-file nature.

Need to shove this on the wiki somewhere, as it helps get a better understanding of how the Oak has been put together.

Pete

Ah, see, that makes so much more sense.  The firmware_v1.bin contains a single Oak image that is divided into two, which is the "setup" and "update" images.  When we run the esptool from OakRestore, we're overwriting those two images. 

Then there are two more user slots.  If you download from Particle Cloud or from Arduino, you're writing into one of those.  So you could download your OakSetup image into one for testing, if you wanted to.

[PeterF]

Exactly jdeon, Only stated making sense when I realised the difference between the OakRestore commands, and the firmware update. All the same memory addresses are blanked, but in the OakRestore case, two files are written, and in the manual firmware update case, one file is written.

I believe from what Erik has posted on the "Find a workaround/fix for update issues" bounty, that the oakupdate.zip is the source for the factory image, as he says that "he source is the same as attached in the second comment zip file (oakupdate.zip) - just comment out #DEBUG_SETUP to get exactly what is on a factory Oak"

Also that "The Oak should respond with {"r":0} if after each three lines (set, length, content) are sent - if you get {"r":-1} back then something is wrong with the input.", and remember that it is expecting only a newline at the end of each line -  no CRLF, CR, etc.

I had assumed that the oaksetup_restore bit was contained in oakupdate.zip, and that oaksetup_restore came from the OakSystem repo, but I could be completely wrong there.

[thinbits]

Okay, I've tried the steps from https://github.com/digistump/OakCore/issues/28, but I'm still not having any luck getting a custom built OakSystem running. Not sure if I'm just being dense, or if there are steps missing...

Here are the steps from GitHub:

• Install Python 2.7 and ensure it is in your path
• Install the latest Oak core via board manager, download this repository and copy over the files found at APP_DATA/packages/digistump/hardware/oak/XXX where APP_DATA is where your system keeps the Arduino data and XXX is the current Oak core version.
• In cores/oak/OakParticle/particle_globals.h uncomment #define OAK_SYSTEM_ROM_4F616B 82 - this tells it to compile the OakSystem sketch as a system image not a user sketch
• Grab the latest OakSystem sketch from https://github.com/digistump/OakSystem
• Make changes to the core and/or sketch to achieve the above task.
• Connect your Oak using a 3.3v USB to Uart adapter, connect Pin2 to GND before powering on to enter Serial bootloader mode (after an upload the code with automatically run and you'll have to power cycle it to go back to bootloader mode. Disconnect Pin2 to GND to have it run the code on next power up)
• ONCE: Grab OakRestore (https://github.com/digistump/OakRestore) and run python esptool.py --baud 115200 --port YOUR_COM_PORT write_flash -fs 32m 0x1000 blank.bin 0x2000 oaksetup_restore.bin 0x0081000 oakupdate_restore.bin 0x101000 blank.bin 0x102000 blank.bin 0x202000 blank.bin  replacing YOUR_COM_PORT with your com port, to restore the Oak to factory setup so that the Serial upload will work and the Oak will boot to it.
• Use "Upload Using" "Serial" in the Arduino tools menu.
• Upload from the IDE and test

After the restore, Oak appears as an access point (but has my problem, this is expected).
Seems like there should be a step 7.5 of "reboot with pin 2 grounded". You can't upload from the IDE once it reboots after the restore.

Build and Upload from the IDE of OakSystem.ino results in an Oak that no-longer shows up as an access point and has the 3-blink pattern (for a little bit, then stops).

I've built it with DEBUG_SETUP defined and re-booted but I don't see any serial output.

I can't help but wonder if what really needs to happen is to rebuild oaksetup_restore.bin. If I run strings on oaksetup_restore.bin from OakRestore, thats the image that has "configure-ap" in it. Or is oaksetup_restore.bin just the export of the OakSystem.ino build?

[digistump]

A few things to clarify:

Serial uploads always go to slot 0 - no matter what is in that slot. A factory Oak boots to 0 looking for the system firmware, which is why before uploading via serial I instruct people to do a factory restore, so that the Oak is booting to 0 which is where you are putting the code. Once your done using serial uploads for testing/debugging/etc you'd want to restore back to factory or at least put a system image on it over serial so that it is at slot 0.

@thinbits

Yes the steps should include grounding pin2 for each upload.

oaksetup_restore.bin is just a build of the oldest version of OakSystem.ino - the factory version. The latest version (which is what you are building and uploading) is the latest beta system image (which you'd get via the first update if you could connect).

Try downloading https://digistump.com/firmware/firmware_v1.bin (ignore the https error) and copying that over serial in place of oaksetup_restore.bin - you should get the 3 blink pattern - do you see an access point?

Now to the original issue you are seeing with your SSID name -

Seeing what the web browser is sending via the inspector in Chrome or Firebug in firefox seems like a great start - the Oak just writes what it gets from the browser, but it does so dumbly so my guess is the browser is converting the & before it sends it. Seems like something we should fix moving forward, but of course we can't for the very first update. A faster way to fix it for you may be to find a way to send it uncommented to the Oak - like a raw post using something like wget/curl/etc.

[thinbits]

@digistump: that link to firmware_v1.bin yields a 404.

At this point I'm giving up and getting back to work with my Huzzah's. Not going away mad, just the build process for this is a bit ridiculous and there are no working instructions. I hope someone else figures it out as being able to re-build and debug the base software seems important to the platform. At the moment it seems like Digistump might be the only one who can do this.