Changing from Oak/Particle to generic ESP8266 code with serial and OTA updates

[postuma]

I've struggled enough with this I figured it deserved its own thread - and when I say struggled, I mean at least a half dozen hours to get it working.

So - why bother?

Despite the difficulties, it's already been worth it:

• Speed! Reliability! Approximately 30 seconds for a simple serial flash that works every time, and 8 seconds for an over-the-air flash ...
• Hours of fail-after-fail-after-fail Particle updates - more than I've invested in the fix thus far - gone now
• Previously, even when the code updated ... you couldn't always tell it failed. Not failing now
• I've had two Oaks that I've never been able to "claim" using Particle - they work fine now
• Code fixes that haven't made it to the Oak have long been implemented into the ESP8266 core
• Compatibility: newer libraries may not be compatible with the Oak, like the newer versions of the Adafruit GFX libraries. They are now!
• Serial output - Particle limited messages to 255 characters, a few times per second. No such limits on Serial output
• Getting code into SPIFFS required copying files to the SD card, then copy from SD card to SPIFFS - and the SD card frequently does not properly initialize - hours wasted here as well. With the change to ESP8266 core, the ESP8266 Sketch Data Upload tool becomes available, and directly and oh-so-easily transfers data to SPIFFS

I was a reluctant to make the change. What if things broke irrepairably? But Particle is so broken I felt I had no choice.

To his credit, Erik has acknowledged these problems - see https://www.kickstarter.com/projects/digistump/oak-by-digistump-wi-fi-for-all-things-arduino-comp/posts/1912567. He's also put a big disclaimer on the Digistump order page - caveat emptor. And I have no doubt he will address these issues in some way, at some time. But family comes first! and I've no doubt he's way over-extended (very excited about the BuildOne printer, BTW).

You'll need a USB to TTL adapter to program the Oak using a serial interface, found el cheapo on Ebay: http://www.ebay.com/itm/USB-to-TTL-CH340G-Converter-Module-Adapter-STC-Dowanloader-Programmer-3-3V-5V-/112215846564.

Follow the Brendan 'nog3' Halliday tutorial at http://nog3.net/2017/06/05/removing-particle-from-oak/. Things to note:

On Python:

• You have to have Python installed on your system (typically native on linux, not on Mac or Windows). Version 3.x may not work with the Arduino IDE; use 2.7. Use the installer
• Windows: make sure that Python is installed on the system PATH (use the installer; this is NOT selected by default)

On esptool.py:

• You'll need the esptool.py script - and the version in your Oak install may not work. No biggie. The one in the generic ESP8266 install does work
  Alternately, the new-and-improved esptool.py can be had from https://github.com/espressif/esptool. Works nicely, as of this writing
  The new install does not overwrite the Oak's original information; it creates new entries for the Oak-as-ESP8266. Don't hesitate to install the new package
  Backing up your firmware after install: do  it!

On the USB-to-TTL adapter:

• The USB-to-TTL serial adapter draws enough power from the USB port to power itself - but often not quite enough to power the Oak (especially using the 3.3V pin out; you may have luck with the 5V pin.) Best to power it separately using the micro-USB port, or using another power source for VIN.
• If powering the Oak independently, you don't need to connect either power-out pin from the adapter to your Oak. I don't - you'll only need to disconnect the power source to power-cycle your Oak, which you may do frequently. And if you're using more than 5V on VIN, you should not - the higher reverse voltage could damage the adapter
• On Windows: always use the same USB port for your serial adapter - if you happen to use another port by mistake, the COM port will change, but there won't be an error message that tells you so
• Windows - with the USB-to-TTL module plugged into the USB port you intend to use, click on Start > Settings > Devices and you should see it. Note the COM port used. With older version of Windows, you'll find it under Device Manager
• Serial port for Mac/linux is typically something like /dev/tty.SLAB_USBtoUART; dmesg | grep ttyUSB should get you the name
• To back up your current entire firmware, including SPIFFS, using Windows on COM4 would look something like
       esptool.py --baud 115200 --port COM4 read_flash 0x000000 0x400000 flash_c70a.bin
  (I use the last four hex values of the Oak's MAC address/Particle device ID to identify them/keep them straight - I write them on the back of the module in indelible marker. Here, I use it to uniquely identify the backup of each bin file as flash_xxxx.bin)
• Wiping flash on my Windows system, on COM4:
       sptool.py --baud 115200 --port COM4 erase_flash
• If you regret your decision and want to restore, on Windows:
       esptool.py --baud 115200 --port COM4 write_flash -fm dio 0x000000 flash_c70a.bin

Modifying the Arduino IDE:
     

• Your new entry in the variants folder is a sub-folder containing the file pins_arduino.h. These remap the ESP8266 pin definitions to our Oak equivalents, with a caveat: 1 still refers to the ESP8266 GPIO pin 1, 2 refers to the ESP8266 2, etc. The new versions are P1, P2, etc. What I like better - also defined are SDA, SCL, SS, MOSI, MISO, SCK, BUILTIN_LED/LED_BUILTIN, and A0. I no longer have to remember which maps to which!
• You may have to update libraries that are no longer compatible with the ESP8266 core. Example: my old Adafruit GFX and ILI9341 TFT libraries don't work with the new core

Using the Arduino IDE:

• Board selection: find esp8266 using Filter your search under Boards Manager, and load
• Traditionally the Oak uses 4 MB flash/1 MB SPIFFS; this is preferred if you intend to use OTA (as OTA updates actually store two sketches in memory). However, if updating only over serial then you're directly overwriting the old sketch with the new - 1 MB for program should be more than ample, and presumably 4M/3M SPIFFS is an option - lots of on-board storage! I haven't tested/confirmed this yet, however
• You need not disconnect pin 2 from ground after flashing to enter run-progam mode; the Oak will reboot automatically and run its program
• IF flashing again after a successful flash without first power-cycling the Oak, you'll get a lengthy, uninformative error message. It's rebooted into run-program mode, not bootloader mode. Disconnect both the adapter and your power source from the Oak (either one provides enough power to keep it turned on), and reconnect. Alternately, if you haven't connected the USB-TTL adapter power pin to your Oak, just disconnecting the Oak's power source will do
• Serial messages work much like Particle (only better). Remember to specify a baud rate when initializing, i.e. Serial.begin(115200). Also, to get useful info from the IDE's Serial Monitor, remember to set it to the same baud rate (mine defaulted to 9600)   
• I've had javascript errors/upload fails that only resolved when I upgraded the Arduino IDE - but can't guarantee that this won't introduce new issues. Created a fresh backup before the upgrade just in case; hope not to need it
• If you're unwilling to take this step but still get the Javascript errors, or if this did not work, Sketch > Export compiled Binary also worked. I just had to manually upload the binary, i.e.
       esptool.py --baud 115200 --port COM4 write_flash -fm dio 0x000000 blink.bin

Assuming all went well - and please reply if it didn't, let's figure out how to make it work consistently and well - click on Sketch > Upload to flash the new core and sketch to your Oak.

As mentioned, I was also (finally) able to load files directly to SPIFFS using the Sketch Data Upload tool. See https://github.com/esp8266/arduino-esp8266fs-plugin.

Anything pertinent that I've missed?

Now, on Serial over OTA:

Lovely, once you get it to work.

Notes thus far:

• Basic information/install info at http://esp8266.github.io/Arduino/versions/2.0.0/doc/ota_updates/ota_updates.html
• I've tried both Classic OTA and ArduinoOTA and don't know what the relative advantages are - but the former is deprecated; I'd go with the latter
• Find and load BasicOTA.ino from Sketch > Examples > ArduinoOTA
• For ssid and pass use the name and password for your local network
• If there's any chance of your Oak getting hacked (its WiFi range is respectable) I would also uncomment and set ArduinoOTA.setPassword((const char *)"somethingunique") to something unique. Not too hard, as you'll need to enter it for every OTA update
• If your Oak connects to the network, you'll see something like
       Ready
       IP address: 192.168.2.18
• Potential problem: you look under Tools > Port, and the expected OTA port does not show. From the command line, ping your IP - this should work. If it does, the problem is probably mDNS packets - either your router's blocking them, or your PC isn't processing them. On Windows systems, the problem is often that the Mac/Apple-specific Bonjour service, used for mDNS messaging, is not installed. Use Task Manager or msconfig to check that it is running. For me, downloading and installing Bonjour Print Services for Windows, then restarting the Arduino IDE, did the trick. (Also, please note that Bonjour may be blocked by some anti-viruses; firewalls will need to have UDP 5353 open for it to work properly)
• Don't worry if you don't see an “Upload Using: OTA” option: I've found that selecting the esp8266-xxxxxx network port is sufficient
• You may see a "missing 'runtime.pwd.192.168.2.18' configuration parameter" error message initially - my sketches upload fine!
• Make sure pin 2 is not connected to ground for an OTA upload - the Oak has to be up and running in program mode
• Finally, unless you embed the OTA update code into your new sketch, you'll get a "[ERROR]: No Answer" message on your next upload - the OTA protocol has to be installed to be ready for it

One thing I haven't figured out yet - whether I can get Serial.print feedback back to me, over the air? Anyone?

So - feedback very welcome. Other problems/solutions?

Paul

[exeng]

Paul,

Thanks for this post and all the effort and detail that you obviously put into it. Also, thanks to Brendan (nog3) for his description of how to remove Oak from Particle.

While I have experienced the occasional fail OTAs and misrepresented fail/success of OTA's, I'm still able (after multiple retries) to get my Oak sketches OTA flashed. At a point where this becomes even less stable, I'll will most likely give this a go. I'm not dissing Particle but I never use their dev environment and only post a few bits of data there from my operational Oaks. Largely, this is due to the lack of (not sure if this is still true) a means to log data. I'm a fan of Thinkspeak. Others may have their personal favorites but I find the ability to keep historical data useful.

So again, thanks for bringing this solution to our attention.

exeng

[PeterF]

Paul, I would also like to thank you for the writeup

Like exeng, I'm still using Particle on my Oaks (bar one, a ESP8266 Arduino OTA guinea pig), but haven't programmed them in a while, so could quite quickly decide to jump ship should they start being 'lil s... ugars I would like to add the following:

* Arduino OTA is indeed FAST! You just have to remember to add the OTA.handle function, and give it the chance to do it's thing... it won't be 'always watching' like the Oak Particle firmware, unless you program it that way.

* True serial on the Oak was never limited AFAIK, just the Particle Serial implementation. You could always Serial.print() and monitor via a Serial UART, but it just wasn't as convenient.

* Unfortunately I don't think they've implemented any form of Serial over WiFi, so you won't be able to get that back in a simple fashion. Will probably have use some form of logging if you want that functionality, else, stick with hard-wired serial.

* Newer version of the docs (2.3.0 vs 2.0.0) is here (annoyingly the 'other versions' link at the top never worked, so you have to specifically look for the matching documentation for the current release) => http://esp8266.github.io/Arduino/versions/2.3.0/doc/ota_updates/readme.html

[postuma]

Good observations - thanks for the additions!

[DrJFM]

I also noted Erik's Kickstarter update acknowledging the issues with Particle.  I also found that after a 2 to 3 month break from Oak use, I could not even log in to Particle Dashboard.  Rather than spend time trying to recover my account at Particle, I decided to follow Brendan 'Nog3' Halliday's tutorial.

To his credit, Erik has acknowledged these problems - see https://www.kickstarter.com/projects/digistump/oak-by-digistump-wi-fi-for-all-things-arduino-comp/posts/1912567. He's also put a big disclaimer on the Digistump order page - caveat emptor. And I have no doubt he will address these issues in some way, at some time. But family comes first! and I've no doubt he's way over-extended (very excited about the BuildOne printer, BTW).

You'll need a USB to TTL adapter to program the Oak using a serial interface, found el cheapo on Ebay: http://www.ebay.com/itm/USB-to-TTL-CH340G-Converter-Module-Adapter-STC-Dowanloader-Programmer-3-3V-5V-/112215846564.

Follow the Brendan 'nog3' Halliday tutorial at http://nog3.net/2017/06/05/removing-particle-from-oak/.

After a moderate amount of extra time getting a never previously powered up Oak turned into a ESP8266Oak, I decided to pop in here and see if others were also making the transition. Wish I had looked in here sooner -- might have saved some time! That said, I really just wanted to encourage others to make the switch.  It is really not bad.  If you have ever done a serial flash to restore the Oak, this is easier.  Paul's further hints, directions and clarifications address some of the few issues that slowed the process for me and should help assure success. 

I will likely move to the OTA loading just to say I have, but I see myself more than content to use the serial upload process for most of my hobby projects.  I have been most satisfied using Blynk to monitor and control my Oaks and spent only about 1 hr (in part because I have not been using Arduino/Oak much lately) porting my Temp Humidity monitor to my newly configured ESP8266Oak.  One of my older registered Oaks had been updating my Blynk app w Temp/RH data for a couple months, most likely with absolutely no Particle connection.  I moved the level shifter shield to the ESP8266Oak and deleted a bunch of Particle overhead.  A couple tweaks and back to knowing when my dehumidifier needed to be emptied using my Blynk Android app.

So some bottom line thought:

Do it, it is not a hard process
There is a robust ESP8266 community that has likely solved most of your future problems
Erik sounded to me like any Cloud server/services he will develop will be based on the ESP8266 Core
It is great to type in a new SSID and Password and upload over serial in about 30 sec
As long as the ESP8266Oak can easily transmit over WiFi, there are several non-particle systems to allow remote interactions

I did a fresh install of Arduino 1.8.3 before I started.  The ESP8266 core is loaded by the Board Manager after adding in the link as described. No python flash needed!  I will likely convert my last "new" Oak and maybe some of the registered ones without worrying about the python backup -- just the erase!  The ESP8266 Core I installed was 2.3.0.  In my Windows 10 system, the files from the board manager end up in my AppData folder ie
C:\Users\UserName\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.3.0\variants   -not in my sketch folder. For anyone using Blynk, I added the most current Blynk Arduino library, which includes BlynkESP8266 version, not the one in the Oak download. I backed the Blynk Kickstart campaign so have lots of juice (ie $) on Blynk but would likely pay for the service as a very easy way to get data and control to you phone.

Finally, should we see if they will set up a ESP8266Oak branch of the forum?
Cheers. 
James

[wickwire]

Hi,


I was a backer for the Oak by Digistump kickstarter campaign, when it happened. I had already bought a Spark Core (now I guess it's called Particle Core) and hadn't done much with it, and for the price tag, once I found out about the Oak, I was on board.

I followed Erik's journey with the Oak and received my order when the time came: a bunch of Oaks, a bunch of Acorns, LCD displays, batteries, 3.3v relays, you name it.

By then, I was already fiddling with ESP8266-12 and ESP8266-01 modules and nodeMCU, which in spite of not having the cloud infra-structure set up, were really cheap to buy and as entertaining and a learning tool as any.

So I tried one Oak, had some troubles connecting it to the cloud, managed to get it going and since I already had stuff going on using MQTT with SSL on the ESP8266, was also going for that - which then lead to this github issue:

https://github.com/digistump/OakCore/issues/71

The Oak was set aside once more, I continued using the ESP8266-12 and nodeMCU, and then this week decided to give the Oak another shot.

After about an hour of failing to set it up with the Particle Cloud, I found an email message from Erik mentioning some possible issues with the particle registration process and the available alternative firmwares that could be loaded on the Oak.

The email contained nog3's instructions on how to set up the ESP8266 Arduino firmware, pretty much like the initial post in this thread, so I set everything up - since I also wanted to try out something other than nodeMCU.

I have to say that the ESP8266 Arduino project on the Oak has been a very positive experience so far, I got my Oak to register and publish timed messages to an MQTT broker online with SSL by updating the new sketch over the air fairly quickly and I'm willing to dedicate more of my time to this alternative. This is something that not even the original Oak firmware has allowed me to do, so far.

I consider myself to be an electronics newbie since my background is in app software development, however I do have an ESP8266 currently controlling on demand power sockets using solid state relays, also with MQTT and SSL but on nodeMCU - and it seems to me that sooner or later I'll be updating that setup code-wise by using ESP8266 Arduino + PubSubClient instead.

That said, and getting back to the Oak and Acorn, would it be fair to say that when compared to the ESP8266-12 dev kit or generic ESP8266-12 module respectively, it would just be extra cash for no real added value at the moment?

I must admit that I didn't check the differences between both hardware implementations, so please feel free to correct me on this and if you also feel that in spite of the noted issues with the cloud platform, this solution is still worth the money extra over the ESP, please note that too.

Finally, I'd like to mention that regardless of any possible issues with the OTA infra-structure with the Oak at the moment, I believe that it will get sorted out with time, as well.

[PeterF]

Yeah, because of the age of the ESP8266 core and because we don't have reference point to determine where exactly Erik branched off the ESP8266 Arduino core (so we could work out how to bring as much of it as possible up to date), things can and will not work properly, and we just have to live with it.

Having said that, I'm liking the ESP8266 Arduino OTA stuff more and more, especially since it is actually faster than doing serial uploads... which I thought were reasonably fast already.

As far as hardware, all I can say is the Oak is a solid and reliable board. Beyond that, you need to make your own determination. The regulator is well spec'd, and is reliable. The ESP8266 ("Acorn") is also well made, and even though I haven't looked under the can, I suspect it is better quality than the cheap ESP-12F boards you can find online. You just need to work out if you want to spend ~US$11 for a single Oak (they do get cheaper if you want quantities) or ~US$7 for a D1 WeMos from eBay or (my preference) the ESP8266 Witty modules for ~US$4 on eBay. The reason I like the latter is not because they're cheaper, but because they have the option of the USB serial or not. So you have it when you want to program it, and when you're finished, or need it back in your project, you just remove the module, and there's no power loss from unused USB serial circuitry (which is why I don't like the nodemcu boards for low power ESP8266 stuff). This is where the Oak really shines... if you have it in deep sleep mode, it draws next to no power... as the regulator is a good low quiescent current one, and there is no extra stuff to draw power (you just cut the trace for the power led as provided for), and it's as low as you can get it.

[wickwire]

I guess you are right, I have no hardware related complaints whatsoever on the Oak so far, whereas I've had some stresses with ESP8266-12 modules in the past. Granted, I fiddled a lot more with the ESP8266-12 than the Oak, so I guess it will still take me some time, effort and acquired knowledge to properly compare the two solutions.

That said, I'm very pleased with ESP8266 Arduino on the Oak, it's just like you said, it will even be slightly smaller than my ESP8266-12 + GPIO expansion boards and includes micro-USB for power, so having bought 5 Oaks and 10 Acorns on the kickstarter campaign, in no way I feel that the Oaks and Acorns will be shelved for long. Would I have to rely solely on the Particle Cloud and original Oak firmware though, I guess that my happen.

As for buying more modules, for now I'm good working with the ones I still have and will take your comments into account as well when the time comes where I'll have to choose which solution to go for.

I'm just a hobbyist wanting to have some fun with some DIY projects and if I could spend more of my time working on my designs and less time maintaining the modules themselves, that would be the best outcome for me - and by the looks of it, right now it looks like money aside, the Oak + ESP8266 Arduino seems to accomplish that just fine

[PeterF]

Sorry, I submitted my post before I was finished, so I've finished it and reposted the edit above.

[kathblair]

I wanted to say thanks for this info. I  had claimed some oaks a while ago (successfully) on the particle cloud, but couldn't use any libraries. This info let me swap to generic ESP8266, which was awesome!

I also wanted to add a note about the OTA updating - I couldn't see my port, and it turned out that I had to enable IPv6 in my router to be able to see the mDNS sent out by the oak. Before I did that, I couldn't see it even with Bonjour Browser, and afterwards I can and it works to upload code - haven't tried adding additional code with the OTA sketch, but baby steps.

I was also able to upload binaries with the Example > ESP8266WebServer > WebUpdate example, even without mDNS working, if I went to [oak IP]/update and used exported bins, so that would have worked if I couldn't have updated router settings.

Just to avoid having to power on/off the oak when uploading new code, but eh, it's less confusing. I still have my serial port connected for debugging, but I don't need to futz with P2 low/high.

[exeng]

It appears that the server hosting nog3's blog which describes how to remove the Oak from Particle is and has been down for awhile. I've reached out to nog3 via PM but no answer. Does anyone have a copy of his or your own instruction for switching the Oak to generic ESP8266?  My Oaks are all down and have continually failed to accept any Particle OTA updates. I'm ready to make the switch and abandon any Particle support.

Thanks,
exeng

[JeffRand]

It appears that the server hosting nog3's blog which describes how to remove the Oak from Particle is and has been down for awhile. I've reached out to nog3 via PM but no answer. Does anyone have a copy of his or your own instruction for switching the Oak to generic ESP8266?  My Oaks are all down and have continually failed to accept any Particle OTA updates. I'm ready to make the switch and abandon any Particle support.

Thanks,
exeng

The Wayback Machine has the page archived. Here:
https://web.archive.org/web/20180223211918/http://nog3.net/2017/06/05/removing-particle-from-oak

[exeng]

Jeff,

You are awesome. Thanks for the quick response and for providing a working link. The old link is referenced in many places, including an old update by Erik on Kickstarter.  Now there is a valid working link to this for anyone else that may be looking.

exeng

[victagayun]

You may use this HW connections to use UART https://arduino-esp8266.readthedocs.io/en/2.5.0-beta2/boards.html#minimal-hardware-setup-for-bootloading-and-usage and still program and erase the OAK.
This will be like any normal ESP8266.

[exeng]

Finally found the time to convert one of my Oaks to using generic ESP8266 code with serial and OTA updates using nog3's archived blog.
Link: https://web.archive.org/web/20180223211918/http://nog3.net/2017/06/05/removing-particle-from-oak

Thank you JeffRand (again) for posting that. I can now upload via serial and OTA reliably.  Goodbye Particle.  Still have four more Oaks to convert that were in service but eventually failed to take new uploads via Particle.  Looking forward to not wasting anymore time hoping that a Particle upload will eventually work.

Another useful reference (which may be linked in the blog) was this one: https://arduino-esp8266.readthedocs.io/en/latest/ota_updates/readme.html#arduino-ide

exeng