Exception Decoding - Any Guidance Notes Available?

[Nevyn]

Are there any guides on working out the cause of an exception within an application?

I have an application that is taking readings from sensors every minute.  The application then posts the results to the Internet and also throws out a fair amount of debug information to the serial port.

I've read through the code several times and I'm confident that I'm deleting any objects I'm making after I've finished with them.  The app does make heavy use of the String class, I'm hoping that's not too much of an issue.

One saving grace is that the application does seem to reset the board after a few minutes.

Regards,
Mark

[PeterF]

I can't see that I have seen anything so far...

So by exception does the Oak actually throw an exception and do some sort of memory dump, or does it just freeze for a while, and then reboot? It might be worth throwing a few yield() or delay(1) statements through out your code, to see if it is actually the particle connect stuff having a hissy fit because it's been unable to maintain a connection. 

Also, another thing to be aware of is code that makes it so the main loop function takes longer than 1 second (I think it's around a second, it could be longer) to run, without inserting any yield() or delay(1)'s, the ESP8266's hardware watchdog may fire, triggering a reset. That doesn't sound like it's your problem, but it could be related.

You're welcome to PM me code if you can't publicly show it, I may just take a day or two to get back do you due to other commitments. I can't really see the String objects being an issue... the ESP8266/Oak has a lot more memory than your average Arduino!

[Nevyn]

The system halts with some diagnostic information and a stack trace.  This is the one that has just happened:

Code: [Select]

05-09-2016 10:29:04: Ground temperature:
Exception (29):
epc1=0x40209c16 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

ctx: cont
sp: 3fff4e90 end: 3fff51c0 offset: 01a0

>>>stack>>>
3fff5030:  00000000 00000000 3fff50a0 4020afcc 
3fff5040:  00000000 00000000 3fff50a0 4020b2fa 
3fff5050:  3ffed5ac 00000000 3fff50a0 4020acc0 
3fff5060:  00000000 00000000 3fff5114 40215ec6 
3fff5070:  41ac6666 428a7100 3fff5114 3fff4190 
3fff5080:  41ac6666 428a7100 3fff3f60 4020b392 
3fff5090:  41ac6666 428a7100 3fff3f60 4020850a 
3fff50a0:  00000000 00000000 00000000 00000000 
3fff50b0:  00000000 3f000050 00000000 00000000 
3fff50c0:  00000000 3fff4000 00000000 00000000 
3fff50d0:  00000000 3fffbc78 0000000f 00000000 
3fff50e0:  00000000 00000000 00000000 00000000 
3fff50f0:  00000000 00000000 ffffffff 40215200 
3fff5100:  00302e30 3f003731 00000001 402152c4 
3fff5110:  3fff5174 00000000 00000000 00000000 
3fff5120:  00000000 00000000 00000000 00000000 
3fff5130:  00000000 00000000 00000000 0000003f 
3fff5140:  0000003c 0000001f 00000013 40215dec 
3fff5150:  41ac6666 428a7100 3fff3f54 402089e5 
3fff5160:  00302e30 3fff4190 4020e738 3fff4190 
3fff5170:  3fffdc20 3fffbc78 0000001f 00000014 
3fff5180:  3fffba88 0000001f 00000012 40211a5e 
3fff5190:  3fffdc20 00000000 3fff4188 40208bb4 
3fff51a0:  4020e70a 00000000 3fff4188 4020e6fa 
3fff51b0:  00000000 00000000 3fff41a0 40100114 
<<<stack<<<
c_?RS?fJ?z??

05-09-2016 10:29:11: -----------------------------
05-09-2016 10:29:11: Weather Station Starting (version 0.18, built: 19:32:26 on Sep  4 2016)

The lines with the date/time at the start comes from my logging to the serial port.  As you can see, the system does reset, sometimes instantly as this one did, sometimes within a minute or two.

The code is freely available on Github (https://github.com/NevynUK/WeatherStation) under an MIT licence.

Regards,
Mark

[PeterF]

Hi Mark,

Ok, it looks like a straight out crash... and I have no idea how to interpret the stack trace dumps... thankfully the only time I encountered that (due to fiddling with something I shouldn't have) was resolved by completely overwriting the firmware via serial.

I've only had a cursory glance at the code, but I'm wondering if it always crashes *roughly* around the "Ground temperature:" message, meaning it is the WDT kicking in... might be worth adding a yield() or a delay(1) in your main WeatherStation.ino file... maybe between 232-233, and either side of 251. That's all that I can think of without some more investigation. Looks like a very, nice project though!

STM8... is that just an 8 bit version of the STM32s used on the Maple Mini-type board? Available as a DIP chip by any chance?

Pete

[Nevyn]

The location looks to be different each time (or at least unpredictable) and assuming that the number in brackets after the Exception is the exception number then I seem to be seeing one of three exceptions, 3, 9 and 29.

I'll try adding some yields and delays into the code to see if that helps.

Glad you like the project.  The Oak just seemed to fit the bill although my sensor list just exceeded the pin capacity of the Oak by one   The STM8 is going to be used to act as a low power timekeeper prompting the Oak into action, eventually.  The current version of the project uses the Oak exclusively and ignores the power issue and the UV sensor.  For those who are interested in why there's more on my latest blog post: http://blog.mark-stevens.co.uk/2016/09/define-minimum-viable-product-ship/

The STM8 is a nice low power processor but not related to the STM32s which are Cotrex-M processors.  I picked up on it from another community that I was an active member of but it has since become dormant.  I like it as it's cheap and only requires one external capacitor to get going.  There are some PDIP variants about but I use the 8K TSSOP20 variant.  ST do have a Discovery board available, in the UK this costs about £5 (http://uk.rs-online.com/web/p/processor-microcontroller-development-kits/7959640/).  This also include the programmer.  All of the development environments I've seen that are easy to set up and use are Windows environments though.  The IAR environment is limited to 8K but Cosmic have just released their environment unlimited for hobbyists.  Something for later investigation...

BTW, I'm using the 1.0.5 firmware and I'm only programming over serial.  OTA was a pain to get going with any degree of reliability.

I'll let you know how I get on with the yield etc.

Thanks for your time,
Mark

[kh]

Are there any guides on working out the cause of an exception within an application?

I haven't tried this, but it looks promising:

https://github.com/me-no-dev/EspExceptionDecoder

[Nevyn]

I've only had a cursory glance at the code, but I'm wondering if it always crashes *roughly* around the "Ground temperature:" message, meaning it is the WDT kicking in... might be worth adding a yield() or a delay(1) in your main WeatherStation.ino file... maybe between 232-233, and either side of 251. That's all that I can think of without some more investigation. Looks like a very, nice project though!

Pete,

So I've had chance to play with this a little more and after a little refactoring I added a yield to the DebugMessage calls as these will be scattered through out the code including in the main loop (via ReadAndPublishData).  I have also reduced the delay in the main loop fro 1s to 0.25s.

Now things are getting even more curious.  Every now and then the main loop itself stops being called but ironically, the interrupts are still being fired.

More debugging required I think,
Mark

[Nevyn]

Are there any guides on working out the cause of an exception within an application?

I haven't tried this, but it looks promising:

https://github.com/me-no-dev/EspExceptionDecoder

Had a look at this and received the message Not supported on Oak.

Regards,
Mark

[PeterF]

I just have it a shot on my lappy as i'm not at my usual machine atm, and I got the same 'Not supported for Oak' error, so tried it with a ESP8266 board selected, and it sort of worked... It seems to work by comparing the stack trace with the compiled code, so it probably isn't going to work unless we can get around the 'Not supported by Oak' error. Just for giggles, this is what it gave me for the stack dump you listed earlier, but I don't expect it to be of any use whatsoever since it's not even against the right code... but then again... would could be pleasantly surprised?

Interesting you say that some yields in the debug messages and changed delays, and the main loop stops responding, but interrupts keep firing... curioser and curioser!

Code: [Select]

Decoding 16 results
0x4020afcc: ipaddr_addr at ?? line ?
0x4020b2fa: ip_route at ?? line ?
0x4020acc0: etharp_query at ?? line ?
0x40215ec6: get_data_from_rtc at ?? line ?
0x4020b392: ip_router at ?? line ?
0x4020850a: node_remove_from_list at ?? line ?
0x40215200: phy_gpio_cfg at ?? line ?
0x402152c4: phy_gpio_cfg at ?? line ?
0x40215dec: phy_afterwake_set_rfoption at ?? line ?
0x402089e5: node_remove_from_list at ?? line ?
0x4020e738: tcp_write at ?? line ?
0x40211a5e: txpwr_offset at ?? line ?
0x40208bb4: node_remove_from_list at ?? line ?
0x4020e70a: tcp_write at ?? line ?
0x4020e6fa: tcp_write at ?? line ?
0x40100114: cont_norm at C:\Users\pfeer\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\2.2.0\cores\esp8266/cont.S line 109

[Nevyn]

I was thinking about trying this on a generic ESP module but then had a look at the source (for the exception decoder) and there were only two lines of code needed changing to get this going on the Oak.  Quick recompile and fed my latest exception through the application:

Code: [Select]

ecoding 14 results
0x4020d4d3: __static_initialization_and_destruction_0 at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/IPAddress.h line 84
:  (inlined by) _GLOBAL__sub_I__ZN7TwoWire8rxBufferE at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\libraries\Wire/Wire.cpp line 246
0x4020d4c4: __static_initialization_and_destruction_0 at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/IPAddress.h line 84
:  (inlined by) _GLOBAL__sub_I__ZN7TwoWire8rxBufferE at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\libraries\Wire/Wire.cpp line 246
0x4020d0a8: Adafruit_BME280::readPressure() at \\Mac\Home\SoftwareDevelopment\Arduino\libraries\Adafruit_BME280_Library/Adafruit_BME280.cpp line 333
0x40215c88: String::String(String const&) at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/WString.cpp line 46
0x4020d2cc: TwoWire::flush() at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\libraries\Wire/Wire.cpp line 240
0x40215aaa: String::changeBuffer(unsigned int) at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/WString.cpp line 156
0x40215b06: String::reserve(unsigned int) at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/WString.cpp line 146
0x40215b90: String::String(char const*) at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/WString.cpp line 41
0x40208a4d: Debugger::FloatToAscii(char*, double, int) at C:\Users\Mark\AppData\Local\Temp\build9022eab8f2eaaa03de3f1490a8667144.tmp\sketch/Debug.cpp line 117
0x402116d2: particle_core::spark_connect() at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak\OakParticle/particle_core.cpp line 298
0x40208cbd: WeatherSensors::WeatherSensors() at C:\Users\Mark\AppData\Local\Temp\build9022eab8f2eaaa03de3f1490a8667144.tmp\sketch/WeatherSensors.cpp line 624
0x4020e43a: HardwareSerial::write(unsigned char) at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/cbuf.h line 90
0x4020e42a: HardwareSerial::write(unsigned char) at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/cbuf.h line 90
0x40100114: cont_norm at C:\Users\Mark\AppData\Local\Arduino15\packages\digistump\hardware\oak\1.0.5\cores\oak/cont.S line 109

At least I now have somewhere to start looking.

Thanks for all the help,
Mark

[Last_Mile]

Nevyn, could you please post the changes you made to EspExceptionDecoder to get it to work for the Oak?

How did you recompile?  I tried replacing esp8266 with oak, but the make.sh file failed?

Thanks