Author Topic: Memory Usage on the PRO  (Read 8178 times)

defragster

  • Sr. Member
  • ****
  • Posts: 467
Memory Usage on the PRO
« on: January 18, 2015, 08:52:44 pm »
RAM is short at 512 bytes.  You get a feel for sketch usage at compile time.  When I've see mine get near 300 bytes things start dropping out - like F() strings stop working.  The 'system' uses other stuff in the background - but it isn't made clear anywhere that I have seen. 

For instance I assume the F() code allocates a stack buffer from the system pool and that may be why it fails.  The libraries involved add up in the compile time number - but there has to be some other expectation after boot that isn't shown.  If anyone has seen a way to track that it would be nice to be able to know. 

I found a RAM MemoryFree library - it just seemed to match the number shown at compile time, but I didn't work much with it.  Making this new thread for Tracking purposes. 

The library I found came from here according to the readme: http://github.com/ribbotson/USB-Host/tree/master/Memory_Free/

defragster

  • Sr. Member
  • ****
  • Posts: 467
Re: Memory Usage on the PRO
« Reply #1 on: January 18, 2015, 09:35:21 pm »
I found these web topics - wondering about trying dynamic allocations versus growing user RAM and recompiling to failure:

This Arduino site details getting RAM specs beyond what the IDE exposes:
http://playground.arduino.cc/Main/CorruptArrayVariablesAndMemory

Available Memory  The microcontrollers used on Arduino boards have a small amount of RAM. Running out of RAM due to the complexity of the sketch or a memory leak may be the cause of bugs. Here are a few options for finding the amount of free RAM in your Arduino.

MemoryFree Library
http://playground.arduino.cc/Code/AvailableMemory

Some other pointers can be found on: Issue 40 - arduino - SRAM Memory size check:
http://code.google.com/p/arduino/issues/detail?id=40

« Last Edit: January 18, 2015, 09:37:49 pm by defragster »

digi_guy

  • Jr. Member
  • **
  • Posts: 87
Re: Memory Usage on the PRO
« Reply #2 on: January 19, 2015, 10:09:30 am »
Thanks for that. I am definitely running out of ram, and always at the worst time. 

defragster

  • Sr. Member
  • ****
  • Posts: 467
Re: Memory Usage on the PRO
« Reply #3 on: January 19, 2015, 08:07:39 pm »
I keep running up against it too - Please post if you find a tool that helps you track it.  Sometime you can't do much about it - but at least knowing why or when it is bad can help.

digi_guy

  • Jr. Member
  • **
  • Posts: 87
Re: Memory Usage on the PRO
« Reply #4 on: February 06, 2015, 10:53:11 pm »
I've been playing around lately with sha-1 hashes and base64 encoding.

Seemed like things were going really well, and I managed to generate hash code for "abc"
Code: [Select]
int  hash_output[20] = {169,153,62,54,71,6,129,106,186,62,37,113,120,80,194,108,156,208,216,157}'
to convert that into base64 I need to take 6 bits at a time to get a number, and from that use
char b64chars[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
to get a code.

I've found it easiest to work from back to front, and I've been able to iterate 6 times but then it all goes to shit.

At this point I'm stumped. Any thoughts how how to break this up so that I can eventually get
"qZk+NkcGgWq6PiVxeFDCbJzQ2J0="



defragster

  • Sr. Member
  • ****
  • Posts: 467
Re: Memory Usage on the PRO
« Reply #5 on: February 07, 2015, 03:16:07 pm »
Show your work? Which vars are in memory? 

These two tips may free 84 bytes of RAM:

This in Ram would be large: " char b64chars[] "
Code: [Select]
const char b64chars[] PROGMEM = ". . . ";
These are small numbers for int - maybe
Code: [Select]
"uint8_t  hash_output[20]"
Since you are already working back to front - you could put the output at the end of your buffer working forward and use a single buffer if you don't mind destroying the input as you go.

digi_guy

  • Jr. Member
  • **
  • Posts: 87
Re: Memory Usage on the PRO
« Reply #6 on: February 07, 2015, 06:33:06 pm »
Here is the current code I'm working with:
Code: [Select]
#include "sha1.h"

// goal message is qZk+NkcGgWq6PiVxeFDCbJzQ2J0=
//  unsigned long hash[]={169, 153, 62, 54, 71, 6, 129, 106, 186, 62, 37, 113, 120, 80, 194, 108, 156, 20, 8, 157, 0};
char b64chars[] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
unsigned long hash_msg;
char output_message[28];
int shift;

void setup() {

  Serial.begin(9600);
  Sha1.init();
  Sha1.print("abc");
  uint8_t* hash = Sha1.result();

  int hash_counter = 0;
  int output_counter = 0;
  for (int i =0; i<7; i++) {
    hash_msg=0;
    for (int j=0; j<3; j++) {
      hash_msg = hash_msg<<8 | hash[hash_counter];
      hash_counter++;
    }
    for (int j=0; j<4; j++) {
      shift = 18-6*j;
      int temp_msg = hash_msg>>shift & 63;
      output_message[output_counter] = b64chars[temp_msg];
      output_counter++;
    }
  }
  output_message[27]='=';
 
  Serial.println(); 
  Serial.println(output_message );
 
}


void loop() {
}

The sha1 function generates 20 x 8 bit words that look like
Code: [Select]
hash[]={169, 153, 62, 54, 71, 6, 129, 106, 186, 62, 37, 113, 120, 80, 194, 108, 156, 20, 8, 157};
The conversion to base64 requires taking 6 bits at a time, which generates an integer {0 ->63} which translate into the letters
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"

For example, the first 3 numbers from the hash is '0b101010011001100100111110' which makes 4 integers [42, 25, 36, 62] which then ultimately becomes the letters qZk+NkcGgWq6PiVxeFDCbJzQ2J0=

I'm going to try deleting the hash integers as I go and see if that frees things up. What sucks is that the conversion takes an array of size 20 and makes it 27.

defragster

  • Sr. Member
  • ****
  • Posts: 467
Re: Memory Usage on the PRO
« Reply #7 on: February 08, 2015, 03:54:28 am »
Thanks, helps seeing the code - though I didn't look at the sha1 code at all so that is still guessing.

Doing this 'const char b64chars' should free that from RAM.

It looks like the code returns ' uint8_t* hash' - so it is only using the needed memory - as long as it is always the fixed 20byte length.

Since it returns '* hash' - the problem may be it is not freeing it between each iteration.  Not sure if you can terminate the sha.init(), or if there is a supported call you need to make to free the RAM.  Compare the * values between calls and if they are the same that isn't it.  The freed RAM from 'const' above might get you another 2-3 iterations - not a fix but would indicate the suspected memory leakage.  You can also see if the MemoryFree noted below gives a changing value between calls.  It may be trying to free the RAM but getting fragmented or not garbage collected - as I recall when I was looking and making the post below it suggested ALLOC wasn't good at this.

Since it allocates the buffer you cannot 'over allocate' as needed to account for the growth.  If you can supply your own buffer you can prevent the leakage and add the bytes needed for growth as it expands from 20 bytes during conversion.

How much RAM is indicated as used on the IDE compile?

Also just noticed you have all your code in setup() - I'm not sure what happens in the single task Arduino world after setup() returns and before loop() - or between subsequent calls to loop().  Try to split your existing code (with const on the b64chars[]) so that the init() and static stuff is done in setup() - and one conversion pass is done each loop.  This code looks like a single iteration?  Not sure what you do differently when you make more passes?

defragster

  • Sr. Member
  • ****
  • Posts: 467
Re: Memory Usage on the PRO
« Reply #8 on: February 10, 2015, 02:52:24 am »
@digi_guy : where did you get the sha1.h file?  I don't have a Pro to test on just now but was wondering.

I found this page :
http://forum.arduino.cc/index.php?topic=119848.0

They have a static buffer in the class code - their examples also run fully in the setup(), but the code seems different, and are FAT with print text.

digi_guy

  • Jr. Member
  • **
  • Posts: 87
Re: Memory Usage on the PRO
« Reply #9 on: February 11, 2015, 04:18:23 pm »
I was using the crypto suit from Adafruit
https://github.com/Adafruit/Cryptosuite

and I just got finished writing my own version, with built in hex conversion, but haven't had a chance to test it. I need to spend a bit more time understanding now PROGMEM works, since I'm pretty sure that's the only way I'm going to get this to work.

To answer your other question, things were in setup() while I was debugging. Having stuff in the main loop got annoying.