Out of memory?
Posted: Mon Aug 27, 2018 8:05 am
Hi All
I am developing a web parsing program and all is working well until I have more than
35 urls to download, this data changes daily so is needed to be parsed daily, each url
generates between 330k - 800k in file size.
It first downloads a list of urls to be parsed and then steps through the list
downloading each web page into a browser object then export to variable that
is then saved to a file, which is then parsed into a Db, as I said, this all works great until it runs out of memory.
I am using a timer object combined with the browsers navigate complete property
setting a delay of 3 secs just to make sure the page is downloaded.
I have check with the task manager and watch as each browser goto url increases the
memory used by about 50k, I have cleared variables and deleted arrays as it steps through
the urls but makes no difference, I also use pc-probe to check physical memory and that
percentage also increases.
I also clear the cache every 10 urls.
Is there a way to release memory used by the browser? or am I missing something?
any help greatly appreciated
I am developing a web parsing program and all is working well until I have more than
35 urls to download, this data changes daily so is needed to be parsed daily, each url
generates between 330k - 800k in file size.
It first downloads a list of urls to be parsed and then steps through the list
downloading each web page into a browser object then export to variable that
is then saved to a file, which is then parsed into a Db, as I said, this all works great until it runs out of memory.
I am using a timer object combined with the browsers navigate complete property
setting a delay of 3 secs just to make sure the page is downloaded.
I have check with the task manager and watch as each browser goto url increases the
memory used by about 50k, I have cleared variables and deleted arrays as it steps through
the urls but makes no difference, I also use pc-probe to check physical memory and that
percentage also increases.
I also clear the cache every 10 urls.
Is there a way to release memory used by the browser? or am I missing something?
any help greatly appreciated