Out of memory?

[paeon09]

Hi All

I am developing a web parsing program and all is working well until I have more than
35 urls to download, this data changes daily so is needed to be parsed daily, each url
generates between 330k - 800k in file size.

It first downloads a list of urls to be parsed and then steps through the list
downloading each web page into a browser object then export to variable that
is then saved to a file, which is then parsed into a Db, as I said, this all works great until it runs out of memory.

I am using a timer object combined with the browsers navigate complete property
setting a delay of 3 secs just to make sure the page is downloaded.

I have check with the task manager and watch as each browser goto url increases the
memory used by about 50k, I have cleared variables and deleted arrays as it steps through
the urls but makes no difference, I also use pc-probe to check physical memory and that
percentage also increases.

I also clear the cache every 10 urls.

Is there a way to release memory used by the browser? or am I missing something?

any help greatly appreciated

[Tony Kroos]

And why are you using WebBrowser object instead of InternetGet ?

[paeon09]

Hi Tony

I was using web browser because I am not sure what headers are sent with InternetGet, some
sites block programs (bots) that do not send correct headers from a web browser or simulated web browser,
is it possible send headers with InternetGet? or is there a better way?

EDIT

Found we can use [HTTPAgent] with InternetGet and check download with [DownloadProgress].

many thanks for the pointer.

[Tony Kroos]

I rarely reboot my laptop and my Chrome tabs can go above 500mb memory consumption after few days of surfing.
So it's not neobook problem but IE engine used by WebBrowser. The only way is to destroy and create fresh WebBrowser object but it's impossible in runtime.

[paeon09]

For anyone looking for User Agent Strings::

http://www.useragentstring.com/pages/us ... string.php

thanks again Tony