Out of memory?

General questions about NeoBook

Moderator: Neosoft Support

Locked
User avatar
paeon09
Posts: 8
Joined: Sun Sep 23, 2012 5:54 am

Out of memory?

Post by paeon09 » Mon Aug 27, 2018 8:05 am

Hi All

I am developing a web parsing program and all is working well until I have more than
35 urls to download, this data changes daily so is needed to be parsed daily, each url
generates between 330k - 800k in file size.

It first downloads a list of urls to be parsed and then steps through the list
downloading each web page into a browser object then export to variable that
is then saved to a file, which is then parsed into a Db, as I said, this all works great until it runs out of memory.

I am using a timer object combined with the browsers navigate complete property
setting a delay of 3 secs just to make sure the page is downloaded.

I have check with the task manager and watch as each browser goto url increases the
memory used by about 50k, I have cleared variables and deleted arrays as it steps through
the urls but makes no difference, I also use pc-probe to check physical memory and that
percentage also increases.

I also clear the cache every 10 urls.

Is there a way to release memory used by the browser? or am I missing something?

any help greatly appreciated

Tony Kroos
Posts: 419
Joined: Thu Oct 15, 2009 3:43 pm

Re: Out of memory?

Post by Tony Kroos » Mon Aug 27, 2018 1:00 pm

And why are you using WebBrowser object instead of InternetGet ?

User avatar
paeon09
Posts: 8
Joined: Sun Sep 23, 2012 5:54 am

Re: Out of memory?

Post by paeon09 » Tue Aug 28, 2018 1:34 am

Hi Tony

I was using web browser because I am not sure what headers are sent with InternetGet, some
sites block programs (bots) that do not send correct headers from a web browser or simulated web browser,
is it possible send headers with InternetGet? or is there a better way?

EDIT

Found we can use [HTTPAgent] with InternetGet and check download with [DownloadProgress].

many thanks for the pointer.

:oops:

Tony Kroos
Posts: 419
Joined: Thu Oct 15, 2009 3:43 pm

Re: Out of memory?

Post by Tony Kroos » Tue Aug 28, 2018 8:06 am

I rarely reboot my laptop and my Chrome tabs can go above 500mb memory consumption after few days of surfing.
So it's not neobook problem but IE engine used by WebBrowser. The only way is to destroy and create fresh WebBrowser object but it's impossible in runtime.

User avatar
paeon09
Posts: 8
Joined: Sun Sep 23, 2012 5:54 am

Re: Out of memory?

Post by paeon09 » Tue Aug 28, 2018 8:53 am

For anyone looking for User Agent Strings::

http://www.useragentstring.com/pages/us ... string.php

thanks again Tony

Locked