Apps Reading a large buffer

[theFestival]

What I'm trying to do is connect to a website and store everything that comes back using this code:

StringBuilder sb = new StringBuilder("");

URLConnection conn = url.openConnection();
        BufferedReader rd = new BufferedReader(new InputStreamReader
                (conn.getInputStream()));
String line = "";
while ((line = rd.readLine()) != null) {    
       sb.append(line);
}

However, the stringbuilder is obviously not big enough because it cuts off quite early and before the information I need. Is there another way to store large buffers?

Thanks.

 

[Boogs]

I would say that you either need a custom class, or you are going to have write that information to a file on the local disk and then process it. I solved a similar problem in one of my programs by doing the latter because I needed to perhaps read more than 16 megabytes of information-which will cause an out of memory error.

 

[theFestival]

I would say that you either need a custom class, or you are going to have write that information to a file on the local disk and then process it. I solved a similar problem in one of my programs by doing the latter because I needed to perhaps read more than 16 megabytes of information-which will cause an out of memory error.

Ok, i was thinking among those lines. Thank you.

 

[punky]

What about creating an ArrayList of StringBuilder/String objects and when you've reaching the StringBuilder maximum size, create another object and append it to the ArrayList?

 

[theFestival]

What about creating an ArrayList of StringBuilder/String objects and when you've reaching the StringBuilder maximum size, create another object and append it to the ArrayList?

I thought about that as well. But wouldn't that take up more more memory in the limited amount the droid has? I'm not sure what's more taxing, parsing big objects or files.

 

[punky]

I thought about that as well. But wouldn't that take up more more memory in the limited amount the droid has? I'm not sure what's more taxing, parsing big objects or files.

Well it depends on how big the webpage you are reading. What it sounds like you have come up against is a size limit for either the String type or a StringBuilder object (which is just a wrapper for a String really)

The sizes shouldn't be unhandlable though in memory.

The AF front page for example has 168,248 chars which is 329kbytes (in unicode) in total. So to hit the 16mbyte limit that Boogs has it, it has to be a massive webpage.

I'm presuming the 16mbyte limit is the JIT compiler stopping programs from crashing the phone due to a memory leak.

 

[cp1]

Do you know where the part you need is? You can discard all the lines that don't have what you want.

 

[theFestival]

Do you know where the part you need is? You can discard all the lines that don't have what you want.

Ok i've figured it out that what I'm looking for is 2146 chars in (not counting spaces). So size shouldn't be an issue. I outputted what i'm getting from logcat to a text file and did a manual search and I could not find what i'm looking for, this is my problem now.

String line;
line = rd.readLine().toString();

I'm guessing that readLine will only read a certain number of characters, in fact its actually ignoring large chunks of text and this is my problem. Is there an alternative to readLine?

URLConnection conn = url.openConnection();
        BufferedReader rd = new BufferedReader(new InputStreamReader
                (conn.getInputStream()));

is not going to work. Anyone know any alternatives to parsing through HTML? Also are there legal implications to searching through html and displaying certain variables?

Thank you all for your help so far.

 

[Scottmc]

I just wanted to mention that when writing software for embedded systems you usually end up doing much more work. One person suggested creating an array of Strings and storing them there. I have to point out that you have very limited RAM and when developing for mobile (or embedded systems) it's important to limit the usage of that valuable resource. Using it to store temporary data you read from a file is not a really good use of that resource. If this is not a time critical requirement I suggest you just stream your data directly to disk. Yes it's more code to write but 30 lines of code takes up way less room than 16MB of string data. Just my observations after spending a better part of my career in embedded development (8+ years).

As for screen scraping HTML - there is no legal precedence against it as far as I know. That's not to say that there aren't moral consequences. You have to measure the purpose (and method) of getting those values against those moral ideals. Unless you don't care then, by all means, scrape away! As far as I know, if it's plain text and publicly facing you shouldn't have any issues displaying those values (they are usually available by viewing a web pages source anyway).

 

[cp1]

I haven't used a buffered reader, so I'm not sure why readline seems to skip things. read() should just read in 1 character, so you can do that however many times you need and then start saving the characters to a string.