Go Back   Android Forums > Android Development > Application Development
Application Development Dev lounge for our application developers.

Get excited for the Samsung Galaxy S5! Find everything you need and discuss it in our Galaxy S5 Forum!

test: Reply
 
LinkBack Thread Tools
Old February 14th, 2011, 09:15 AM   #1 (permalink)
New Member
Thread Author (OP)
 
Join Date: Feb 2011
Posts: 7
 
Device(s):
Carrier: Not Provided

Thanks: 1
Thanked 1 Time in 1 Post
Default SAX RSS streaming fine on Wifi not so good on 3G

Hello

I've checked everywhere for an answer to this but I just can't find it. I can't believe I'm the only one to have it.

I have written an RSS reader using SAX, like this

Code:
try
        {
            // setup the url
           URL url = new URL(urlToRssFeed);

           // create the factory
           SAXParserFactory factory = SAXParserFactory.newInstance();
           // create a parser
           SAXParser parser = factory.newSAXParser();

           // create the reader (scanner)
           XMLReader xmlreader = parser.getXMLReader();
           // instantiate our handler
           RSSHandler theRssHandler = new RSSHandler();
           // assign our handler
           xmlreader.setContentHandler(theRssHandler);
           // get our data through the url class
           InputSource is = new InputSource(url.openStream());
           // perform the synchronous parse           
           xmlreader.parse(is);
           // get the results - should be a fully populated RSSFeed instance, 
		   // or null on error
           return theRssHandler.getFeed();
        }
        catch (Exception ee)
        {
            // if you have a problem, simply return null
            return null;
        }

The RSS is sorted and stored in a database through the RSSHandler class.

The problem :
This works fine. Running under Wifi I get all the data back from the RSS. However, If I'm running using 3G and the signal strength is not so good, it returns maybe a few feeds from the RSS or in some cases no feeds at all.

The question :
Is there away to check is the entire RSS feed has been sorted and stored before it quits? I'm guessing that it loses connection during the RSSHandler class and so just returns what it's received, or maybe it's in the openStream(). Maybe even actually download the RSS to the cache and then use that?

Anyway any help would be appreciated.

Thanks
James

jimthechimp is offline  
Reply With Quote
sponsored links
Old February 16th, 2011, 01:27 AM   #2 (permalink)
Over Macho Grande?
 
alostpacket's Avatar
 
Join Date: Nov 2009
Location: NY
Posts: 7,873
 
Device(s): GlassXE, MotoX, N5, N4, N7'12, GNex, N1, SGT10.1, Revue, Xoom, Eris, OG Droid
Carrier: TMO

Thanks: 4,582
Thanked 3,563 Times in 1,522 Posts
Default

SAX is event based (or callback-based depending on the lingo you use)... so you need to show the code where you override startElement, characters, endElement.

If you want something to download the entire document I'd just use the DOMparser it's MUCH easier to code for and the reports of how slow it is are a bit exagerrated.


Actually, you probably know all that -- and now that I think about it, I think I found the error....

If you're having a problem where the callbacks are never getting called i'd consider buffering the input and making sure you understand the data flow from the URL to the XMLParser (see my code below)



your code went:

URL -> input stream -> SAX xml parse


what I did differently below was

URL -> input stream -> buffer -> reader -> SAX xml parse


Code:

     SAXParser sp = spf.newSAXParser();
     XMLReader xr = sp.getXMLReader();
     InputSource is = new InputSource();

     BufferedInputStream buffer = new BufferedInputStream( url.openStream() );
     InputStreamReader reader = new InputStreamReader( buffer );
     is.setCharacterStream( reader );
     xr.parse( is );
alostpacket is offline  
Reply With Quote
Old February 19th, 2011, 01:44 PM   #3 (permalink)
New Member
Thread Author (OP)
 
Join Date: Feb 2011
Posts: 7
 
Device(s):
Carrier: Not Provided

Thanks: 1
Thanked 1 Time in 1 Post
Default

Thanks alostpacket.

I actually did something similar to this before I read your reply. What I did differently was I read about HttpClient, that it's a better of connecting. so my code now looks like this. Apparently HttpClient trys to connect 3 times?

Code:
 try {
               HttpGet httpGet = new HttpGet(url);
               HttpClient httpclient = new DefaultHttpClient();
               HttpResponse response = httpclient.execute(httpGet);
               
               xmlreader.parse(new InputSource(new BufferedInputStream(response.getEntity().getContent())));
               } catch (Exception e) {
            	   Log.d("RSS Reader", "Network Error");
               }
It seems that this is working better than before, but it's not perfect.
Thanks for your help.
jimthechimp is offline  
Reply With Quote
Old February 27th, 2011, 09:36 AM   #4 (permalink)
Over Macho Grande?
 
alostpacket's Avatar
 
Join Date: Nov 2009
Location: NY
Posts: 7,873
 
Device(s): GlassXE, MotoX, N5, N4, N7'12, GNex, N1, SGT10.1, Revue, Xoom, Eris, OG Droid
Carrier: TMO

Thanks: 4,582
Thanked 3,563 Times in 1,522 Posts
Default

Did you ever get this figured out? Sorry for the late reply I was super busy working on BlueMuze
alostpacket is offline  
Reply With Quote
Old February 28th, 2011, 04:42 AM   #5 (permalink)
New Member
Thread Author (OP)
 
Join Date: Feb 2011
Posts: 7
 
Device(s):
Carrier: Not Provided

Thanks: 1
Thanked 1 Time in 1 Post
Default

Hi, I'm really not sure if I fixed this or not. I've tried your method and it seems to be working fine. I really need some help here because I'm running out of ideas.

The App, I've designed an app for the online Sim Racing community that reads in 4 RSS feeds from 4 different servers. Because some of them are not particularly well formatted I've had to read and store the data one by one in the SQLite database. I've released this onto the market as I wanted to test it. I have made the Sim Racing Community that there are errors and it needs testing so here's the basic structure of the code.

Start App
---runs inside AsyncTask
---loop through string of RSS feeds (4 times)
---parse RSS
---store RSS
---change to Feeds Activity
---inside thread to allow progress bar to display.
---load from db feeds into list.

The problem is that when I use it on my HTC Desire, it works, I've put catches in place so that if there are any download errors it tells you. I've even got it to work in areas where my reception has been less than perfect. the problem is that some people are simply complaining of never getting any results back, even on WiFi. Of course these people don't give proper descriptions of what errors are recieved etc. so it's very difficult to diagnose.

ok, so here's some code

this function is called from the doinbackground inside the Asynctask
Code:
private int storeRSS() {
    	
    	int count = 0;
    	
    	this.droidDB = new SimRacingNewsDB(this);
			this.droidDB.deleteAll("articles");
			for (int i = 0; i < RSSFEEDS.length; i++) {
				String[] items = null;
				String author = null;
				if (i == 0)
					author = "www.virtualr.net";
				else if (i == 1)
					author = "www.racesimcentral.com";
				
				feed = getFeed(RSSFEEDS[i]);					
				if (feed != null) {
					count = count + feed.getItemCount();
					feeds = feed.getAllItems();
						for (int j = 0; j < feeds.size(); j++) {
							items = new String[6];
							items[0] = feed.getItem(j).getTitle();
							items[1] = feed.getItem(j).getLink();
							items[2] = feed.getItem(j).getPubDate();
							items[3] = feed.getItem(j).getDescription();
							items[4] = "Sim Racing News";
							items[5] = author;
							// Now we need to insert this into the database.
							droidDB.insert(items, "articles");
						}	
				}
			}
			return count;
    }
which calls this,
Code:
 private RSSFeed getFeed(String urlToRssFeed)
    {
        try
        {
            // setup the url
           URL url = new URL(urlToRssFeed);
           String u = url.toString();

           // create the factory
           SAXParserFactory factory = SAXParserFactory.newInstance();
           // create a parser
           SAXParser parser = factory.newSAXParser();

           // create the reader (scanner)
           XMLReader xmlreader = parser.getXMLReader();
           // instantiate our handler
           RSSHandler theRssHandler = new RSSHandler();
           // assign our handler
           xmlreader.setContentHandler(theRssHandler);
           // get our data through the url class
        //   HttpURLConnection con = null;
        //   InputStream is = null;
        //   con = (HttpURLConnection)url.openConnection();
           
           InputSource is = new InputSource();
           InputStreamReader reader = null;
           
 //          try {

        	   BufferedInputStream buffer = new BufferedInputStream( url.openStream() );
        	   reader = new InputStreamReader( buffer );  
        	   is.setCharacterStream( reader );
        	   xmlreader.parse(is);
 //          } catch (IOException e) {
 //       	   exception = e.toString();
 //          } catch (ParseException e) {
 //       	   exception = e.toString();
 //          } catch (SAXException e) {
			// TODO Auto-generated catch block
//			e.printStackTrace();
//		}
           
   /*        
           try {
               HttpGet httpGet = new HttpGet(u);
               HttpClient httpclient = new DefaultHttpClient();
               HttpResponse response = httpclient.execute(httpGet);
              // InputStream content = response.getEntity().getContent();
               
               xmlreader.parse(new InputSource(new BufferedInputStream(response.getEntity().getContent())));
               } catch (Exception e) {
            	   Log.d("RSS Reader", "Network Error");
            	   Toast.makeText(getApplicationContext(), "Network error", Toast.LENGTH_LONG).show();
               }
               
    */
           return theRssHandler.getFeed();
        }
        catch (IOException ee)
        {
            // if you have a problem, simply return null
        	
        	exception = ee.toString();
        	Log.e("Sim Racing News", exception);
        	downloaded = 2;
            return null;
        } catch (SAXException e) {
			// TODO Auto-generated catch block
        	exception = e.toString();
        	Log.e("Sim Racing News", exception);
        	downloaded = 3;
            return null;
		} catch (ParserConfigurationException e) {
			// TODO Auto-generated catch block
			exception = e.toString();
			Log.e("Sim Racing News", exception);
			downloaded = 4;
            return null;
		} catch (ParseException e) {
			exception = e.toString();
			Log.e("Sim Racing News", exception);
			downloaded = 4;
            return null;
		}
    }
Main bulk of the RSSHandler looks like this.
Code:
public void startElement(String namespaceURI, String localName,String qName, 
                                             Attributes atts) throws SAXException
    {
        depth++;
        if (localName.equals("channel"))
        {
            currentstate = 0;
            return;
        }
//        if (localName.equals("image"))
//        {
            // record our feed data - you temporarily stored it in the item :)
//            _feed.setTitle(_item.getTitle());
//            _feed.setPubDate(_item.getPubDate());
//       }
        if (localName.equals("item"))
        {
            // create a new item
        	itemFound = true;
            _item = new RSSItem();
            return;
        }
        if (localName.equals("title"))
        {
            currentstate = RSS_TITLE;
            return;
        }
        if (localName.equals("description"))
        {
            currentstate = RSS_DESCRIPTION;
            return;
        }
        if (localName.equals("link"))
        {
            currentstate = RSS_LINK;
            return;
        }
        /**
        if (localName.equals("category"))
        {
            currentstate = RSS_CATEGORY;
            return;
        }
        */
        if (localName.equals("pubDate"))
        {
            currentstate = RSS_PUBDATE;
            return;
        }
        // if you don't explicitly handle the element, make sure you don't wind 
               // up erroneously storing a newline or other bogus data into one of our 
               // existing elements
        currentstate = 0;
    }
    
    public void endElement(String namespaceURI, String localName, String qName) 
                                                               throws SAXException
    {
        depth--;
        if (localName.equals("item"))
        {
            // add our item to the list!
        	itemFound = false;
            titleFound = false;
            _feed.addItem(_item);
           // bFoundCategory = false;
            return;
        }
    }
     
    public void characters(char ch[], int start, int length)
    {
        String theString = new String(ch,start,length);
        Log.i("RSSReader","characters[" + theString + "]");
        
        switch (currentstate)
        {
            case RSS_TITLE:
            	if (itemFound != false) {
            		if (titleFound != true) {
            			if (theString.length() > 4)
            			{
            			_lastTitle = theString;
            			_item.setTitle(theString);
            			currentstate = 0;
            			titleFound = true;
            			}
            		}
            	}
                break;
            case RSS_LINK:
                _item.setLink(theString);
                currentstate = 0;
                break;
            case RSS_DESCRIPTION:
            	_lastDesc = theString;
                _item.setDescription(theString);
                currentstate = 0;
                break;
                /**
            case RSS_CATEGORY:
            	if (bFoundCategory != true){
            	theString = sortCat(theString);
                _item.setCategory(theString);
                currentstate = 0;
            	}
                break;
                */
            case RSS_PUBDATE:
                _item.setPubDate(theString);
                currentstate = 0;
                break;
            default:
                return;
        }
        
    }

So that's it. Like I say I can get it to work, and I know others can too as I've been told, it's just that some people can't and unfortunately these are the people that comment and give bad star ratings.

Any help I would be more than grateful. The app is available under the Android market, by searching Sim Racing News.

Thanks
jimthechimp is offline  
Reply With Quote
Old February 28th, 2011, 09:52 AM   #6 (permalink)
Over Macho Grande?
 
alostpacket's Avatar
 
Join Date: Nov 2009
Location: NY
Posts: 7,873
 
Device(s): GlassXE, MotoX, N5, N4, N7'12, GNex, N1, SGT10.1, Revue, Xoom, Eris, OG Droid
Carrier: TMO

Thanks: 4,582
Thanked 3,563 Times in 1,522 Posts
Default

is StoreRSS() getting called after all the SAX parsing/downloading is done? If it is, I might try downloading the RSS and writing it to a cache file first, then reading it with SAX after all the network operations have finished.

So get file from server -> write file to disk -> read file with SAX (or DOM parser) -> enter stuff in db.

Really, unless you need to read the RSS on the fly because the feeds are huge or something, you may want to consider the DOM parser instead too.

SAX is designed for low memory consumption and speed but they achieve that by breaking the process down into an element by element callback setup. This is great for old Android 1 phones with tiny amounts of RAM but what you're downloading is small, (like 30 KB) so not a real issue...

Also the thing about the SAX characters method is that it is not garunteed to be called with the whole contents of that element

so say you have

<item>Hello World</item>

You get 1 call to startElement -> "<item>"
You COULD get 2 calls to characters
-> Hello
then later
-> World
then you get one call to endElement "</item>"

make sense?
alostpacket is offline  
Reply With Quote
The Following User Says Thank You to alostpacket For This Useful Post:
jimthechimp (February 28th, 2011)
Old February 28th, 2011, 11:16 AM   #7 (permalink)
New Member
Thread Author (OP)
 
Join Date: Feb 2011
Posts: 7
 
Device(s):
Carrier: Not Provided

Thanks: 1
Thanked 1 Time in 1 Post
Default

Actually, yes that makes perfect sense. And I have experienced problems where only half of a tag's contents have been downloaded which would probably be due to what you say.
The storeRSS function runs the whole process, so it,
---clears the db
---loops through an array of urls,
---gets the RSS
---saves to db.
---finish loop
Well I didn't want to do it but I think you're right. I'm going to re-write the code in DOM and save the file to the cache before reading it.

Thanks for your help and I'll keep you updated
jimthechimp is offline  
Reply With Quote
Old February 28th, 2011, 02:44 PM   #8 (permalink)
Over Macho Grande?
 
alostpacket's Avatar
 
Join Date: Nov 2009
Location: NY
Posts: 7,873
 
Device(s): GlassXE, MotoX, N5, N4, N7'12, GNex, N1, SGT10.1, Revue, Xoom, Eris, OG Droid
Carrier: TMO

Thanks: 4,582
Thanked 3,563 Times in 1,522 Posts
Default

I just think DOM is easier to understand but if you want to try the process of saving to the SD/cache first and still use SAX you can do that too.

I mean you might as well code it so that it processes the XML last and if you still have trouble, then swap out your parser for the DOM one.

This is a key concept to learn when organizing your programs I've always thought. Try and make the code modular so if you need to redo a section you can just pop it out and pop in the new one. (you probably already know that I'm just tired and rambling )

Anyways, if you want to fix the chracters() method, just make sure you use a temp StringBuffer field and append to it every time characters() gets called, and then add your "_item" only when endElement() is called, then clear your StringBuffer in your endElement() method for the next pass.

Code:
/////////////////////////////////////
//  assuming xml:
//
//  <item>Hello World</item>
/////////////////////////////////////

public class MYParser extends DefaultHandler
{
    
    Boolean  parsing = false;
    StringBuffer stringBuffer  = new StringBuffer ( );

@Override
public void startElement( String uri, String localName, String qName, Attributes attributes) throws SAXException 
{
       // your code here was fine...
       // just removed to show example
       // we got <item>
       parsing = true;
       stringBuffer  = new StringBuffer();
}


@Override
public void characters(char[] ch, int start, int length) throws SAXException 
{
        
        if( parsing )
        {
            // there are lots of ways to do this but i used a for loop
            // just out of habit
            // on first call to chracters we get Hello
            // on second call to chracters we get World

            for(int i = start; i < length ; i++) 
            {
                stringBuffer.append ( ch[i] );
            }
         }
}

@Override
public void endElement(String uri, String localName, String qName)    throws SAXException 
{
       // your code here was fine too..
       // now we got </item>

       String myStringValueFromCharacters  = stringBuffer.toString();
       parsing = false;
       stringBuffer = null;

       // now we have  myStringValueFromCharacters = "Hello World"
       // now add "myStringValueFromCharacters"   to your DB or whatever you need here
}
alostpacket is offline  
Reply With Quote
Old February 28th, 2011, 02:46 PM   #9 (permalink)
Over Macho Grande?
 
alostpacket's Avatar
 
Join Date: Nov 2009
Location: NY
Posts: 7,873
 
Device(s): GlassXE, MotoX, N5, N4, N7'12, GNex, N1, SGT10.1, Revue, Xoom, Eris, OG Droid
Carrier: TMO

Thanks: 4,582
Thanked 3,563 Times in 1,522 Posts
Default

oh and feel free to try out my apps if I helped you out I'm glad to help but I need beta testers too
alostpacket is offline  
Reply With Quote
Old March 3rd, 2011, 08:32 AM   #10 (permalink)
New Member
Thread Author (OP)
 
Join Date: Feb 2011
Posts: 7
 
Device(s):
Carrier: Not Provided

Thanks: 1
Thanked 1 Time in 1 Post
Default

Wow, DOM is a much nicer way to work with XML. Much more logical.

So I've re-written the entire code, it downloads all 4 rss feeds and then parses using DOM. It's working, but then the other way worked for me too. So I've just got to hope that this new way works.

I'm happy to download and test your apps, where can I get hold of them?
jimthechimp is offline  
Reply With Quote
The Following User Says Thank You to jimthechimp For This Useful Post:
alostpacket (March 3rd, 2011)
sponsored links
Old March 3rd, 2011, 07:59 PM   #11 (permalink)
Over Macho Grande?
 
alostpacket's Avatar
 
Join Date: Nov 2009
Location: NY
Posts: 7,873
 
Device(s): GlassXE, MotoX, N5, N4, N7'12, GNex, N1, SGT10.1, Revue, Xoom, Eris, OG Droid
Carrier: TMO

Thanks: 4,582
Thanked 3,563 Times in 1,522 Posts
Default

Here are the announcements: My apps: BlueMuze | Listables

You wanna hear something funny too, I'm having trouble with my own SAX parser at the moment heh Though I think it's mostly being tired and not following my own logic. I cant code for my life when I'm tired...
alostpacket is offline  
Reply With Quote
Reply


Go Back   Android Forums > Android Development > Application Development
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -5. The time now is 02:39 PM.
Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.