• After 15+ years, we've made a big change: Android Forums is now Early Bird Club. Learn more here.

Apps SAX RSS streaming fine on Wifi not so good on 3G

jimthechimp

Lurker
Feb 11, 2011
7
1
Hello

I've checked everywhere for an answer to this but I just can't find it. I can't believe I'm the only one to have it.

I have written an RSS reader using SAX, like this

Code:
try
        {
            // setup the url
           URL url = new URL(urlToRssFeed);

           // create the factory
           SAXParserFactory factory = SAXParserFactory.newInstance();
           // create a parser
           SAXParser parser = factory.newSAXParser();

           // create the reader (scanner)
           XMLReader xmlreader = parser.getXMLReader();
           // instantiate our handler
           RSSHandler theRssHandler = new RSSHandler();
           // assign our handler
           xmlreader.setContentHandler(theRssHandler);
           // get our data through the url class
           InputSource is = new InputSource(url.openStream());
           // perform the synchronous parse           
           xmlreader.parse(is);
           // get the results - should be a fully populated RSSFeed instance, 
		   // or null on error
           return theRssHandler.getFeed();
        }
        catch (Exception ee)
        {
            // if you have a problem, simply return null
            return null;
        }


The RSS is sorted and stored in a database through the RSSHandler class.

The problem :
This works fine. Running under Wifi I get all the data back from the RSS. However, If I'm running using 3G and the signal strength is not so good, it returns maybe a few feeds from the RSS or in some cases no feeds at all.

The question :
Is there away to check is the entire RSS feed has been sorted and stored before it quits? I'm guessing that it loses connection during the RSSHandler class and so just returns what it's received, or maybe it's in the openStream(). Maybe even actually download the RSS to the cache and then use that?

Anyway any help would be appreciated.

Thanks
James
 
SAX is event based (or callback-based depending on the lingo you use)... so you need to show the code where you override startElement, characters, endElement.

If you want something to download the entire document I'd just use the DOMparser it's MUCH easier to code for and the reports of how slow it is are a bit exagerrated.


Actually, you probably know all that -- and now that I think about it, I think I found the error....

If you're having a problem where the callbacks are never getting called i'd consider buffering the input and making sure you understand the data flow from the URL to the XMLParser (see my code below)



your code went:

URL -> input stream -> SAX xml parse


what I did differently below was

URL -> input stream -> buffer -> reader -> SAX xml parse


Code:
     SAXParser sp = spf.newSAXParser();
     XMLReader xr = sp.getXMLReader();
     InputSource is = new InputSource();

     BufferedInputStream buffer = new BufferedInputStream( url.openStream() );
     InputStreamReader reader = new InputStreamReader( buffer );
     is.setCharacterStream( reader );
     xr.parse( is );
 
Upvote 0
Thanks alostpacket.

I actually did something similar to this before I read your reply. What I did differently was I read about HttpClient, that it's a better of connecting. so my code now looks like this. Apparently HttpClient trys to connect 3 times?

Code:
 try {
               HttpGet httpGet = new HttpGet(url);
               HttpClient httpclient = new DefaultHttpClient();
               HttpResponse response = httpclient.execute(httpGet);
               
               xmlreader.parse(new InputSource(new BufferedInputStream(response.getEntity().getContent())));
               } catch (Exception e) {
            	   Log.d("RSS Reader", "Network Error");
               }

It seems that this is working better than before, but it's not perfect.
Thanks for your help.
 
Upvote 0
Hi, I'm really not sure if I fixed this or not. I've tried your method and it seems to be working fine. I really need some help here because I'm running out of ideas.

The App, I've designed an app for the online Sim Racing community that reads in 4 RSS feeds from 4 different servers. Because some of them are not particularly well formatted I've had to read and store the data one by one in the SQLite database. I've released this onto the market as I wanted to test it. I have made the Sim Racing Community that there are errors and it needs testing so here's the basic structure of the code.

Start App
---runs inside AsyncTask
---loop through string of RSS feeds (4 times)
---parse RSS
---store RSS
---change to Feeds Activity
---inside thread to allow progress bar to display.
---load from db feeds into list.

The problem is that when I use it on my HTC Desire, it works, I've put catches in place so that if there are any download errors it tells you. I've even got it to work in areas where my reception has been less than perfect. the problem is that some people are simply complaining of never getting any results back, even on WiFi. Of course these people don't give proper descriptions of what errors are recieved etc. so it's very difficult to diagnose.

ok, so here's some code

this function is called from the doinbackground inside the Asynctask
Code:
private int storeRSS() {
    	
    	int count = 0;
    	
    	this.droidDB = new SimRacingNewsDB(this);
			this.droidDB.deleteAll("articles");
			for (int i = 0; i < RSSFEEDS.length; i++) {
				String[] items = null;
				String author = null;
				if (i == 0)
					author = "www.virtualr.net";
				else if (i == 1)
					author = "www.racesimcentral.com";
				
				feed = getFeed(RSSFEEDS[i]);					
				if (feed != null) {
					count = count + feed.getItemCount();
					feeds = feed.getAllItems();
						for (int j = 0; j < feeds.size(); j++) {
							items = new String[6];
							items[0] = feed.getItem(j).getTitle();
							items[1] = feed.getItem(j).getLink();
							items[2] = feed.getItem(j).getPubDate();
							items[3] = feed.getItem(j).getDescription();
							items[4] = "Sim Racing News";
							items[5] = author;
							// Now we need to insert this into the database.
							droidDB.insert(items, "articles");
						}	
				}
			}
			return count;
    }

which calls this,
Code:
 private RSSFeed getFeed(String urlToRssFeed)
    {
        try
        {
            // setup the url
           URL url = new URL(urlToRssFeed);
           String u = url.toString();

           // create the factory
           SAXParserFactory factory = SAXParserFactory.newInstance();
           // create a parser
           SAXParser parser = factory.newSAXParser();

           // create the reader (scanner)
           XMLReader xmlreader = parser.getXMLReader();
           // instantiate our handler
           RSSHandler theRssHandler = new RSSHandler();
           // assign our handler
           xmlreader.setContentHandler(theRssHandler);
           // get our data through the url class
        //   HttpURLConnection con = null;
        //   InputStream is = null;
        //   con = (HttpURLConnection)url.openConnection();
           
           InputSource is = new InputSource();
           InputStreamReader reader = null;
           
 //          try {

        	   BufferedInputStream buffer = new BufferedInputStream( url.openStream() );
        	   reader = new InputStreamReader( buffer );  
        	   is.setCharacterStream( reader );
        	   xmlreader.parse(is);
 //          } catch (IOException e) {
 //       	   exception = e.toString();
 //          } catch (ParseException e) {
 //       	   exception = e.toString();
 //          } catch (SAXException e) {
			// TODO Auto-generated catch block
//			e.printStackTrace();
//		}
           
   /*        
           try {
               HttpGet httpGet = new HttpGet(u);
               HttpClient httpclient = new DefaultHttpClient();
               HttpResponse response = httpclient.execute(httpGet);
              // InputStream content = response.getEntity().getContent();
               
               xmlreader.parse(new InputSource(new BufferedInputStream(response.getEntity().getContent())));
               } catch (Exception e) {
            	   Log.d("RSS Reader", "Network Error");
            	   Toast.makeText(getApplicationContext(), "Network error", Toast.LENGTH_LONG).show();
               }
               
    */
           return theRssHandler.getFeed();
        }
        catch (IOException ee)
        {
            // if you have a problem, simply return null
        	
        	exception = ee.toString();
        	Log.e("Sim Racing News", exception);
        	downloaded = 2;
            return null;
        } catch (SAXException e) {
			// TODO Auto-generated catch block
        	exception = e.toString();
        	Log.e("Sim Racing News", exception);
        	downloaded = 3;
            return null;
		} catch (ParserConfigurationException e) {
			// TODO Auto-generated catch block
			exception = e.toString();
			Log.e("Sim Racing News", exception);
			downloaded = 4;
            return null;
		} catch (ParseException e) {
			exception = e.toString();
			Log.e("Sim Racing News", exception);
			downloaded = 4;
            return null;
		}
    }

Main bulk of the RSSHandler looks like this.
Code:
public void startElement(String namespaceURI, String localName,String qName, 
                                             Attributes atts) throws SAXException
    {
        depth++;
        if (localName.equals("channel"))
        {
            currentstate = 0;
            return;
        }
//        if (localName.equals("image"))
//        {
            // record our feed data - you temporarily stored it in the item :)
//            _feed.setTitle(_item.getTitle());
//            _feed.setPubDate(_item.getPubDate());
//       }
        if (localName.equals("item"))
        {
            // create a new item
        	itemFound = true;
            _item = new RSSItem();
            return;
        }
        if (localName.equals("title"))
        {
            currentstate = RSS_TITLE;
            return;
        }
        if (localName.equals("description"))
        {
            currentstate = RSS_DESCRIPTION;
            return;
        }
        if (localName.equals("link"))
        {
            currentstate = RSS_LINK;
            return;
        }
        /**
        if (localName.equals("category"))
        {
            currentstate = RSS_CATEGORY;
            return;
        }
        */
        if (localName.equals("pubDate"))
        {
            currentstate = RSS_PUBDATE;
            return;
        }
        // if you don't explicitly handle the element, make sure you don't wind 
               // up erroneously storing a newline or other bogus data into one of our 
               // existing elements
        currentstate = 0;
    }
    
    public void endElement(String namespaceURI, String localName, String qName) 
                                                               throws SAXException
    {
        depth--;
        if (localName.equals("item"))
        {
            // add our item to the list!
        	itemFound = false;
            titleFound = false;
            _feed.addItem(_item);
           // bFoundCategory = false;
            return;
        }
    }
     
    public void characters(char ch[], int start, int length)
    {
        String theString = new String(ch,start,length);
        Log.i("RSSReader","characters[" + theString + "]");
        
        switch (currentstate)
        {
            case RSS_TITLE:
            	if (itemFound != false) {
            		if (titleFound != true) {
            			if (theString.length() > 4)
            			{
            			_lastTitle = theString;
            			_item.setTitle(theString);
            			currentstate = 0;
            			titleFound = true;
            			}
            		}
            	}
                break;
            case RSS_LINK:
                _item.setLink(theString);
                currentstate = 0;
                break;
            case RSS_DESCRIPTION:
            	_lastDesc = theString;
                _item.setDescription(theString);
                currentstate = 0;
                break;
                /**
            case RSS_CATEGORY:
            	if (bFoundCategory != true){
            	theString = sortCat(theString);
                _item.setCategory(theString);
                currentstate = 0;
            	}
                break;
                */
            case RSS_PUBDATE:
                _item.setPubDate(theString);
                currentstate = 0;
                break;
            default:
                return;
        }
        
    }


So that's it. Like I say I can get it to work, and I know others can too as I've been told, it's just that some people can't and unfortunately these are the people that comment and give bad star ratings.

Any help I would be more than grateful. The app is available under the Android market, by searching Sim Racing News.

Thanks
 
Upvote 0
is StoreRSS() getting called after all the SAX parsing/downloading is done? If it is, I might try downloading the RSS and writing it to a cache file first, then reading it with SAX after all the network operations have finished.

So get file from server -> write file to disk -> read file with SAX (or DOM parser) -> enter stuff in db.

Really, unless you need to read the RSS on the fly because the feeds are huge or something, you may want to consider the DOM parser instead too.

SAX is designed for low memory consumption and speed but they achieve that by breaking the process down into an element by element callback setup. This is great for old Android 1 phones with tiny amounts of RAM but what you're downloading is small, (like 30 KB) so not a real issue...

Also the thing about the SAX characters method is that it is not garunteed to be called with the whole contents of that element

so say you have

<item>Hello World</item>

You get 1 call to startElement -> "<item>"
You COULD get 2 calls to characters
-> Hello
then later
-> World
then you get one call to endElement "</item>"

make sense?
 
  • Like
Reactions: jimthechimp
Upvote 0
Actually, yes that makes perfect sense. And I have experienced problems where only half of a tag's contents have been downloaded which would probably be due to what you say.
The storeRSS function runs the whole process, so it,
---clears the db
---loops through an array of urls,
---gets the RSS
---saves to db.
---finish loop
Well I didn't want to do it but I think you're right. I'm going to re-write the code in DOM and save the file to the cache before reading it.

Thanks for your help and I'll keep you updated
 
Upvote 0
I just think DOM is easier to understand but if you want to try the process of saving to the SD/cache first and still use SAX you can do that too.

I mean you might as well code it so that it processes the XML last and if you still have trouble, then swap out your parser for the DOM one.

This is a key concept to learn when organizing your programs I've always thought. Try and make the code modular so if you need to redo a section you can just pop it out and pop in the new one. (you probably already know that I'm just tired and rambling :) )

Anyways, if you want to fix the chracters() method, just make sure you use a temp StringBuffer field and append to it every time characters() gets called, and then add your "_item" only when endElement() is called, then clear your StringBuffer in your endElement() method for the next pass.

Code:
/////////////////////////////////////
//  assuming xml:
//
//  <item>Hello World</item>
/////////////////////////////////////

public class MYParser extends DefaultHandler
{
    
    Boolean  parsing = false;
    StringBuffer stringBuffer  = new StringBuffer ( );

@Override
public void startElement( String uri, String localName, String qName, Attributes attributes) throws SAXException 
{
       // your code here was fine...
       // just removed to show example
       // we got <item>
       parsing = true;
       stringBuffer  = new StringBuffer();
}


@Override
public void characters(char[] ch, int start, int length) throws SAXException 
{
        
        if( parsing )
        {
            // there are lots of ways to do this but i used a for loop
            // just out of habit
            // on first call to chracters we get Hello
            // on second call to chracters we get World

            for(int i = start; i < length ; i++) 
            {
                stringBuffer.append ( ch[i] );
            }
         }
}

@Override
public void endElement(String uri, String localName, String qName)    throws SAXException 
{
       // your code here was fine too..
       // now we got </item>

       String myStringValueFromCharacters  = stringBuffer.toString();
       parsing = false;
       stringBuffer = null;

       // now we have  myStringValueFromCharacters = "Hello World"
       // now add "myStringValueFromCharacters"   to your DB or whatever you need here
}
 
Upvote 0
Wow, DOM is a much nicer way to work with XML. Much more logical.

So I've re-written the entire code, it downloads all 4 rss feeds and then parses using DOM. It's working, but then the other way worked for me too. So I've just got to hope that this new way works.

I'm happy to download and test your apps, where can I get hold of them?
 
  • Like
Reactions: alostpacket
Upvote 0

BEST TECH IN 2023

We've been tracking upcoming products and ranking the best tech since 2007. Thanks for trusting our opinion: we get rewarded through affiliate links that earn us a commission and we invite you to learn more about us.

Smartphones