SAX RSS streaming fine on Wifi not so good on 3G


Last Updated:

  1. jimthechimp

    jimthechimp Member

    Hello

    I've checked everywhere for an answer to this but I just can't find it. I can't believe I'm the only one to have it.

    I have written an RSS reader using SAX, like this

    Code (Text):
    1. try
    2.         {
    3.             // setup the url
    4.            URL url = new URL(urlToRssFeed);
    5.  
    6.            // create the factory
    7.            SAXParserFactory factory = SAXParserFactory.newInstance();
    8.            // create a parser
    9.            SAXParser parser = factory.newSAXParser();
    10.  
    11.            // create the reader (scanner)
    12.            XMLReader xmlreader = parser.getXMLReader();
    13.            // instantiate our handler
    14.            RSSHandler theRssHandler = new RSSHandler();
    15.            // assign our handler
    16.            xmlreader.setContentHandler(theRssHandler);
    17.            // get our data through the url class
    18.            InputSource is = new InputSource(url.openStream());
    19.            // perform the synchronous parse          
    20.            xmlreader.parse(is);
    21.            // get the results - should be a fully populated RSSFeed instance,
    22.            // or null on error
    23.            return theRssHandler.getFeed();
    24.         }
    25.         catch (Exception ee)
    26.         {
    27.             // if you have a problem, simply return null
    28.             return null;
    29.         }

    The RSS is sorted and stored in a database through the RSSHandler class.

    The problem :
    This works fine. Running under Wifi I get all the data back from the RSS. However, If I'm running using 3G and the signal strength is not so good, it returns maybe a few feeds from the RSS or in some cases no feeds at all.

    The question :
    Is there away to check is the entire RSS feed has been sorted and stored before it quits? I'm guessing that it loses connection during the RSSHandler class and so just returns what it's received, or maybe it's in the openStream(). Maybe even actually download the RSS to the cache and then use that?

    Anyway any help would be appreciated.

    Thanks
    James

    Advertisement
  2. alostpacket

    alostpacket Over Macho Grande? VIP Member

    SAX is event based (or callback-based depending on the lingo you use)... so you need to show the code where you override startElement, characters, endElement.

    If you want something to download the entire document I'd just use the DOMparser it's MUCH easier to code for and the reports of how slow it is are a bit exagerrated.


    Actually, you probably know all that -- and now that I think about it, I think I found the error....

    If you're having a problem where the callbacks are never getting called i'd consider buffering the input and making sure you understand the data flow from the URL to the XMLParser (see my code below)



    your code went:

    URL -> input stream -> SAX xml parse


    what I did differently below was

    URL -> input stream -> buffer -> reader -> SAX xml parse


    Code (Text):
    1.  
    2.  
    3.  
    4.      SAXParser sp = spf.newSAXParser();
    5.      XMLReader xr = sp.getXMLReader();
    6.      InputSource is = new InputSource();
    7.  
    8.      BufferedInputStream buffer = new BufferedInputStream( url.openStream() );
    9.      InputStreamReader reader = new InputStreamReader( buffer );
    10.      is.setCharacterStream( reader );
    11.      xr.parse( is );
    12.  
  3. jimthechimp

    jimthechimp Member

    Thanks alostpacket.

    I actually did something similar to this before I read your reply. What I did differently was I read about HttpClient, that it's a better of connecting. so my code now looks like this. Apparently HttpClient trys to connect 3 times?

    Code (Text):
    1.  try {
    2.                HttpGet httpGet = new HttpGet(url);
    3.                HttpClient httpclient = new DefaultHttpClient();
    4.                HttpResponse response = httpclient.execute(httpGet);
    5.                
    6.                xmlreader.parse(new InputSource(new BufferedInputStream(response.getEntity().getContent())));
    7.                } catch (Exception e) {
    8.                    Log.d("RSS Reader", "Network Error");
    9.                }
    It seems that this is working better than before, but it's not perfect.
    Thanks for your help.
  4. alostpacket

    alostpacket Over Macho Grande? VIP Member

    Did you ever get this figured out? Sorry for the late reply I was super busy working on BlueMuze
  5. jimthechimp

    jimthechimp Member

    Hi, I'm really not sure if I fixed this or not. I've tried your method and it seems to be working fine. I really need some help here because I'm running out of ideas.

    The App, I've designed an app for the online Sim Racing community that reads in 4 RSS feeds from 4 different servers. Because some of them are not particularly well formatted I've had to read and store the data one by one in the SQLite database. I've released this onto the market as I wanted to test it. I have made the Sim Racing Community that there are errors and it needs testing so here's the basic structure of the code.

    Start App
    ---runs inside AsyncTask
    ---loop through string of RSS feeds (4 times)
    ---parse RSS
    ---store RSS
    ---change to Feeds Activity
    ---inside thread to allow progress bar to display.
    ---load from db feeds into list.

    The problem is that when I use it on my HTC Desire, it works, I've put catches in place so that if there are any download errors it tells you. I've even got it to work in areas where my reception has been less than perfect. the problem is that some people are simply complaining of never getting any results back, even on WiFi. Of course these people don't give proper descriptions of what errors are recieved etc. so it's very difficult to diagnose.

    ok, so here's some code

    this function is called from the doinbackground inside the Asynctask
    Code (Text):
    1. private int storeRSS() {
    2.        
    3.         int count = 0;
    4.        
    5.         this.droidDB = new SimRacingNewsDB(this);
    6.             this.droidDB.deleteAll("articles");
    7.             for (int i = 0; i < RSSFEEDS.length; i++) {
    8.                 String[] items = null;
    9.                 String author = null;
    10.                 if (i == 0)
    11.                     author = "www.virtualr.net";
    12.                 else if (i == 1)
    13.                     author = "www.racesimcentral.com";
    14.                
    15.                 feed = getFeed(RSSFEEDS[i]);                   
    16.                 if (feed != null) {
    17.                     count = count + feed.getItemCount();
    18.                     feeds = feed.getAllItems();
    19.                         for (int j = 0; j < feeds.size(); j++) {
    20.                             items = new String[6];
    21.                             items[0] = feed.getItem(j).getTitle();
    22.                             items[1] = feed.getItem(j).getLink();
    23.                             items[2] = feed.getItem(j).getPubDate();
    24.                             items[3] = feed.getItem(j).getDescription();
    25.                             items[4] = "Sim Racing News";
    26.                             items[5] = author;
    27.                             // Now we need to insert this into the database.
    28.                             droidDB.insert(items, "articles");
    29.                         }  
    30.                 }
    31.             }
    32.             return count;
    33.     }
    which calls this,
    Code (Text):
    1.  private RSSFeed getFeed(String urlToRssFeed)
    2.     {
    3.         try
    4.         {
    5.             // setup the url
    6.            URL url = new URL(urlToRssFeed);
    7.            String u = url.toString();
    8.  
    9.            // create the factory
    10.            SAXParserFactory factory = SAXParserFactory.newInstance();
    11.            // create a parser
    12.            SAXParser parser = factory.newSAXParser();
    13.  
    14.            // create the reader (scanner)
    15.            XMLReader xmlreader = parser.getXMLReader();
    16.            // instantiate our handler
    17.            RSSHandler theRssHandler = new RSSHandler();
    18.            // assign our handler
    19.            xmlreader.setContentHandler(theRssHandler);
    20.            // get our data through the url class
    21.         //   HttpURLConnection con = null;
    22.         //   InputStream is = null;
    23.         //   con = (HttpURLConnection)url.openConnection();
    24.            
    25.            InputSource is = new InputSource();
    26.            InputStreamReader reader = null;
    27.            
    28.  //          try {
    29.  
    30.                BufferedInputStream buffer = new BufferedInputStream( url.openStream() );
    31.                reader = new InputStreamReader( buffer );  
    32.                is.setCharacterStream( reader );
    33.                xmlreader.parse(is);
    34.  //          } catch (IOException e) {
    35.  //            exception = e.toString();
    36.  //          } catch (ParseException e) {
    37.  //            exception = e.toString();
    38.  //          } catch (SAXException e) {
    39.             // TODO Auto-generated catch block
    40. //          e.printStackTrace();
    41. //      }
    42.            
    43.    /*        
    44.            try {
    45.                HttpGet httpGet = new HttpGet(u);
    46.                HttpClient httpclient = new DefaultHttpClient();
    47.                HttpResponse response = httpclient.execute(httpGet);
    48.               // InputStream content = response.getEntity().getContent();
    49.                
    50.                xmlreader.parse(new InputSource(new BufferedInputStream(response.getEntity().getContent())));
    51.                } catch (Exception e) {
    52.                    Log.d("RSS Reader", "Network Error");
    53.                    Toast.makeText(getApplicationContext(), "Network error", Toast.LENGTH_LONG).show();
    54.                }
    55.                
    56.     */
    57.            return theRssHandler.getFeed();
    58.         }
    59.         catch (IOException ee)
    60.         {
    61.             // if you have a problem, simply return null
    62.            
    63.             exception = ee.toString();
    64.             Log.e("Sim Racing News", exception);
    65.             downloaded = 2;
    66.             return null;
    67.         } catch (SAXException e) {
    68.             // TODO Auto-generated catch block
    69.             exception = e.toString();
    70.             Log.e("Sim Racing News", exception);
    71.             downloaded = 3;
    72.             return null;
    73.         } catch (ParserConfigurationException e) {
    74.             // TODO Auto-generated catch block
    75.             exception = e.toString();
    76.             Log.e("Sim Racing News", exception);
    77.             downloaded = 4;
    78.             return null;
    79.         } catch (ParseException e) {
    80.             exception = e.toString();
    81.             Log.e("Sim Racing News", exception);
    82.             downloaded = 4;
    83.             return null;
    84.         }
    85.     }
    Main bulk of the RSSHandler looks like this.
    Code (Text):
    1. public void startElement(String namespaceURI, String localName,String qName,
    2.                                              Attributes atts) throws SAXException
    3.     {
    4.         depth++;
    5.         if (localName.equals("channel"))
    6.         {
    7.             currentstate = 0;
    8.             return;
    9.         }
    10. //        if (localName.equals("image"))
    11. //        {
    12.             // record our feed data - you temporarily stored it in the item :)
    13. //            _feed.setTitle(_item.getTitle());
    14. //            _feed.setPubDate(_item.getPubDate());
    15. //       }
    16.         if (localName.equals("item"))
    17.         {
    18.             // create a new item
    19.             itemFound = true;
    20.             _item = new RSSItem();
    21.             return;
    22.         }
    23.         if (localName.equals("title"))
    24.         {
    25.             currentstate = RSS_TITLE;
    26.             return;
    27.         }
    28.         if (localName.equals("description"))
    29.         {
    30.             currentstate = RSS_DESCRIPTION;
    31.             return;
    32.         }
    33.         if (localName.equals("link"))
    34.         {
    35.             currentstate = RSS_LINK;
    36.             return;
    37.         }
    38.         /**
    39.         if (localName.equals("category"))
    40.         {
    41.             currentstate = RSS_CATEGORY;
    42.             return;
    43.         }
    44.         */
    45.         if (localName.equals("pubDate"))
    46.         {
    47.             currentstate = RSS_PUBDATE;
    48.             return;
    49.         }
    50.         // if you don't explicitly handle the element, make sure you don't wind
    51.                // up erroneously storing a newline or other bogus data into one of our
    52.                // existing elements
    53.         currentstate = 0;
    54.     }
    55.    
    56.     public void endElement(String namespaceURI, String localName, String qName)
    57.                                                                throws SAXException
    58.     {
    59.         depth--;
    60.         if (localName.equals("item"))
    61.         {
    62.             // add our item to the list!
    63.             itemFound = false;
    64.             titleFound = false;
    65.             _feed.addItem(_item);
    66.            // bFoundCategory = false;
    67.             return;
    68.         }
    69.     }
    70.      
    71.     public void characters(char ch[], int start, int length)
    72.     {
    73.         String theString = new String(ch,start,length);
    74.         Log.i("RSSReader","characters[" + theString + "]");
    75.        
    76.         switch (currentstate)
    77.         {
    78.             case RSS_TITLE:
    79.                 if (itemFound != false) {
    80.                     if (titleFound != true) {
    81.                         if (theString.length() > 4)
    82.                         {
    83.                         _lastTitle = theString;
    84.                         _item.setTitle(theString);
    85.                         currentstate = 0;
    86.                         titleFound = true;
    87.                         }
    88.                     }
    89.                 }
    90.                 break;
    91.             case RSS_LINK:
    92.                 _item.setLink(theString);
    93.                 currentstate = 0;
    94.                 break;
    95.             case RSS_DESCRIPTION:
    96.                 _lastDesc = theString;
    97.                 _item.setDescription(theString);
    98.                 currentstate = 0;
    99.                 break;
    100.                 /**
    101.             case RSS_CATEGORY:
    102.                 if (bFoundCategory != true){
    103.                 theString = sortCat(theString);
    104.                 _item.setCategory(theString);
    105.                 currentstate = 0;
    106.                 }
    107.                 break;
    108.                 */
    109.             case RSS_PUBDATE:
    110.                 _item.setPubDate(theString);
    111.                 currentstate = 0;
    112.                 break;
    113.             default:
    114.                 return;
    115.         }
    116.        
    117.     }

    So that's it. Like I say I can get it to work, and I know others can too as I've been told, it's just that some people can't and unfortunately these are the people that comment and give bad star ratings.

    Any help I would be more than grateful. The app is available under the Android market, by searching Sim Racing News.

    Thanks
  6. alostpacket

    alostpacket Over Macho Grande? VIP Member

    is StoreRSS() getting called after all the SAX parsing/downloading is done? If it is, I might try downloading the RSS and writing it to a cache file first, then reading it with SAX after all the network operations have finished.

    So get file from server -> write file to disk -> read file with SAX (or DOM parser) -> enter stuff in db.

    Really, unless you need to read the RSS on the fly because the feeds are huge or something, you may want to consider the DOM parser instead too.

    SAX is designed for low memory consumption and speed but they achieve that by breaking the process down into an element by element callback setup. This is great for old Android 1 phones with tiny amounts of RAM but what you're downloading is small, (like 30 KB) so not a real issue...

    Also the thing about the SAX characters method is that it is not garunteed to be called with the whole contents of that element

    so say you have

    <item>Hello World</item>

    You get 1 call to startElement -> "<item>"
    You COULD get 2 calls to characters
    -> Hello
    then later
    -> World
    then you get one call to endElement "</item>"

    make sense?
    jimthechimp likes this.
  7. jimthechimp

    jimthechimp Member

    Actually, yes that makes perfect sense. And I have experienced problems where only half of a tag's contents have been downloaded which would probably be due to what you say.
    The storeRSS function runs the whole process, so it,
    ---clears the db
    ---loops through an array of urls,
    ---gets the RSS
    ---saves to db.
    ---finish loop
    Well I didn't want to do it but I think you're right. I'm going to re-write the code in DOM and save the file to the cache before reading it.

    Thanks for your help and I'll keep you updated
  8. alostpacket

    alostpacket Over Macho Grande? VIP Member

    I just think DOM is easier to understand but if you want to try the process of saving to the SD/cache first and still use SAX you can do that too.

    I mean you might as well code it so that it processes the XML last and if you still have trouble, then swap out your parser for the DOM one.

    This is a key concept to learn when organizing your programs I've always thought. Try and make the code modular so if you need to redo a section you can just pop it out and pop in the new one. (you probably already know that I'm just tired and rambling :) )

    Anyways, if you want to fix the chracters() method, just make sure you use a temp StringBuffer field and append to it every time characters() gets called, and then add your "_item" only when endElement() is called, then clear your StringBuffer in your endElement() method for the next pass.

    Code (Text):
    1.  
    2. /////////////////////////////////////
    3. //  assuming xml:
    4. //
    5. //  <item>Hello World</item>
    6. /////////////////////////////////////
    7.  
    8. public class MYParser extends DefaultHandler
    9. {
    10.    
    11.     Boolean  parsing = false;
    12.     StringBuffer stringBuffer  = new StringBuffer ( );
    13.  
    14. @Override
    15. public void startElement( String uri, String localName, String qName, Attributes attributes) throws SAXException
    16. {
    17.        // your code here was fine...
    18.        // just removed to show example
    19.        // we got <item>
    20.        parsing = true;
    21.        stringBuffer  = new StringBuffer();
    22. }
    23.  
    24.  
    25. @Override
    26. public void characters(char[] ch, int start, int length) throws SAXException
    27. {
    28.        
    29.         if( parsing )
    30.         {
    31.             // there are lots of ways to do this but i used a for loop
    32.             // just out of habit
    33.             // on first call to chracters we get Hello
    34.             // on second call to chracters we get World
    35.  
    36.             for(int i = start; i < length ; i++)
    37.             {
    38.                 stringBuffer.append ( ch[i] );
    39.             }
    40.          }
    41. }
    42.  
    43. @Override
    44. public void endElement(String uri, String localName, String qName)    throws SAXException
    45. {
    46.        // your code here was fine too..
    47.        // now we got </item>
    48.  
    49.        String myStringValueFromCharacters  = stringBuffer.toString();
    50.        parsing = false;
    51.        stringBuffer = null;
    52.  
    53.        // now we have  myStringValueFromCharacters = "Hello World"
    54.        // now add "myStringValueFromCharacters"   to your DB or whatever you need here
    55. }
    56.  
  9. alostpacket

    alostpacket Over Macho Grande? VIP Member

    oh and feel free to try out my apps if I helped you out ;) I'm glad to help but I need beta testers too :)
  10. jimthechimp

    jimthechimp Member

    Wow, DOM is a much nicer way to work with XML. Much more logical.

    So I've re-written the entire code, it downloads all 4 rss feeds and then parses using DOM. It's working, but then the other way worked for me too. So I've just got to hope that this new way works.

    I'm happy to download and test your apps, where can I get hold of them?
    alostpacket likes this.
  11. alostpacket

    alostpacket Over Macho Grande? VIP Member

    Here are the announcements: My apps: BlueMuze | Listables

    You wanna hear something funny too, I'm having trouble with my own SAX parser at the moment heh :) Though I think it's mostly being tired and not following my own logic. I cant code for my life when I'm tired... :)

Share This Page