1. Check out our app, Forums for Android! Download and leave feedback here!

Handling large JSON streams (OOM Error)

Discussion in 'Application Development' started by DoddyUK, May 14, 2010.

  1. DoddyUK

    DoddyUK Member
    Thread Starter
    35

    Jan 31, 2010
    12
    0
    35
    Hi,
    I'm currently in the process of creating an application that uses a
    large amount of data streamed from a web service. In this case, I'm
    using The Guardian's Politics API to make an application about UK
    General Elections. In some cases, I need to stream and parse about
    750KB of data - one example of this is when I'm loading data for all
    constituencies. (Example: http://www.guardian.co.uk/politics/api/general-election/1997/results/json)

    When I'm parsing this data, I occasionally get an OutOfMemoryError
    problem when creating the JSON object from the Buffer Reader -
    admittedly this is happening a lot less often than when I was using
    the streamToString function, but I still want to eliminate the
    possibility of an OOM Error wherever I can.

    So I'd like to ask, what's the best way of parsing large amounts of
    JSON data streamed over HTTP into a single JSON object?

    Here's the error log:
    Code (Text):
    1. [color=red]
    2. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862): Uncaught handler:
    3. thread Thread-16 exiting due to uncaught exception
    4. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):
    5. java.lang.OutOfMemoryError
    6. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    7. java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:
    8. 99)
    9. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    10. java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:
    11. 139)
    12. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    13. java.lang.StringBuilder.append(StringBuilder.java:282)
    14. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    15. java.io.BufferedReader.readLine(BufferedReader.java:415)
    16. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    17. com.mdodd.ElectionMonitor.resources.RestHandler.connect(RestHandler.java:
    18. 189)
    19. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    20. com.mdodd.ElectionMonitor.resources.ConstituenciesThread.run(ConstituenciesThread.java:
    21. 49)
    22. [/color]
    And here's RestHandler.java:
    Code (Text):
    1.  
    2. package com.mdodd.ElectionMonitor.resources;
    3.  
    4. import java.io.*;
    5. import java.net.URI;
    6. import java.net.URISyntaxException;
    7.  
    8.  
    9. import org.apache.http.HttpEntity;
    10. import org.apache.http.HttpResponse;
    11. import org.apache.http.client.HttpClient;
    12. import org.apache.http.client.methods.HttpGet;
    13. import org.apache.http.impl.client.DefaultHttpClient;
    14. import org.apache.http.params.BasicHttpParams;
    15. import org.apache.http.params.HttpConnectionParams;
    16. import org.apache.http.params.HttpParams;
    17. import org.json.*;
    18.  
    19.  
    20. /**
    21.  * Handles the REST interface provided by the web service.
    22.  *
    23.  * @author michael
    24.  *
    25.  */
    26. public class RestHandler {
    27.  
    28.        URI                     url;                            // Address of the service
    29.        HttpClient              hc;                                     // The HTTP client used to manage the connection
    30.        int                     statusCode;                     // The HTTP status code of the response
    31.        JSONObject              json;
    32.  
    33.        // Constants
    34.        public static final String DELETE = "DELETE";
    35.        public static final String GET = "GET";
    36.        public static final String POST = "POST";
    37.        public static final String PUT = "PUT";
    38.  
    39.        // Server constants
    40.        public static final String SERVER = "http://www.guardian.co.uk";
    41.        public static final String PORT = ":80";
    42.        public static final String RESOURCE_DIR = "/politics/api/general-
    43. election/";
    44.        public static final int    CONNECTION_TIMEOUT = 5000;
    45.        public static final int    SOCKET_TIMEOUT = 15000;
    46.  
    47.        public static final String REST_URI =
    48.                RestHandler.SERVER + RestHandler.RESOURCE_DIR;
    49.  
    50.        /**
    51.         * Constructor
    52.         * @param url RESTful URL to connect to
    53.         */
    54.        public RestHandler(String url)
    55.        {
    56.                try
    57.                {
    58.                        this.url = new URI(url);
    59.                }
    60.                catch(URISyntaxException murl)
    61.                {
    62.                        this.url = null;
    63.                }
    64.  
    65.                this.statusCode = -1;
    66.  
    67.                System.out.println("Creates REST resouce at " + this.url);
    68.        }
    69.  
    70.        //
    71.        // GET and SET methods
    72.        //
    73.  
    74.    /**
    75.    * Retrieves the JSON object populated from the connect() function
    76.    * @return The retrieved JSON object
    77.    */
    78.        public JSONObject getJSON()
    79.        {
    80.                return json;
    81.        }
    82.  
    83.        /**
    84.         * Get the HTTP status code of the connection
    85.         * @return The HTTP status code of the connection
    86.         */
    87.        public int getStatusCode()
    88.        {
    89.                return this.statusCode;
    90.        }
    91.  
    92.        /**
    93.         * Converts an inputStream into a readable string.
    94.         * @param in The InputStream to be converted
    95.         * @return The converted String
    96.         */
    97.        /*
    98.        public String streamToString(InputStream in)
    99.        {
    100.                // Set up vars for reading
    101.                BufferedReader buffer = new BufferedReader(new
    102. InputStreamReader(in));
    103.                StringBuilder sb = new StringBuilder();
    104.                String line;
    105.  
    106.                try
    107.                {
    108.                        //TODO - Getting an OutOfMemoryError problem when loading the 1997
    109. election
    110.                        while((line = buffer.readLine()) != null)
    111.                        {
    112.                                // Add the current line to the buffer
    113.                                sb.append(line + "\n");
    114.                        }
    115.                }
    116.                catch(OutOfMemoryError me)
    117.                {
    118.                        System.err.println("Out of memory when downloading information");
    119.                        me.printStackTrace();
    120.                }
    121.                catch(Exception e)
    122.                {
    123.                        System.err.println("Error converting in function streamToString");
    124.                        e.printStackTrace();
    125.                }
    126.                finally
    127.                {
    128.                        // Close the stream
    129.                        try
    130.                        {
    131.                                in.close();
    132.                        }
    133.                        catch(IOException e)
    134.                        {
    135.                                System.err.println("Error closing stream in function
    136. streamToString");
    137.                                e.printStackTrace();
    138.                        }
    139.                }
    140.  
    141.                return sb.toString();
    142.        }
    143.        */
    144.  
    145.        /**
    146.         * Establishes a connection to the requested service
    147.         */
    148.        public void connect() throws JSONException
    149.        {
    150.                // Local vars
    151.                HttpResponse    response;
    152.                HttpEntity              entity;
    153.  
    154.                // Set up the connection
    155.  
    156.                HttpGet hGet = new HttpGet(url);
    157.                hGet.addHeader("Accept", "application/json");
    158.  
    159.                // Set the timeouts
    160.                HttpParams hp = new BasicHttpParams();
    161.                HttpConnectionParams.setConnectionTimeout(hp, 5000);    // 5 seconds to
    162. establish connection
    163.                HttpConnectionParams.setSoTimeout(hp, 10000);                   // 10 seconds to
    164. transfer data
    165.  
    166.                // Create the client
    167.                hc = new DefaultHttpClient(hp);
    168.  
    169.  
    170.                // try the connection
    171.                try
    172.                {
    173.  
    174.                        // Make the request
    175.                        response = hc.execute(hGet);
    176.  
    177.                        entity = response.getEntity();
    178.                        this.statusCode = response.getStatusLine().getStatusCode();
    179.  
    180.                        if(entity != null)
    181.                        {
    182.                                // Get the contents of the entity
    183.                                InputStream in = entity.getContent();
    184.  
    185.                                /*
    186.                                 * Use JSON to parse
    187.                                 *
    188.                                 * The buffer is fed directly into the constructor
    189.                                 * as this prevents an OutOfMemoryError from
    190.                                 * occurring. Can't use a separate string!
    191.                                 */
    192.                                BufferedReader buffer = new BufferedReader(new
    193. InputStreamReader(in));
    194.                                json = new JSONObject(buffer.readLine());
    195.  
    196.                        }
    197.  
    198.  
    199.                }
    200.                catch(Exception e)
    201.                {
    202.                        System.err.println("Unable to connect to REST
    203. service due to " + e.getMessage());
    204.                        e.printStackTrace();
    205.                }
    206.  
    207.        }
    208.  
    209.  
    210.        /**
    211.         * Generic method for getting a particular attribute from a REST
    212. service
    213.         * @param uri The RESTful URL to be accessed
    214.         * @param tag The attribute to be retrieved by the
    215.         * @return returns a URI object
    216.         */
    217.        public static Object getFromURI(String uri, String tag)
    218.        {
    219.                // initial setup
    220.                RestHandler r = new RestHandler(uri);
    221.                Object result = new Object();
    222.  
    223.                try
    224.                {
    225.                        r.connect();
    226.                }
    227.                catch(JSONException je)
    228.                {
    229.                        je.printStackTrace();
    230.                        return null;
    231.                }
    232.  
    233.                // Get the arrays
    234.  
    235.                JSONObject tmp = r.getJSON();
    236.  
    237.                try
    238.                {
    239.                        // Try getting the object with that tag.
    240.                        // Fails if tag not found
    241.                        result = tmp.get(tag);
    242.                }
    243.                catch(JSONException joe)
    244.                {
    245.                        // tag not found
    246.                        result = null;
    247.  
    248.                }
    249.  
    250.                // Return our match, if there is one
    251.                return result;
    252.  
    253.  
    254.        }
    255.  
    256.  
    257.        /**
    258.         * Outputs the structure of the JSON object to System.out, for
    259. debugging purposes
    260.         */
    261.        public void printJSON()
    262.        {
    263.                try
    264.                {
    265.                        System.out.print(json.toString(2));
    266.                }
    267.                catch(JSONException je)
    268.                {
    269.                        System.out.println("Unable to print JSON layout due to " +
    270. je.getMessage());
    271.                        je.printStackTrace();
    272.                }
    273.        }
    274. }
    275.  
    Any advice would be very much appreciated. Many thanks.
     

    Advertisement

  2. albert_pooholes

    albert_pooholes New Member
    5

    May 10, 2010
    3
    0
    5
    Did you ever find a good solution for constructing JSON objects from large files?
     
  3. DoddyUK

    DoddyUK Member
    Thread Starter
    35

    Jan 31, 2010
    12
    0
    35
  4. mrmanishs

    mrmanishs New Member
    5

    Oct 5, 2010
    1
    0
    5
    I'm trying to process a large JSON too, and am having a hard time how to parse it with Jackson. Can you let me know how you ended up parsing the data?
     

Share This Page

Loading...