Handling large JSON streams (OOM Error)


Last Updated:

  1. DoddyUK

    DoddyUK Member

    Hi,
    I'm currently in the process of creating an application that uses a
    large amount of data streamed from a web service. In this case, I'm
    using The Guardian's Politics API to make an application about UK
    General Elections. In some cases, I need to stream and parse about
    750KB of data - one example of this is when I'm loading data for all
    constituencies. (Example: http://www.guardian.co.uk/politics/api/general-election/1997/results/json)

    When I'm parsing this data, I occasionally get an OutOfMemoryError
    problem when creating the JSON object from the Buffer Reader -
    admittedly this is happening a lot less often than when I was using
    the streamToString function, but I still want to eliminate the
    possibility of an OOM Error wherever I can.

    So I'd like to ask, what's the best way of parsing large amounts of
    JSON data streamed over HTTP into a single JSON object?

    Here's the error log:
    Code (Text):
    1. [color=red]
    2. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862): Uncaught handler:
    3. thread Thread-16 exiting due to uncaught exception
    4. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):
    5. java.lang.OutOfMemoryError
    6. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    7. java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:
    8. 99)
    9. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    10. java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:
    11. 139)
    12. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    13. java.lang.StringBuilder.append(StringBuilder.java:282)
    14. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    15. java.io.BufferedReader.readLine(BufferedReader.java:415)
    16. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    17. com.mdodd.ElectionMonitor.resources.RestHandler.connect(RestHandler.java:
    18. 189)
    19. 05-14 02:15:33.381: ERROR/AndroidRuntime(21862):     at
    20. com.mdodd.ElectionMonitor.resources.ConstituenciesThread.run(ConstituenciesThread.java:
    21. 49)
    22. [/color]
    And here's RestHandler.java:
    Code (Text):
    1.  
    2. package com.mdodd.ElectionMonitor.resources;
    3.  
    4. import java.io.*;
    5. import java.net.URI;
    6. import java.net.URISyntaxException;
    7.  
    8.  
    9. import org.apache.http.HttpEntity;
    10. import org.apache.http.HttpResponse;
    11. import org.apache.http.client.HttpClient;
    12. import org.apache.http.client.methods.HttpGet;
    13. import org.apache.http.impl.client.DefaultHttpClient;
    14. import org.apache.http.params.BasicHttpParams;
    15. import org.apache.http.params.HttpConnectionParams;
    16. import org.apache.http.params.HttpParams;
    17. import org.json.*;
    18.  
    19.  
    20. /**
    21.  * Handles the REST interface provided by the web service.
    22.  *
    23.  * @author michael
    24.  *
    25.  */
    26. public class RestHandler {
    27.  
    28.        URI                     url;                            // Address of the service
    29.        HttpClient              hc;                                     // The HTTP client used to manage the connection
    30.        int                     statusCode;                     // The HTTP status code of the response
    31.        JSONObject              json;
    32.  
    33.        // Constants
    34.        public static final String DELETE = "DELETE";
    35.        public static final String GET = "GET";
    36.        public static final String POST = "POST";
    37.        public static final String PUT = "PUT";
    38.  
    39.        // Server constants
    40.        public static final String SERVER = "http://www.guardian.co.uk";
    41.        public static final String PORT = ":80";
    42.        public static final String RESOURCE_DIR = "/politics/api/general-
    43. election/";
    44.        public static final int    CONNECTION_TIMEOUT = 5000;
    45.        public static final int    SOCKET_TIMEOUT = 15000;
    46.  
    47.        public static final String REST_URI =
    48.                RestHandler.SERVER + RestHandler.RESOURCE_DIR;
    49.  
    50.        /**
    51.         * Constructor
    52.         * @param url RESTful URL to connect to
    53.         */
    54.        public RestHandler(String url)
    55.        {
    56.                try
    57.                {
    58.                        this.url = new URI(url);
    59.                }
    60.                catch(URISyntaxException murl)
    61.                {
    62.                        this.url = null;
    63.                }
    64.  
    65.                this.statusCode = -1;
    66.  
    67.                System.out.println("Creates REST resouce at " + this.url);
    68.        }
    69.  
    70.        //
    71.        // GET and SET methods
    72.        //
    73.  
    74.    /**
    75.    * Retrieves the JSON object populated from the connect() function
    76.    * @return The retrieved JSON object
    77.    */
    78.        public JSONObject getJSON()
    79.        {
    80.                return json;
    81.        }
    82.  
    83.        /**
    84.         * Get the HTTP status code of the connection
    85.         * @return The HTTP status code of the connection
    86.         */
    87.        public int getStatusCode()
    88.        {
    89.                return this.statusCode;
    90.        }
    91.  
    92.        /**
    93.         * Converts an inputStream into a readable string.
    94.         * @param in The InputStream to be converted
    95.         * @return The converted String
    96.         */
    97.        /*
    98.        public String streamToString(InputStream in)
    99.        {
    100.                // Set up vars for reading
    101.                BufferedReader buffer = new BufferedReader(new
    102. InputStreamReader(in));
    103.                StringBuilder sb = new StringBuilder();
    104.                String line;
    105.  
    106.                try
    107.                {
    108.                        //TODO - Getting an OutOfMemoryError problem when loading the 1997
    109. election
    110.                        while((line = buffer.readLine()) != null)
    111.                        {
    112.                                // Add the current line to the buffer
    113.                                sb.append(line + "\n");
    114.                        }
    115.                }
    116.                catch(OutOfMemoryError me)
    117.                {
    118.                        System.err.println("Out of memory when downloading information");
    119.                        me.printStackTrace();
    120.                }
    121.                catch(Exception e)
    122.                {
    123.                        System.err.println("Error converting in function streamToString");
    124.                        e.printStackTrace();
    125.                }
    126.                finally
    127.                {
    128.                        // Close the stream
    129.                        try
    130.                        {
    131.                                in.close();
    132.                        }
    133.                        catch(IOException e)
    134.                        {
    135.                                System.err.println("Error closing stream in function
    136. streamToString");
    137.                                e.printStackTrace();
    138.                        }
    139.                }
    140.  
    141.                return sb.toString();
    142.        }
    143.        */
    144.  
    145.        /**
    146.         * Establishes a connection to the requested service
    147.         */
    148.        public void connect() throws JSONException
    149.        {
    150.                // Local vars
    151.                HttpResponse    response;
    152.                HttpEntity              entity;
    153.  
    154.                // Set up the connection
    155.  
    156.                HttpGet hGet = new HttpGet(url);
    157.                hGet.addHeader("Accept", "application/json");
    158.  
    159.                // Set the timeouts
    160.                HttpParams hp = new BasicHttpParams();
    161.                HttpConnectionParams.setConnectionTimeout(hp, 5000);    // 5 seconds to
    162. establish connection
    163.                HttpConnectionParams.setSoTimeout(hp, 10000);                   // 10 seconds to
    164. transfer data
    165.  
    166.                // Create the client
    167.                hc = new DefaultHttpClient(hp);
    168.  
    169.  
    170.                // try the connection
    171.                try
    172.                {
    173.  
    174.                        // Make the request
    175.                        response = hc.execute(hGet);
    176.  
    177.                        entity = response.getEntity();
    178.                        this.statusCode = response.getStatusLine().getStatusCode();
    179.  
    180.                        if(entity != null)
    181.                        {
    182.                                // Get the contents of the entity
    183.                                InputStream in = entity.getContent();
    184.  
    185.                                /*
    186.                                 * Use JSON to parse
    187.                                 *
    188.                                 * The buffer is fed directly into the constructor
    189.                                 * as this prevents an OutOfMemoryError from
    190.                                 * occurring. Can't use a separate string!
    191.                                 */
    192.                                BufferedReader buffer = new BufferedReader(new
    193. InputStreamReader(in));
    194.                                json = new JSONObject(buffer.readLine());
    195.  
    196.                        }
    197.  
    198.  
    199.                }
    200.                catch(Exception e)
    201.                {
    202.                        System.err.println("Unable to connect to REST
    203. service due to " + e.getMessage());
    204.                        e.printStackTrace();
    205.                }
    206.  
    207.        }
    208.  
    209.  
    210.        /**
    211.         * Generic method for getting a particular attribute from a REST
    212. service
    213.         * @param uri The RESTful URL to be accessed
    214.         * @param tag The attribute to be retrieved by the
    215.         * @return returns a URI object
    216.         */
    217.        public static Object getFromURI(String uri, String tag)
    218.        {
    219.                // initial setup
    220.                RestHandler r = new RestHandler(uri);
    221.                Object result = new Object();
    222.  
    223.                try
    224.                {
    225.                        r.connect();
    226.                }
    227.                catch(JSONException je)
    228.                {
    229.                        je.printStackTrace();
    230.                        return null;
    231.                }
    232.  
    233.                // Get the arrays
    234.  
    235.                JSONObject tmp = r.getJSON();
    236.  
    237.                try
    238.                {
    239.                        // Try getting the object with that tag.
    240.                        // Fails if tag not found
    241.                        result = tmp.get(tag);
    242.                }
    243.                catch(JSONException joe)
    244.                {
    245.                        // tag not found
    246.                        result = null;
    247.  
    248.                }
    249.  
    250.                // Return our match, if there is one
    251.                return result;
    252.  
    253.  
    254.        }
    255.  
    256.  
    257.        /**
    258.         * Outputs the structure of the JSON object to System.out, for
    259. debugging purposes
    260.         */
    261.        public void printJSON()
    262.        {
    263.                try
    264.                {
    265.                        System.out.print(json.toString(2));
    266.                }
    267.                catch(JSONException je)
    268.                {
    269.                        System.out.println("Unable to print JSON layout due to " +
    270. je.getMessage());
    271.                        je.printStackTrace();
    272.                }
    273.        }
    274. }
    275.  
    Any advice would be very much appreciated. Many thanks.

    Advertisement
  2. albert_pooholes

    albert_pooholes New Member

    Did you ever find a good solution for constructing JSON objects from large files?
  3. DoddyUK

    DoddyUK Member

  4. mrmanishs

    mrmanishs New Member

    I'm trying to process a large JSON too, and am having a hard time how to parse it with Jackson. Can you let me know how you ended up parsing the data?

Share This Page