Apps Custom video streaming to MediaPlayer

[ArthurGray]

Hi folks, I really need your help, having a hard time trying to debug this..
I'm building a custom video streaming service. I have a server application written in Python which is able to send a stream to my Android app. To display the contents, I'm using the native MediaPlayer class, and I've written my own media source class (called VideoDataSource) for it by extending MediaDataSource. I'll add the source code to the end of this post, but I think it's rather tedious so first I'll explain it so that you may just get some intuition to what is it I'm doing wrong here.

Operation --
The operating principle is simple: in the main activity creates an instance of VideoDataSource. within its constructor, this instance starts a new thread which uses the socket library to communicate with my server. When a new chunk of stream data is received by this thread, it proceeds to lock (or wait for unlocking) of a local file and flushes there all the data.
In the meantime, my main activity calls prepareAsync method to prepare the mediaplayer. This uses my VideoDataSource methods getSize and readAt. I have getSize always return -1 (size unknown), while readAt sleeps until the local file is unlocked, then proceeds to lock it and check if the data requested by the mediaplayer is ready. If so, it dumps it to the buffer provided by readAt call. If not, it keeps sleeping untill the other thread has downloaded it.

I have all of this run in the emulator with an mp4 video sample and it seems to be working fine. Whenever I try the same on my smartphone, the MediaPlayer throws the preparation error exception, as if the data is wrong and it doesen't recognize the type of data.

-- Debug attempts --
If I go and look at the downloaded files with the emulator and the smartphone, they are exactly the same.
I started suspecting that it was a synchronization problem of some kind, since the emulator was in the same machine as the server it received the packets immediately, so I tried to delay the prepareAsync call by 10 seconds as to allow the smartphone to gather all info, with no success. Then I tried to log manually all the calls to readAt - I logged the input parameter values, the returned value (number of bytes read) as well as the raw data retrieved - to text files with both emulator and smartphone, then compare them one by one. Turns out the first 74 calls to readAt are exactly the same. Same input args, same output value, same data. The 75 is different in that in the smartphone requests a higher size of data. the following calls also are different, up to the 112th, when the smartphone just gives up and throws the exception. Now i thought that maybe my smartphone had a different version of the MediaPlayer for some reason, even if they both have API version 29 (my smartphone is a oneplus 6 which just recently received Pie update). Finally, I tried to manually crop the sample video file in the server to a few KB, so that it could be sent entirely with a single tcp packet... and the smartphone now visualizes it correctly. To this, i don't know exactly what to think. When the smarpthone failed, it was reading only data from the first packet either way, thus the reception of a second packet does not affect the functioning of this. I even tried back with the normal length sequence and a modified version of the server which only sends the first packet. And the video wasn't displayed. The app has no way to know how big is the source file..this is driving me crazy.

Now for the code, I'll post the major classes. If you need more, i'll upload the whole project..

MainActivity.java

package com.example.andre.videoviewer2;

import android.app.Activity;
import android.os.Bundle;
import android.media.MediaPlayer;
import android.util.Log;
import android.view.SurfaceHolder;
import android.view.SurfaceView;

import com.example.andre.videoviewer2.StreamServices.VideoDataSource;


public class MainActivity extends Activity
        implements SurfaceHolder.Callback, MediaPlayer.OnPreparedListener
{
    private MediaPlayer mp = null;
    private SurfaceHolder surfaceHolder = null;
    private SurfaceView surfaceView = null;

    @Override
    protected void onCreate(Bundle savedInstanceState)
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        surfaceView = (SurfaceView)findViewById(R.id.surfaceView);
        surfaceHolder = surfaceView.getHolder();
        surfaceHolder.addCallback(MainActivity.this);
    }

    @Override
    public void surfaceCreated(SurfaceHolder holder)
    {
        mp = new MediaPlayer();
        mp.setDisplay(surfaceHolder);
        VideoDataSource vds = new VideoDataSource(18892, this);
        mp.setDataSource(vds);
        mp.setOnPreparedListener(this);
        try {
            Thread.sleep(10000);
            mp.prepareAsync();
        }
        catch(Exception e)
        {
            Log.e("ANTANI",e.getMessage()+"\nMainActivity.java:"+e.getStackTrace()[0].getLineNumber());
        }
    }

    @Override
    protected void onResume() {
        super.onResume();
    }

    @Override
    protected void onPause() {
        super.onPause();
        releaseMediaPlayer();
    }

    @Override
    protected void onDestroy() {
        super.onDestroy();
        releaseMediaPlayer();
    }

    @Override
    public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {

    }

    @Override
    public void surfaceDestroyed(SurfaceHolder holder) {

    }

    @Override
    public void onPrepared(MediaPlayer mp) {
        Log.d("ANTANI","onPrepared received");
        this.mp.start();
    }

    private void releaseMediaPlayer()
    {
        if( mp != null )
        {
            mp.release();
            mp = null;
        }
    }
}

VideoDataSource.java

package com.example.andre.videoviewer2.StreamServices;


import android.content.Context;
import android.media.MediaDataSource;
import android.os.Environment;
import android.util.Log;

import com.example.andre.videoviewer2.Const;
import com.example.andre.videoviewer2.Utils;

import java.io.File;
import java.io.FileOutputStream;
import java.io.OutputStreamWriter;
import java.io.RandomAccessFile;
import java.net.Socket;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;


public class VideoDataSource extends MediaDataSource
{
    public dataBox db = new dataBox();
    private Thread t = null;

    private List getChunkChain( int position, int size ) {
        // position and size describe a chunk of data from the original source file.
        // If the chunk of data has been entirely downloaded, this method returns a list
        // of ChunkInfo objects describing the locations inside the resource file where to
        // look for the data. If the chunk has not been downloaded, returns null
        int count = db.chunkTable.size();
        if( count == 0 )
            return null;
        boolean completed = false;
        List chunkChain = new ArrayList();
        ChunkInfo c;
        int bytesToRead = size;
        while( !completed ) {
            boolean added = false;
            for (int i = 0; i < count; i++) {
                c = ((ChunkInfo) db.chunkTable.get(i));
                if (position >= c.position && position <= c.position+c.length-1) {
                    chunkChain.add(c);
                    int availableBytes = c.position+c.length-position;
                    if( availableBytes >= bytesToRead )
                        completed = true;
                    bytesToRead -= availableBytes;
                    added = true;
                    break;
                }
            }
            if (! added ) return null;
        }
        return chunkChain;
    }
    private int chunkChainCount = 0;
    public int readAt(long position, byte[] buffer, int offset, int size) {
        // Read size bytes from source at position pos inside buffer at position offset
        try {
            int pos = (int) position;
            Log.d("ANTANI-readAt",String.format(Locale.US,"call readAt(offs:%d,size%d)",pos,size));

            // wait for size packet reception
            int sleepCount = 0;
            while(db.waitingForVideoSize) {
                Thread.sleep(50 );
                if(sleepCount == 0) {
                    Log.d("ANTANI-readAt","waiting for video size");
                    sleepCount = 1;
                }
            }

            // no data
            if(db.videoSize == 0) return(0);

            // end of stream
            if(position >= db.videoSize) return (-1);

            // Wait untill data is completely received
            List chunkChain = getChunkChain(pos, size);

            sleepCount = 0;
            while (chunkChain == null){
                Thread.sleep(30);
                chunkChain = getChunkChain(pos, size);
                if(sleepCount == 0) {
                    Log.d("ANTANI-readAt","waiting for data to be ready");
                    sleepCount = 1;
                }
                //Log.d("ANTANI","readAt: waiting for data to be available");
            }
            chunkChainCount += 1;

            // Wait while file is locked
            sleepCount = 0;
            while( db.fileLocked ) {
                Thread.sleep(20);
                if( sleepCount == 0) {
                    sleepCount = 1;
                    Log.d("ANTANI-readAt","waiting for file to be free");
                }
            }

            // Lock file and follow the chain to read all the data into buffer
            db.fileLocked = true;
            int bytesToRead, readBytes = 0, availableBytes;
            RandomAccessFile raf = new RandomAccessFile(db.trackResource,"r");
            for(int i=0;i<chunkChain.size();i++) {
                ChunkInfo c = (ChunkInfo)chunkChain.get(i);
                if(i == 0) {
                    raf.seek(c.offset + pos - c.position);
                    availableBytes = c.length - (pos - c.position);
                    bytesToRead = availableBytes < size ? availableBytes : size;
                }
                else {
                    raf.seek(c.offset);
                    bytesToRead = size-readBytes < c.length ? size-readBytes : c.length;
                }
                raf.read(buffer,offset+readBytes, bytesToRead);
                readBytes += bytesToRead;
            }
            /*if( chunkChainCount == 73 ) {
                String msg = "";
                File logg = new File(
                        db.context.getExternalFilesDir(Environment.DIRECTORY_MOVIES),
                        "bad"+Integer.toString(chunkChainCount)+".txt"
                );
                FileOutputStream out = new FileOutputStream(logg);
                int doneBytes = 0, doAmount, pass = 0;
                while (true) {
                    pass += 1;
                    msg = "fulldata" + Integer.toString(pass) + ":[";
                    doAmount = 100 < size - doneBytes ? 100 : size - doneBytes;
                    if (doAmount == 0) break;
                    for (int jj = 0; jj < doAmount; jj++)
                        msg += Byte.toString(buffer[offset + doneBytes + jj]) + ",";
                    doneBytes += doAmount;
                    msg += "]\n";
                    out.write(msg.getBytes());
                }
                out.close();
            }*/

            // free the file and return
            raf.close();
            db.fileLocked = false;
            Log.d("ANTANI-readAt","read "+Integer.toString(readBytes)+" bytes.");
            return readBytes;
        }
        catch(Exception ioe)
        {
            Log.e("ANTANI",ioe.getMessage());
            return 0;
        }
    }
    public long getSize() {
        return -1;
    }
    public void close() {
        dispose();
    }

    public VideoDataSource(int trackID, Context context) {
        dispose();
        try {
            db.trackID = trackID;
            db.context = context;

            // Delete track resource file if present, create file resource for later access
            File ff = new File(context.getExternalFilesDir(Environment.DIRECTORY_MOVIES),
                    Integer.toString(db.trackID)+".res");
            if(ff.exists())
                if(!ff.delete())
                    Log.e("ANTANI","unable to delete resource file!");
            db.trackResource = ff;

            // Start new thread to receive the stream
            t = new Thread( new receiveThread(db));
            t.start();
        }
        catch( Exception e ) {
            Log.e("ANTANI",e.getMessage()+"\nVideoDataSource.java:"+e.getStackTrace()[0].getLineNumber());
        }
    }

    private void dispose() {
        try {
            if (db.sock != null)
            {
                Log.d("ANTANI","called dispose..");
                //Utils.sendPacket(db.sock,new byte[] {(byte)Const.PACKET_QUIT},0,1);
                //db.sock.close();
                //db.sock = null;
            }
            if (t != null)
            {
                //t.interrupt();
                //t = null;
            }
        }
        catch( Exception e )
        {
            Log.e("ANTANI",e.getMessage()+"\nVideoDataSource.java:"+e.getStackTrace()[0].getLineNumber());
        }
    }
}

receiveThread.java

package com.example.andre.videoviewer2.StreamServices;

import android.util.Log;
import com.example.andre.videoviewer2.Const;
import com.example.andre.videoviewer2.Utils;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.net.Socket;
import java.nio.ByteBuffer;



class receiveThread implements Runnable {
    private Socket sock = null;
    private InputStream inStream = null;
    private byte[] inBuff = new byte[1024];
    private byte[] packetBuffer = new byte[8000000];
    private dataBox db;

    receiveThread(dataBox db) {
        this.db = db;
    }
    private int globalFileOffset;
    public void run() {
        try {

            if(sock == null) {
                globalFileOffset = 0;
                String host = "192.168.1.70"; //"10.0.2.2";
                sock = new Socket(host, 9934);
                db.sock = sock;
                inStream = db.sock.getInputStream();
                // Send the first packet to the server requesting the desired track

                byte[] intArr = ByteBuffer.allocate(4).putInt(db.trackID).array();
                Utils.sendPacket(db.sock, new byte[]{(byte) Const.PACKET_REQUEST, intArr[0], intArr[1], intArr[2], intArr[3]}, 0, 5);
            }

            int bytesRead = 0;
            int offset = 0;
            while ((bytesRead = inStream.read(inBuff,0, inBuff.length)) != -1)
            {
                // Collect phase: collect the received bytes inside packetBuffer until the
                System.arraycopy(inBuff,0,packetBuffer, offset, bytesRead);
                offset += bytesRead;
                int dataLen, surplus;
                if(offset > 1) {
                    switch ((int) packetBuffer[0]) {
                        case Const.PACKET_CHUNK:
                            if (offset < 9) break;
                            dataLen = ByteBuffer.wrap(packetBuffer, 5, 4).getInt();
                            if(offset >= dataLen + 9) {
                                // packet complete, do what you need
                                int pos = ByteBuffer.wrap(packetBuffer, 1, 4).getInt();

                                // Wait for file to be writable
                                while (db.fileLocked) Thread.sleep(1);
                                // Lock the file for my write operation
                                db.fileLocked = true;

                                FileOutputStream out = new FileOutputStream(db.trackResource, true);
                                out.write(packetBuffer, 9, dataLen);
                                out.close();

                                db.chunkTable.add(new ChunkInfo(pos, dataLen, globalFileOffset));
                                globalFileOffset += dataLen;
                                db.fileLocked = false;

                                surplus = offset - (9+dataLen);
                                for(int i=0; i<surplus; i++)
                                    packetBuffer[i] = packetBuffer[9+dataLen+i];
                                offset = surplus;
                                Log.d("ANTANI","received chunk of data!");
                                Utils.sendPacket(db.sock,new byte[]{(byte)Const.PACKET_ACK},0,1);
                                break;
                            }
                            break;
                        // note: db.videoSize and db.waitingForVideoSize are not actually used
                        case Const.PACKET_SIZE:
                            if( offset < 5 ) break;
                            // packet complete: do what you need to do
                            db.videoSize = ByteBuffer.wrap(packetBuffer,1,4).getInt();
                            db.waitingForVideoSize = false;
                            // reset offset
                            Log.d("ANTANI","received size");
                            surplus = offset - (5);
                            for (int i = 0; i < surplus; i++)
                                packetBuffer[i] = packetBuffer[5 + i];
                            offset = surplus;
                            break;

                        default:
                            Log.e("ANTANI","unknown type packet received");
                    }
                }
            }
        }
        catch( Exception e ) {
            Log.e("ANTANI",e.getMessage()+"\nreceiveThread.java:"+e.getStackTrace()[0].getLineNumber());
        }
    }
}

Thanks in advance to anyone who'll have the patience to help me figure out what's wrong

 

[Deleted User]

Firstly, rather than using the emulator to debug, connect the phone to your computer, and deploy the app to it from Android Studio. Run the app in debug mode, and use breakpoints in your code, rather than Log statements.

Also, can you show the exception stack trace?

 

[ArthurGray]

Ok when I run the app in the phone i get a simple error and not an exception. I must have confused it. Here it is:

E/MediaPlayerNative: error (100, 1)
E/MediaPlayer: Error (100,1)
E/MediaPlayerNative: error (1, -2147483648)
E/MediaPlayer: Error (1,-2147483648)

It's quite hard to debug with breakpoints a stream. i tried a little bit but never found wrong data or variables...
I tried to run the app in the phone and have the readAt function called by mediaPlayer log once again to text file all the data to log files. I then moved those files in my computer, wronte a simple python program to compare the data to the actual video file and they are seamless. In other words:
when mediaPlayer requests some data through readAt( position, buffer, offset, size), it is requesting size bytes from the source starting at position, and saved to buffer starting at offset.
I have checked that the bytes I put in buffer with the app streaming to my phone from my desktop are exactly the bytes that i can find at the right position in my source mp4 file, and yet mediaplayer cannot
display my source..

Perhaps the mediaplayer is susceptible to the sleep function I use when the data is not ready yet?
This is odd...

 

[Dannydet]

Use VLC media player instead

 

[ArthurGray]

Dunno about that, I wanted to try my best with the stock mediaplayer since it is designed to allow the user to create its own media source class.. and I'm still in troubles, I can't imagine how hard it would be for me to reverse engineer some code and add my own source.. or does also vlc provide the explicit possibility to create your own source?

 

[Dannydet]

That I can't answer.
But it is hands down the best video player for PCs and android devices

 

[ArthurGray]

Ok thank you I'll consider using something else if I really cannot get out of this.
For now I'm making some progress.
It seems to be correct that MediaPlaer is not able to handle sleep or strange timings in readAt function. After the large amount of testing I did, it appears that everything goes allright if the data is immediately available when the call to readAt arrives. To test this, I implemented the onError listener and I'm resetting the mediaplayer everytime it gets an error. While the MediaPlayer is reset, my network stream receiver thread goes on to collect more data, once it receives enough packets that readAt needs not to sleep, and after a good amount of resets, the mediaplayer finally displays the video correctly.

It's strange because it feels quite a "fragile" class prone to errors even for playback of local files, but what do you know.. maybe it's supposed to be reset.

edit: Now i have more problems because for some reason after a certain number of resets, the mediaplayer gets stuck at some point in the stream and the picture is still, yet it doesent generate errors so it is not reset..