Take photo in background - ARCore + Unity3D

[Rich3030]

I have built an Android plugin (.AAR file) for a Unity3d Augmented Reality app. The Augmented Reality Unity3d app is the standard HelloAR demo app. This AR app uses Google AR Core. Information on developing native Andriod plugins for Unity3D can be found here.

The Unity3D AR app loads the .AAR plugin and tries to open the camera, take a picture and save it to file, without any input from the user: no UI, no button press, and no preview of the capture to the user.

When my plugin attempts to take a picture, I get the following exception:

2020/08/06 18:47:15.857 19906 21180 Info CameraManagerGlobal Camera 1 facing CAMERA_FACING_FRONT state now CAMERA_STATE_OPEN for client com.cambridge.helloar API Level 1
2020/08/06 18:47:15.877 19906 19906 Warn System.err java.io.IOException: setPreviewTexture failed
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at android.hardware.Camera.setPreviewTexture(Native Method)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at com.cambridge.helloar.PluginManager.takePhoto(PluginManager.java:242)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at com.cambridge.helloar.PluginManager.access$100(PluginManager.java:38)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at com.cambridge.helloar.PluginManager$4.run(PluginManager.java:351)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at android.os.Handler.handleCallback(Handler.java:883)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at android.os.Handler.dispatchMessage(Handler.java:100)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at android.os.Looper.loop(Looper.java:237)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at android.app.ActivityThread.main(ActivityThread.java:8107)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at java.lang.reflect.Method.invoke(Native Method)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:496)
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1100)
2020/08/06 18:47:15.877 19906 19906 Debug TAKEPHOTO Opened camera
2020/08/06 18:47:15.877 19906 19906 Debug TAKEPHOTO Started preview
2020/08/06 18:47:15.877 19906 19906 Warn System.err java.lang.RuntimeException: takePicture failed
2020/08/06 18:47:15.877 19906 19906 Warn System.err    at android.hardware.Camera.native_takePicture(Native Method)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at android.hardware.Camera.takePicture(Camera.java:1564)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at android.hardware.Camera.takePicture(Camera.java:1506)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at com.cambridge.helloar.PluginManager.takePhoto(PluginManager.java:250)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at com.cambridge.helloar.PluginManager.access$100(PluginManager.java:38)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at com.cambridge.helloar.PluginManager$4.run(PluginManager.java:351)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at android.os.Handler.handleCallback(Handler.java:883)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at android.os.Handler.dispatchMessage(Handler.java:100)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at android.os.Looper.loop(Looper.java:237)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at android.app.ActivityThread.main(ActivityThread.java:8107)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at java.lang.reflect.Method.invoke(Native Method)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:496)
2020/08/06 18:47:15.878 19906 19906 Warn System.err    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1100)

I've tried both the front and back camera but I get this exception in both cases.

I have also noticed that I get a warning on my AR app saying something like "another app is using the camera. Tap this app or try closing the other app", so it's definitely something to do with the camera being blocked by ARCore. I do not want to prevent the Unity3D AR app from using the camera... of course. Instead, I want the AR app and my plugin to BOTH have access to the camera.

I know my Plugin code works because doing this in a non-AR app works and a picture is taken in the background.

Here is the code:

@SuppressWarnings("deprecation")
private static void takePhoto() {

    Log.d("TAKEPHOTO", "Surface created");

    SurfaceTexture surfaceTexture = new SurfaceTexture(10);
    Camera camera = Camera.open(0);
    camera.getParameters().setPreviewSize(1, 1);
    try {
        camera.setPreviewTexture(surfaceTexture);
    } catch (IOException e) {
        e.printStackTrace();
    }

    Log.d("TAKEPHOTO", "Opened camera");

    Log.d("TAKEPHOTO", "Started preview");
    camera.takePicture(null, null, new Camera.PictureCallback() {

        @Override
        public void onPictureTaken(byte[] data, Camera camera) {
            Log.d("TAKEPHOTO", "Took picture");

            File pictureFileDir = new File("/sdcard/CaptureByService");
            if (!pictureFileDir.exists() && !pictureFileDir.mkdirs()) {
                return;
            }
            SimpleDateFormat dateFormat = new SimpleDateFormat("yyyymmddhhmmss");
            String date = dateFormat.format(new Date());
            String photoFile = "PictureFront_" + "_" + date + ".jpg";
            String filename = pictureFileDir.getPath() + File.separator + photoFile;
            File mainPicture = new File(filename);
            try {
                FileOutputStream fos = new FileOutputStream(mainPicture);
                fos.write(data);
                fos.close();
                Log.d("TAKEPHOTO", "image saved");
            } catch (Exception error) {
                Log.d("TAKEPHOTO", "Image could not be saved");
                error.printStackTrace();
            }
            camera.release();
        }
    });
}

Does anyone have any suggestions/solutions?

 

[Daveroonie]

Have you found a solution for this? I am trying to make an app that visualizes the trackable points in the scene, but takes a photo every time the phone stops moving. Tried some plugins in Unity, but did not work with my phone. Then I tried building it in Android Studio. I have it set up so that it can either take a photo when the camera stops OR visualize the trackable points, but not both. For the AR points, I am using an ArFragment. In order for the camera to work, I need to comment out this fragment and all references to it. Any ideas?

 

[Rich3030]

Have you found a solution for this? I am trying to make an app that visualizes the trackable points in the scene, but takes a photo every time the phone stops moving. Tried some plugins in Unity, but did not work with my phone. Then I tried building it in Android Studio. I have it set up so that it can either take a photo when the camera stops OR visualize the trackable points, but not both. For the AR points, I am using an ArFragment. In order for the camera to work, I need to comment out this fragment and all references to it. Any ideas?

No, I never did. I read something about "shared camera access", but AFAIK, it's basically pausing the camera, letting another process use it, then retaking control. Also, in Android 11 we will be able to access both the front and back cameras at the same time (probably doesn't help you, though!).