How to Build a Surveillance Robot With Twitter, Android Things, and Raspberry Pi - Owlcation - Education
Updated date:

How to Build a Surveillance Robot With Twitter, Android Things, and Raspberry Pi

Davinder Singh is a Polymath. He loves reading, learning and writing technical cookbooks about new technologies.

What You Will Learn in This Article?

  • You will learn how to use the camera module for taking pictures and videos.
  • You will learn how to connect and then program the camera module with Raspberry Pi.
  • You will learn how to use and implement the Twitter Api.
  • You will learn the internals of Android Things such as permissions, manifest and how to add external libraries in the project.

Lastly, you will learn how to handle Camera through Application Program Interface(API) framework provided by Android and thus you can take knowledge from here and make your own twitter client for Android Mobile Application.

What Will This Article Not Teach You?

  • This is surely not a “How to code in java” article. Hence, you won't learn Java in this one.
  • This is also not a “How to code?” article.

Prerequisites

Before we start you will require following things by your side

  • A computer running Mac, Linux or Windows.
  • A stable internet connection.
  • A raspberry Pi 3 with Android Things installed (How to do it?).
  • A raspberry Pi compatible camera module.
  • Android Studio (Installing Android Studio)
  • Beginner or greater level of experience in programming.

Step 1: Download Twitter Java API

API or Application Program Interface is like a bridge between client (we) and service (in this case twitter). We will use twitter4j to access the twitter. Twitter4j is written in and for Java programming language hence the name. All android applications are written in Java or Kotlin (which in turn gets compiled to Java). Goto twitter4j's site and download latest version of library. It should be a zip file. There will be many directories inside the zip (Don't get panic!). We only require lib directory.

Step 2: Create a New Android Things Project

Let's create a new project. At this point I am assuming that you have already installed Android studio and Android software development kit(SDK) and that it's working. Start the studio and create a new project. If you are running studio version > 3.0 then goto Android Things tabs and select Android Things Empty Activity and click next. Otherwise, check Android Things checkbox right at the bottom of creating a new project dialog or window.

Android Things

Android Things

Step 3: Configure the Project

Configure the project

Configure the project

Configure the activity

Configure the activity

Step 4: Importing the Twitter4j

Before we can use twitter4j, We first have to import it into our project.

  • Goto lib directory in twitter4j's zip folder and copy all the files except twitter4j-examples-4.0.7.jar and Readme.txt.
  • Switch back to android studio and change project view type from android to project tree.
Project Tree view type

Project Tree view type

  • In the directory tree look for the lib directory and right click and then select paste and then OK. It will copy all the jar files in the lib folder.
Lib folder

Lib folder

Step 5: Adding Permissions in Manifest

Android operating system is very serious about security and hence it requires declaration of every hardware or features being used by application in application's manifest. Manifest is like a summary of android application. It contains features used by application, name of application, package name other metadata. We will be using Internet and Camera so application manifest must contain these two.

  • Goto Manifest file under manifest directory.
  • Add following lines after “<manifest .... >” tags.
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="io.vendator.dav.twittbot">
  
 <!--Copy and paste these lines -->
    <uses-permission android:name="android.permission.INTERNET"/>
    <uses-permission android:name="android.permission.CAMERA"/>
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
  <!--Don't alter the rest -->

    <application android:label="@string/app_name">
        <uses-library android:name="com.google.android.things" />

        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
            <!-- Make this the first activity that is displayed when the device boots. -->
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />

                <category android:name="android.intent.category.HOME" />
                <category android:name="android.intent.category.DEFAULT" />
            </intent-filter>
        </activity>
    </application>
</manifest>

At this point try building the project (Build -> Rebuild project). Make sure everything is error free.

Step 6: Adding a Camera Handler Class

In this step we will add a new class to the project containing all the code to manage the Camera for us.

  • Goto File and then New and click on create new java class
  • Give this class name CameraHandler

Note: In java name of class must match with the name of file otherwise java compiler will complain.

At this point your project should contain two files MainActivity and CameraHandler. We will alter MainActivity later. Let's add camera handling code in CameraHandler. I am assuming that you have at least beginner level experience in object oriented programming language that is not necessarily in Java.

  • Add following fields in the class. (As you typing these fields you will get error from IDE that following symbol is not found that's because the required library is not imported. Just hit ctrl + Enter or alt + Enter (Mac) and that should do the trick)
public class CameraHandler {
     //TAG for debugging purpose
    private static final String TAG = CameraHandler.class.getSimpleName();

    //You can change these parameters to the required resolution
    private static final int IMAGE_WIDTH = 1024;
    private static final int IMAGE_HEIGHT = 720;

   //Number of images per interval
    private static final int MAX_IMAGES = 1;
    
    private CameraDevice mCameraDevice;
   //Every picture capture event is handled by this object
    private CameraCaptureSession mCaptureSession;

    /**
     * An {@link ImageReader} that handles still image capture.
     */
    private ImageReader mImageReader;
}
  • Now let's add a few constructors to the class and logic to initialise the camera. A constructor is a special function or method or block of code which contain the logic for creating the object out of class (A class is analogous to blueprint of building while an object is actual building)
//Add following after mImageReader
  

  //Private constructor means this class cannot be constructed from outside
  //This is part of Singleton pattern. Where only a single object can be made from class
    private CameraHandler() {
    }

   //This is nested static class, used to hold the object that we've created
   //so that it can be returned when required and we don't have to create a new object everytime
    private static class InstanceHolder {
        private static CameraHandler mCamera = new CameraHandler();
    }

    //This returns the actual object
    public static CameraHandler getInstance() {
        return InstanceHolder.mCamera;
    }

    /**
     * Initialize the camera device
     */
    public void initializeCamera(Context context /*Context is android specific object*/,
                                 Handler backgroundHandler,
                                 ImageReader.OnImageAvailableListener imageAvailableListener) {
        // Discover the camera instance
        CameraManager manager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
        String[] camIds = {};
        try {
            camIds = manager.getCameraIdList();
        } catch (CameraAccessException e) {
            Log.e(TAG, "Cam access exception getting IDs", e);
        }
        if (camIds.length < 1) {
            Log.e(TAG, "No cameras found");
            return;
        }
        String id = camIds[0];
        Log.d(TAG, "Using camera id " + id);

        // Initialize the image processor
        mImageReader = ImageReader.newInstance(IMAGE_WIDTH, IMAGE_HEIGHT,
                ImageFormat.YUY2, MAX_IMAGES);
        mImageReader.setOnImageAvailableListener(
                imageAvailableListener, backgroundHandler);

        // Open the camera resource
        try {
            manager.openCamera(id, mStateCallback, backgroundHandler);
        } catch (CameraAccessException cae) {
            Log.d(TAG, "Camera access exception", cae);
        }
    }

//Make sure code is between starting and closing curly brackets of CameraHandler
  • After camera has been initialised, We need to add methods to control various other camera related tasks such as Image Capture, Saving Captured file and Shutting down the camera. These method uses code which is highly dependent on Android Framework and thus I'll not try to go in depth in it as this article is not about explaining the internals of framework. However you can see the android documentation here for further learning and research. For now just copy and paste the code.
//Full code for camera handler
public class CameraHandler {
    private static final String TAG = CameraHandler.class.getSimpleName();

    private static final int IMAGE_WIDTH = 1024;
    private static final int IMAGE_HEIGHT = 720;
    private static final int MAX_IMAGES = 1;

    private CameraDevice mCameraDevice;

    private CameraCaptureSession mCaptureSession;

    /**
     * An {@link ImageReader} that handles still image capture.
     */
    private ImageReader mImageReader;

    // Lazy-loaded singleton, so only one instance of the camera is created.
    private CameraHandler() {
    }

    private static class InstanceHolder {
        private static CameraHandler mCamera = new CameraHandler();
    }

    public static CameraHandler getInstance() {
        return InstanceHolder.mCamera;
    }

    /**
     * Initialize the camera device
     */
    public void initializeCamera(Context context,
                                 Handler backgroundHandler,
                                 ImageReader.OnImageAvailableListener imageAvailableListener) {
        // Discover the camera instance
        CameraManager manager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
        String[] camIds = {};
        try {
            camIds = manager.getCameraIdList();
        } catch (CameraAccessException e) {
            Log.e(TAG, "Cam access exception getting IDs", e);
        }
        if (camIds.length < 1) {
            Log.e(TAG, "No cameras found");
            return;
        }
        String id = camIds[0];
        Log.d(TAG, "Using camera id " + id);

        // Initialize the image processor
        mImageReader = ImageReader.newInstance(IMAGE_WIDTH, IMAGE_HEIGHT,
                ImageFormat.YUY2, MAX_IMAGES);
        mImageReader.setOnImageAvailableListener(
                imageAvailableListener, backgroundHandler);

        // Open the camera resource
        try {
            manager.openCamera(id, mStateCallback, backgroundHandler);
        } catch (CameraAccessException cae) {
            Log.d(TAG, "Camera access exception", cae);
        }
    }

    /**
     * Callback handling device state changes
     */
    private final CameraDevice.StateCallback mStateCallback = new CameraDevice.StateCallback() {
        @Override
        public void onOpened(CameraDevice cameraDevice) {
            Log.d(TAG, "Opened camera.");
            mCameraDevice = cameraDevice;
        }

        @Override
        public void onDisconnected(CameraDevice cameraDevice) {
            Log.d(TAG, "Camera disconnected, closing.");
            cameraDevice.close();
        }

        @Override
        public void onError(CameraDevice cameraDevice, int i) {
            Log.d(TAG, "Camera device error, closing.");
            cameraDevice.close();
        }

        @Override
        public void onClosed(CameraDevice cameraDevice) {
            Log.d(TAG, "Closed camera, releasing");
            mCameraDevice = null;
        }
    };

    /**
     * Begin a still image capture
     */
    public void takePicture() {
        if (mCameraDevice == null) {
            Log.e(TAG, "Cannot capture image. Camera not initialized.");
            return;
        }

        // Here, we create a CameraCaptureSession for capturing still images.
        try {
            mCameraDevice.createCaptureSession(
                    Collections.singletonList(mImageReader.getSurface()),
                    mSessionCallback,
                    null);
        } catch (CameraAccessException cae) {
            Log.e(TAG, "access exception while preparing pic", cae);
        }
    }

    /**
     * Callback handling session state changes
     */
    private CameraCaptureSession.StateCallback mSessionCallback =
            new CameraCaptureSession.StateCallback() {
                @Override
                public void onConfigured(CameraCaptureSession cameraCaptureSession) {
                    // The camera is already closed
                    if (mCameraDevice == null) {
                        return;
                    }

                    // When the session is ready, we start capture.
                    mCaptureSession = cameraCaptureSession;
                    triggerImageCapture();
                }

                @Override
                public void onConfigureFailed(CameraCaptureSession cameraCaptureSession) {
                    Log.e(TAG, "Failed to configure camera");
                }
            };

    /**
     * Execute a new capture request within the active session
     */
    private void triggerImageCapture() {
        try {
            final CaptureRequest.Builder captureBuilder =
                    mCameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_STILL_CAPTURE);
            captureBuilder.addTarget(mImageReader.getSurface());
            captureBuilder.set(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_ON);
            Log.d(TAG, "Session initialized.");
            mCaptureSession.capture(captureBuilder.build(), mCaptureCallback, null);
        } catch (CameraAccessException cae) {
            Log.e(TAG, "camera capture exception", cae);
        }
    }

    /**
     * Callback handling capture session events
     */
    private final CameraCaptureSession.CaptureCallback mCaptureCallback =
            new CameraCaptureSession.CaptureCallback() {

                @Override
                public void onCaptureProgressed(CameraCaptureSession session,
                                                CaptureRequest request,
                                                CaptureResult partialResult) {
                    Log.d(TAG, "Partial result");
                }

                @Override
                public void onCaptureCompleted(CameraCaptureSession session,
                                               CaptureRequest request,
                                               TotalCaptureResult result) {
                    if (session != null) {
                        session.close();
                        mCaptureSession = null;
                        Log.d(TAG, "CaptureSession closed");
                    }
                }
            };


    /**
     * Close the camera resources
     */
    public void shutDown() {
        if (mCameraDevice != null) {
            mCameraDevice.close();
        }
    }

    /**
     * Helpful debugging method:  Dump all supported camera formats to log.  You don't need to run
     * this for normal operation, but it's very helpful when porting this code to different
     * hardware.
     */
    public static void dumpFormatInfo(Context context) {
        CameraManager manager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
        String[] camIds = {};
        try {
            camIds = manager.getCameraIdList();
        } catch (CameraAccessException e) {
            Log.d(TAG, "Cam access exception getting IDs");
        }
        if (camIds.length < 1) {
            Log.d(TAG, "No cameras found");
        }
        String id = camIds[0];
        Log.d(TAG, "Using camera id " + id);
        try {
            CameraCharacteristics characteristics = manager.getCameraCharacteristics(id);
            StreamConfigurationMap configs = characteristics.get(
                    CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
            for (int format : configs.getOutputFormats()) {
                Log.d(TAG, "Getting sizes for format: " + format);
                for (Size s : configs.getOutputSizes(format)) {
                    Log.d(TAG, "\t" + s.toString());
                }
            }
            int[] effects = characteristics.get(CameraCharacteristics.CONTROL_AVAILABLE_EFFECTS);
            for (int effect : effects) {
                Log.d(TAG, "Effect available: " + effect);
            }
        } catch (CameraAccessException e) {
            Log.d(TAG, "Cam access exception getting characteristics.");
        }
    }
}

Step 7: Take a Rest

Seriously, at this point you should take a moment to understand the code. Read the comment or have a sip of coffee. You have come a long way, and we are very close to our final thing.

Step 8: Creating a Twitter Application

Before we can access Twitter using twitter api we require some keys or secret passcodes which lets twitter's server know that we are legitimate developers and not here to abuse their's api. To get those passcodes we need to create an application in twitter's developer registry.

  • Goto Twitter developer site and login with your twitter credentials.
  • Create a new twitter developer request. Answer all the questions being asked by twitter and in the confirm your email address.
  • After confirming you will be forwarded to developer's dashboard. Click on create a new application.
  • Give app a name. In description write anything that you want (I wrote, “A bot that tweets images periodically.”) and lastly in website's url give name of website if you have otherwise type anything that qualifies as website url. And lastly at the end give 100 word description of application again use your creativity here. Once done click create app.

Step 9: The Twitter API

I am assuming you have correctly imported the twitter4j jars in lib directory inside the android things project. And project still builds fine without any errors (comment them if you have any I'll be happy to help). Now it's time to finally code the juicy part of application MainActivity (or whatever you named it).

  • Double click activity class to open it up in editor. Add the following fields inside the class.
public class MainActivity extends Activity {
       //Type these
        private Handler mCameraHander;  //A handler for camera thread
        private HandlerThread mCameraThread;  //CameraThread
        private Handler captureEvent; //EventHandler (imageCaptured etc.)
        private CameraHandler mCamera; //reference to CameraHandler object 
        private Twitter mTwitterClient; //reference to the twitter client
        private final String TAG = "TwitterBot";
       //Take image after every 4 second
        private final int IMAGE_CAPTURE_INTERVAL_MS = 4000;

//---Other methods
} //End of MainActivity
  • Now let's complete the twitter part. Add following code inside your activity
private Twitter setupTwitter() {
        ConfigurationBuilder configurationBuilder = new ConfigurationBuilder();
        configurationBuilder.setDebugEnabled(true)
            .setOAuthConsumerKey("") //Copy Consumer key from twitter application
            .setOAuthConsumerSecret("") //Copy Consumer secret from twitter application
            .setOAuthAccessToken("") //Copy Access token from twitter application
            .setOAuthAccessTokenSecret("") //Copy Access token secret from twitter application
            .setHttpConnectionTimeout(100000); //Maximum Timeout time

        TwitterFactory twitterFactory = new TwitterFactory(configurationBuilder.build());
        return twitterFactory.instance;
   }
Where to find keys

Where to find keys

  • Inside activity's onCreate method add the following code to get the twitter's instance and setup camera module.
 @Override protected void onCreate(Bundle savedInstanceState) {
            super.onCreate(savedInstanceState);
            setContentView(R.layout.activity_main);
            //Write following lines
			//To get rid of Networking on  main thread error
			//Note: This should not be done in production application
            StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build();
            StrictMode.setThreadPolicy(policy);
			
			//Just a harmless permission check
            if(checkSelfPermission(Manifest.permission.CAMERA)
                    != PackageManager.PERMISSION_GRANTED){
                Log.e(TAG,"No Permission");
                return;
            }

	    //Running camera in different thread so as not to block the main application
		
            mCameraThread = new HandlerThread("CameraBackground");
            mCameraThread.start();
            mCameraHander = new Handler(mCameraThread.getLooper());
            captureEvent = new Handler();
            captureEvent.post(capturer);

            mCamera = CameraHandler.getInstance();
            mCamera.initializeCamera(this,mCameraHander, mOnImageAvailableListener);
            mTwitterClient = setupTwitter();
        }
  • You probably have errors at the moment. Let's solve them by adding more code or I should say missing code.
 //Release the camera when we are done
  @Override public void onDestroy(){
            super.onDestroy();
            mCamera.shutDown();

            mCameraThread.quitSafely();
        }

//A listener called by camera when image has been captured
        private ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {
            @Override
            public void onImageAvailable(ImageReader imageReader) {
                Image image = imageReader.acquireLatestImage();

                ByteBuffer imageBuf = image.getPlanes()[0].getBuffer();
                final byte[] imageBytes = new byte[imageBuf.remaining()];
                imageBuf.get(imageBytes);
                image.close();
                onPictureTaken(imageBytes);
            }
        };

//Here we will post the image to twitter
        private void onPictureTaken(byte[] imageBytes) {
            //TODO:Add code to upload image here.
            Log.d(TAG,"Image Captured");
           
        }

//Runnable is section of code which runs on different thread.
//We are scheduling take picture after every 4th second
        private Runnable capturer = new Runnable() {
            @Override
            public void run() {
                mCamera.takePicture();
                captureEvent.postDelayed(capturer,IMAGE_CAPTURE_INTERVAL_MS);
            }
        };

Step 10: Finalising the TwitterBot

And we are just few lines of code away from having our very own Twitter bot. We have Camera capturing images and twitter api we just have to bridge the both. Let's do this.

 private void onPictureTaken(byte[] imageBytes) {
     Log.d(TAG,"Image Captured");
     String statusMessage = "Twitting picture from TwitterBot!! made by %your name%";
     StatusUpdate status = new StatusUpdate(message);
     status.setMedia(Date().toString(), new ByteArrayInputStream(imageBytes));  		 
     Log.e(TAG, mTwitterClient.updateStatus(status).toString());
    //here you can add a blinking led code to indicate successful tweeting.
}

Conclusion

Connect the raspberry pi and camera module through the interface wires. Follow the instruction came with the camera module. Lastly connect raspberry pi with computer and run the project (green arrow over the top right side). Select your raspberry pi in the list. Wait for build and restarting. Camera module should start blinking and hopefully you will see some weird images on your twitter account wall. If you ran into problems, just comment and I'll help you out. Thank you for reading.

Related Articles