Wednesday, February 4, 2009

J2ME: Thumbnails Extraction of JPEG (Exif) Images made with Mobile Phone Camera

....continuation
Following are the code snippets for extracting the thumbnail from JPEG image/picture created from mobile phone camera. I have put comments inside the code where necessary, I guess they should be enough to explain the code, however, I would strongly recommend to study this link for understanding the Exif format. This will really help you understand the code. Please the read the comments carefully!!

Disclaimer: The code provided has been written for exploratory purposes (that’s why there are so many console output and debug statements), not following the official guideline of Software Engineering and therefore has not been tested thoroughly but it does provide a good foundation and good starting point if you intend to work on the topic. However, you are FREE to use the code for any purpose and be aware to test it before using for production uses!!!

To start with you need an object of ‘InputStream’ opened to the JPEG file. (We will discuss at some other time how to open a connection to a file). Pass this object to the extractThumbnail(). Rest of stuff is pretty much clear from the comments so I leave you here with it!!!

/*
This method is the starting point, it takes an 'InputStream' object as an
argument to the JPEG file
*/
public int extractThumbnail(InputStream in) throws Exception {
int current;
if (in == null) {
return 0;
}
current = read16Bytes(in);
if (current != 0xFFD8) { //SOI
error("Not a JPEG file");
return 0;
}
current = read16Bytes(in);

// Reading until 'APP' marker is reached
while (!(current == 0xFFE0 || current == 0xFFE1 || current == 0xFFE2 ||
current == 0xFFE3 || current == 0xFFE4 || current == 0xFFE5 ||
current == 0xFFE6 || current == 0xFFE7 || current == 0xFFE8 ||
current == 0xFFE9 || current == 0xFFEA || current == 0xFFEB ||
current == 0xFFEC || current == 0xFFED || current == 0xFFEE ||
current == 0xFFEF)) {
current = read16Bytes(in);
if (current == 0xFFD9) { // End of Image
error("ERROR: File Malformed");
return 0;
}
}
return readAppMarker(in);
}

// Utility Method to throw exceptions
private void error(String message) throws Exception {
throw new Exception(message);
}

// Utility Method to read 2 bytes from the file*/
private final int read16Bytes(InputStream in) throws Exception {
int temp;
try {
temp = in.read();
temp <<= 8;
return temp | in.read();
} catch (IOException e) {
error("read16Bytes() read error: " + e.toString());
return -1;
}
}

// Utility Method to read 1 bytes from the file
private final int read8Bytes(InputStream in) throws Exception {
try {
return in.read();
} catch (IOException e) {
error("read8Bytes() read error: " + e.toString());
return -1;
}
}

// Method that reads the App Marker from the file to the point where
// thumbnail exists...
private int readAppMarker(InputStream in) throws Exception {
boolean debug = false;
// 'Lp' holds the lenght of the marker
int Lp;
// 'content' contains the bytes read from the file
int content = -1;
//'count' for the bytes read from the file
int count = 0;
int offset = 0;
int entries = 0;
// Readin the lenght for marker
Lp = read16Bytes(in);
count += 2;
System.out.println("Length of App1 marker: " + Lp);
if (debug)
System.out.println("Reading App1 identifier\nIdentifier: ");

// Reading the identifier i.e.6 bytes
int iden = 0;
String id = null;
content = read8Bytes(in); //1st byte read of the identifier
count++;
id = Integer.toHexString(content); // This byte should be '45' for Exif or '4a' for JFIF
if (!id.equals("45")) {
while (iden < 5) { // Reading the remaining 5 bytes which are 78, 69, 66, 00, 00
content = read8Bytes(in);
count++; // Incrementing each time a byte is read
iden++;
if (debug)
System.out.print(Integer.toHexString(content));
}
} else {
// Not an exif image
return 0;
}

/* If the image is found to be ''Exif', remaining
* bytes will be following 'TIFF' image format
*/

if (debug) {
System.out.println("\nBytes read for identifier: " + (count - 2));
System.out.println("\n\nReading TIFF Header");
}
// 'tiffCounter' contains the bytes read from the file
// Though i could have used 'count'.....
int tiffCounter = 0;
// 'intelAlign' used as a flag indiacte byte alignment
// either intel (little endian) or
// non-intel(motorola-big endian)
boolean intelAlign = false;

// Reading first 8 bytes for TIFF header
while (tiffCounter < 8) {
if (tiffCounter == 0) {
content = read16Bytes(in);
// count+=2;
tiffCounter += 2;
if (Integer.toHexString(content).equals("4949")) {
intelAlign = true;
}
if (debug) {
System.out.println("Byte Align: " + Integer.toHexString(content));
}
} else {
if (tiffCounter == 2) {
content = read16Bytes(in);
// count += 2;
tiffCounter += 2;
if (debug) {
System.out.println("Tag mark: " + Integer.toHexString(content));
}
} else if (tiffCounter == 4) {
// Reading offset for IFD0 which conatins
// offset for IFD1 which in turn conatins the
// thumbnail
content = read16Bytes(in);
content <<= 16;
content |= read16Bytes(in);
if (intelAlign) {
content = convertIntegerToBigEndian(content);
}
offset = content;
// count += 4;
tiffCounter += 4;
if (debug) {
System.out.println("IFD0 Offset: " + Integer.toHexString(offset));
}
}
}
}
if (debug) {
System.out.println("Bytes read for header: " + tiffCounter);
System.out.println("\n\nBytes to be read for IFD0: " + (offset - tiffCounter));
}

// Reading bytes until reached IFD0
while (offset - tiffCounter != 0) {
content = read8Bytes(in);
tiffCounter++;
if (debug) {
System.out.println("Bytes to be read for IFD0: " + (offset - tiffCounter));
}

}

// Reading IFD0 .......to get the offset for FD01 which
// contains Thumbnail
content = 0;
// reading the number of entries in IFD0
content = read16Bytes(in);
if (intelAlign) {
content = convertShortToBigEndian(content);
}

tiffCounter += 2;
entries = content;
int bytesReadSoFar = tiffCounter;
if (debug) {
System.out.println("\n\nNo. of entries in IFD0: " + entries);
}

// Reading all the entries in IFD0
// 12 bytes for each entry
// The loop is bit elaborated although i could have just skim
// all those byte for all the entries
// I was just exploring it:)
for (int i = 0; i < entries; i++) {

int entryCounter = tiffCounter;
if (debug) {
System.out.println("\n\nScaning entry in IFD0 no. : " + (i + 1));
}
content = read16Bytes(in);
tiffCounter += 2;
if (debug) {
System.out.println("Tag no. : " + content);
}

content = read16Bytes(in);
tiffCounter += 2;
if (debug) {
System.out.println("Data Format : " + content);
}

content = read16Bytes(in);
content |= read16Bytes(in);
tiffCounter += 4;
if (debug) {
System.out.println("no. of Components : " + content);
}

content = read16Bytes(in);
content |= read16Bytes(in);
tiffCounter += 4;
if (debug) {
System.out.println("Either data value or offset to data value : " + content);
System.out.println("Bytes read for entry : " + (tiffCounter - entryCounter));
}
}

if (debug) {
System.out.println("Bytes read for IFD0 : " + (tiffCounter - bytesReadSoFar));
}

// Reading offset for IFD1
content = 0;
content = read16Bytes(in);
content <<= 16;
content |= read16Bytes(in);
if (intelAlign) {
content = convertIntegerToBigEndian(content);
}
tiffCounter += 4;
offset = content;
if (debug) {
System.out.println("\n\nBytes to be read for IFD1 " + (offset - tiffCounter));
}

// Reading bytes until reached IFD1
while (offset - tiffCounter != 0) {
content = read8Bytes(in);
tiffCounter++;
if (debug) {
System.out.println("Bytes to be read for IFD1: " + (offset - tiffCounter));
}
}

// Reading number of entries for IFD1
content = read16Bytes(in);
if (intelAlign) {
content = convertShortToBigEndian(content);
}
tiffCounter += 2;
entries = content;
bytesReadSoFar = tiffCounter;
if (debug) {
System.out.println("\n\nNo. of entries in IFD1: " + entries);
}

// 'data' array contains the info about the thumbnail
int data[] = new int[3];
// Reading the entried for IFD1
// 12 bytes for each entry
for (int i = 0; i < entries; i++) {
int desiredTag = -1;
int entryCounter = tiffCounter;
if (debug) {
System.out.println("\n\nScaning entry in IFD1 no. : " + (i + 1));
}
content = read16Bytes(in);
content = convertShortToBigEndian(content);
tiffCounter += 2;
// tag '103' containd about the format of thumbnail i.e. Jpeg compression
// or TIFF format howeve, we are interested in JPEg compression
// JPEG compression has been standardised by DCS so we expect it to
// JPEG compression for mobile phone cameras
if (Integer.toHexString(content).equals("103")) {
desiredTag = 0;
}
// if JPEG compression tag '201' contains the offset to the thumbnail image
if (Integer.toHexString(content).equals("201")) {
desiredTag = 1;
}
// if JPEG compression tag '202' contains the size of the thumbnail
if (Integer.toHexString(content).equals("202")) {
desiredTag = 2;
}

if (debug) {
System.out.println("Tag no. : " + Integer.toHexString(content));
}

content = read16Bytes(in);
content = convertShortToBigEndian(content);
tiffCounter += 2;
if (debug) {
System.out.println("Data Format : " + content);
System.out.println("Data Format Hex : " + Integer.toHexString(content));
}

content = read16Bytes(in);
content <<= 16;
content |= read16Bytes(in);
content = convertIntegerToBigEndian(content);
tiffCounter += 4;
if (debug) {
System.out.println("no. of Components : " + content);
System.out.println("no. of Components : " + Integer.toHexString(content));
}

content = read16Bytes(in);
content <<= 16;
content |= read16Bytes(in);
content = convertIntegerToBigEndian(content);
tiffCounter += 4;
if (desiredTag != -1) {
data[desiredTag] = content;
}
if (debug) {
System.out.println("Either data value or offset to data value : " + content);
System.out.println("Either data value or offset to data value : " + Integer.toHexString(content));
System.out.println("Bytes read for entry : " + (tiffCounter - entryCounter));
}
}
if (debug) {
System.out.println("Bytes read for IFD1 : " + (tiffCounter - bytesReadSoFar));
}

/*
'data[0]' conatins the thumbnail format
'data[1]' contains the offset to the thumbnail
'data[2]' contains the size of the thumbnail in bytes (which we might not need as we'll pass
to a JPEG Decoder which will take care of it...in fact the length is already there in JPEG
thumbnail image!!!)
*/

// we only proceed only if the thumbnail is JPED compressed
// for which data[0] = 6
if (data[0] != 6) {
error("Not a JPEG file");
return 0;
} else {
// reading bytes until reached the thumbnail
while (data[1] - tiffCounter != 0) {
content = read8Bytes(in);
tiffCounter++;
if (debug) {
System.out.println("Bytes to be read for IFD1: " + (data[1] - tiffCounter));
}
}
}
if (debug) {
System.out.println("Total Length for the marker: " + Lp);
System.out.println("Total bytes read so far: " + (tiffCounter + count));
System.out.println("Bytes left for this marker");
}

/*
At this point, we need a JPEG decoder which starts reading the file
* from this point in the file, that decodes the thumbnail. Remember
* thumbnail is reduced version of actual image but still a JPEG image
* if JPEG Compressed!!
*/
return 1;

}

// Utility Method to convert Short (2bytes) from Little Endian to
// Big Endian
int convertShortToBigEndian(int i) {
return ((i >> 8) & 0xff) + ((i << 8) & 0xff00);
}

// Utility Method to convert Integer (4bytes) from Little Endian to
// Big Endian
int convertIntegerToBigEndian(int i) {
return ((i & 0xff) << 24) + ((i & 0xff00) << 8) + ((i & 0xff0000) >> 8) + ((i >> 24) & 0xff);
}



After the execution of extractThumbnail() method you are at the point where thumbnail exists in the file, it should be noted that thumbnail is scaled down version of the actual image, but some of the markers are missing here. You need a JPEG decoder that decodes the thumbnail for display; there is one free JPEG Decoder available here. It also instructs how to incorporate this decoder in your code.
Well, you might be wondering why we are extracting thumbnail if we still need a JPEG decoder!!! Why don’t we just decode the actual image and produced a reduced version of the image using above mentioned decoder?? The answer is, as I experienced it, it takes very long to decode the full image and producing a thumbnail, the situation is even worst while dealing with image of very high dimension e.g. 2048x1680, which is not good from your application point of view. So it’s a good idea to use the already embedded thumbnail which is far quicker to decode for the decoder, moreover, we should use this if already there because that’s the whole purpose of it!!!!!

10 comments:

mike said...

can you use the MIDP Image.CreateImage call to decode the thumbnail?

Mike said...

NetBeans is choking on this - specifically the <<>> part - any ideas?


// Utility Method to convert Integer (4bytes) from Little Endian to
// Big Endian
int convertIntegerToBigEndian(int i) {
return ((i & 0xff) <<>> 8) + ((i >> 24) & 0xff);
}

Mike said...

I hate to bombard you with questions. The convertIntegerToBigEndian method is used when Intel Alignment is in the image - most palces wher eyou call convertIntegerToBigEndian are protected by an if (intelAlign) - but there are a couple of cases where intelAlign is not considered - was that your intent?

Atif said...

Issue 1: I dont think that MIDP's Image class method createImage(InputStream stream) can be used in this case. As MIDP documentation says that this method creates Image object from decode image data. May be you can give it a bash!!

Issue 2: There was some problem while publishing that post due to HTML tagging, that part of the code was wrongly shown, I have changed it, please see the updated post.

Issue 3: You are right, at some points i might have not considered intelAlign, this is simply because i was not intesrest in those contents to make them understandable for me to proceed further. For example, if offset to thumbnail has been read then at this point i need to consider Intel Alignment to find exact number of bytes to get to the thumbnail.
Hope that helps!!!

Mike said...

thanks for the quick reply. I'm not having much luck at this - your routine is working fine - the JPEG-Decoder is choking on 0xFFC4 - onward thru the fog, I always say

Mike said...

a word to the wise about the JPEG-Decoder from Dersch - don't use the one listed as a separate download - download the PTViewerME program and use the JPEGDecoder.java file from it.

I've spent 2 days chasing bugs trying to get the separate download to work.

As a last resort, I used the one from the PTViewerME download and it worked right out of the box.

Atif - thanks again for the very helpful site and posts about this topic.

Atif said...

Taking Mike's comment ahead, I would like to share my findings on this. While working on this stuff, I used the JPEG-Decoder java file listed separately (as I didnt know, that time, about the file mentioned in the last comment by Mike from PTViewerME), I didnt find it working, so during deugging that file, I found an issue in decode(InputStream in, PixelArray out) method. To resolve it, I changed the condition in the very first while loop of the method, from while(current>>4 != 0x0FFC) to while(current != 0xFFC0). This fixed my problem!!
Moreover, I explored it further and compared the two files: the one I used and the one mentioned by Mike in his last comment from PTViewerME. The only difference I found between the two files was exactly the same while condition I described above. For my fix, I changed the condition to while(current != 0xFFC0) while in PTViewerME's decoder file this condition has been changed to while(current>>4 != 0x0FFC || current == 0xFFC4). I need to find the out the reason for this difference if I find time. You guys can use either PTViewerME's decoder file or the other file (with the change I did) whichever works for you!

Mike said...

that while loop is the place I was trying to troubleshoot this, but I never tried your specific condition. As it was written, some of the case values would never be reached.

Thanks again for all the help.

shyam said...

Hi, I think this post would help me in developing my mobile app. I need to show the thumbnails of the images, and found this post while looking for 'how to-s' on exif. But I don't understand the code here, i.e, how to use it. I don't understand what's returned by the function, etc. I'm new to J2ME, so could you help me use this?

Atif said...

Well, its already explained in the post,actually the Inputstream's Ojbect passed as an argument is opened to some JPEG file. After this method's execution has finished, you are at the point in the stream where the thumbnail starts. In other words, the method leads you to the location of thumbnail within the file. Please use comments in the code to understand it!