In this article we’ll describe the complete workflow for image and video forensics. In fact, just like computer forensics is not only simply copying and looking at files, forensic video analysis is broad and complex and there are many steps that are commonly missed and rarely taken into account. It can be quite overwhelming if we think of all the tasks related to analysis. As a forensic video analyst, it is important to be aware of all the possible steps needed for a really complete analysis. This way, you can stay organized and minimize the possibility of skipping or missing steps. Also, if you do have to go to court, you have an outline that serves as the basis of your presentation.
It is important to remember that the job of a forensic video analyst does not start and end with viewing and enhancing a video. It’s more complex than that. You must identify the data, decode it properly, document the process, compare it with other material, and then go to court. Since digital data is really just a collection of bits, below is an outline of a process around working with these bits what you need to do with the bits.
Step 1. Retrieve the data: get the bits!
This step may be the most important and can be very complex and critical for evidence integrity. Unfortunately, this step is usually not taken by an expert analyst or by those who know how important it is to take care when retrieving data. Most times, the original evidence is not gathered by a police investigator but rather a patrol officer who is the first on the scene, and often a low-res copy of an original is output to a disk or thumb-drive, and the original DVR is left to be copied over by the next day’s footage. This happens far too frequently, and gets everyone off to a pretty bad start.
Another way you may get the data is if it comes from colleagues on the digital forensics side of things. If they are meticulous and take the steps necessary to document their process, you can be in a good position to start. Often analysts start working on a video burned on a DVD or on some images received by email, however we need to be conscious and document the previous steps, which may be, for example:
- Analyze a disk image for complete and deleted files
- Export a video from a DVR
- Copy a file from a hard disk
- Capture a video from an analog device (e.g. VHS)
Even a simple operation like a copy should be done in the most scientifically relevant way by verifying hash codes and so on.
Police investigators often work with evidence gathered by a non-specialized first responder who may not follow best practices. However, with better communication, training, and diplomacy, the situation can be improved. Let them know what you need them to gather and document and how to avoid damage to the evidence, like overwriting the original hard drive. For example, explain to them how to do an export without introducing re-encoding quality loss and to document the make and model of DVR. And make sure they call you if they have any questions.
Step 2. Decoding the data: what are these bits?
Now that you have the bits, you must get something from them to analyze. If you have some plain video files, it is not a big problem, given you have the proper codecs. Since there are hundreds of proprietary video files out in the wild, that in itself may turn into a challenge. To determine which approach you take, you need to know what you are looking at. This step generally defines the challenges that you may face:
- You have a disk image and you need to reconstruct images and videos, even the ones that may have been deleted (this is common in child pornography cases, for example).
- You have a dump of a DVR drive and don’t know how the data is encoded.
- You have an export of surveillance footage, but the video is in a proprietary format (a very common situation). Far too often the video player given by the system producer is full of bugs, unusable, incompatible with modern versions of Windows and videos can’t be properly exported.
With this step, you may need to do some research to really define what you need in order to complete the job. Do you need to focus first on data recovery? Do you need to find a better decoder? Do you need to find a better player for the video or means to export it properly?
Sometimes, this is an easy step, other times it is far more difficult and time consuming. Remember, this is a scientific pursuit and science sometimes can’t be rushed.
Step 3. Finding the useful data: where are the right bits?
During this step you should be able to view the videos or the images, but you need to find the right ones! Two examples of what you may face:
- Finding images of interest in a large database
- Looking for an event of interest in hours and hours of video
This step can be helped with communication from other team members working on this case. For most cases, the basic thing you need to understand is: what happened, and when did it happen? And of course, technology may help too, with technologies such as video content analysis and face recognition.
Step 4. Finding the source of data: where do these bits come from?
Depending on the situation, you may need to understand how the original files have been generated. With some generalization this may be called image ballistics. Understanding the type of file and the source can help you understand several things in a case. Some analysis that may be done could be:
- Identify the type of source (digital camera, scanned image, computer generated…)
- Identify the camera model used for taking the picture
- Identify the specific device that has taken the picture
You need to document the source so that you can maintain the integrity of your evidence. This will help you if you have to go to court later.
Step 5. Verify the integrity of the data: has someone tampered with the bits?
At this point you may be interested in understanding if you can trust the data you have gathered. Is there a probability that someone altered it? This can be done on various levels:
- Verify if the file has been manipulated, for example, altering the metadata
- Verify if the image has been manipulated, for example, converting the format, resizing or cropping it
- Verify if the content has been manipulated, for example, removing or adding a subject
Tampering is becoming more commonplace. It can be done innocently (like converting formats from original to a low-res media file) or purposely “photoshopping” to manipulate facts. In this digital age, it is something that should be addressed. You should always work with an original if possible and you can easily verify this by using forensic tools such as Amped Software’s Amped Authenticate.
Step 6. Estimate the quality of the data: do you have enough bits?
At this point you can see something in the image, but you must understand if the quality is good enough for your purposes. For example, if you see a car, are you able to read the license plate? Or if you have a face, do you have enough pixels for a reliable identification?
- Does the image effectively contain the information you need (e.g. the license plate has enough pixels)?
- If not, can the information be recovered or viewed better with image enhancement or image restoration techniques?
- What are the specific defects in the image? Can they be recovered?
Technical knowledge and experience is very important to estimate quickly if you have enough quality or not. It is not always easy to estimate the minimum quality to get useful results. A shortcut for things like faces and license plates is to zoom in and count pixels. If you only have six or eight pixels to draw all the characters in a license plate, the probability of success is pretty low. Based on our experience it usually is not possible to get anything from a license plate if its vertical resolution is less than 12-15 pixels.
Step 7. Enhance the data: get out the good bits!
Once you have identified the problems affecting the images or videos, given the right tools, such as Amped Software’s Amped FIVE, you can enhance and restore the data. This step is actually pretty vast, and can involve processes like:
- Image enhancement techniques: emphasize (or reduce) some features of interest of the image (contrast enhancement, histogram equalization, sharpening…)
- Image restoration techniques: understand the mathematical model of a known disturbance and try to invert the model to recover the image without the defect (deblurring, Fourier filtering, frame integration…)
For hardcore video enhancement people, this is the most fun part of the process. It does involve a bit of trial and error, but it is at least the fun part. While no one can guarantee the great results found with Hollywood magic on the CSI shows, you can often see some amazing results.
One thing that is really important to remember in this step is to document the enhancement process, so you have a scientific record to take to court to validate all the processing steps you took.
Step 8. Analyze and compare the data: what do the bits represent?
This is where you see what you have gained. The enhancement step would be useless if there’s no improvement to the content of the image so you can understand and classify it. In this step you can do the following:
- Compare a face in two different images
- Compare a face with a known subject
- Read the license of a vehicle
- Identify the place where a picture is taken
- Measure the height of a subject
- Find the corresponding fingerprints in a database
If you don’t get the results you need, you can go back and repeat the steps until you do; or you determine that you can’t get what you need with the data provided. Remember that this is a scientific process and it is too easy to get bent out of shape over a ton of work without results; but that is sometimes the cards that are dealt.
Step 9. Validation: did you get the right bits?
Validation isn’t just focused on the quality of the result. It is also about the quality of the process used to gain the result. You must always maintain that the techniques used must be valid from a scientific point of view and follow a procedural set of standards accepted by the courts that have jurisdiction over what you are doing. This is extremely important for the verification of image integrity and for documenting the enhancement workflow.
A few things to consider are:
- State of the art techniques must be validated by peer review and accepted by the scientific communities
- The results must be scientific and repeatable
- A detailed audit trail must be kept to explain how you go from the original image to the enhanced one
This may be seen (and actually is) as manipulation of evidence, and thus you must be able to justify it properly from a scientific point of view. In this case, documentation is key.
Step 10. Presentation: I’ll show you the right bits!
Getting the results is not enough. You must explain them to the court and the jury: you must be able to make them understand and accept the techniques you used. Scientists, engineers, attorneys and common people speak different languages. It is really important to organize the facts you present with the idea that you must explain the terms you use in your context, and be clear and open.
A good defense attorney or prosecutor will try to trip you up with the “has this image been photoshopped?” question and questions about certainty, possibility, etc. That is their job, and frankly, would you respect them if they didn’t ask tough questions? An attorney isn’t a scientist. The courtroom is a stage and the attorneys are actors. Their questioning tends to be less focused on science, more focused on emotion. In court, emotions are often charged and an attack on your process can be presented as a personal attack with the idea to get you to deviate from facts. That is the game-plan for attorneys when they can’t debate the facts.
The key to overcoming this type of questioning is to document your workflow and stick to the science. This is a scientific process, but it has to be explained to non-scientific people. Can you present the facts of the case and explain the science in plain language? If you are organized, clear, and concise this will help. At the very minimum, you need to show:
- The original
- Where it came from
- How you got it
- What steps were taken to get the result
- How that result relates to the case
- What scientific methods were used to validate the process
It’s not an easy task, as very often complex matters are oversimplified to be understood by laymen and at the end of the day the work of the expert witness can amount to nothing.
The bottom line
You may not use all of these steps in all cases. Often, you may work as a team and only concentrate on a couple of them. But, the most important thing to take away from this sample workflow is organization and methodology of your process.
Depending on what you are working with, you may not formally define these steps so clearly. Critical and systematic thinking needs to be the foundation for what you do. When documenting the workflow, remember the old Gen. Dwight Eisenhower maxim: “those who fail to plan, plan to fail”.
This article is an edit of the post that appeared on the Amped Blog on 2011-11-09.