What is Multimedia Data Mining?

C.B. Fox

Multimedia data mining refers to the analysis of large amounts of multimedia information in order to find patterns or statistical relationships. Once data is collected, computer programs are used to analyze it and look for meaningful connections. This information is often used by governments to improve social systems. It can also be used in marketing to discover consumer habits.

Multimedia data mining involves the collection of significant amounts of data.
Multimedia data mining involves the collection of significant amounts of data.

Multimedia data mining requires the collection of huge amounts of data. The sample size is important when analyzing data because predicted trends and patterns are more likely to be inaccurate with a smaller sample. This data can be collected from a number of different media, including videos, sound files, and images. Some experts also consider spatial data and text to be multimedia. Information from one or more of these media is the focus of data collection.

Multimedia data mining includes collecting and reviewing data with the use of various media, such as video or sound files.
Multimedia data mining includes collecting and reviewing data with the use of various media, such as video or sound files.

Whereas an analysis of numerical data can be straightforward, multimedia data analysis requires sophisticated computer programs which can turn it into useful numerical data. There are a number of computer programs available that make sense of the information gathered from multimedia data mining. These computer programs are used to search for relationships that may not be apparent or logically obvious.

When multimedia is mined for information, one of the most common uses for this information is to anticipate behavior patterns or trends. Information can be divided into classes as well, which allows different groups, such as men and women or Sundays and Mondays, to be analyzed separately. Data can be clustered, or grouped by logical relationship, which can help track consumer affinity for a certain brand over another, for example.

Multimedia data mining has a number of uses in today’s society. An example of this would be the use of traffic camera footage to analyze traffic flow. This information can be used when planning new streets, expanding existing streets, or diverting traffic. Government organizations and city planners can use the information to help traffic flow more smoothly and quickly.

While the term data mining is relatively new, the practice of mining data has been around for a long time. Grocery stores, for example, have long used data mining to track consumer behavior by collecting data from their registers. The numerical data relating to sales information can be used by a computer program to learn what people are buying and when they are likely to buy certain products. This information is often used to determine where to place certain products and when to put certain products on sale.

Government officials can use multimedia data mining to help traffic flow more smoothly and quickly.
Government officials can use multimedia data mining to help traffic flow more smoothly and quickly.

You might also Like

Discussion Comments


Hey guys. Nice article. I have founded a website and our main occupation is to extract "business" data out of multimedia.

Indeed, the range possible uses is wide. From face recognition to tumor recognition, if one can extract "business" information from a image or video then indeed this can be used for data mining. One can predict behavior or identify patterns.

Let's take the example of a store filming customers. We analyze every frame and identify person(s) in it. Now in every next frame, we check for these persons. Then we compare it with the last five frames and now we have the person's behavior.

So at this point, we are monitoring the person's behavior and we can run data mining algorithms comparing it with predefined data sets or even run-time behavior sets of other persons currently on the floor.

Let's say you have a data set of 1,000 purchases (person taking a product of the shell and putting it in his basket) and 1,000 sets of persons stealing. Using data mining you can now say this person is probably buying this product or this person is probably stealing it.

This is a very basic example of how you can extract "business" data out of unstructured multimedia data.

@NathanG: I think you are correct concerning video on YouTube.

Now you can watch a video on YouTube and you are shown commercial information related to the tags of the video. Most of the times completely unrelated to what you are watching. These days Youtube even stops the video to show commercials. So you watch a data mining video and in the middle you are interrupted for 60 seconds to watch a commercial about, let's say, some bank.

The decision to show you this commercial was taken from maybe your age, gender, location and some clicks you did on google a few days ago and not on the last five frames of the video you are watching.


@MrMoody - I used to do some website optimization and I think I have a clue as to how the programs could work with video files.

These days, a lot of video files posted on websites have tags attached to them. The tags contain the keywords which tell you what the video is about.

Obviously this is not as descriptive a piece of information as watching the whole video itself, but it does tell you something; I’ve often had to add tags to videos when adding them to websites, so that the videos would show up in search engine result pages.


@SkyWhisperer - I like the example that the article gives of visual traffic patterns that are captured by cameras near a traffic light or someplace like that.

I am sure that satellite imagery can be used in such analysis, when it gets down to the level of detail where it shows where people are traveling.

I don’t know how such computer algorithms would work however. I work in the software industry and admit that any program that can parse an image and provide “trends” is a sophisticated product indeed. Text data is much easier to work with and can be sucked into a lot of different applications.


@NathanG - I would say yes it would, but not for the reasons you described.

I think the strictest definition of multimedia data mining involves looking at videos and images themselves to see patterns. You don’t have to do that with YouTube or other file sharing sites.

All you need to do is look at the number of hits a web page receives. You don’t have to parse the video, itself, so to speak.

However, the article says that text can fall into the category of multimedia data mining, so I guess you could argue that the links to the YouTube videos would be part of the data mining operation. Frankly, I think that multimedia data mining is a very broad term that can encompass a lot of things.


I wonder if multimedia data mining also involves tracking what videos people are watching on YouTube or on some of the other video sharing sites.

So many people are watching and uploading videos these days that I think this has become a better barometer of what people are doing online than what websites they visit.

For that matter, tracking the songs that people download could be another metric, too, that marketers use to figure what to sell to the general public.

Post your comments
Forgot password?