Till this point, I have a system which works well for burned-in white subtitles and generates a timed output file. The next step is to add the same support for colored subtitles too.
The HSV Color Space
The HSV color space, and the Hue component (H) in particular, is an excellent representation of the exact color value of a pixel. The normal RGB space requires 3 values to represent the color, whereas the H component takes a value in the range of 0-360 and gives the necessary color information.
You can read more about the color space here.
The chart below shows how the values of H vary for different types of colors.
I exploited a conversion from the RGB to the HSV space in order to detect colored subtitles. Just like there was a luminance threshold in order to detect white subtitles, there is a threshold around the range of the user-specified hue value in order to detect subtitles of a particular color.
This hue based thresholding, along with the existing vertical edge dilation was used to detect subtitles of a particular color.
Color options in the program
The program has 7 predefined color names. The first and most prevalent case is White, the detection of which is luminance based. The other 6 are equally spaced in the hue value range. The colors, along with their hue values, are:-
- Yellow – 60
- Green – 120
- Cyan – 180
- Blue – 240
- Magenta – 300
- Red – 0
Each of these colors can be specified along with the -subcolor option. For example:-
ccextractor video.mp4 -hardsubx -subcolor yellow
In addition to these preset values, there is also the possibility to supply a custom hue value. This value is a custom value between 0 and 360 (not included) which can be supplied to the subcolor option, and could be of help to users who want to extract subtitles of the precise hue value in their stream if it fails to meet one of the presets.
Local Adaptive Thresholding
In addition to detecting colored subtitles, I was also able to improve the detection of white subtitles using local adaptive thresholding algorithms, and Sauvola Binarization in particular. This was an additional step which marginally improved the quality of results for white subtitles (which always have a pixel value greater than their surroundings), however could not be applied to colored subtitles in all cases due to a wide variety of contrasting backgrounds.
Abhinav Shukla 