A Machine Learning Method for Detecting the Trace of Seam Carving

1 Abstract —Image retargeting is a manipulation approach for resizing the images while aiming to keep the image distortion at a low level. Detecting image retargeting is of importance in image forensics or sometimes of importance in checking the originality. The aim of this paper is to introduce a new blind detection method for identifying retargeted images based on seam carving. For this purpose, a new method based on stripes at various numbers, Local Binary Pattern (LBP) transform, and energy map is introduced. The sub-images were obtained from square root of the energy map of LBP transform in the form of stripes for the feature extraction and these were evaluated in terms of several statistical features. The features extracted both from the natural and the seam carved images were used to train a Support Vector Machine (SVM) as a binary classifier. Experimental results were obtained using four-fold cross validation to improve the validity of the results during the evaluation process. According to the experiments, the proposed method produces improved accuracies when compared with the state-of-the-art solutions for the image retargeting detection based on seam carving.


I. INTRODUCTION
Digital image forensics is attracting increasing interest as image manipulation methods continue to be developed. Although seam carving algorithm in itself has no intended malevolent purpose, it can easily be used for the digital image forgery since it alters the images with few or no defects. Seam carving achieves a content-aware image resizing or object removal leaving very few or no traces behind. Its success comes from evaluating the images in a semantic manner. It considers the energy maps (maps with edges, salient objects, user-specified regions, etc.) of the images and adds or removes the least important pixel path without causing visual deterioration. Most of the defects caused by seam carving cannot be distinguished by the human eyes alone. Therefore, the method is open to the forgery applications. Detection of the images scaled by seam carving has been studied soon after the method was proposed. Machine learning-based detection methods have been frequently used for the seam carving problem. The Manuscript received 7 April, 2021; accepted 2 August, 2021. feature extraction and using these features as an input of a classifier is the most common solution for the problem. There are various examples of these features, such as 324dimensional Markov feature and 2D difference histograms in Discrete Cosine Transform (DCT) domain [1], gradient histogram features of the image [2], the features of the seam carving process [3], transition probabilities of several patches [4], additional noise features [5], Markov features and high-dimensional features of DCT coefficients [6], features of Blocking Artefact Characteristics Matrix (BACM) [7], statistical features in Local Binary Pattern (LBP) domain [8], hybrid features using LBP [11], histogram features of uniform LBP [9], features of Markov transition probabilities and subtractive pixel adjacency model in Local Derivative Pattern (LDP) domain [10], histogram features of Weber Local Descriptor (WLD) and LBP [12], statistical features after applying Sobel filter [13], features of LDP encoders [14], features indicating the variation in energy and pixel [15]. Most of the machine learning-based solutions uses Support Vector Machine (SVM) classifier for the classification of the images whether they are seam carved or not based on the extracted features mentioned above.
Apart from the machine learning-based methods, there are some other techniques for seam carving detection, such as visual word representations of the Scale Invariant Feature Transform (SIFT) [16] and forensic hash composed by the invariant speed-up robust feature points [17]. However, these methods need the original image to make the analyses. Further, Convolutional Neural Networks, one of the most popular methods in computer science, was also used for the detection of seam carving-based resizing ( [18] and [19]). The effects of smoothness, noise, and compression attacks [2] and second seam carving or insertion operation [20] were also evaluated.
In addition to proposing a detection method, proposing detection method in a more efficient way was also investigated in a couple of studies. Using Linear SVM [10], and Kruscal-Wallis statistics [12] are the examples of this case.
The notable deficiencies of the seam carving detection methods in the literature were demonstrated by their computational complexity and low detection accuracy in the A Machine Learning Method for Detecting the Trace of Seam Carving images scaled at low ratios (e.g., 3 % and 6 %). In this paper, we intended to overcome these deficiencies and provide a more accurate and robust detection method at low computational cost. Contributions of the paper are as follows:  The detection of the seam carving-based image resizing was achieved by a stripe-based approach. Images were investigated as a collection of sub-images and the effect of seam carving on each sub-image was deeply analysed in terms of statistics. Therefore, the effects of seam carving on different parts of the images were analysed.  The effective features were used for the seam carvingbased resizing detection. Only seam-based features were taken into account during the classification.  The ideal number of stripes (sub-images) was determined and only the features of those stripes were shown to be the most effective on the detection of seam carving, which enables efficient feature extraction and training processes.  A feature vector of size 60 gave the best detection performance, which implies a lightweight pre-training, training, and testing processes.  In the proposed method, by considering all possible pixel paths instead of the optimal one via Algorithm 1, feature selection was made more robust and method independent, and taking the square root of the energy map restricted the data interval and made the features more determinant.  The power of LBP method for revealing local pixel variations has been effectively used on the sub-images to reveal the trace of seam carving.  The results for the proposed method show that the overall number of misclassified images, that is the number of the original images, which were classified as seam carved and the number of the seam carved images which were classified as original, was highly reduced when compared with the previous methods.
Rest of the paper is organized as follows. Section II gives a brief background information about the topic. The proposed method is explained in Section III. The experimental evaluations are given in Section IV, and conclusions are made in Section V.

A. Seam Carving Algorithm
Seam carving [21] is a content-aware image retargeting technique that can be used for different purposes, including image resizing [22]- [28], image enhancement [29], and object removal. In this method, an optimal seam is found and removed or inserted one by one until the desired image size or other stopping conditions are satisfied. A vertical seam (s x ) for an image of n×m is defined as given by (1) where a mapping is defined as x: In other words, a vertical seam is an 8-connected path of pixels from top to bottom in an image [21]. In each row of the image, there is only and exactly one pixel that belongs to the seam path and all pixels in rows are connected. Taking one pixel in each row preserves the rectangular shape of the resultant image. Pixels (I s ) that form the seam (s) path can then be described as Removing a pixel path from the image will have a local effect. All pixels which are on the right of the seam path will be shifted left to fulfil the removed pixels.
Energy map of the input image is computed by the energy function (e) where E(s) is the cost of a seam. An optimal seam path should have the lowest total energy. Therefore, the optimal seam (s * ) where the cost is the minimum is computed as (4) stands for the cost of a seam path and it is calculated as the summation of the energies of each pixel in the path.
Once the energy map (e) of the image is obtained, its Minimum Cumulative Energy Map (MCEM) can be computed. The energy map shows most of the edges, foreground or salient objects inside the image. Different energy functions can be used to obtain the energy map, including the L1, L2-norm of gradient, saliency measure [30], Harris-corners measure [31], e 1 error, entropy, segmentation, histogram of gradients, and many other edgedetection filters [21]. A seam is an 8-connected path of pixels along the top to bottom or the left to right of an image. Hence, MCEM is composed of the summation of the pixel values of the minimal upper-left, upper-middle, and upper-right neighbours for a vertical seam. Similarly, the upper, middle, and lower values are used cumulatively for a horizontal seam. Figure 1 shows the energy map and corresponding MCEM with an optimal vertical seam (green connected pixel path) for the input image.
The optimal seam is chosen as the path with minimum cumulative energy in to prevent removal of the pixels inside the important regions. An example of an image, its energy map with an optimal seam, and the seams to be removed from the original image are given in Fig. 2.

B. Local Binary Patterns (LBP)
LBP was proposed in 1994 as a visual descriptor to be used for the classification purposes [32]. Images are generally divided into 3×3 cells and every pixel along the edges of the cells is compared to the central pixel. An 8-bit binary number is established by this comparison. A "0" (zero) is assigned if the intensity value of the neighbourhood pixel is greater than the central pixel and a "1" (one) if it is less.
Comparison results are written in a clockwise direction to form the binary number. Cells can be considered in any size instead of 3×3 providing that the central pixel is exactly in the centre, i.e., every pixel on the cell edge has the same distance to the central pixel. In this paper, experiments are conducted with the cell size of 3×3.
Equation 5 represents the computation of an LBP value for a pixel at (x, y): Neighbourhood of pix c is described by K number of pixels within the radius Rad. Pixel at the centre is compared with the neighbour pixels using f(pix c , pix n ) comparison function. Since the LBP represents neighbour relations, the LBP values of the pixels directly neighbouring the seam path change perceptibly when an image is seam carved. Figure 3 shows the LBP transform as it occurs for a pixel with the intensity value 111.

III. STRIPE-BASED SEAM CARVING DETECTION (SSCD) METHOD
As classified by Ansari, Ghrera, and Tyagi [33], image resizing is a kind of pixel-based image forgery. Seam carving-based resizing can also be included in this type, although the detection of seam carving-based image resizing was not included in the mentioned review paper. The detection of seam carving-based resizing is much more difficult than other pixel-based resampling methods since not only the geometrical properties, but also the semantic properties of the images are considered. Although energybased, noise-based, and half seam-based features have been proposed in the literature, the seam-based features are the most effective for the seam carving detection [8]. For this reason, in this study, we used only seam-based features for the training and testing. The stripe-based seam carving detection method partitions the images into several stripes and evaluates each stripe in terms of these features. Upon measuring, the distance between the seams found in some portions of the image may be greater than zero. This means that the image is seam carved [2].
Yin, Yang, Li, Zhang, and Sun [8] observed that an optimal seam found in the upper half of the image was different than that was found in the lower half.
In this study, a new method based on multiple stripes is proposed. As shown by the example stripes given by Fig. 4, the images were considered as a collection of stripes in the horizontal and vertical directions. After considering the images as two stripes, four stripes, six stripes, and eight stripes, each stripe was evaluated in terms of the ten statistical features of Ryu, Lee, and Lee [5], which are given in Table I. Example seams found in the four stripes are shown in Fig. 5. Through experimental observations, it is shown that the features extracted from the image stripes (sub-images) reveal seam carving-based image resizing forgeries better than the previous approaches. Those features are given in Table I for an n×m image.
There are several methods to determine the optimal one for resizing by seam carving. Algorithm 1 shows how we selected a seam. Thanks to this algorithm, we considered all possible seams and evaluated them in terms of total energy. Evaluating all possible seams instead of the optimal seam makes the detection method more robust since it is independent from the way the optimal seam is selected.
In Table I, C hs and C vs stand for the MCEM of a stripe for a horizontal seam and a vertical seam, respectively. The C vs is calculated as mentioned in Section IV and the C hs is calculated based on the 90-degree left rotated image. The minimum energy (minimum total seam energy in horizontal and vertical directions), maximum energy (maximum total seam energy in horizontal and vertical directions), mean and standard deviation in both directions and the difference between the minimum and maximum in both directions are calculated for each stripe. The features were obtained after the image was transformed into LBP form to see the local artefacts more explicitly [8]. A 210-dimensional feature vector was constituted for every image in the database and these constitute the input for the SVM. A flow chart of the mentioned steps of the proposed method is summarized in Fig. 6. Fig. 4. 2 stripes (a and c) and 4 stripes (b and d) in the horizontal and vertical directions.     SVM classifier was trained and tested on Octave, which is a free high-level software designed especially for numerical computations [34]. Utilized SVM classifier version was LibSVM 3.21 [35]. The radial basis function (RBF) was chosen as a kernel function. The grid search method was used to find the optimal parameters of the RBF. Detection accuracy is very sensitive to the c (Cost) and g (Gamma) of the RBF; therefore, the best c and g were selected from the results of the grid search for better classification performance.
In addition, a database was established to be used in the training and testing phases of the detection approach. For this purpose, 1338 images were downloaded from a publicly available dataset (UCID) [36] which was also used in the most of the similar SVM-based studies to carry out a comparison.
These original images were scaled by seam carving at 3 %, 6 %, 10 %, 20 %, and 30 %. There are 1338 original + 1338 × 5 tampered images (a total of 8028 images) in the established database for the experiments related to the SSCD method. Vertical seam removal was considered in all experiments, i.e., width reduction by seam carving is performed. Height reduction can also be applied easily. In this case, the same operations would be applied to the transposed image. To compare with other studies, same scaling ratios are used in the training and testing images. In other words, for detecting whether the image is scaled 30 % with seam carving, we used 30 % scaled images and their originals together to train the SVM. Success of the classifier was measured using four-fold cross validation. Threefourths of the images were randomly selected for training and the remaining one-fourth of the images were used for testing, i.e., there are 2006 images in the training set and 670 images in the test set for each scaling ratio. Four possible training-testing pairs were tried and their average was taken as the accuracy of the method for one run. This procedure was repeated ten times and the average was calculated to obtain more accurate detection results.
As Figure 6 summarizes the experimental process of the proposed blind detection method, all the images in the database, including the originals and the carved ones, were transformed into LBP domain. Feature extraction was realized using the square root of the energy map for each image. Taking the square root of the energy map restricted the data interval and made the features more determinant. The images are handled as a whole (one stripe), as two stripes, as four stripes, as six stripes, and as eight stripes. Each stripe was evaluated in terms of the ten statistical features stated in Table I. In total, 210 features were extracted from every image and an 8028×210 feature matrix was established. Four-fold cross validation determined the training and testing data sets in to guarantee the randomness of the experiments by keeping the testing images different from the training images. All the features extracted for the detection of seam carving resizing in this paper were related to the seam behaviour. Since the seam-carving operation removes or inserts seams for size reduction or expansion, the seams are the most affected items in the images. Their average energy increases by the seam removals and so the energy evaluations reveal the operation. The most effective features for the LBP-based seam carving detection method were shown to be seam-based features [8], hence only these features were used for the proposed blind detection method.
In addition, the effects of dividing the image into different numbers of stripes were also evaluated. To this end, five more experiments were realized. In the first one, the image itself as a whole was considered as a stripe and ten statistical features were extracted. The performance was measured using ten features as an input. The image was then handled as two stripes and ten features for each stripe (twenty features in total) were extracted. The other three types of stripe evaluation were also done in a similar way. Table II shows the detection accuracies of the method with respect to the number of stripes used. It can be seen that dividing the image into six stripes and extracting 6×10 features and training the SVM with those 60 features give the best solution among the evaluations. For every scaling ratio, six stripes produced the best detection accuracy. The recent literature presents considerable amount of papers on the detection of seam carving. In this part of the study, we present a comparison of the proposed detection approach with the state-of-the-art papers. Table III shows the detection accuracies of the recent methods for the attempted scaling ratios. Although we aimed to improve the detection accuracy of the seam carving operation, especially in low scaling ratios, we also tested our approach for higher scaling ratios too. For this purpose, we also tested the proposed method on 40 % and 50 % scaled images.
As seen from Table III, the proposed approach outperforms the state-of-the-art papers on seam carving detection. Accuracies in almost all scaling ratios are well ahead when compared with the existing methods. Although the detection performance for 3 % and 5 % scaling ratios are some less than [18] and [19], the performance of the proposed method is satisfactory when the overall scheme of the experiments and the remaining scaling ratios are considered. Besides, we provide these performances with a lighter training process. In addition, the proposed method provides a robust approach since it considers all possible seams instead of the optimal one alone. Selecting the optimal seam may change, e.g., if there are three minimum elements in the last row of the MCEM, three different seam selections are possible. The results show that the proposed detector produces good results when compared to state-ofthe-art studies. V. CONCLUSIONS In this paper, seam carving-based image resizing was detected by using a stripes-based detection approach. According to the proposed method, the images were considered as a combination of multiple sub-images. Several statistical features of the stripes were extracted after LBP transformation and they were given to a SVM as an input and the machine was trained and tested. In the experiments, up to a number of eight stripes were investigated. The statistical features of every seam in each stripe were evaluated for the detection of seam carving-based image resizing. In general, the proposed method achieves good results for scaling ratios between 10 % and 50 % when compared with the recent studies. For the scaling ratios of 20 % or greater, it provides at least 99 % accuracy. Although the success for scaling ratios of 3 % and 6 % is lower than the results with deep learning based approach, it still provides better results for overall performance among the machine learning methods which are usually light weight to implement when compared with the deep learning approaches.
The future studies will focus on high-resolution images for which the optimal number of stripes may change for the detection of seam carving-based image resizing. In addition, unsupervised methodologies like deep learning will be investigated.

CONFLICTS OF INTEREST
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.