OpenCV C Obj-C Detecting a sheet of paper Square Detection

Question

I successfully implemented the OpenCV square-detection example in my test application  but now need to filter the output  because it s quite messy - or is my code wrong   I m interested in the four corner points of the paper for skew reduction  like that  and further processing       Input  amp  Output    Original image   click  Code   double angle  cv  Point pt1  cv  Point pt2  cv  Point pt0         double dx1   pt1 x - pt0 x      double dy1   pt1 y - pt0 y      double dx2   pt2 x - pt0 x      double dy2   pt2 y - pt0 y      return  dx1 dx2   dy1 dy2  sqrt  dx1 dx1   dy1 dy1   dx2 dx2   dy2 dy2    1e-10      -  std  vector lt std  vector lt cv  Point gt   gt  findSquaresInImage  cv  Mat  image       std  vector lt std  vector lt cv  Point gt   gt  squares      cv  Mat pyr  timg  gray0  image size    CV 8U   gray      int thresh   50  N   11      cv  pyrDown  image  pyr  cv  Size  image cols 2   image rows 2        cv  pyrUp pyr  timg   image size         std  vector lt std  vector lt cv  Point gt   gt  contours      for  int c   0  c  lt  3  c               int ch      c  0           mixChannels  amp timg  1   amp gray0  1  ch  1           for  int l   0  l  lt  N  l                   if  l    0                     cv  Canny gray0  gray  0  thresh  5                   cv  dilate gray  gray  cv  Mat    cv  Point -1 -1                              else                   gray   gray0  gt    l 1  255 N                            cv  findContours gray  contours  CV RETR LIST  CV CHAIN APPROX SIMPLE               std  vector lt cv  Point gt  approx              for  size t i   0  i  lt  contours size    i                                   cv  approxPolyDP cv  Mat contours i    approx  arcLength cv  Mat contours i    true  0 02  true                   if  approx size      4  amp  amp  fabs contourArea cv  Mat approx     gt  1000  amp  amp  cv  isContourConvex cv  Mat approx                          double maxCosine   0                       for  int j   2  j  lt  5  j                                                   double cosine   fabs angle approx j 4   approx j-2   approx j-1                             maxCosine   MAX maxCosine  cosine                                              if  maxCosine  lt  0 3                             squares push back approx                                                                             return squares      EDIT 17 08 2012   To draw the detected squares on the image use this code   cv  Mat debugSquares  std  vector lt std  vector lt cv  Point gt   gt  squares  cv  Mat image         for   int i   0  i lt  squares size    i                  draw contour         cv  drawContours image  squares  i  cv  Scalar 255 0 0   1  8  std  vector lt cv  Vec4i gt     0  cv  Point                 draw bounding rect         cv  Rect rect   boundingRect cv  Mat squares i             cv  rectangle image  rect tl    rect br    cv  Scalar 0 255 0   2  8  0               draw rotated rect         cv  RotatedRect minRect   minAreaRect cv  Mat squares i             cv  Point2f rect points 4           minRect points  rect points            for   int j   0  j  lt  4  j                   cv  line  image  rect points j   rect points  j 1  4   cv  Scalar 0 0 255   1  8       blue                      return image

User · Answer

Unless there is some other requirement not specified, I would simply convert your color image to grayscale and work with that only (no need to work on the 3 channels, the contrast present is too high already). Also, unless there is some specific problem regarding resizing, I would work with a downscaled version of your images, since they are relatively large and the size adds nothing to the problem being solved. Then, finally, your problem is solved with a median filter, some basic morphological tools, and statistics (mostly for the Otsu thresholding, which is already done for you).

Here is what I obtain with your sample image and some other image with a sheet of paper I found around:

enter image description here

The median filter is used to remove minor details from the, now grayscale, image. It will possibly remove thin lines inside the whitish paper, which is good because then you will end with tiny connected components which are easy to discard. After the median, apply a morphological gradient (simply dilation - erosion) and binarize the result by Otsu. The morphological gradient is a good method to keep strong edges, it should be used more. Then, since this gradient will increase the contour width, apply a morphological thinning. Now you can discard small components.

At this point, here is what we have with the right image above (before drawing the blue polygon), the left one is not shown because the only remaining component is the one describing the paper:

enter image description here

Given the examples, now the only issue left is distinguishing between components that look like rectangles and others that do not. This is a matter of determining a ratio between the area of the convex hull containing the shape and the area of its bounding box; the ratio 0.7 works fine for these examples. It might be the case that you also need to discard components that are inside the paper, but not in these examples by using this method (nevertheless, doing this step should be very easy especially because it can be done through OpenCV directly).

For reference, here is a sample code in Mathematica:

f = Import["http://thwartedglamour.files.wordpress.com/2010/06/my-coffee-table-1-sa.jpg"]
f = ImageResize[f, ImageDimensions[f][[1]]/4]
g = MedianFilter[ColorConvert[f, "Grayscale"], 2]
h = DeleteSmallComponents[Thinning[
     Binarize[ImageSubtract[Dilation[g, 1], Erosion[g, 1]]]]]
convexvert = ComponentMeasurements[SelectComponents[
     h, {"ConvexArea", "BoundingBoxArea"}, #1 / #2 > 0.7 &], 
     "ConvexVertices"][[All, 2]]
(* To visualize the blue polygons above: *)
Show[f, Graphics[{EdgeForm[{Blue, Thick}], RGBColor[0, 0, 1, 0.5], 
     Polygon @@ convexvert}]]

If there are more varied situations where the paper's rectangle is not so well defined, or the approach confuses it with other shapes -- these situations could happen due to various reasons, but a common cause is bad image acquisition -- then try combining the pre-processing steps with the work described in the paper "Rectangle Detection based on a Windowed Hough Transform".

User · Answer

This is a recurring subject in Stackoverflow and since I was unable to find a relevant implementation I decided to accept the challenge   I made some modifications to the squares demo present in OpenCV and the resulting C   code below is able to detect a sheet of paper in the image   void find squares Mat amp  image  vector lt vector lt Point gt   gt  amp  squares           blur will enhance edge detection     Mat blurred image       medianBlur image  blurred  9        Mat gray0 blurred size    CV 8U   gray      vector lt vector lt Point gt   gt  contours          find squares in every color plane of the image     for  int c   0  c  lt  3  c                  int ch      c  0           mixChannels  amp blurred  1   amp gray0  1  ch  1               try several threshold levels         const int threshold level   2          for  int l   0  l  lt  threshold level  l                             Use Canny instead of zero threshold level                 Canny helps to catch squares with gradient shading             if  l    0                                Canny gray0  gray  10  20  3                           Dilate helps to remove potential holes between edge segments                 dilate gray  gray  Mat    Point -1 -1                              else                                   gray   gray0  gt    l 1    255   threshold level                                Find contours and store them in a list             findContours gray  contours  CV RETR LIST  CV CHAIN APPROX SIMPLE                   Test contours             vector lt Point gt  approx              for  size t i   0  i  lt  contours size    i                                         approximate contour with accuracy proportional                        to the contour perimeter                     approxPolyDP Mat contours i    approx  arcLength Mat contours i    true  0 02  true                           Note  absolute value of an area is used because                        area may be positive or negative - in accordance with the                        contour orientation                     if  approx size      4  amp  amp                              fabs contourArea Mat approx     gt  1000  amp  amp                              isContourConvex Mat approx                                                      double maxCosine   0                               for  int j   2  j  lt  5  j                                                                      double cosine   fabs angle approx j 4   approx j-2   approx j-1                                         maxCosine   MAX maxCosine  cosine                                                              if  maxCosine  lt  0 3                                      squares push back approx                                                           After this procedure is executed  the sheet of paper will be the largest square in vector lt vector lt Point gt   gt      I m letting you write the function to find the largest square

User · Answer

Well  I m late     In your image  the paper is white  while the background is colored  So  it s better to detect the paper is Saturation      channel in HSV color space  Take refer to wiki HSL and HSV first  Then I ll copy most idea from my answer in this Detect Colored Segment in an image     Main steps    Read into BGR Convert the image from bgr to hsv space Threshold the S channel Then find the max external contour or do Canny  or HoughLines as you like  I choose findContours   approx to get the corners      This is my result       The Python code Python 3 5   OpenCV 3 3       usr bin python3   2017 12 20 10 47 28 CST   2017 12 20 11 29 30 CST  import cv2 import numpy as np     1  read into  bgr-space img   cv2 imread  test2 jpg       2  convert to hsv-space  then split the channels hsv   cv2 cvtColor img  cv2 COLOR BGR2HSV  h s v   cv2 split hsv      3  threshold the S channel using adaptive method  THRESH OTSU   or fixed thresh th  threshed   cv2 threshold s  50  255  cv2 THRESH BINARY INV      4  find all the external contours on the threshed S     cnts      cv2 findContours threshed  cv2 RETR EXTERNAL  cv2 CHAIN APPROX SIMPLE  cnts   cv2 findContours threshed  cv2 RETR EXTERNAL  cv2 CHAIN APPROX SIMPLE  -2   canvas    img copy    cv2 drawContours canvas  cnts  -1   0 255 0   1      sort and choose the largest contour cnts   sorted cnts  key   cv2 contourArea  cnt   cnts -1      approx the contour  so the get the corner points arclen   cv2 arcLength cnt  True  approx   cv2 approxPolyDP cnt  0 02  arclen  True  cv2 drawContours canvas   cnt   -1   255 0 0   1  cv2 LINE AA  cv2 drawContours canvas   approx   -1   0  0  255   1  cv2 LINE AA      Ok  you can see the result as tag 6  cv2 imwrite  detected png   canvas      Related answers    How to detect colored patches in an image using OpenCV  Edge detection on colored background using OpenCV OpenCV C   Obj-C  Detecting a sheet of paper   Square Detection How to use  cv2 findContours  in different OpenCV versions

User · Answer

What you need is a quadrangle instead of a rotated rectangle  RotatedRect will give you incorrect results  Also you will need a perspective projection   Basicly what must been done is    Loop through all polygon segments and connect those which are almost equel  Sort them so you have the 4 most largest line segments  Intersect those lines and you have the 4 most likely corner points  Transform the matrix over the perspective gathered from the corner points and the aspect ratio of the known object    I implemented a class Quadrangle which takes care of contour to quadrangle conversion and will also transform it over the right perspective   See a working implementation here  Java OpenCV deskewing a contour

User · Answer

Once you have detected the bounding box of the document  you can perform a four-point perspective transform to obtain a top-down birds eye view of the image  This will fix the skew and isolate only the desired object      Input image     Detected text object    Top-down view of text document    Code  from imutils perspective import four point transform import cv2 import numpy    Load image  grayscale  Gaussian blur  Otsu s threshold image   cv2 imread  1 png   gray   cv2 cvtColor image  cv2 COLOR BGR2GRAY  blur   cv2 GaussianBlur gray   7 7   0  thresh   cv2 threshold blur  0  255  cv2 THRESH BINARY   cv2 THRESH OTSU  1     Find contours and sort for largest contour cnts   cv2 findContours thresh  cv2 RETR EXTERNAL cv2 CHAIN APPROX SIMPLE  cnts   cnts 0  if len cnts     2 else cnts 1  cnts   sorted cnts  key cv2 contourArea  reverse True  displayCnt   None  for c in cnts        Perform contour approximation     peri   cv2 arcLength c  True      approx   cv2 approxPolyDP c  0 02   peri  True      if len approx     4          displayCnt   approx         break    Obtain birds  eye view of image warped   four point transform image  displayCnt reshape 4  2    cv2 imshow  thresh   thresh  cv2 imshow  warped   warped  cv2 imshow  image   image  cv2 waitKey

User · Answer

Detecting sheet of paper is kinda old school  If you want to tackle skew detection then it is better if you straightaway aim for text line detection  With this you will get the extremas left  right  top and bottom  Discard any graphics in the image if you dont want and then do some statistics on the text line segments to find the most occurring angle range or rather angle  This is how you will narrow down to a good skew angle  Now after this you put these parameters the skew angle and the extremas to deskew and chop the image to what is required   As for the current image requirement  it is better if you try CV RETR EXTERNAL instead of CV RETR LIST    Another method of detecting edges is to train a random forests classifier on the paper edges and then use the classifier to get the edge Map  This is by far a robust method but requires training and time   Random forests will work with low contrast difference scenarios for example white paper on roughly white background

[c++] OpenCV C++/Obj-C: Detecting a sheet of paper / Square Detection

Examples related to c++

Examples related to objective-c

Examples related to image-processing

Examples related to opencv

Examples related to computer-vision