By Abid Rahman K


2012-02-23 12:37:22 8 Comments

I am trying to implement a "Digit Recognition OCR" in OpenCV-Python (cv2). It is just for learning purposes. I would like to learn both KNearest and SVM features in OpenCV.

I have 100 samples (i.e. images) of each digit. I would like to train with them.

There is a sample letter_recog.py that comes with OpenCV sample. But I still couldn't figure out on how to use it. I don't understand what are the samples, responses etc. Also, it loads a txt file at first, which I didn't understand first.

Later on searching a little bit, I could find a letter_recognition.data in cpp samples. I used it and made a code for cv2.KNearest in the model of letter_recog.py (just for testing):

import numpy as np
import cv2

fn = 'letter-recognition.data'
a = np.loadtxt(fn, np.float32, delimiter=',', converters={ 0 : lambda ch : ord(ch)-ord('A') })
samples, responses = a[:,1:], a[:,0]

model = cv2.KNearest()
retval = model.train(samples,responses)
retval, results, neigh_resp, dists = model.find_nearest(samples, k = 10)
print results.ravel()

It gave me an array of size 20000, I don't understand what it is.

Questions:

1) What is letter_recognition.data file? How to build that file from my own data set?

2) What does results.reval() denote?

3) How we can write a simple digit recognition tool using letter_recognition.data file (either KNearest or SVM)?

3 comments

@Abid Rahman K 2012-03-08 15:35:49

Well, I decided to workout myself on my question to solve above problem. What I wanted is to implement a simpl OCR using KNearest or SVM features in OpenCV. And below is what I did and how. ( it is just for learning how to use KNearest for simple OCR purposes).

1) My first question was about letter_recognition.data file that comes with OpenCV samples. I wanted to know what is inside that file.

It contains a letter, along with 16 features of that letter.

And this SOF helped me to find it. These 16 features are explained in the paperLetter Recognition Using Holland-Style Adaptive Classifiers. ( Although I didn't understand some of the features at end)

2) Since I knew, without understanding all those features, it is difficult to do that method. I tried some other papers, but all were a little difficult for a beginner.

So I just decided to take all the pixel values as my features. (I was not worried about accuracy or performance, I just wanted it to work, at least with the least accuracy)

I took below image for my training data:

enter image description here

( I know the amount of training data is less. But, since all letters are of same font and size, I decided to try on this).

To prepare the data for training, I made a small code in OpenCV. It does following things:

  1. It loads the image.
  2. Selects the digits ( obviously by contour finding and applying constraints on area and height of letters to avoid false detections).
  3. Draws the bounding rectangle around one letter and wait for key press manually. This time we press the digit key ourselves corresponding to the letter in box.
  4. Once corresponding digit key is pressed, it resizes this box to 10x10 and saves 100 pixel values in an array (here, samples) and corresponding manually entered digit in another array(here, responses).
  5. Then save both the arrays in separate txt files.

At the end of manual classification of digits, all the digits in the train data( train.png) are labeled manually by ourselves, image will look like below:

enter image description here

Below is the code I used for above purpose ( of course, not so clean):

import sys

import numpy as np
import cv2

im = cv2.imread('pitrain.png')
im3 = im.copy()

gray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray,(5,5),0)
thresh = cv2.adaptiveThreshold(blur,255,1,1,11,2)

#################      Now finding Contours         ###################

contours,hierarchy = cv2.findContours(thresh,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)

samples =  np.empty((0,100))
responses = []
keys = [i for i in range(48,58)]

for cnt in contours:
    if cv2.contourArea(cnt)>50:
        [x,y,w,h] = cv2.boundingRect(cnt)

        if  h>28:
            cv2.rectangle(im,(x,y),(x+w,y+h),(0,0,255),2)
            roi = thresh[y:y+h,x:x+w]
            roismall = cv2.resize(roi,(10,10))
            cv2.imshow('norm',im)
            key = cv2.waitKey(0)

            if key == 27:  # (escape to quit)
                sys.exit()
            elif key in keys:
                responses.append(int(chr(key)))
                sample = roismall.reshape((1,100))
                samples = np.append(samples,sample,0)

responses = np.array(responses,np.float32)
responses = responses.reshape((responses.size,1))
print "training complete"

np.savetxt('generalsamples.data',samples)
np.savetxt('generalresponses.data',responses)

Now we enter in to training and testing part.

For testing part I used below image, which has same type of letters I used to train.

enter image description here

For training we do as follows:

  1. Load the txt files we already saved earlier
  2. create a instance of classifier we are using ( here, it is KNearest)
  3. Then we use KNearest.train function to train the data

For testing purposes, we do as follows:

  1. We load the image used for testing
  2. process the image as earlier and extract each digit using contour methods
  3. Draw bounding box for it, then resize to 10x10, and store its pixel values in an array as done earlier.
  4. Then we use KNearest.find_nearest() function to find the nearest item to the one we gave. ( If lucky, it recognises the correct digit.)

I included last two steps ( training and testing) in single code below:

import cv2
import numpy as np

#######   training part    ############### 
samples = np.loadtxt('generalsamples.data',np.float32)
responses = np.loadtxt('generalresponses.data',np.float32)
responses = responses.reshape((responses.size,1))

model = cv2.KNearest()
model.train(samples,responses)

############################# testing part  #########################

im = cv2.imread('pi.png')
out = np.zeros(im.shape,np.uint8)
gray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY)
thresh = cv2.adaptiveThreshold(gray,255,1,1,11,2)

contours,hierarchy = cv2.findContours(thresh,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPLE)

for cnt in contours:
    if cv2.contourArea(cnt)>50:
        [x,y,w,h] = cv2.boundingRect(cnt)
        if  h>28:
            cv2.rectangle(im,(x,y),(x+w,y+h),(0,255,0),2)
            roi = thresh[y:y+h,x:x+w]
            roismall = cv2.resize(roi,(10,10))
            roismall = roismall.reshape((1,100))
            roismall = np.float32(roismall)
            retval, results, neigh_resp, dists = model.find_nearest(roismall, k = 1)
            string = str(int((results[0][0])))
            cv2.putText(out,string,(x,y+h),0,1,(0,255,0))

cv2.imshow('im',im)
cv2.imshow('out',out)
cv2.waitKey(0)

And it worked, below is the result I got:

enter image description here


Here it worked with 100% accuracy. I assume this is because all the digits are of same kind and same size.

But any way, this is a good start to go for beginners ( I hope so).

@karlphillip 2012-04-17 17:50:04

+1 Long post, but very educational. This should go to opencv tag info

@goncalopp 2012-10-14 03:01:36

in case anyone's interested, I made a proper OO engine from this code, along with some bells and whistles: github.com/goncalopp/simple-ocr-opencv

@Ricardo 2012-11-12 10:34:52

Hi, the google docs link refered by this post doesn't work for me.

@Abid Rahman K 2012-11-12 11:12:56

@Ricardo : edited the link, check if it works, or search for the name of the paper, first link in google turns up.

@Edgar Andrés Margffoy Tuay 2012-12-16 16:02:57

When you're extracting the features, it is possible to do it in order? Or the border detection is always random? Thanks in advance

@Abid Rahman K 2012-12-16 19:13:27

I didn't get you. Are you asking the order in which contours are found in opencv? then it is not in a good order. I don't know if it is random, but not in any order we know, like left-to-right-top-to-bottom, vice versa etc. It causes some difficulty in some cases. We have to manually order it according to our criteria. (if this is not what you meant to ask, please clarify )

@mmgp 2013-01-28 05:23:39

Note that there is no need for using SVM and KNN when you have a well defined perfect font. For instance, the digits 0, 4, 6, 9 form one group, the digits 1, 2, 3, 5, 7 form another, and 8 another. This group is given by the euler number. Then "0" has no endpoints, "4" has two, and "6" and "9" are distinguished by centroid position. "3" is the only one, in the other group, with 3 endpoints. "1" and "7" are distinguished by the skeleton length. When considering the convex hull together with the digit, "5" and "2" have two holes and they can be distinguished by the centroid of largest hole.

@Abid Rahman K 2013-01-28 09:47:28

Well, thanks for this information on euler number. I didn't know that. Anyway, in this case I know it is pretty straight forward problem. But my aim was to understand how to use kNearest function and how to develop a simple OCR, in the most basic level. That is what all I wanted.

@rash 2013-08-19 11:13:41

@AbidRahmanK, First code runs perfect. But unfortunately I am getting this error while running second code.OpenCV Error: Bad argument (train data must be floating-point matrix) in cvCheckTrainData, file /build/buildd/opencv-2.3.1/modules/ml/src/inner_functions.cp‌​p, line 857 Traceback (most recent call last): File "num1.py", line 10, in <module> model.train(samples,responses) cv2.error: /build/buildd/opencv-2.3.1/modules/ml/src/inner_functions.cp‌​p:857: error: (-5) train data must be floating-point matrix in function cvCheckTrainData How can i fix this?

@Abid Rahman K 2013-08-19 13:48:27

@rash : check if samples and responses are floating point. If not, convert them to floating point.

@rash 2013-08-20 05:14:07

@AbidRahmanK : I am getting generalresponses.data and generalsamples.data files as empty. Actually first program gives responses a null list in my case and gives 1048586 as key.

@Abid Rahman K 2013-08-20 05:17:19

@rash : did you use the same images I used? First part of code is set for this image only.

@rash 2013-08-20 06:42:09

@AbidRahmanK : Yes brother. And it showed as you said. But the files are empty :( .

@shalki 2013-10-10 06:11:04

@rash: did you get the solution for your problem? I was doing this thing and getting the same issue.

@shalki 2013-10-10 06:22:36

Got the problem.. Thank you. It was a great tutorial. I was making a small mistake. If anyone else faces same issue in this like me and @rash then that is because you are pressing wrong key. For each number in box, you have to enter that no so that it gets trained on it. Hope that helps.

@pdm 2015-08-12 09:33:24

In case for guys who installed OpenCV(version 2.4+), there are several API has been changed in OpenCV. I just made a repo, which works fine for me. Hope it helps.

@Johannes Brodwall 2016-08-09 11:44:00

A stellar tutorial. Thank you! There are a few changes needed to get this to work with the latest (3.1) versjon of OpenCV: contours,hierarchy = cv2.findContours(thresh,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPL‌​E) => _,contours,hierarchy = cv2.findContours(thresh,cv2.RETR_LIST,cv2.CHAIN_APPROX_SIMPL‌​E), model = cv2.KNearest() => model = cv2.ml.KNearest_create(), model.train(samples,responses) => model.train(samples,cv2.ml.ROW_SAMPLE,responses), retval, results, neigh_resp, dists = model.find_nearest(roismall, k = 1) => retval, results, neigh_resp, dists = model.find_nearest(roismall, k = 1)

@Abid Rahman K 2016-08-09 11:58:17

Thanks. Yeah, this has become pretty old and I haven't explored 3.x version recently. I think I should revisit this once.

@Thomas 2016-11-14 11:56:12

@JohannesBrodwall Thanks for your update, quick note - your last correction is slightly off and should read: retval, results, neigh_resp, dists = model.find_nearest(roismall, k = 1) => retval, results, neigh_resp, dists = model.findNearest(roismall, k = 1)

@siddhesh 2017-01-31 10:10:06

it's really good explaination but how we can find number iteself in image at the above image every contour will be the number. but in realworld there are contours which are not [email protected]

@Macumbaomuerte 2017-04-29 11:52:25

I'm curious to know how fast is this aproach compared to tesseract OCR regarding only numbers. Does anybody tested both aproaches?

@Macumbaomuerte 2017-04-29 17:22:54

I tested myself, so I'll give my results if anyone wants to have a clear idea on how fast this aproach can be. I used a small (38x24px) image with the characters 123 I counted from when the image to analyse is loaded till i get the string var. Using OpenCV 3 on Windows 10 x64 with an Intel i-73540M CPU and Tesseract 4.0 alpha (untrained). - AbidRahmanK code took: **0.0510** seconds - Tesseract with "-psm 6" attributes took: **0.8970** seconds So @AbidRahmanK wins by far on speed, haven't compared accuracy beyond this very simple test wich gave 100% accuracy

@Yingqiang Gao 2018-01-21 19:30:29

I tried your code and indeed the digits was perfectly recognized. But how do you recognize the dot?

@Neji Soltani 2018-01-29 08:47:54

Can we make the same process using Java ?

@Dark 2018-09-27 07:31:57

I tried your second code and get the following error message at line model.train(samples,responses): TypeError: only size-1 arrays can be converted to Python scalars I don't know what I am doing wrong. Any ideas?

@shan 2019-01-28 13:42:07

@Dark Did you solve your problem? I am getting the same error.

@Cees Timmerman 2019-06-07 08:37:24

@Dark Use model.train(samples, cv2.ml.ROW_SAMPLE, responses)

@Azhar Hussain 2019-10-14 06:20:13

I updated the code and added more images. For every character i trained on about 500 images and files are created but the size of those files is extremely large. classifications.txt is about 12mb and flattened_images files is about 11.2gb. what am i doing wrong?

@Yonatan Simson 2016-05-16 10:39:04

If you are interested in the state of the art in Machine Learning, you should look into Deep Learning. You should have a CUDA supporting GPU or alternatively use the GPU on Amazon Web Services.

Google Udacity has a nice tutorial on this using Tensor Flow. This tutorial will teach you how to train your own classifier on hand written digits. I got an accuracy of over 97% on the test set using Convolutional Networks.

@Haris 2014-01-03 11:13:20

For those who interested in C++ code can refer below code. Thanks Abid Rahman for the nice explanation.


The procedure is same as above but, the contour finding uses only first hierarchy level contour, so that the algorithm uses only outer contour for each digit.

Code for creating sample and Label data

//Process image to extract contour
Mat thr,gray,con;
Mat src=imread("digit.png",1);
cvtColor(src,gray,CV_BGR2GRAY);
threshold(gray,thr,200,255,THRESH_BINARY_INV); //Threshold to find contour
thr.copyTo(con);

// Create sample and label data
vector< vector <Point> > contours; // Vector for storing contour
vector< Vec4i > hierarchy;
Mat sample;
Mat response_array;  
findContours( con, contours, hierarchy,CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE ); //Find contour

for( int i = 0; i< contours.size(); i=hierarchy[i][0] ) // iterate through first hierarchy level contours
{
    Rect r= boundingRect(contours[i]); //Find bounding rect for each contour
    rectangle(src,Point(r.x,r.y), Point(r.x+r.width,r.y+r.height), Scalar(0,0,255),2,8,0);
    Mat ROI = thr(r); //Crop the image
    Mat tmp1, tmp2;
    resize(ROI,tmp1, Size(10,10), 0,0,INTER_LINEAR ); //resize to 10X10
    tmp1.convertTo(tmp2,CV_32FC1); //convert to float
    sample.push_back(tmp2.reshape(1,1)); // Store  sample data
    imshow("src",src);
    int c=waitKey(0); // Read corresponding label for contour from keyoard
    c-=0x30;     // Convert ascii to intiger value
    response_array.push_back(c); // Store label to a mat
    rectangle(src,Point(r.x,r.y), Point(r.x+r.width,r.y+r.height), Scalar(0,255,0),2,8,0);    
}

// Store the data to file
Mat response,tmp;
tmp=response_array.reshape(1,1); //make continuous
tmp.convertTo(response,CV_32FC1); // Convert  to float

FileStorage Data("TrainingData.yml",FileStorage::WRITE); // Store the sample data in a file
Data << "data" << sample;
Data.release();

FileStorage Label("LabelData.yml",FileStorage::WRITE); // Store the label data in a file
Label << "label" << response;
Label.release();
cout<<"Training and Label data created successfully....!! "<<endl;

imshow("src",src);
waitKey();

Code for training and testing

Mat thr,gray,con;
Mat src=imread("dig.png",1);
cvtColor(src,gray,CV_BGR2GRAY);
threshold(gray,thr,200,255,THRESH_BINARY_INV); // Threshold to create input
thr.copyTo(con);


// Read stored sample and label for training
Mat sample;
Mat response,tmp;
FileStorage Data("TrainingData.yml",FileStorage::READ); // Read traing data to a Mat
Data["data"] >> sample;
Data.release();

FileStorage Label("LabelData.yml",FileStorage::READ); // Read label data to a Mat
Label["label"] >> response;
Label.release();


KNearest knn;
knn.train(sample,response); // Train with sample and responses
cout<<"Training compleated.....!!"<<endl;

vector< vector <Point> > contours; // Vector for storing contour
vector< Vec4i > hierarchy;

//Create input sample by contour finding and cropping
findContours( con, contours, hierarchy,CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE );
Mat dst(src.rows,src.cols,CV_8UC3,Scalar::all(0));

for( int i = 0; i< contours.size(); i=hierarchy[i][0] ) // iterate through each contour for first hierarchy level .
{
    Rect r= boundingRect(contours[i]);
    Mat ROI = thr(r);
    Mat tmp1, tmp2;
    resize(ROI,tmp1, Size(10,10), 0,0,INTER_LINEAR );
    tmp1.convertTo(tmp2,CV_32FC1);
    float p=knn.find_nearest(tmp2.reshape(1,1), 1);
    char name[4];
    sprintf(name,"%d",(int)p);
    putText( dst,name,Point(r.x,r.y+r.height) ,0,1, Scalar(0, 255, 0), 2, 8 );
}

imshow("src",src);
imshow("dst",dst);
imwrite("dest.jpg",dst);
waitKey();

Result

In the result the dot in the first line is detected as 8 and we haven’t trained for dot. Also I am considering every contour in first hierarchy level as the sample input, user can avoid it by computing the area.

Results

@skm 2014-02-14 19:41:22

I tired to run this code. I was able to create sample and label data. But when i run the test-training file, it runs with an error *** stack smashing detected ***: and hence i am not getting a final proper image as you are getting above (digits in green color)

@skm 2014-02-14 19:57:30

i change char name[4]; in your code to char name[7]; and i didn't get the stack related error but still i am not getting the correct results. I am getting a image like here < i.imgur.com/qRkV2B4.jpg >

@Haris 2014-02-20 04:48:35

@skm Make sure that you are getting number of contour same as the number of digits in the image, also try by printing the result on console.

@yode 2018-07-11 06:47:51

Hello, could we load a trained net to use?

Related Questions

Sponsored Content

26 Answered Questions

[SOLVED] Does Python have a ternary conditional operator?

44 Answered Questions

22 Answered Questions

[SOLVED] What are metaclasses in Python?

8 Answered Questions

[SOLVED] Manually raising (throwing) an exception in Python

  • 2010-01-12 21:07:40
  • TIMEX
  • 1806852 View
  • 2328 Score
  • 8 Answer
  • Tags:   python exception

42 Answered Questions

[SOLVED] How to get the current time in Python

  • 2009-01-06 04:54:23
  • user46646
  • 3369630 View
  • 2970 Score
  • 42 Answer
  • Tags:   python datetime time

61 Answered Questions

[SOLVED] Calling an external command from Python

10 Answered Questions

[SOLVED] Does Python have a string 'contains' substring method?

24 Answered Questions

[SOLVED] Image Processing: Algorithm Improvement for 'Coca-Cola Can' Recognition

Sponsored Content