face.hpp

00001 /*
00002 By downloading, copying, installing or using the software you agree to this
00003 license. If you do not agree to this license, do not download, install,
00004 copy or use the software.
00005 
00006                           License Agreement
00007                For Open Source Computer Vision Library
00008                        (3-clause BSD License)
00009 
00010 Copyright (C) 2013, OpenCV Foundation, all rights reserved.
00011 Third party copyrights are property of their respective owners.
00012 
00013 Redistribution and use in source and binary forms, with or without modification,
00014 are permitted provided that the following conditions are met:
00015 
00016   * Redistributions of source code must retain the above copyright notice,
00017     this list of conditions and the following disclaimer.
00018 
00019   * Redistributions in binary form must reproduce the above copyright notice,
00020     this list of conditions and the following disclaimer in the documentation
00021     and/or other materials provided with the distribution.
00022 
00023   * Neither the names of the copyright holders nor the names of the contributors
00024     may be used to endorse or promote products derived from this software
00025     without specific prior written permission.
00026 
00027 This software is provided by the copyright holders and contributors "as is" and
00028 any express or implied warranties, including, but not limited to, the implied
00029 warranties of merchantability and fitness for a particular purpose are
00030 disclaimed. In no event shall copyright holders or contributors be liable for
00031 any direct, indirect, incidental, special, exemplary, or consequential damages
00032 (including, but not limited to, procurement of substitute goods or services;
00033 loss of use, data, or profits; or business interruption) however caused
00034 and on any theory of liability, whether in contract, strict liability,
00035 or tort (including negligence or otherwise) arising in any way out of
00036 the use of this software, even if advised of the possibility of such damage.
00037 */
00038 
00039 #ifndef __OPENCV_FACE_HPP__
00040 #define __OPENCV_FACE_HPP__
00041 
00042 /**
00043 @defgroup face Face Recognition
00044 
00045 - @ref face_changelog
00046 - @ref tutorial_face_main
00047 
00048 */
00049 
00050 #include "opencv2/core.hpp"
00051 #include "face/predict_collector.hpp"
00052 #include <map>
00053 
00054 namespace cv { namespace face {
00055 
00056 //! @addtogroup face
00057 //! @{
00058 
00059 /** @brief Abstract base class for all face recognition models
00060 
00061 All face recognition models in OpenCV are derived from the abstract base class FaceRecognizer, which
00062 provides a unified access to all face recongition algorithms in OpenCV.
00063 
00064 ### Description
00065 
00066 I'll go a bit more into detail explaining FaceRecognizer, because it doesn't look like a powerful
00067 interface at first sight. But: Every FaceRecognizer is an Algorithm, so you can easily get/set all
00068 model internals (if allowed by the implementation). Algorithm is a relatively new OpenCV concept,
00069 which is available since the 2.4 release. I suggest you take a look at its description.
00070 
00071 Algorithm provides the following features for all derived classes:
00072 
00073 -   So called “virtual constructor”. That is, each Algorithm derivative is registered at program
00074     start and you can get the list of registered algorithms and create instance of a particular
00075     algorithm by its name (see Algorithm::create). If you plan to add your own algorithms, it is
00076     good practice to add a unique prefix to your algorithms to distinguish them from other
00077     algorithms.
00078 -   Setting/Retrieving algorithm parameters by name. If you used video capturing functionality from
00079     OpenCV highgui module, you are probably familar with cv::cvSetCaptureProperty,
00080 ocvcvGetCaptureProperty, VideoCapture::set and VideoCapture::get. Algorithm provides similar
00081     method where instead of integer id's you specify the parameter names as text Strings. See
00082     Algorithm::set and Algorithm::get for details.
00083 -   Reading and writing parameters from/to XML or YAML files. Every Algorithm derivative can store
00084     all its parameters and then read them back. There is no need to re-implement it each time.
00085 
00086 Moreover every FaceRecognizer supports the:
00087 
00088 -   **Training** of a FaceRecognizer with FaceRecognizer::train on a given set of images (your face
00089     database!).
00090 -   **Prediction** of a given sample image, that means a face. The image is given as a Mat.
00091 -   **Loading/Saving** the model state from/to a given XML or YAML.
00092 -   **Setting/Getting labels info**, that is stored as a string. String labels info is useful for
00093     keeping names of the recognized people.
00094 
00095 @note When using the FaceRecognizer interface in combination with Python, please stick to Python 2.
00096 Some underlying scripts like create_csv will not work in other versions, like Python 3. Setting the
00097 Thresholds +++++++++++++++++++++++
00098 
00099 Sometimes you run into the situation, when you want to apply a threshold on the prediction. A common
00100 scenario in face recognition is to tell, whether a face belongs to the training dataset or if it is
00101 unknown. You might wonder, why there's no public API in FaceRecognizer to set the threshold for the
00102 prediction, but rest assured: It's supported. It just means there's no generic way in an abstract
00103 class to provide an interface for setting/getting the thresholds of *every possible* FaceRecognizer
00104 algorithm. The appropriate place to set the thresholds is in the constructor of the specific
00105 FaceRecognizer and since every FaceRecognizer is a Algorithm (see above), you can get/set the
00106 thresholds at runtime!
00107 
00108 Here is an example of setting a threshold for the Eigenfaces method, when creating the model:
00109 
00110 @code
00111 // Let's say we want to keep 10 Eigenfaces and have a threshold value of 10.0
00112 int num_components = 10;
00113 double threshold = 10.0;
00114 // Then if you want to have a cv::FaceRecognizer with a confidence threshold,
00115 // create the concrete implementation with the appropiate parameters:
00116 Ptr<FaceRecognizer> model = createEigenFaceRecognizer(num_components, threshold);
00117 @endcode
00118 
00119 Sometimes it's impossible to train the model, just to experiment with threshold values. Thanks to
00120 Algorithm it's possible to set internal model thresholds during runtime. Let's see how we would
00121 set/get the prediction for the Eigenface model, we've created above:
00122 
00123 @code
00124 // The following line reads the threshold from the Eigenfaces model:
00125 double current_threshold = model->getDouble("threshold");
00126 // And this line sets the threshold to 0.0:
00127 model->set("threshold", 0.0);
00128 @endcode
00129 
00130 If you've set the threshold to 0.0 as we did above, then:
00131 
00132 @code
00133 //
00134 Mat img = imread("person1/3.jpg", CV_LOAD_IMAGE_GRAYSCALE);
00135 // Get a prediction from the model. Note: We've set a threshold of 0.0 above,
00136 // since the distance is almost always larger than 0.0, you'll get -1 as
00137 // label, which indicates, this face is unknown
00138 int predicted_label = model->predict(img);
00139 // ...
00140 @endcode
00141 
00142 is going to yield -1 as predicted label, which states this face is unknown.
00143 
00144 ### Getting the name of a FaceRecognizer
00145 
00146 Since every FaceRecognizer is a Algorithm, you can use Algorithm::name to get the name of a
00147 FaceRecognizer:
00148 
00149 @code
00150 // Create a FaceRecognizer:
00151 Ptr<FaceRecognizer> model = createEigenFaceRecognizer();
00152 // And here's how to get its name:
00153 String name = model->name();
00154 @endcode
00155 
00156  */
00157 class CV_EXPORTS_W FaceRecognizer : public Algorithm
00158 {
00159 public:
00160     /** @brief Trains a FaceRecognizer with given data and associated labels.
00161 
00162     @param src The training images, that means the faces you want to learn. The data has to be
00163     given as a vector<Mat>.
00164     @param labels The labels corresponding to the images have to be given either as a vector<int>
00165     or a
00166 
00167     The following source code snippet shows you how to learn a Fisherfaces model on a given set of
00168     images. The images are read with imread and pushed into a std::vector<Mat>. The labels of each
00169     image are stored within a std::vector<int> (you could also use a Mat of type CV_32SC1). Think of
00170     the label as the subject (the person) this image belongs to, so same subjects (persons) should have
00171     the same label. For the available FaceRecognizer you don't have to pay any attention to the order of
00172     the labels, just make sure same persons have the same label:
00173 
00174     @code
00175     // holds images and labels
00176     vector<Mat> images;
00177     vector<int> labels;
00178     // images for first person
00179     images.push_back(imread("person0/0.jpg", CV_LOAD_IMAGE_GRAYSCALE)); labels.push_back(0);
00180     images.push_back(imread("person0/1.jpg", CV_LOAD_IMAGE_GRAYSCALE)); labels.push_back(0);
00181     images.push_back(imread("person0/2.jpg", CV_LOAD_IMAGE_GRAYSCALE)); labels.push_back(0);
00182     // images for second person
00183     images.push_back(imread("person1/0.jpg", CV_LOAD_IMAGE_GRAYSCALE)); labels.push_back(1);
00184     images.push_back(imread("person1/1.jpg", CV_LOAD_IMAGE_GRAYSCALE)); labels.push_back(1);
00185     images.push_back(imread("person1/2.jpg", CV_LOAD_IMAGE_GRAYSCALE)); labels.push_back(1);
00186     @endcode
00187 
00188     Now that you have read some images, we can create a new FaceRecognizer. In this example I'll create
00189     a Fisherfaces model and decide to keep all of the possible Fisherfaces:
00190 
00191     @code
00192     // Create a new Fisherfaces model and retain all available Fisherfaces,
00193     // this is the most common usage of this specific FaceRecognizer:
00194     //
00195     Ptr<FaceRecognizer> model =  createFisherFaceRecognizer();
00196     @endcode
00197 
00198     And finally train it on the given dataset (the face images and labels):
00199 
00200     @code
00201     // This is the common interface to train all of the available cv::FaceRecognizer
00202     // implementations:
00203     //
00204     model->train(images, labels);
00205     @endcode
00206      */
00207     CV_WRAP virtual void train(InputArrayOfArrays src, InputArray labels) = 0;
00208 
00209     /** @brief Updates a FaceRecognizer with given data and associated labels.
00210 
00211     @param src The training images, that means the faces you want to learn. The data has to be given
00212     as a vector<Mat>.
00213     @param labels The labels corresponding to the images have to be given either as a vector<int> or
00214     a
00215 
00216     This method updates a (probably trained) FaceRecognizer, but only if the algorithm supports it. The
00217     Local Binary Patterns Histograms (LBPH) recognizer (see createLBPHFaceRecognizer) can be updated.
00218     For the Eigenfaces and Fisherfaces method, this is algorithmically not possible and you have to
00219     re-estimate the model with FaceRecognizer::train. In any case, a call to train empties the existing
00220     model and learns a new model, while update does not delete any model data.
00221 
00222     @code
00223     // Create a new LBPH model (it can be updated) and use the default parameters,
00224     // this is the most common usage of this specific FaceRecognizer:
00225     //
00226     Ptr<FaceRecognizer> model =  createLBPHFaceRecognizer();
00227     // This is the common interface to train all of the available cv::FaceRecognizer
00228     // implementations:
00229     //
00230     model->train(images, labels);
00231     // Some containers to hold new image:
00232     vector<Mat> newImages;
00233     vector<int> newLabels;
00234     // You should add some images to the containers:
00235     //
00236     // ...
00237     //
00238     // Now updating the model is as easy as calling:
00239     model->update(newImages,newLabels);
00240     // This will preserve the old model data and extend the existing model
00241     // with the new features extracted from newImages!
00242     @endcode
00243 
00244     Calling update on an Eigenfaces model (see createEigenFaceRecognizer), which doesn't support
00245     updating, will throw an error similar to:
00246 
00247     @code
00248     OpenCV Error: The function/feature is not implemented (This FaceRecognizer (FaceRecognizer.Eigenfaces) does not support updating, you have to use FaceRecognizer::train to update it.) in update, file /home/philipp/git/opencv/modules/contrib/src/facerec.cpp, line 305
00249     terminate called after throwing an instance of 'cv::Exception'
00250     @endcode
00251 
00252     @note The FaceRecognizer does not store your training images, because this would be very
00253     memory intense and it's not the responsibility of te FaceRecognizer to do so. The caller is
00254     responsible for maintaining the dataset, he want to work with.
00255      */
00256     CV_WRAP virtual void update(InputArrayOfArrays src, InputArray labels);
00257 
00258     /** @overload */
00259     CV_WRAP_AS(predict_label) int predict(InputArray src) const;
00260 
00261 
00262     /** @brief Predicts a label and associated confidence (e.g. distance) for a given input image.
00263 
00264     @param src Sample image to get a prediction from.
00265     @param label The predicted label for the given image.
00266     @param confidence Associated confidence (e.g. distance) for the predicted label.
00267 
00268     The suffix const means that prediction does not affect the internal model state, so the method can
00269     be safely called from within different threads.
00270 
00271     The following example shows how to get a prediction from a trained model:
00272 
00273     @code
00274     using namespace cv;
00275     // Do your initialization here (create the cv::FaceRecognizer model) ...
00276     // ...
00277     // Read in a sample image:
00278     Mat img = imread("person1/3.jpg", CV_LOAD_IMAGE_GRAYSCALE);
00279     // And get a prediction from the cv::FaceRecognizer:
00280     int predicted = model->predict(img);
00281     @endcode
00282 
00283     Or to get a prediction and the associated confidence (e.g. distance):
00284 
00285     @code
00286     using namespace cv;
00287     // Do your initialization here (create the cv::FaceRecognizer model) ...
00288     // ...
00289     Mat img = imread("person1/3.jpg", CV_LOAD_IMAGE_GRAYSCALE);
00290     // Some variables for the predicted label and associated confidence (e.g. distance):
00291     int predicted_label = -1;
00292     double predicted_confidence = 0.0;
00293     // Get the prediction and associated confidence from the model
00294     model->predict(img, predicted_label, predicted_confidence);
00295     @endcode
00296      */
00297     CV_WRAP void predict(InputArray src, CV_OUT int &label, CV_OUT double &confidence) const;
00298 
00299 
00300     /** @brief - if implemented - send all result of prediction to collector that can be used for somehow custom result handling
00301     @param src Sample image to get a prediction from.
00302     @param collector User-defined collector object that accepts all results
00303 
00304     To implement this method u just have to do same internal cycle as in predict(InputArray src, CV_OUT int &label, CV_OUT double &confidence) but
00305     not try to get "best@ result, just resend it to caller side with given collector
00306     */
00307     CV_WRAP_AS(predict_collect) virtual void predict(InputArray src, Ptr<PredictCollector>  collector) const = 0;
00308 
00309     /** @brief Saves a FaceRecognizer and its model state.
00310 
00311     Saves this model to a given filename, either as XML or YAML.
00312     @param filename The filename to store this FaceRecognizer to (either XML/YAML).
00313 
00314     Every FaceRecognizer overwrites FaceRecognizer::save(FileStorage& fs) to save the internal model
00315     state. FaceRecognizer::save(const String& filename) saves the state of a model to the given
00316     filename.
00317 
00318     The suffix const means that prediction does not affect the internal model state, so the method can
00319     be safely called from within different threads.
00320      */
00321     CV_WRAP virtual void save(const String& filename) const;
00322 
00323     /** @brief Loads a FaceRecognizer and its model state.
00324 
00325     Loads a persisted model and state from a given XML or YAML file . Every FaceRecognizer has to
00326     overwrite FaceRecognizer::load(FileStorage& fs) to enable loading the model state.
00327     FaceRecognizer::load(FileStorage& fs) in turn gets called by
00328     FaceRecognizer::load(const String& filename), to ease saving a model.
00329      */
00330     CV_WRAP virtual void load(const String& filename);
00331 
00332     /** @overload
00333     Saves this model to a given FileStorage.
00334     @param fs The FileStorage to store this FaceRecognizer to.
00335     */
00336     virtual void save(FileStorage& fs) const = 0;
00337 
00338     /** @overload */
00339     virtual void load(const FileStorage& fs) = 0;
00340 
00341     /** @brief Sets string info for the specified model's label.
00342 
00343     The string info is replaced by the provided value if it was set before for the specified label.
00344      */
00345     CV_WRAP virtual void setLabelInfo(int label, const String& strInfo);
00346 
00347     /** @brief Gets string information by label.
00348 
00349     If an unknown label id is provided or there is no label information associated with the specified
00350     label id the method returns an empty string.
00351      */
00352     CV_WRAP virtual String getLabelInfo(int label) const;
00353 
00354     /** @brief Gets vector of labels by string.
00355 
00356     The function searches for the labels containing the specified sub-string in the associated string
00357     info.
00358      */
00359     CV_WRAP virtual std::vector<int> getLabelsByString(const String& str) const;
00360     /** @brief threshold parameter accessor - required for default BestMinDist collector */
00361     virtual double getThreshold() const = 0;
00362     /** @brief Sets threshold of model */
00363     virtual void setThreshold(double val) = 0;
00364 protected:
00365     // Stored pairs "label id - string info"
00366     std::map<int, String> _labelsInfo;
00367 };
00368 
00369 //! @}
00370 
00371 }}
00372 
00373 #include "opencv2/face/facerec.hpp"
00374 
00375 #endif
00376