The process consists of four main modules:

The results obtained depend on the features extracted to represent the face pattern and the classification methods used to distinguish the faces, but to extract these features properly, it is necessary to localize and normalize the face properly.[2]​.

Traditional Techniques

Traditional facial recognition methods can be divided into two large groups:

They recognize the entire facial image. They are methods based on correlation. The simplest classification scheme, where comparison models are used for recognition, is template matching. people. This is what is called facial subspace"). Examples of methods that work from subspaces are Principal Component Analysis (PCA - Principal Component Analysis) from eigenfaces, Linear Discriminant Analysis (LDA - Linear Discriminant Analysis) or Fisher Linear Discriminant (FLD - Fisher Linear Discriminant) "from fisherfaces").

The PCA technique is considered one of those that provides the highest performance. It works by projecting facial images onto a feature space that encompasses significant variations between known facial images. The significant features are called eigenfaces, since they are the eigenvectors, or principal components, of the set of faces. The projection characterizes the facial image of an individual as the sum of the different weights of all the features and, in the same way, to recognize a given facial image it will only be necessary to compare these weights with those of previously known individuals. It does not take into account information about which images belong to the same individual. It is very sensitive to changes in lighting conditions in different images of the same person.

The LDA method allows using information between members of the same class (images of the same person) to develop a set of feature vectors where variations between different faces are emphasized while changes due to lighting, facial expression and face orientation are not. That is, it maximizes the variance of samples between classes, and minimizes it between samples of the same class.

The FLD technique is equivalent to LDA. The results obtained with FLD are much better than those we can obtain with PCA, especially when the lighting conditions vary between the set of training and test images, and also with changes in facial expression, giving more weight to areas such as the eyes, nose or cheeks than to the mouth, because they are more invariable areas in the different expressions that a person can have.

Other methods, instead of using facial subspaces, follow a classification by neural networks and deformable templates, such as EGM - Elastic graph matching").

Different geometric features of faces are compared. There are two divisions, the one based on the characteristic vectors extracted from the profile, and the one based on those extracted from a frontal view. It was used a lot previously but its results are not optimal.

3D techniques

Lately, the trend of three-dimensional facial recognition has increased, where 3D images are used in both training and recognition. This technique uses 3D sensors to capture information about the shape of the face. This information is subsequently used to identify characteristic features of the face such as the chin, eye contour, nose or cheekbones, retaining spatial information apart from texture and depth. An advantage of 3D facial recognition is that it is not affected by changes in lighting, as is the case with other techniques. Additionally, another plus point is that they can recognize a face at different angles, even in profile. The problem is that it is difficult to obtain reliable 3D images in the recognition phase, since the 3D sensors have to be very well calibrated and synchronized to acquire the information correctly. That is why the Partial Principal Component Analysis (PCA) method is used, derived from PCA, where 3D images are used in the training phase and in the database, but in the test phase it can use both 2D and 3D images. The technique attempts to reconstruct facial models in 3D from multiple images of the same person acquired through a multi-camera system or from 3D devices. The images 3D are 180° images in cylindrical coordinates. Other examples of 3D techniques are 3-D Morphable Model") and 3-D Face Recognition").

Skin texture analysis techniques

This trend uses the visual details of the skin. Analyze the unique lines, patterns and obvious details such as blemishes and/or scars on the subject's face. By using this algorithm we avoid having to go through the entire database since we can easily discard images. There are studies that show that using this technique, together with facial recognition, performance can increase up to 25 percent.[3]​.

Some video-based facial recognition systems have been developed,[4]​ for example, smart rooms that can recognize people and automatically initiate appropriate actions. Another example is systems that detect a driver's fatigue, monitoring facial expressions and head movements. But in video surveillance applications, facial recognition and identification is still a difficult task due to:

There are applications where it gives good results, such as access control systems and ATMs, where the video is acquired in a relatively controlled environment and the size of the face is acceptable. In these cases, video-based recognition offers some advantages over image-based recognition:

Weaknesses

The facial recognition system is not perfect and sometimes difficult to implement in certain conditions. One of the main weaknesses of this system is due to the angle at which the face we want to recognize is located. Studies have confirmed that recognition acts correctly up to 20º,[3]​ once this angle is exceeded, problems begin to arise. It is for this reason that 3D recognition is being investigated with which this problem would disappear.

Another drawback is poor functioning in low light situations. In addition, wearing long hair, sunglasses or other objects that cover part of the face makes the task very difficult. The algorithm is not always able to distinguish faces if their expression is different from the one stored in the database.

To solve some of these problems, apart from doing recognition on video, multimodal techniques can also be used[5]​ where, apart from the image of the person, voice and audio information is also incorporated, if it is found within the content.

Mainly, we can distinguish two problems that cause serious performance degradation in most existing systems:

The changes caused by lighting are typically larger than the differences between people, causing comparison-based systems to make mistakes when classifying input images. Some knowledge-based solutions have been proposed, in particular taking into account that all faces belong to the same class. These techniques are divided into 4 types:

The performance of a facial recognition system also drops significantly when changes in pose are present. There are different methods proposed to solve it:

Effectiveness

According to a recent study, automatic facial recognition gives better results than in humans, but there is still no technique that provides a robust solution for all types of situations and for the different applications that may need it.

According to 'Science' magazine, using images composed of several photographs acquired at different angles of the same person allows the algorithm's performance to reach up to 100% in the best of cases. This is the result of studies by psychologists at the University of Glasgow, where the issue is being discussed and the conclusion has been reached that this new system emulates one of the keys to the human ability to recognize faces. Even so, he continues to be unable to distinguish between different facial expressions.

Privacy

Although this technology brings us great benefits, the subject's privacy is morally questioned. What at first glance is considered a technological advance can become a weapon. Depending on its use, an entire society can be controlled, knowing at all times where they are and what they are doing. Some of the concerns include the use of technology by the military of the law without consent and the use of this technology as a form of surveillance.[6]​.

Given the large number of theories and techniques applicable to facial recognition, a clear evaluation and comparison of these algorithms are necessary. To be feasible, large quantities of images must be used for adequate evaluation. It is also very important that the sample be as statistically similar as possible to the images arising in the application under consideration. Scoring must be done in a way that reflects the cost of recognition errors. Let us remember that the results depend a lot on the application we are dealing with, so we cannot extrapolate the results to other applications.

There are extensive public databases available, as well as test protocols, to test facial recognition applications. One of them is the FERET protocol, whose objective is to provide a framework that models a configuration in real time and gathers an extensive database with images of faces in order to develop algorithms and evaluate them. It is the one used in the FRVT (Face Recognition Vendor Test).

The following table shows a series of databases currently available, where you can see the number of elements used and the recognition time:

In mid-2006, the latest facial recognition algorithms were evaluated in the Face Recognition Grand Challenge (FRGC)&action=edit&redlink=1 "Face Recognition Grand Challenge (FRGC) (not yet written)". 3D facial scanners, high-definition facial images and iris images were used in the tests. The results indicated that these new algorithms are 10 times more accurate than algorithms dated before 2002, and 100 times more accurate than those from 1995. These algorithms are so precise that they are capable of recognizing two identical twins.

Another recent improvement is that low-resolution images are currently no longer a problem because they can be treated using the super-resolution of the face, known as Face Hallucination. Likewise, the installation of high definition cameras is in process.

Although this system has numerous advantages, the main risks that have been discussed in recent years are:

Some consider that privacy in the century is a myth,[7][8][9]​ especially if you take into account the Cambridge Analytica scandal of 2018[10]​ related to the commercialization of personal information and the consequent changes, in the same year, in the "Privacy Policy".[11]​.

However, there is still a majority that defends its desire to preserve privacy, something that is endangered by, among other things, advances in the facial recognition system. Although the idea of ​​using the facial recognition system to strengthen legal systems and make countries safer is accepted by the vast majority, the problem lies in what can be achieved from this.

The information that one can have through the facial recognition system (Location or personal data, among other things[12]​) allows its use for both positive[13]​ (The reasons why it is used) and negative purposes. In the hands of the wrong person, facial recognition can be used as a weapon of harassment and extortion, among other things, by having exact information about location and personal data.

Examples.

FBI.

Through an investigation carried out by the American Government Accountability Office"), in March 2018, it was discovered that the FBI, through its database Next Generation Identification-Interstate Photo System") (NGI-IPS), has 412,000,000 photographs, among which are people who have not been the subject of any type of investigation.[12][14]​.

Furthermore, the average use of this data was deliberately concealed by the FBI so as not to go against the requirements of the privacy impact assessment.[15]​.

Moscow.

The implementation of FaceN technology, providing the Find code) Face (Service that allows you to search for people using their photos) allows you to obtain the images and data of millions of people in Russia. This service allows you to do up to 200 searches and thus look at at least 2 million faces in less than a second.[16]​.

Social networks such as Facebook, Instagram or Google + have many more images of faces than the FBI, and also have a facial recognition solution that is constantly developed and improved.[17]​.

Specifically, Facebook's facial recognition system is one of the largest in the world.[18]​ The problem with this is the discreet introduction of the system to the platform, so that your image is entered into the database almost imperceptibly.

Since it implemented this system in its database in 2012, it has received numerous complaints regarding the problems that having their data saved in the Facebook or Google database may cause users. [1][2].

The possibility of accessing a person's social networks through facial recognition in an image makes a wide range of personal information available to anyone.

Example.

Starting from a photography project carried out by student Yegor Tsvertkov in St. Petersburg, consisting of photographing unknown people and finding them on social networks through FindFace,[19]​ a group of trolls "Trol (Internet)") began to harass Russian porn actresses.

The images of these actresses were uploaded to the FindFace application by these trolls "Trol (Internet)") and through it they managed to find their social networks, once harassing family and friends of the actresses.[20]​.

Contenido

En relación con esta polémica han surgido diferentes opiniones, ofreciendo cada una de ellas un balance relacionado con los potenciales beneficios que el sistema de reconocimiento facial ofrece contrapuesto con los aspectos negativas de este.

Al Now

According to the report made by Al Now (Institute for the study of the social implications of Artificial Intelligence) in 2018, which discusses numerous incidents that occurred in 2018 due to new technologies, systems such as facial recognition could lead to facilitating manipulation and surveillance of governments and large corporations.[21]​.

Concretamente, este grupo afirmará que el reconocimiento facial necesitará ser regulado a fin de proteger los intereses públicos.

Forbes Technology Council

On the other hand, a group of 10 members of the Forbes Technology Council prepared a report regarding the dangers that the use of facial recognition for legal issues would generate. This council consists of a non-profit organization made up of leaders from various technology companies.[7]​ In this report they established the following:.

[7]​.

Carnegie Mellon study

A study conducted by researchers at Carnegie Mellon University in 2011 demonstrated how the identification of strangers allowed them to obtain their personal information and sometimes security numbers, using facial recognition systems.[22] This research team was led by Alessandro Adquisti, also a professor of information technology and public policy at Heinz University, and his team, among which were doctors Ralph Gross and Fred Stutztman.

The objective of this study was to determine the potential danger that facial recognition could generate in a society, given the ease with which a person's personal data can be accessed through this system. For this, three experiments were carried out:

This study was also complemented by the creation of a smartphone application that allowed information about subjects to be discovered from images of their faces.

The study seeks to demonstrate how this recognition capacity will continue to improve and become more affordable as there are more ways to identify each other. In this situation, Acquisti will consider the following:

"Ultimately, all this access will force us to rethink our notion of privacy. It may also affect how we interact with each other. Through natural evolution, humans have developed mechanisms to manage trust in face-to-face interpersonal relationships. Will we trust our instincts when our mobile phones can predict information about other people?"[23]​.

The process consists of four main modules:

The results obtained depend on the features extracted to represent the face pattern and the classification methods used to distinguish the faces, but to extract these features properly, it is necessary to localize and normalize the face properly.[2]​.

Traditional Techniques

Traditional facial recognition methods can be divided into two large groups:

They recognize the entire facial image. They are methods based on correlation. The simplest classification scheme, where comparison models are used for recognition, is template matching. people. This is what is called facial subspace"). Examples of methods that work from subspaces are Principal Component Analysis (PCA - Principal Component Analysis) from eigenfaces, Linear Discriminant Analysis (LDA - Linear Discriminant Analysis) or Fisher Linear Discriminant (FLD - Fisher Linear Discriminant) "from fisherfaces").

The PCA technique is considered one of those that provides the highest performance. It works by projecting facial images onto a feature space that encompasses significant variations between known facial images. The significant features are called eigenfaces, since they are the eigenvectors, or principal components, of the set of faces. The projection characterizes the facial image of an individual as the sum of the different weights of all the features and, in the same way, to recognize a given facial image it will only be necessary to compare these weights with those of previously known individuals. It does not take into account information about which images belong to the same individual. It is very sensitive to changes in lighting conditions in different images of the same person.

The LDA method allows using information between members of the same class (images of the same person) to develop a set of feature vectors where variations between different faces are emphasized while changes due to lighting, facial expression and face orientation are not. That is, it maximizes the variance of samples between classes, and minimizes it between samples of the same class.

The FLD technique is equivalent to LDA. The results obtained with FLD are much better than those we can obtain with PCA, especially when the lighting conditions vary between the set of training and test images, and also with changes in facial expression, giving more weight to areas such as the eyes, nose or cheeks than to the mouth, because they are more invariable areas in the different expressions that a person can have.

Other methods, instead of using facial subspaces, follow a classification by neural networks and deformable templates, such as EGM - Elastic graph matching").

Different geometric features of faces are compared. There are two divisions, the one based on the characteristic vectors extracted from the profile, and the one based on those extracted from a frontal view. It was used a lot previously but its results are not optimal.

3D techniques

Lately, the trend of three-dimensional facial recognition has increased, where 3D images are used in both training and recognition. This technique uses 3D sensors to capture information about the shape of the face. This information is subsequently used to identify characteristic features of the face such as the chin, eye contour, nose or cheekbones, retaining spatial information apart from texture and depth. An advantage of 3D facial recognition is that it is not affected by changes in lighting, as is the case with other techniques. Additionally, another plus point is that they can recognize a face at different angles, even in profile. The problem is that it is difficult to obtain reliable 3D images in the recognition phase, since the 3D sensors have to be very well calibrated and synchronized to acquire the information correctly. That is why the Partial Principal Component Analysis (PCA) method is used, derived from PCA, where 3D images are used in the training phase and in the database, but in the test phase it can use both 2D and 3D images. The technique attempts to reconstruct facial models in 3D from multiple images of the same person acquired through a multi-camera system or from 3D devices. The images 3D are 180° images in cylindrical coordinates. Other examples of 3D techniques are 3-D Morphable Model") and 3-D Face Recognition").

Skin texture analysis techniques

This trend uses the visual details of the skin. Analyze the unique lines, patterns and obvious details such as blemishes and/or scars on the subject's face. By using this algorithm we avoid having to go through the entire database since we can easily discard images. There are studies that show that using this technique, together with facial recognition, performance can increase up to 25 percent.[3]​.

Some video-based facial recognition systems have been developed,[4]​ for example, smart rooms that can recognize people and automatically initiate appropriate actions. Another example is systems that detect a driver's fatigue, monitoring facial expressions and head movements. But in video surveillance applications, facial recognition and identification is still a difficult task due to:

There are applications where it gives good results, such as access control systems and ATMs, where the video is acquired in a relatively controlled environment and the size of the face is acceptable. In these cases, video-based recognition offers some advantages over image-based recognition:

Weaknesses

The facial recognition system is not perfect and sometimes difficult to implement in certain conditions. One of the main weaknesses of this system is due to the angle at which the face we want to recognize is located. Studies have confirmed that recognition acts correctly up to 20º,[3]​ once this angle is exceeded, problems begin to arise. It is for this reason that 3D recognition is being investigated with which this problem would disappear.

Another drawback is poor functioning in low light situations. In addition, wearing long hair, sunglasses or other objects that cover part of the face makes the task very difficult. The algorithm is not always able to distinguish faces if their expression is different from the one stored in the database.

To solve some of these problems, apart from doing recognition on video, multimodal techniques can also be used[5]​ where, apart from the image of the person, voice and audio information is also incorporated, if it is found within the content.

Mainly, we can distinguish two problems that cause serious performance degradation in most existing systems:

The changes caused by lighting are typically larger than the differences between people, causing comparison-based systems to make mistakes when classifying input images. Some knowledge-based solutions have been proposed, in particular taking into account that all faces belong to the same class. These techniques are divided into 4 types:

The performance of a facial recognition system also drops significantly when changes in pose are present. There are different methods proposed to solve it:

Effectiveness

According to a recent study, automatic facial recognition gives better results than in humans, but there is still no technique that provides a robust solution for all types of situations and for the different applications that may need it.

According to 'Science' magazine, using images composed of several photographs acquired at different angles of the same person allows the algorithm's performance to reach up to 100% in the best of cases. This is the result of studies by psychologists at the University of Glasgow, where the issue is being discussed and the conclusion has been reached that this new system emulates one of the keys to the human ability to recognize faces. Even so, he continues to be unable to distinguish between different facial expressions.

Privacy

Although this technology brings us great benefits, the subject's privacy is morally questioned. What at first glance is considered a technological advance can become a weapon. Depending on its use, an entire society can be controlled, knowing at all times where they are and what they are doing. Some of the concerns include the use of technology by the military of the law without consent and the use of this technology as a form of surveillance.[6]​.

Given the large number of theories and techniques applicable to facial recognition, a clear evaluation and comparison of these algorithms are necessary. To be feasible, large quantities of images must be used for adequate evaluation. It is also very important that the sample be as statistically similar as possible to the images arising in the application under consideration. Scoring must be done in a way that reflects the cost of recognition errors. Let us remember that the results depend a lot on the application we are dealing with, so we cannot extrapolate the results to other applications.

There are extensive public databases available, as well as test protocols, to test facial recognition applications. One of them is the FERET protocol, whose objective is to provide a framework that models a configuration in real time and gathers an extensive database with images of faces in order to develop algorithms and evaluate them. It is the one used in the FRVT (Face Recognition Vendor Test).

The following table shows a series of databases currently available, where you can see the number of elements used and the recognition time:

In mid-2006, the latest facial recognition algorithms were evaluated in the Face Recognition Grand Challenge (FRGC)&action=edit&redlink=1 "Face Recognition Grand Challenge (FRGC) (not yet written)". 3D facial scanners, high-definition facial images and iris images were used in the tests. The results indicated that these new algorithms are 10 times more accurate than algorithms dated before 2002, and 100 times more accurate than those from 1995. These algorithms are so precise that they are capable of recognizing two identical twins.

Another recent improvement is that low-resolution images are currently no longer a problem because they can be treated using the super-resolution of the face, known as Face Hallucination. Likewise, the installation of high definition cameras is in process.

Although this system has numerous advantages, the main risks that have been discussed in recent years are:

Some consider that privacy in the century is a myth,[7][8][9]​ especially if you take into account the Cambridge Analytica scandal of 2018[10]​ related to the commercialization of personal information and the consequent changes, in the same year, in the "Privacy Policy".[11]​.

However, there is still a majority that defends its desire to preserve privacy, something that is endangered by, among other things, advances in the facial recognition system. Although the idea of ​​using the facial recognition system to strengthen legal systems and make countries safer is accepted by the vast majority, the problem lies in what can be achieved from this.

The information that one can have through the facial recognition system (Location or personal data, among other things[12]​) allows its use for both positive[13]​ (The reasons why it is used) and negative purposes. In the hands of the wrong person, facial recognition can be used as a weapon of harassment and extortion, among other things, by having exact information about location and personal data.

Examples.

FBI.

Through an investigation carried out by the American Government Accountability Office"), in March 2018, it was discovered that the FBI, through its database Next Generation Identification-Interstate Photo System") (NGI-IPS), has 412,000,000 photographs, among which are people who have not been the subject of any type of investigation.[12][14]​.

Furthermore, the average use of this data was deliberately concealed by the FBI so as not to go against the requirements of the privacy impact assessment.[15]​.

Moscow.

The implementation of FaceN technology, providing the Find code) Face (Service that allows you to search for people using their photos) allows you to obtain the images and data of millions of people in Russia. This service allows you to do up to 200 searches and thus look at at least 2 million faces in less than a second.[16]​.

Social networks such as Facebook, Instagram or Google + have many more images of faces than the FBI, and also have a facial recognition solution that is constantly developed and improved.[17]​.

Specifically, Facebook's facial recognition system is one of the largest in the world.[18]​ The problem with this is the discreet introduction of the system to the platform, so that your image is entered into the database almost imperceptibly.

Since it implemented this system in its database in 2012, it has received numerous complaints regarding the problems that having their data saved in the Facebook or Google database may cause users. [1][2].

The possibility of accessing a person's social networks through facial recognition in an image makes a wide range of personal information available to anyone.

Example.

Starting from a photography project carried out by student Yegor Tsvertkov in St. Petersburg, consisting of photographing unknown people and finding them on social networks through FindFace,[19]​ a group of trolls "Trol (Internet)") began to harass Russian porn actresses.

The images of these actresses were uploaded to the FindFace application by these trolls "Trol (Internet)") and through it they managed to find their social networks, once harassing family and friends of the actresses.[20]​.

Contenido

En relación con esta polémica han surgido diferentes opiniones, ofreciendo cada una de ellas un balance relacionado con los potenciales beneficios que el sistema de reconocimiento facial ofrece contrapuesto con los aspectos negativas de este.

Al Now

According to the report made by Al Now (Institute for the study of the social implications of Artificial Intelligence) in 2018, which discusses numerous incidents that occurred in 2018 due to new technologies, systems such as facial recognition could lead to facilitating manipulation and surveillance of governments and large corporations.[21]​.

Concretamente, este grupo afirmará que el reconocimiento facial necesitará ser regulado a fin de proteger los intereses públicos.

Forbes Technology Council

On the other hand, a group of 10 members of the Forbes Technology Council prepared a report regarding the dangers that the use of facial recognition for legal issues would generate. This council consists of a non-profit organization made up of leaders from various technology companies.[7]​ In this report they established the following:.

[7]​.

Carnegie Mellon study

A study conducted by researchers at Carnegie Mellon University in 2011 demonstrated how the identification of strangers allowed them to obtain their personal information and sometimes security numbers, using facial recognition systems.[22] This research team was led by Alessandro Adquisti, also a professor of information technology and public policy at Heinz University, and his team, among which were doctors Ralph Gross and Fred Stutztman.

The objective of this study was to determine the potential danger that facial recognition could generate in a society, given the ease with which a person's personal data can be accessed through this system. For this, three experiments were carried out:

This study was also complemented by the creation of a smartphone application that allowed information about subjects to be discovered from images of their faces.

The study seeks to demonstrate how this recognition capacity will continue to improve and become more affordable as there are more ways to identify each other. In this situation, Acquisti will consider the following:

"Ultimately, all this access will force us to rethink our notion of privacy. It may also affect how we interact with each other. Through natural evolution, humans have developed mechanisms to manage trust in face-to-face interpersonal relationships. Will we trust our instincts when our mobile phones can predict information about other people?"[23]​.