Unpacking the EU Proposal for an AI Act: Implications for AI Systems Used in the Context of Migration, Asylum and Border Control Management

The proposed Regulation distinguishes AI systems on the basis of the risk they pose to the fundamental rights of individuals or EU values. In light of this, according to Article 5 of the proposal, particularly harmful AI practices should be prohibited, such as those that can manipulate vulnerable individuals through subliminal techniques. These practices may cause harm to the manipulated individual or others. For the purpose of law enforcement, social scoring and real-time remote biometric identification are not permitted in publicly accessible areas due to the 'unacceptable risk' they create.[1] 'High risk' AI systems impact people's safety or fundamental rights, and are therefore considered high-risk. Systems listed in Annex III include those used for biometric identification and categorisation of individuals, critical infrastructures (such as transportation), formal education or vocational training, essential private and public services (such as credit scoring which denies citizens a loan), law enforcement, migration, asylum, border control, management and administration of justice and democratic processes. The deployment of 'high-risk AI' systems will need to undergo a conformity assessment before being placed on the market and comply with a range of safety requirements (regarding, for instance, risk management, human oversight and data governance).[2] In addition, an ex-post market surveillance and supervision must be put in place to ensure compliance with the obligations and requirements for all high-risk AI systems already placed on the market.[3]

Zooming into Migration, Asylum and Border Control Management

Under migration, asylum and border management, Annex III details four different types of AI systems. These are: 

(a) AI systems intended to be used by competent public authorities as polygraphs and similar tools or to detect the emotional state of a natural person; 

(b) AI systems intended to be used by competent public authorities to assess a risk, including a security risk, a risk of irregular immigration, or a health risk, posed by a natural person who intends to enter or has entered into the territory of a Member State; 

(c) AI systems intended to be used by competent public authorities for the verification of the authenticity of travel documents and supporting documentation of natural persons and detect non-authentic documents by checking their security features; and 

(d) AI systems intended to assist competent public authorities for the examination of applications for asylum, visa and residence permits and associated complaints with regard to the eligibility of the natural persons applying for a status.

The first type essentially refers to emotion recognition systems that claim to infer people’s emotions and mental states from physical, physiological, behavioural, as well as biometric data. Their use worldwide is currently in its early stages, but not prohibited. In the U.S, the Department of Homeland Security funded research of the virtual border agent technology known as the Automated Virtual Agent for Truth Assessments in Real-Time (AVATAR) and allowed it to be tested it at the U.S-Mexico border on travelers who volunteered to participate.[4] Perhaps the most prominent application of emotion recognition in the EU was Frontex's highly contentious iBorderCtrl project (Intelligent Portable Border Control System) from September 2016 through August 2019. Detecting mental states and emotions by examining a person's facial expressions as well as other physiognomic indicators has attracted considerable attention. Hungary, Latvia, and Greece piloted an automated lie-detection test analysing facial micro-gestures at three undisclosed airports in order to identify ‘biomarkers of deceit’.[5] By scanning refugees' and migrants' facial expressions as they answer questions, the machine is able to determine whether they have lied to the machine, and then pass on that information to border officers. The project has been challenged before the CJEU on transparency grounds. MEP Patrick Breyer has requested that the EU Research Agency (REA) release classified documents evaluating the 4.5 million euro trial of AI lie detectors to bolster EU border control to determine its ethical and legal justifiability, as well as the results of the technology, on the ethical justification and legal admissibility.[6] In December 2021, the General Court of Justice opined that REA may no longer keep these documents completely secret. The ethical and legal evaluation of technologies for ‘automated deception detection’ or automated risk assessment must be published, as long as they do not relate specifically to the iBorderCtrl project.[7] Nevertheless, to protect commercial interests, the examination of ethical risks (e.g. stigmatization and false positives) and legal admissibility of the concrete technology of iBorderCtrl, as well as reports of project results, may be kept confidential. 

Considering that the EU has already been testing AI polygraphs and lie detectors for years, it is unsurprising that the proposal does not wish to ban them entirely. This is notwithstanding the fact that the programme has reported an accuracy rate of 73-75 percent and has raised (well-deserved) criticism about its validity that its applicability in a real-life context.[8] Similarly, the U.S. AVATAR as a deception-detection judge has a success rate of 60 to 75 percent and sometimes up to 80 percent.[9] Expressions and facial expressions differ greatly between cultures and situations. As a result, emotion detection has limited reliability;[10] emotion categories are neither reliably expressed through, nor unequivocally associated with a common set of facial movements. Furthermore, facial expressions do not perfectly match emotion categories, so generalizability is generally limited. In addition, the effectiveness of such tools has yet to be proved, whereas the risk of racial profiling is significant.[11] From that perspective, the judgment by the General Court is somewhat disappointing, as the results of the project could support the claim that emotion detection systems posed such ethical and fundamental rights challenges to the extent that they should not be classified as high risk and should be banned altogether, as advocated by civil society actors.[12] At least, it is obligatory for project participants to make a scientific publication about the project within four years and therefore it is possible to get more answers on the project.

Whenever automated assessments are made as to whether an individual may pose a security, immigration, or health risk, these are rightly classified as high-risk AI systems. A subset of automatic assessments falls under this category of AI, which incorporate algorithmic profiling against risk indicators developed through statistical analysis of rates of overstaying and refusals of entry. It also includes the use of AI tools to identify irregular traveling patterns as an additional piece of risk analysis and identify so-called ‘mala fide’ travelers. This category also includes AI systems that determine whether an asylum seeker poses a risk of absconding and should therefore be detained. The EU legislature has already adopted legal instruments that will require in the near future such automated assessments in the cases of the European Travel Information and Authorisation System (ETIAS)[13] and the Visa Information System (VIS).[14] Visa-free nationals will have to register for ETIAS online in order to provide a wide array of personal data, which will then be cross-checked against databases, both European and some Interpol ones, according to specific screening rules, in order to enable profiling on the basis of risk indicators. An ETIAS authorization will be automatically provided in the event that there is no problem with the data provided, otherwise the application will go through manual processing by the responsible Member State. Similarly, data provided by short or long-stay visa and residence permits applicants will be subject to comparable rules, except that the automated processing will never lead to automatic issuance of a visa or a residence permit and there is always manual processing of the application by visa authorities.[15] As argued elsewhere, the risk of discrimination both direct and indirect in the design of the algorithms is particularly high. As they are trained using pre-existing data and past decisions, they run the risk of repeating all of the implicit and inherent biases of those earlier decisions. Such initiatives are already tried (unsuccessfully) in the UK, where the Home Secretary decided to withdraw an algorithm used to filter visa applications after being called out as racially discriminatory.[16] The system took some information provided by visa applicants and automatically processed it, giving each person a colour code based on a "traffic light" system - green, amber, or red. Similarly, in Canada the Government tested AI solutions to triage temporary resident visa applications.[17] Regrettably, these systems will not fall within the scope of the AI Act. According to article 83, AI systems which are components of large-scale IT systems governed by a series of EU legal instruments (listed in Annex IX) and implemented before 12 months after the application date of the AI Act - after its enactment - are excluded from the AI Act, unless the replacement or amendment of those legal acts results in a significant change to the design or intended purpose of the AI system or AI systems concerned. As ETIAS is slated for implementation in 2023, and the VIS rules will be in place not long after, it is almost certain that algorithmic profiling of travelers and migrants will fall outside the scope of the AI Act, which is still being negotiated and will take two years to enforce in the Member States. That said, the principle that automated risk assessments should be classified as high risk will still be applicable, but the deployment of AI systems in relation to these databases will not be subject to the safeguards encompassed in the forthcoming AI Act.[18]

The exclusion clause of Article 83 is particularly problematic for a series of other reasons. In addition to ETIAS and VIS, the remaining information systems that process personal data of third-country nationals, Schengen Information System (SIS), Eurodac, Entry/Exit System and European Criminal Record Information System for Third-Country Nationals (ECRIS-TCN) are also under development (EES and ECRIS-TCN) or refurbishment (SIS and Eurodac) to encompass among others facial recognition technology, which enables biometric identification and is also considered as high risk type of AI systems. Consequently, Article 83 essentially excludes all information systems for third-country nationals from the protective scope of the AI Act. This exclusion applies unless those systems are subject to ‘significant changes‘ in design or intended purpose. As mentioned in the Joint Opinion of the European Data Protection Supervisor (EDPS) and the European Data Protection Board (EDPB), the threshold for ‘significant changes’ is not clear. Recital 66 of the Proposal specifies a lower threshold for conformity re-assessment ‘whenever a change occurs which may affect the compliance’ and it has been rightly argued, a similar threshold would be appropriate for Article 83 as well, at least for high risk AI systems.[19] According to the Commission report on the opportunities and challenges for the use of AI in border control, migration and security, additional initiatives are underway, including application triaging, along the lines of the UK failed experiment. [20] In the case of ETIAS, the report notes that there will be in the future individual risk assessment by an AI tool to assist in manual processing.[21] These initiatives should be considered as ‘significant changes’ to the functioning of the systems to subject them to the forthcoming AI Act. Therefore, if these initiatives are to be adopted and implemented within the next few years, it makes no sense to keep the exclusion clause. Overall, it is unclear why this exclusion clause, which will create protection gaps, has been included in the first place. One possible explanation is the fact that the AI systems that will be included in the information systems for third-country nationals are already under development and awaiting for the adoption of the AI Act could potentially change the timeline for implementation as well as change the procedures for testing the algorithms that will be used, which are to be developed by the ETIAS Central Unit, within the European Border and Coast Guard. In any case, and as a minimum, considering that the entry into application is envisaged for 24 months following the entry into force of the future Regulation, exempting AI systems already placed on the market for an even longer period of time is not appropriate.[22] Moreover, while the Proposal also provides that the requirements of the Regulation shall be taken into account when evaluating each large-scale IT system as provided by the legal acts listed in Annex IX, both EDPB and EDPS believe that conditions for putting AI systems into service should be applicable from the date of application of the future Regulation. That said, Article 83 contains a protection clause; the requirements of the AI Act must be taken into account, where applicable, in the evaluation of large-scale IT systems. This is an important safeguard, but it must be noted that in the past, the legal bases of information systems have been amended before the conduct of an evaluation and therefore it is possible that the AI Act prescriptions will not be taken into account at that phase. Thus, it is imperative that the threshold of placing existing AI systems within the scope of the Act is lowered to minimize gaps in protection. 

Under the third type of AI systems, AI systems might enable tools that detect forgeries of travel documents or other supporting documents. Furthermore, these systems are rightly regarded as high risk, since they are bound to significantly affect individuals if the results (false positives and negatives) are inaccurate. For example, various unjustified claims could be made against an individual (e.g., document fraud) on the basis of inaccurate underlying data. Meanwhile, individuals may face detention and degrading treatment when suspected of criminal conduct.

In regard to the last type of AI systems classified as high risk, the proposal refers to assistance in examining requests for asylum, visas, and residence permits, as well as complaints regarding eligibility of the natural person applying for such a status. AI systems of this type seem closely related to the second, but they may encompass broader initiatives such as personalized application forms that tailor questions to the applicant using AI. In this case, augmented application forms could be developed, or artificial intelligence systems could be used to perform vulnerability assessments of asylum seekers to determine if they need to be further investigated by a social worker or granted special procedural rights. The repercussions of severe decisions and actions in this context can have particularly severe consequences on individuals; the remedies to reverse the harm caused by these actions and decisions are unclear. 

Finally, there are three further reasons why the list of migration-related AI systems is problematic. The first one concerns the lack of references to the use of AI systems for predictive analytics of migration.[23] As mentioned by Beduschi, AI technologies to foresee arrivals and prepare more efficiently for large influxes of people.[24] In February 2022, the Commission along with the European Union Asylum Agency (former European Asylum Support Office) announced the development of a forecasting model for asylum flows.[25] In addition, the use of chatbots and intelligent agents, which would take in information, answer questions posed by the applicant, and ensure data quality, also falls outside the scope of the AI Act. Chatbots use natural language processing to conduct human-like conversations with users and they have been criticized for embedding biases and using discriminatory language.[26] Therefore, these should also be considered as high risk. Furthermore, individuals should know in advance whether they are dealing with a human or a machine. Finally, the proposal refers to AI systems used by public authorities only, whereas increasingly there are instances of border management entrusted to private companies, for example, in the management of camps in the Greek islands.[27] Due to the blurred lines between public and private security it was imperative that the use of AI system for immigration control purposes would fall within the scope of the AI Act regardless of the actor involved. In fact, the latest Council documents consolidating the amendments proposed during the Slovenian presidency address this concern.[28]

This contribution pointed out that the proposed EU AI Act categorizes as high risk several AI systems that may be used for immigration-related purposes; however, there are significant flaws in the current approach, which the Council has only partially rectified. Negotiations are still ongoing, so it will be up to the Parliament to push for a better balance between taking advantage of the opportunities offered by AI and protecting fundamental rights.