Algorithmic bias: A first look at discrimination and the fintech sector in Peru

Introduction

This report considers the use of artificial intelligence (AI) in the so-called “fintech” sector in Peru. Fintech refers to emerging industries that use technology to improve activities in the financial market, for example, to offer loans or mobile bank services. The report frames the discussion in terms of the potential of AI to unfairly discriminate against potential customers, laying the basis for more detailed studies in the sector.

A set of recommendations aimed at reducing potential discrimination in the fintech sector is also proposed, in order to exploit the potential of fintech to ensure the inclusion of vulnerable groups in the financial system. 

Background

In 2018, the Inter-American Development Bank (IDB) and Finnovista, an organisation that promotes fintech ventures in the Latin American region, published the Report on Fintech in Latin America 2018: Growth and Consolidation, a follow-up study on the development of fintech in the region. It was reported that Peru was ranked sixth out of a total of 18 countries in the region, with 5% of the total fintech market and a total of 57 fintech ventures in the country. Compared to the measurement taken in 2017, the Peruvian fintech market experienced growth of 256%, the second largest growth experienced in Latin America.[1]

In May 2019 the Lima Fintech Forum 2019 was held in the capital, the third such event that, according to its own description, "brings together the main representatives in fintech, banking, insurance, regulation, cybersecurity, digital transformation and innovation in Peru.”[2] The event had the participation of a high authority in the Peruvian government, the vice minister of economy, who expressed his satisfaction with the growth of the fintech companies and his commitment to support a regulatory framework favourable to their interests.

However, despite what seems like universal enthusiasm with which this new business segment has been received, the understanding of the impact of fintech in Peru seems to be limited, even within the Peruvian financial industry. There are several elements that confirm this. Despite its exponential growth, the information available on fintech is scarce in the media, and when it is available, it is usually self-promotional, and basically limited to the websites where they offer products and services. In addition, many of the traditional financial actors still do not recognise fintech ventures as important agents of their ecosystem, which translates into a low level of strategic cooperation and “coopetition”.[3] Finally, studies conducted by universities on the subject are scarce and are focused almost exclusively on describing the construction and operation of the business models of these companies.[4]

In Peru, discussions on fintech tend to revolve around economics and financial regulation, but do not touch on other topics that are already widely discussed in more mature ecosystems, such as the ethics of the use of technology or its implications for human rights. In this scenario, trying to diagnose the use of AI in the Peruvian fintech sector to establish the existence of discrimination seems premature, and can be compared to talking about rocket science to a pre-industrial audience. However, given the rapid growth of the sector, it is at least necessary to frame the discussion for further debate and investigation.

Algorithmic bias and discrimination

Nowadays, it is an indisputable fact that technologies such as AI can generate or reproduce situations of discrimination against people as a product of a design contaminated by the prejudices and unfair biases of its creators. This "contamination" can be conscious or unconscious and occur in different stages. It includes definitions of concepts such as "beauty" or "creditworthiness" under which an algorithm will look for patterns that allow it to assign values to certain attributes such as age, sex, address, etc., that help the AI to make decisions.[5]

What is meant by discrimination in Peru?

Discrimination, in its broadest sense, is an act of differentiating between individuals or groups of people based on attributes of subjective evaluation whose result is the benefit of one individual or group (better valued) at the expense of another (less valued). As a social phenomenon, discrimination has been widely studied and its traces can be found throughout Peruvian history. In recent years, discrimination has been understood as a negative action that undermines people's rights to equality and that must therefore be discouraged through laws.

In the Peruvian constitution of 1993, currently in force, discrimination of any kind is prohibited.[6] Over the last 26 years, different laws have been developed in order to reinforce this prohibition, covering almost all areas of social life. For example, performing an act considered discriminatory constitutes a crime that can currently be punished by up to four years in prison. Likewise, this act may be subject to an administrative sanction in some jurisdictions, which may involve the imposition of economic penalties or even the closure of the establishment where the event occurred. Discrimination can also be subject to sanctions in the field of work, education and even at the level of consumer relations. In all these scenarios, judges and administrative entities have over the years made different rulings and decisions that have helped define the scope of the laws and codes of conduct. Some cases are very relevant to this report and will be presented later.

Why will the fintech sector make a good basis for further research?

In terms of technological ventures that have started up in Peru in the last five years, fintech companies are positioned in a particularly interesting place for three reasons. The first is their intensive use of all types of technologies, including AI tools. The second is that more than half of the local fintech ventures offer B2C (business-to-consumer) services, which means that they have a direct impact on the users of financial services. The third is the dominant narrative that affirms that these companies will contribute to the greater financial inclusion of vulnerable and historically excluded groups. These make fintech ventures an obvious choice if we want to research whether the use of AI generates or reproduces situations of discrimination against Peruvians.

Although they have in common that they all use technology, fintech ventures can be sub-categorised according to the type of services they offer. The most popular at the regional level are those that provide payment and remittance services, those that offer loans and those offering personal or corporate finance management. Fintech ventures that grant loans can serve as a useful reference for study because it is widely recognised that they use AI tools in their business models, compared to others focused on activities such as credit scores or online payments that do not necessarily involve the use of AI.[7] In these “lending fintechs”, AI usually interacts directly with customers through application forms, which allows it to collect, contrast and analyse the data received to make decisions. Depending on how the algorithms have been designed or how the AI ​has been trained, these decisions can exhibit acts of discrimination and, in certain cases, could be sanctioned.

Is it illegal (or even possible) for an algorithm to discriminate?

While discrimination of any kind is prohibited in Peru, there are cases in which a person can be "discriminated" against legally. For example, a restaurant that has a policy of not serving people who are in shorts and flip-flops may refuse to serve a customer that dresses in that way without discriminating against him or her. Another case, closer to the topic of this report, is a person who asks for bank loans and never bothers to pay them, which generates a negative credit record. If a new bank receives this person's loan application, they can reject it based on the negative record and not commit an act of discrimination even though the person has never been a client of the bank.

These two examples are almost common sense. The freedom of contract is a concept that determines that two parties can agree their own rules when contracting. In Peru, freedom of contract is a right recognised in the constitution. If a bank can establish a policy to not give loans to clients with a history of defaulting, it seems logical that an algorithm can perform the same discriminatory behaviour and not break any law.

But what is the limit? Below are two cases that, with 15 years of separation between each, seem to answer this question.

In 1999, a woman requested a credit card from Ripley (a popular department store), for which she delivered the required documentation consisting of her personal data, a copy of her identification document, and the credit card of another bank. Two weeks later, the woman received the news that the credit card request had been denied and when she asked why, she was told that the district in which she lived (La Victoria) was "not verifiable", which meant that it was difficult to carry out actions of verification and debt collection in La Victoria, which was labelled as “dangerous”. The woman decided to report Ripley for discrimination before the Peruvian consumer protection authority (INDECOPI), arguing that the store had only considered where she lived and not her credit record or income. In the first decision by INDECOPI, the complaint was dismissed. However, on appeal, the complaint was accepted based on an interpretation of discrimination currently in force in law.

The rationale of the case was that there are two forms of treatment in consumer relations: differentiated treatment and discriminatory treatment. Differentiated treatment is legal and is based on the existence of objective and justified causes that allow restricting and even denying the provision of services or products to a consumer. On the contrary, discriminatory treatment is based on purely subjective and arbitrary reasons, hence it is illegal. In this case, when it comes to credit, having a restrictive policy for customers that live in certain places can be an objective cause for differentiated treatment, but it should not be the only one. If it was the only one, it can become discriminatory, as has happened in this case.[8]

Fourteen years later, in 2013, a retired woman filed a lawsuit before the Constitutional Court against Banco de la Nacion (the state bank) for a similar reason. The bank had denied her a loan because she was 85 years old and its policy was to only grant loans to clients up to 83 years of age. The woman claimed that this was a violation of her right to equality and non-discrimination, while the bank argued that it was a case of differentiated treatment based on objective causes. To resolve this case, the court used an argument similar to that of INDECOPI. Based on international treaties and local legislation, it determined that age could not be the only requirement to deny access to credit and that doing so constituted discriminatory treatment. In addition, they pointed out that the vulnerable quality of the applicant, who was a senior citizen, had to be considered.[9]

So if an AI decides not to grant a loan to a client solely and exclusively because of his or her age, address, gender, etc. without considering other factors, is it committing an act of discrimination? Using the doctrine of the type of treatment (differentiated or discriminatory), the answer seems to be affirmative. Although in the cases cited the discriminatory behaviour was carried out by people, there is no reason why this doctrine could not be applied to machine-learning systems that have algorithms for decision making. However, while it is possible to recognise discrimination when interacting with a person, it is not so simple when it is the result of AI, because the way these tools work is not usually transparent or self-explanatory to those who interact with it.

Given that these systems collect and process huge amounts of data, identifying how and when a discriminatory act occurs seems an impossible task. For instance, a fintech service provider could have configured its algorithms to deny loans to all women, young people in rural areas or anyone with a surname of indigenous origin and it would be very difficult to prove that this has happened. Perhaps an indicative test could be the terms and conditions that appear on the websites of these companies and contain discriminatory clauses, which would help predict the discriminatory behaviour of the AI. At least two fintech companies that we looked at showed discrimination in terms of age, but it is likely that others also have built-in discriminations, without making these public.[10]

An important aspect that should also be part of this reasoning is how fintech ventures are different from other actors in the financial system in terms of their obligations to not discriminate. Could they argue that their need to increase restrictions on consumers is greater because they are more vulnerable and face greater risks? There is still plenty of room to debate.

Conclusion

It is clear that the fintech sector offers useful opportunities to understand the application of AI in Peru, and its potential impact on citizen rights. In particular, we suggest above that there is scope to explore potential bias and discrimination in the algorithms used by companies operating in the sector, including those that offer loans to vulnerable and disadvantaged groups. We have outlined the difficulties that are likely to be faced by such an analysis, including a lack of transparency in the algorithms used, and that there might be cause to allow a greater degree of differentiation in the sector given its vulnerability. At times this may be considered discriminatory. We have also highlighted, however, how legislation governing discrimination in the country, which is well defined, should be brought to bear on the business practices of fintech companies. Below we suggest some ways in which civil society can engage with the sector in order to strengthen its positive benefits for providing financial services to the vulnerable and marginalised in Peru.

Action steps

The following recommendations can be made for civil society organisations in Peru:

• Peruvian civil society organisations should begin to participate in the spaces related to the use and development of AI in order to acquire skills and capacities that allow them to form an opinion from the perspective of human rights.
• They must demand that ethical practices in the use of AI are incorporated into the development of business models in the local fintech ecosystem, especially in terms of transparency in the use of AI tools.
• Civil society organisations must demand that the Peruvian government adhere to international guidelines for the creation of public policies on AI that promote the development of the human being.
• They should encourage universities to conduct in-depth studies on the use of AI in the economic, social and regulatory fields, among others.
• Civil society organisations should support the financial inclusion of vulnerable populations through fintech and other technological ventures, but with respect for human rights, especially in terms of preventing discrimination.

Footnotes

[1] Inter-American Development Bank (IDB), IDB Invest, & Finnovista. (2018). Report on Fintech in Latin America 2018: Growth and Consolidation. https://publications.iadb.org/en/fintech-latin-america-2018-growth-and-consolidation

[2] https://limafintechforum.com

[3] ASBANC. (2017). Una mirada al fenómeno fintech en el Perú y el mundo. https://www.asbanc.com.pe/publicaciones/asbanc-semanal-242.pdf

[4] When the term "fintech" is used to search the National Repository of Research Papers (RENATI), there are only eight matches: renati.sunedu.gob.pe/simple-search?query=fintech

[5] Hao, K. (2019, 4 February). This is how AI bias really happens—and why it’s so hard to fix. MIT Technology Review. https://www.technologyreview.com/s/612876/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix

[6] Political Constitution of Peru, Article 2: Every person has the right: (...) To equality before the law. No person shall be discriminated against on the basis of origin, race, sex, language, religion, opinion, economic status, or any other distinguishing feature.

[7] Schueffel, P. (2016). Taming the Beast: A Scientific Definition of Fintech. Journal of Innovation Management, 4(4), 32-54. https://journals.fe.up.pt/index.php/IJMAI/article/view/2183-0606_004.004_0004/221

[8] See Final Resolution Nº 747-2000-CPC and Final Resolution Nº 0517-2001/TDC-INDECOPI for the case file Nº 307-1999-CPC. https://www.scribd.com/document/344636065/0517-2001-1-Cecilia-Reynosa-Contra-Ripley-y-Recaudadora-Discriminacion

[9] See the final ruling of the Constitutional Court for case file N° 05157 2014-PA/TC. http://tc.gob.pe/jurisprudencia/2017/05157-2014-AA.pdf

[10] The Fintech companies who carry out discrimination based on age are Fio (https://www.fio.pe) and Tappoyo (https://www.tappoyo.com). In the case of Fio, in the frequently asked questions section of its website, it is indicated that applicants must be between the ages of 21 and 65. In the case of Tappoyo, the restriction is between 20 and 70 years of age.

Notes:
This report was originally published as part of a larger compilation: “Global Information Society Watch 2019: Artificial intelligence: Human rights, social justice and development"
Creative Commons Attribution 4.0 International (CC BY 4.0) - Some rights reserved.
ISBN 978-92-95113-12-1
APC Serial: APC-201910-CIPP-R-EN-P-301
978-92-95113-13-8
ISBN APC Serial: APC-201910-CIPP-R-EN-DIGITAL-302