Using Politikus for Name Matching PEPs
As more and more data on beneficial ownership is published or leaked, there is ever increasing data on available names, company addresses and so on.
But they're only useful, if you know what you're looking for, and usually what we're mostly interested in looking for are the business dealings of people in power and their associates.
In finance risky people are defined as Politically Exposed Persons or PEPs. Each country has more specific definitions, Malaysia's central bank Bank Negara defines it as the following:
PEPs
(a) foreign PEPs – individuals who are or who have been entrusted with prominent public functions by a foreign country.
For example, Heads of State or Government, senior politicians, senior government, judicial or military officials, senior executives of state-owned corporations and important political party officials;
(b) domestic PEPs – individuals who are or have been entrusted domestically with prominent public functions. For example, Heads of State or Government, senior politicians, senior government (includes federal, state and local government), judicial or military officials, senior executives of state-owned corporations and important political party officials; or
(c) persons who are or have been entrusted with a prominent function by an international organisation which refers to members of senior management.
For example, directors, deputy directors and members of the Board or equivalent functions.
The definition of PEPs is not intended to cover middle ranking or more junior individuals in the foregoing categories.
When you're looking for possible abuse of power, you need to also look at family members and also close associates. BNM defines these as:
Family Members
Refers to individuals who are related to a PEP either directly (consanguinity) or through marriage. A family member in this context, includes:
(a) parent;
(b) sibling;
(c) spouse;
(d) child; or
(e) spouse's parent,
for both biological or non-biological relationships.
Close Associates
Refers to any individual closely connected to a politically exposed person (PEP), either socially or professionally.
A close associate in this context includes:
(a) extended family members, such as relatives (biological and non-biological relationship);
(b) financially dependent individuals (e.g. persons salaried by the PEP such as drivers, bodyguards, secretaries);
(c) business partners or associates of the PEP;
(d) prominent members of the same organisation as the PEP;
(e) individuals working closely with the PEP (e.g. work colleagues); or
(f) close friends.
Getting Politikus Data
Sinar Project's Politikus project, continuously adds and updates data on PEPs, family members, associates as well as their relationships and publishes it as open data, via an API.
A simple example to download a CSV of all the latests PEPs
import pandas
import requests
from tqdm import tqdm,trange
HEADERS = {'Accept': 'application/json'}
BASE_URL = "https://politikus.sinarproject.org/@search?" \
"portal_type=Person&" \
"fullobjects=1"
# default batch is 25 items
BATCH = 25
total_items = requests.get(BASE_URL,
headers=HEADERS).json()['items_total']
peps = []
for i in tqdm(range(0, total_items, 25), desc="Download Progress"):
b_start = i+1
URL = BASE_URL + "&b_start=" + str(b_start)
r = requests.get(URL, headers=HEADERS)
for person in r.json()['items']:
pep = {}
pep['name'] = person['name']
pep['summary'] = person['summary']
pep['pepStatusDetails'] = person['pepStatusDetails']
pep['birth_date'] = person['birth_date']
if person['gender']:
pep['gender'] = person['gender']['title']
else:
pep['gender'] = None
#tax residencies
taxResidencies = []
if person['taxResidencies']:
for tax_country in person['taxResidencies']:
taxResidencies.append(tax_country['title'])
if taxResidencies:
pep['taxResidencies'] = ",".join(taxResidencies)
else:
pep['taxResidencies'] = None
nationalities = []
if person['nationalities']:
for nationality in person['nationalities']:
nationalities.append(nationality['title'])
if nationalities:
pep['nationalities'] = ",".join(taxResidencies)
else:
pep['nationalities'] = None
#ids
pep['UID'] = person['UID']
pep['rdf_id'] = person['@id']
peps.append(pep)
df = pandas.DataFrame(peps)
print("Exporting politikus-persons.csv in current directory")
df.to_csv('politikus-persons.csv')
This gets us enough contextual information after matching names, such as basic description of the person.
name summary rdf_id
0 Adly Kamarudin Datuk Malaysian businessman charged with abet... https://politikus.sinarproject.org/persons/adl...
1 Nasrudin Hassan Nasrudin Hassan is a politician https://politikus.sinarproject.org/persons/nas...
2 Shahrir Abdul Samad Shahrir bin Abdul Samad is a Malaysian politic... https://politikus.sinarproject.org/persons/sha...
3 Syed Mohd Fazmi Syed Mohd Fazmi Bin Sayid Mohammad is the aide... https://politikus.sinarproject.org/persons/sye...
4 Hafizuddin Ariff Mohd Hafizuddin Bin Mohd Ariff is the aide to ... https://politikus.sinarproject.org/persons/haf...
... ... ... ...
1549 Elizabeth Kay Nyigor Elizabeth Kay Nyigor is the aide to a politician https://politikus.sinarproject.org/persons/eli...
1550 Farahana Hussain Farahana binti Hussain is the aide to a politi... https://politikus.sinarproject.org/persons/far...
1551 Abdul Khaliq Ruzain Abdul Jalil Abdul Khaliq Ruzain Abdul Jalil is the aide to... https://politikus.sinarproject.org/persons/abd...
1552 Mohd Fadil Muskon Mohd Fadil Muskon is a Malaysian politician an... https://politikus.sinarproject.org/persons/moh...
1553 Noraini Roslan YDH. TPr. Hajah Noraini binti Haji Roslan is t... https://politikus.sinarproject.org/persons/nor...
ICIJ Offshore Leaks
Now that we have some names, we can use some of these beneficial ownership data to match names with. ICIJ Offshore Leaks is a good one, because each time a new PEP or associate is added, there is a possibility that there is match for some that was missed before.
You can download the database of past leaks from ICIJ website.
It's not difficult to quickly import all of it into pandas
files =[
'bahamas_leaks.nodes.entity.csv',
'bahamas_leaks.nodes.intermediary.csv',
'bahamas_leaks.nodes.officer.csv',
'offshore_leaks.nodes.entity.csv',
'offshore_leaks.nodes.intermediary.csv',
'offshore_leaks.nodes.officer.csv',
'panama_papers.nodes.entity.csv',
'panama_papers.nodes.intermediary.csv',
'panama_papers.nodes.officer.csv',
'paradise_papers.nodes.entity.csv',
'paradise_papers.nodes.intermediary.csv',
'paradise_papers.nodes.officer.csv']
icij_dfs = []
for f in files:
icij_dfs.append(pd.read_csv('/storage/datasets/offshore/' + f, usecols=['name','node_id']))
df_icij = pd.concat(icij_dfs)
This should be enough to get some names (1,531,731) some unique ID we can use later to refer to ICIJ database for other details.
A simple matching exercise
fuzzymatcher is useful for quick name matching and joining two Pandas dataframes.
Some basic cleaning up, to improve matching, you usually want to clean up the data a bit like make it all uppercase, trim etc.
df_icij['name'] = df_icij['name'].str.upper()
df_politikus['name'] = df_politikus['name'].str.upper()
Then match politikus names against ICIJ:
left_on = ['name']
right_on = ['name']
df_match = fuzzymatcher.fuzzy_left_join(df_politikus,df_icij,left_on, right_on)
Update: Note this matches only one ICIJ match, to one Politikus PEP or associate. Your probably want the other way around, icij on left, politikus on right, which try score every ICIJ name against Politikus names. This would need, an additional step of additional filtering for say a certain threshold, so you export only say anything over 0.65 and have a thousand or so rows of the best results, and throw away the 1.4 million or so rows of obviously different names. I'll cover this in upcoming note covering latest data with Pandora papers.
... get some coffee
Once done you can do analysis in Pandas, or export to CSV, maybe for others to then reuse it in whatever tools they're familiar with. Here I loaded it up into spreadsheet, and then sorted it by the best matching scores.You can download this CSV file here.
With some leads, you can now start searching and exploring the highest scoring matches in this case Raja Nong Chik on ICIJ, and Politikus data already provides you with some basic info and url for additional information.
These exercises help me build snippets of code, and explore methods. Instead of putting it all into one single DataFrame, I could explore doing it concurrently for each separate list of names using all CPU threads, refactor some parts into reusable scripts and so on. Politikus also has other data points such as addresses, identification numbers, faces and so on that could also be used as additional signals for scoring and matching.
Politikus also stores relationships, which provide other interesting methods of uncovering possible abuses of power, such as using network graph to identify cliques.