World Bank Data with pandas_datareader

The World Bank publishes lots of data. They make data available for download as csv, xml, or xls files. Additionally, they make the data available through API calls, which is the civilized route. Tariq Khokhar wrote an excellent survey of libraries for accessing World Bank data in Python, R, Ruby, and Stata. We'll use one of the libraries discussed, pandas_datareader. This will be a quick look at per capita GDP trends for a few countries using pandas_datareader. In another blog post, we'll look at the same exercise using the csv files to show how much harder it is.

World Bank Data

Scanning the list of indicators, we see the link to "GDP per capita, PPP...", and the text for the link gives the indicator code we are looking for, "NY.GDP.PCAP.PP.CD". Additionally we can note from the chart that the populated data starts in 1990. Following the link, we get the option to download the series as csv, xml, or xls, and can download a file (now named API_NY.GDP.PCAP.PP.CD_DS2_en_csv_v2_9908727.zip, but it could be something else another day) to use in a later blog.

Installing pandas_datareader

A pandas_datareader can be installed using conda:

$ conda install -c anaconda pandas-datareader

Using pandas_datareader

The following can be downloaded as a notebook. The chart created in the notebook can be seen below.

world bank data with pandas_datareader¶

Downloaded ppp data csv. Workaround neded for version incompatability in datareader. See discussion</a href>.

In [33]:

# imports
import os
import pandas as pd
pd.core.common.is_list_like = pd.api.types.is_list_like # workaround
from pandas_datareader import wb as WB

# CONST
indCode = "NY.GDP.PCAP.PP.CD" # code for  per capita ppp gdp

Downloading with pandas_datareader¶

Arguments are

• indicator= Code for series to grab
• country= "all" or list of 2 byte country codes. Defailt is to "CA MX US".split(). We should prefer to filter in Python rather than in the pull.
• start= Desired start year. Better to filter in Python so set early.
• stop= Desired stop year. Better to filter in Python so set late.

In [50]:

# grab all data for 
pppDF = WB.download(indicator=indCode, country="all", start=1980, end=2020)
pppDF.head()

Out[50]:

		NY.GDP.PCAP.PP.CD
country	year
Arab World	2017	NaN
	2016	16726.722185
	2015	16302.363760
	2014	15934.202070
	2013	15548.200905

Reshaping the data¶

• Removing rows with null entries.
• Reseting indexes because you never want indexes.
• Giving a normal whitespace free name to the series.
• Sorting just to make table display read better.
• Making numeric copy of the string year.

In [51]:

pppDF = pppDF.dropna().reset_index()
pppDF.columns = "country year GDPpc".split()
pppDF.sort_values("year country".split(), inplace=True)
pppDF["yearN"] = pppDF.year.apply(pd.to_numeric)
pppDF.head()

Out[51]:

	country	year	GDPpc	yearN
1263	Albania	1990	2722.280344	1990
1290	Algeria	1990	6616.408352	1990
1317	Angola	1990	2840.200763	1990
1344	Antigua and Barbuda	1990	10587.593409	1990
26	Arab World	1990	6759.785391	1990

Filtering for top African economies¶

The spelling of Egypt is a little unfortunate. Here we need exact matches.

In [52]:

# top Africa economies
aL = "Nigeria|South Africa|Egypt, Arab Rep.|Morocco|Ethiopia".split("|")
africaDF = pppDF[pppDF.country.isin(aL)]
africaDF.head()

Out[52]:

	country	year	GDPpc	yearN
2571	Egypt, Arab Rep.	1990	3819.286370	1990
2694	Ethiopia	1990	421.378824	1990
4261	Morocco	1990	2528.458556	1990
4514	Nigeria	1990	1965.827996	1990
5256	South Africa	1990	6267.091465	1990

plotting¶

In [54]:

# the next line is needed to diaplay the plot in the notebook
%matplotlib inline  
import matplotlib.pyplot as plt
fig, ax = plt.subplots() 
africaDF.groupby("country").plot(x="yearN", y="GDPpc", ax=ax)
ax.legend("Egypt|Ethiopia|Morocco|Nigeria|South Africa".split("|"))
fig.savefig("africaGDPpc.png")

Simple Equations in Sympy

Simple Equation Solution with sympy¶

Thanks to sympy, even if you forgot highschool math, you can still do highschool math. Sympy is part of the basic Anaconda distribution, so there is no prep work needed to try it out in an Anaconda environment. Below, a simple quadratic equation with integer solutions, solved with sympy. A notebook can be downloaded for the code below.

In [5]:

# setup
from sympy.solvers import solve
from sympy import Symbol
# prepare to work with single unknown
x = Symbol('x')
type(x)

Out[5]:

sympy.core.symbol.Symbol

So that we can be sure the sympy solution is correct, we'll start with something where we know the answer. This quadratic should have solutions at -3 and 2.

(x + 3) * (x - 2) = 0

x^2 + x - 6 = 0

The solve function from sympy.solver looks for 2 arguments, the equation to solve and the symbol to solve for.

In [7]:

solve (x**2 + x - 6, x)

Out[7]:

[-3, 2]

Flowcharts in Graphviz

Flowcharts are great but WYSIWYG flowcharting software is horrible. Fortunately, there us Graphviz. This free tool allows for the creation of network diagrams, including flowcharts, using The Dot Language. We'll use the pydot python library, though there are other choices. Start by installing graphviz. Use conda to install pydot by typing

> conda install pydot

into the shell (Linux/OSX) or the Anaconda shell (Windows). Then you are ready to create nodes, connect them with edges, and build a typical flowchart.

A notebook is available with the code below. Download and open in jupyter.

# imports
import os
import pydot
from IPython.display import Image, display
# setup
ng = pydot.Dot(graph_type='digraph')
# nodes
startN = pydot.Node("start", style="filled", fillcolor="yellow")
stopN = pydot.Node("stop", style="filled", fillcolor="red")
# modifiers
makeChartN = pydot.Node("make\nchart", shape="box")
dullChartN = pydot.Node("remove\ndetail", shape="box")
noiseChartN = pydot.Node("add\nnoise", shape="box")
# tests
infoCapN = pydot.Node("useful items\n >? N + random", shape="diamond")
noiseFloorN = pydot.Node("distractions\n <? N + random", shape="diamond")
# collection
nodeL = [startN, makeChartN, dullChartN, noiseChartN, infoCapN, noiseFloorN, stopN]
# add nodes to chart
for nodeN in nodeL:
    ng.add_node(nodeN)
# add edges
ng.add_edge(pydot.Edge(startN, makeChartN))
ng.add_edge(pydot.Edge(makeChartN, infoCapN))
 
ng.add_edge(pydot.Edge(infoCapN, dullChartN, label="Y"))
ng.add_edge(pydot.Edge(infoCapN, noiseFloorN, label="N"))
 
ng.add_edge(pydot.Edge(dullChartN, noiseFloorN))
 
ng.add_edge(pydot.Edge(noiseFloorN, noiseChartN, label="Y"))
ng.add_edge(pydot.Edge(noiseFloorN, stopN, label="N"))
 
ng.add_edge(pydot.Edge(noiseChartN, infoCapN))
# display
ng.write_png("typicalFlowchart.png")
Image("typicalFlowchart.png")

It's Past Time to Switch to Python 3

For a long time, it made sense to stick with Python 2 over Python 3 because important libraries worked best for Python 2. Then even after libraries worked correctly in Python 3, a user who had a large collection of libraries installed might not want to start all over with Python 3. Anaconda has changed all that with a Python distribution that includes all of the fundamental data science libraries, a package manager that assists in keeping packages up to date, and other goodies. So all the examples here will be Python 3 from now on.

SQL on pandas Tables with sqlite3

It can be extremely convenient to use SQL with pandas dataframes. Maybe you are working with pandas dataframes and you want to implement logic that you are borrowing from another environment. Or you could be trying to demonstrate something about SQL, and want a database populated with data that can work on your audience desktops. The sqlite3 module makes this simple. The writeup on python.org is very good but more examples are always better. Here we'll populate some dataframes with wikiHelp, load them into an sqlite3 database, and join them using sql. Of course, we can also join dataframes in pandas, so we will compare the results of of the joins.

Loading dataframes

Here we populate dataframes with extracts from Wikipedia List of 100 wealthiest countries and List of countries by continent.

from wikiHelp import WIKI, getWtable

# CONST
WEALTH = "List_of_countries_by_wealth_per_adult"
CONTINENTS = """List of sovereign states
and dependent territories by continent""" # make narrow for show
CONTINENT = "_".join(CONTINENTS.split()) # string for wiki

# get 1th table from wealth per adult
wealthDF = getWtable("%s%s" %(WIKI, WEALTH), tabNum=1) 
print (wealthDF.iloc[0])
# get 3th continent table
noAmDF = getWtable("%s%s" %(WIKI, CONTINENT), tabNum=3)
print (noAmDF.iloc[0])

The first record from each dataframe is below.

wealthDF

• country World
• totalWealth 280,289
• wealthPerAdult 56,541
• medianWealthPerAdult 3,582
• Name: 0, dtype: object

noAmDF

• flag
• name Anguilla
• capital The Valley
• status Overseas territory of the United Kingdom
• Name: 0, dtype: object

Loading dataframes into sqlite3 database Below, we instantiate an sqlite3 connection, and load our dataframes into it using the pandas to_sql method. The method requires a string table name and a connection, and we supply also an index=False. The database is created in RAM just so that there is no file to clean up, but the code to create instead on disk is shown.

import sqlite3
conn = sqlite3.connect(':memory:') # create connection in ram
# conn = sqlite3.connect('example.db') # create conn on disk
wealthDF.to_sql("wealth", conn, index=False) # load wealthDF
noAmDF.to_sql("northAmerica", conn, index=False) # load noAmDF

We can now access the northAmerica and wealth tables in the database with sql, or use pandas to access the same data he noAmDF and wealthDF. We will use the pandas read_sql method, which we will supply with a string of sql and a connection. For some sql that's quick and easy to check, we will simply count the records in the tables with "select count(*) from tablename ", which we can compare to the len of the dataframe.

qS = 'select count(*) as n from %s' # query string
for tableName, df in zip("wealth northAmerica".split() \
                                 , (wealthDF, noAmDF)):
    tableLen = pd.read_sql(qS %(tableName), conn).iloc[0].n
    print ("%s\t%d\t%d" %(tableName, tableLen, len(df)))

• wealth 107 107
• northAmerica 45 45

Which shows matching lengths. The simplicity of the read_sql call is really amazing, so we'll look at it again with a join. Let's look at the mean wealth and capital only for countries in North America. The sql would be

select wealth.country, wealth.wealthPerAdult
    , continent.capital
from northAmerica continent
inner join wealth wealth
    on wealth.country = continent.name

In Python, we send that sql through the read_sql and view the resulting dataframe.

qS = """select wealth.country, wealth.wealthPerAdult
    , continent.capital
from northAmerica continent
inner join wealth wealth
    on wealth.country = continent.name """
joinDF = pd.read_sql(qS, conn) # join 2 tables in sql
print(joinDF.to_string(index=False))

The Python to perform the same join on the original dataframes is below, were we also check that the resulting dataframe is identical to the one from above. The output shows the same entried in a different order because the database is unordered like a Python dictionary and we made no effort to impose an order on the output.

keepL = "country wealthPerAdult capital".split() # ordered field list 
mergeDF = wealthDF.merge(noAmDF, left_on="country", right_on="name" \
                         , how="inner")[keepL] # merge 2 dataframe
print(mergeDF.to_string(index=False))

Reading Wikipedia Tables

This module, wikiHelp.py, is a small group of functions for loading a table from a Wikipedia page into a Pandas dataframe. The motivation is to have a tool that provides consistent results loading small dataframes for practice exercises. Once the module is imported, we can easily read in the Nth table from a Wikipedia page as a dataframe simply as below:

from wikiHelp import WIKI, getWtable
CONTINENT = "_".join(["List_of_sovereign_states" \
    , "and_dependent_territories_by_continent"])
africaDF = getWtable("%s%s" %(WIKI, CONTINENT), tabNum=0)
print africaDF.shape