Profile picture

Co-founder @ RMOTR

Shared Twitter Friends

Last updated: November 8th, 20182018-11-08Project preview

Let's play with Python and the python-twitter library to figure out how many Twitter followers I share with my friends.

The analysis is summarized in this Notebook, but you have the full Flask app available to run it yourself if you fork this project.

RequirementsΒΆ

First things first, let's install the library. Alternatively, you can set the library as a custom requirement for this project in the settings πŸ˜‰

InΒ [1]:
!pip install python-twitter==3.4.2
Collecting python-twitter==3.4.2
  Downloading https://files.pythonhosted.org/packages/e6/2c/9fc6565b57ce6f3cc8e20b6c4bde8960dd0857629d41654bce46a6dd0bf9/python_twitter-3.4.2-py2.py3-none-any.whl (61kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 71kB 31.1MB/s 
Collecting requests (from python-twitter==3.4.2)
  Downloading https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl (60kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 61kB 31.2MB/s 
Collecting future (from python-twitter==3.4.2)
  Downloading https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz (829kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 829kB 1.7MB/s 
Collecting requests-oauthlib (from python-twitter==3.4.2)
  Downloading https://files.pythonhosted.org/packages/94/e7/c250d122992e1561690d9c0f7856dadb79d61fd4bdd0e598087dce607f6c/requests_oauthlib-1.0.0-py2.py3-none-any.whl
Collecting idna<2.8,>=2.5 (from requests->python-twitter==3.4.2)
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 61kB 39.0MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests->python-twitter==3.4.2)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 143kB 50.1MB/s 
Collecting urllib3<1.25,>=1.21.1 (from requests->python-twitter==3.4.2)
  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 122kB 51.3MB/s 
Collecting certifi>=2017.4.17 (from requests->python-twitter==3.4.2)
  Downloading https://files.pythonhosted.org/packages/56/9d/1d02dd80bc4cd955f98980f28c5ee2200e1209292d5f9e9cc8d030d18655/certifi-2018.10.15-py2.py3-none-any.whl (146kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 153kB 48.4MB/s 
Collecting oauthlib>=0.6.2 (from requests-oauthlib->python-twitter==3.4.2)
  Downloading https://files.pythonhosted.org/packages/e6/d1/ddd9cfea3e736399b97ded5c2dd62d1322adef4a72d816f1ed1049d6a179/oauthlib-2.1.0-py2.py3-none-any.whl (121kB)
    100% |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 122kB 39.0MB/s 
Building wheels for collected packages: future
  Running setup.py bdist_wheel for future ... done
  Stored in directory: /root/.cache/pip/wheels/0c/61/d2/d6b7317325828fbb39ee6ad559dbe4664d0896da4721bf379e
Successfully built future
Installing collected packages: idna, chardet, urllib3, certifi, requests, future, oauthlib, requests-oauthlib, python-twitter
Successfully installed certifi-2018.10.15 chardet-3.0.4 future-0.17.1 idna-2.7 oauthlib-2.1.0 python-twitter-3.4.2 requests-2.20.0 requests-oauthlib-1.0.0 urllib3-1.24.1
You are using pip version 18.0, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

The Twitter clientΒΆ

To run this experiment, you will need to have a Twitter App set up. If you still don't, visit this page and follow the instructions: https://apps.twitter.com/

Once you get your App credentials, you can now initialize the Twitter client like this:

NOTE: I'm reading credentials from env vars. You can do the same by defining custom env vars in the Project settings.

InΒ [2]:
import os
import twitter

api = twitter.Api(
    consumer_key=os.environ['TWITTER_CONSUMER_KEY'],
    consumer_secret=os.environ['TWITTER_CONSUMER_SECRET'],
    access_token_key=os.environ['TWITTER_ACCESS_TOKEN_KEY'],
    access_token_secret=os.environ['TWITTER_ACCESS_TOKEN_SECRET'],
    sleep_on_rate_limit=True  # we will talk about this later
)
InΒ [3]:
api.VerifyCredentials()
Out[3]:
User(ID=172480399, ScreenName=martinzugnoni)

We are good to go! πŸ‘ Our Twitter API client is set up. Now we can start requesting the Twitter API as we need.

AccountsΒΆ

Let's define the list of accounts we want to use. That means, the list of friends we want to analyze shared friends with.

I will list my friends below, but you probably want to replace the list with your own friends.

InΒ [5]:
ACCOUNTS = [
    'santiagobasulto',
    'martinzugnoni',
    'ivanzugnoni',
    'yosoymatias',
    'jperelli',
    'bruno_dimartino',

    # add more friends here.
]

Twitter API rate limits noteΒΆ

It's important to mention that, if you use accounts with a large amount of followers (>75.000), the app might take a long time to load.

The reason is Twitter API's rate limits. We're using the GetFollowerIDs API method, which returns pages of 5000 account ID's at max. As explained here, Twitter allows 15 API requests per 15 minutes. Meaning that, if all your listed accounts sum up to 75000 followers (15 requests, 5000 each), you will hit the API rate limit and the app will sleep for 15 minutes until the next request.

That's what the sleep_on_rate_limit option is used for. When you reach the Twitter API rate limit, the app will sleep for a while instead of raising an exception.

Shared friendsΒΆ

Alright! Without further introduction, let's compute the list of shared followers.

InΒ [7]:
from itertools import permutations, combinations

FOLLOWERS_CACHE = {}

# warm up followers cache.
# requesting Twitter followers (depending on the amount) might be very slow
# save them in a local cache to avoid successive requests of the same data.
for account in ACCOUNTS:
    if not account in FOLLOWERS_CACHE:
        FOLLOWERS_CACHE[account] = api.GetFollowerIDs(screen_name=account)

result = []
for i in range(1, len(ACCOUNTS) + 1):
    # compute combinations of all accounts listed above.
    # the result should look something like:
    #    ('santiagobasulto',)
    #    ('santiagobasulto', 'martinzugnoni')
    #    ('santiagobasulto', 'martinzugnoni', 'ivanzugnoni')
    #    ('santiagobasulto', 'martinzugnoni', 'ivanzugnoni', 'yosoymatias')
    #    ...
    for comb in combinations(ACCOUNTS, i):
        sets = [set(FOLLOWERS_CACHE[account]) for account in comb]
        # when the set contains more than one account, compute the followers
        # intersection. Otherwise count the amount of followers it has.
        if len(sets) > 1:
            intersec = sets[0].intersection(*sets[1:])
            result.append({'sets': list(comb), 'size': len(intersec)})
        else:
            result.append({'sets': list(comb), 'size': len(sets[0])})

Let's take a look at the result list, and try to understand what it means.

The list will look something similar to this:

[{'sets': ['santiagobasulto'], 'size': 542},
 {'sets': ['martinzugnoni'], 'size': 216},
 {'sets': ['ivanzugnoni'], 'size': 86},
 {'sets': ['yosoymatias'], 'size': 244},
 {'sets': ['jperelli'], 'size': 124},
 {'sets': ['bruno_dimartino'], 'size': 129},
 {'sets': ['santiagobasulto', 'martinzugnoni'], 'size': 65},
 {'sets': ['santiagobasulto', 'ivanzugnoni'], 'size': 9},
 {'sets': ['santiagobasulto', 'yosoymatias'], 'size': 9},
 ...

Each dict in the list will represent the amount of followers shared between accounts listed under the 'sets' key.

For example: {'sets': ['santiagobasulto', 'martinzugnoni'], 'size': 65}, means that santiagobasulto and I share 65 followers on Twitter.

Why this format? Simply because we want to use Venn.js library to plot our results.

Plot the resultΒΆ

As I've mentioned above, we will be using https://github.com/benfred/venn.js/ library to plot our results in the HTML template. The library is pretty straignforward to use, that's why I won't get into much details about how to do it.

For futher details you may want to take a look at the venn_friends.html template.

If you did everything right, your plot result should look similar to this:

twitter-friends

Running the Flask appΒΆ

Wanna run the app yourself? Just make sure the requirements.txt are installed, and execute this in the terminal:

python app.py

You should see the output saying

Running on http://0.0.0.0:5000/

That's all! πŸ‘ Now open the webview down below and see if the app is working πŸ˜‰

image

RMOTR Inc.
RMOTR Profile20060