PeTi Blog

If you just need help to solve your problem you’re best off just reading the code-snippets and avoid the rest of this blog entry which merely describes my motivations behind the code.

So this started when I wanted to use the GitHub API to upload automated builds as releases yesterday. I haven’t used REST APIs much before so I read a little from their documentation and it seemed easy enough.

At first I thought I’d just use cmd or Powershell to do the deed. But the logic seemed a little too complex for cmd and I’d need something like curl. The effort to do it in Powershell seemed like a waste, since I might use this for UNIX systems later too. Since other parts of the build required Python anyway I decided to just do it in Python. I perused the web about the basics to use Python as a REST client. This excellent blog post was a helpful starting point: http://isbullsh.it/2012/06/Rest-api-in-python/ . Although it deals with Python 2 and I had Python 3 installed.

I really resented the suggestion that I’d have to add another dependency for this simple task and thought that Python had enough “batteries included” to just do it with the standard libs. So I researched a bit more about urllib2 (or urllib.request in Python 3) and it seemed to suffice for my use-case. It can’t send anything else than POST or GET requests. So if you need to use DELETE or other http methods you’ll likely need to use the http.client module.

So I clobbered together some basic script to dispatch GET and POST requests for the GitHub API for use with personal access tokens as authentication method (which you can generate here: https://github.com/settings/tokens for your account). This doesn’t work if you use two-factor authentication but rewriting the authentication part to work with OAUTH tokens instead shouldn’t be hard and work with two factor auth.

Anyway, long story short, here’s the snippet (sorry for the terrible formatting, you’ll be better off viewing it on GitHub):

import urllib.request
import base64
import sys
import os

def GitHubRequest(repository, credentials, url, data=None, datatype=None, useRawURL=False ):
    """ GitHubRequest(repository, credentials, url, data=None, datatype=None, useRawURL=False ) -> response, returncode
    
    This function is thoroughly unpythonic and you should probably just use it
    as a starting point.
    It dispatches a request to the GitHub API (written with v3 in mind).
    A GET request if data is not specified, a POST request with the string in
    data if datatype is not specified and a POST request with the contents of
    the filename in data if datatype is specified.
    
    Arguments:
        repository  - GitHub with in the format 'User/Repository'
        credentials - GitHub username and personal access token in the format
                      'username:accesstoken'
        url         - the GitHub API url, for example 'issues/3' or a complete
                      url if useRawURL is set to True
        data        - string specifying the data to be send, if datatype is
                      None the string is send, otherwise a file with the name
                      is opened and sent
        datatype    - the type of data to be, if it's a str it will be used as
                      MIME type for the POST request, if it't not a str or 
                      NoneType then the MIME type will be 'application/octet-stream'
        useRawURL   - specify whether the url is the full request URL (when
                      True) or just a partial url to append to 'apiurl/repo/'
    """
    if isinstance(credentials,str):
        credentials = bytes(credentials,'UTF-8')
    if useRawURL == True:
        requesturl = url
    else:
        requesturl = "https://api.github.com/repos/%s/%s" % (repository, url)
    
    print("request: %s" % requesturl)
    #GET request
    if data==None:
        req = urllib.request.Request(requesturl)
    else: #POST request
        if datatype==None:#JSON POST request
            req = urllib.request.Request(requesturl, data=bytes(data,'UTF-8'))
        else: #File POST request
            filehandle = open(data,'rb')
            req = urllib.request.Request(requesturl, data=filehandle)
            filesize = os.path.getsize(data)
            req.add_header("Content-Length", "%d" % filesize)
            if isinstance(datatype,str):
                req.add_header("Content-Type", datatype)
            else:
                req.add_header("Content-Type", "application/octet-stream")
    if credentials!=None:
        base64str = base64.b64encode(credentials)
        req.add_header("Authorization", "Basic %s" % base64str.decode("utf-8"))
    
    try:
        handle = urllib.request.urlopen(req)
    except IOError as e:
        code = -1
        if hasattr(e,'code'):
            code = e.code
        message = str()
        if hasattr(e,'fp'):
            message = e.fp.read()
        if 'filehandle' in locals():
            filehandle.close()
        return message, code
    response = handle.read()
    if 'filehandle' in locals():
        filehandle.close()
    return response, handle.getcode()

And here’s my main usage case (similar version better viewable at GitHub):

#!/usr/bin/env python

from GitHubRequest import GitHubRequest
import urllib.request
import base64
import sys
import os
import json

def main():
    credentials = bytes(sys.argv[1],'UTF-8') #in the format 'User:privateaccesstoken'
    repository  = "Bigpet/rpcs3-buildbot-tools"
    filename    = "some.zip"
    releasename = "sometag"
    commitish   = "3d2659fb20061d43a0057830fca30101c329e06a"
    
    #Check if the tag already exists
    response, code = GitHubRequest(repository,credentials,"releases/tags/%s"%releasename)
    print("code: %d"%code)
    if code == 200: #already a release with this tag there
        #expected
        code = 200 #do nothing
    elif code == 404: #no release with this tag yet
        #Create release
        requestdict = {'tag_name': releasename, 'prerelease': True}
        if commitish != None:
            requestdict['target_commitish'] = commitish
        response, code = GitHubRequest(repository,credentials,'releases',json.dumps(requestdict))
        if code != 201:
            print("got unexpected return code %d while creating a release: %s"%(code,response),file=sys.stderr)
            sys.exit(1)
    else:
        print("got unexpected return code %d while looking for release: %s"%(code,response),file=sys.stderr)
        sys.exit(1)
    
    #Get upload_url
    resdict = json.loads(response.decode('utf-8'))
    upload_url = resdict['upload_url']
    upload_url = upload_url.replace('{?name}',"?name=%s"%filename)
    
    assets = resdict['assets']
    for asset in assets:
        if asset["name"]==filename:
            print("File %s already exists in tag %s"%(filename,releasename),file=sys.stderr)
            sys.exit(1)
    
    #consider just using "application/octet-stream" for generic files
    response, code = GitHubRequest(repository,credentials,upload_url,filename,"application/zip",True)
    if code != 201:
        print("got unexpected return code %d while trying to upload asset to release: %s"%(code,response),file=sys.stderr)
        sys.exit(1)

if __name__ == "__main__":
    main()