Tag: Python

Simple Hollow Knight Save Game Editor – HollowEdit.com

I made a simple Hollow Knight save game editor website at https://hollowedit.com. This is a small postmortem on that project.

https://hollowedit.com/

Initial Inspiration

Sometime near the start of this year I had a discussion with a friend and we decided that we wanted to do a small side-project together.

We decided to create a Hollow Knight save game editor website. The project came to mind because:

  • Hollow Knight’s a cool game
  • the technologically hard problem was already solved (there were existing save game editors)
  • existing editors either require a desktop application or manually editing some json

This meant to us that there was a niche that could be filled by a simple web-based editor that doesn’t require users to be very technical. All that would be required is using the systems file-manager and basic web-browser interaction.

Tech Stack consideration

My friend was mostly focusing on python and had used flask previously. I’m familiar enough with python from writing mostly small utilities and using it as a “portable shell script”. So that’s what we used for the backend.

For the frontend we looked around what was trending around that time. React, Vue and Svelte were the contenders. I thought React was too un-intuitive for such a small project, because when previously working with it I sometimes felt a bit lost about the control-flow. Vue and Svelte both seemed like a fine choice. I was a bit more familiar with Vue and it seemed to have better TypeScript support (and as a die-hard static typing fan I always try to use TS when JS is required). So we chose Vue.

Do we even need a backend? Not really, but we wanted to get coding as quickly as possible and it seemed easier solve the decryption/encryption done in a backend.

That was about as sophisticated as our research was before getting started.

Implementation Learnings

Most of the implementation went fairly smooth. We had the site running locally in dev mode within the weekend we started. Some polishing and the deployment setup took a lot longer and was stretched over a handful of sessions afterwards.

Encryption
I ported the symmetric encryption/decryption already reversed by KayDeeTee/Hollow-Knight-SaveManager in Java to python. Which initially worked, but I had to fix a bug in my port later on in the padding. I fixed it by examining the Hollow Knight de-compilation (dnSpy is really neat when dealing with Unity games).
So I guess some testing with more varied input would’ve been in order if this was for a more serious project.

The Vue Options/Composition APIs
While we did get something running in the first weekend we were very unfortunate with the timing.
Unbeknownst to us the new Vue3 documentation wasn’t out yet. It would be released a week later and includes a very handy little switch to view all examples in either the Options API or the Composition API styles. Not having used the vue composition API before but seeing examples of it mixed in with the documentation led to quite some confusion. Some very deeply confused code resulted that mixed the two in unfortunate ways.
Some weeks after the documentation was releases I ported the handful of Components we had to be very clearly in the Composition API style and it cleaned up a lot of the very ad-hoc bolted together code.
This really underscored the importance of robust documentation for me once again. It also highlighted the value of using technology you’re already familiar with at a conceptual level when trying to get something done quickly. Learning Vue 3 enough to directly apply it to a project without taking time to understand the underlying concepts turned out to be a fool’s errant.

Deployment can look very different than dev
When preparing to publish the site somewhere I knew how to deploy the Vue frontend, but did not previously publish anything flask-based. In development we just used flask run for quick testing. But for deployment it’s very understandably not recommended. So I looked at the official documentation which assumes the code is in a python package. This was not the case yet, since we just used a collection of local python Modules up to this point.
I thought “this should be easy to add, it’s already running with the dev server”. Never having created a python package before I headed to the setuptools documentation. I followed their guide and created a simple pyproject.toml. But then I kept running into import errors which I could not explain to myself. After hours of tinkering and trying to find out what I had done differently now I found out that I didn’t know much about how python handles imports in detail. It turns out when running code inside a package the import statement behaves a little differently.
The proper thing to do would be to just always treat the code as if it’s running in a package and use editable packages. But I was somewhat annoyed by having wasted so much time on diagnosing the issue. So as a punishment to the pythonic gods I instead opted to do this ugly hack to import the same module two different ways depending on whether it’s imported from within the package or not.


# try both file-local and package local imports
# THIS IS A HACK, don't do this in actual code
try:
  from SaveEncoder import SaveEncoder
except:
  from .SaveEncoder import SaveEncoder

The final deployment setup then was made way too complicated. Having learned my lesson by now to just use what I was at least somewhat familiar with I opted to just build two separate Docker images and glue them together in a docker-compose config. But enough text for now, time for you to look at an unnecessary an ugly graph:

None of this needed to be this complicated

When I saw there was a sale on .com domains I snapped up hollowedit.com , setup the Let’s Encrypt certificate for the domain et voil`a the site was online.

Closing thoughts

This was a fun little project and I’m glad I did it. I learned a lot about Vue specifically and it’s a neat tool to have in my back pocket.

I don’t usually do websites for fun. I’m more drawn to native development in my free time. But this was a great change up and being able to simply pass around a link is always a plus of working with web-tech.

In addition to it being a good learning experience I hope it can be useful to a couple of people.

GitHub API basic usage with Python 3

If you just need help to solve your problem you’re best off just reading the code-snippets and avoid the rest of this blog entry which merely describes my motivations behind the code.

So this started when I wanted to use the GitHub API to upload automated builds as releases yesterday. I haven’t used REST APIs much before so I read a little from their documentation and it seemed easy enough.

At first I thought I’d just use cmd or Powershell to do the deed. But the logic seemed a little too complex for cmd and I’d need something like curl. The effort to do it in Powershell seemed like a waste, since I might use this for UNIX systems later too. Since other parts of the build required Python anyway I decided to just do it in Python. I perused the web about the basics to use Python as a REST client. This excellent blog post was a helpful starting point: http://isbullsh.it/2012/06/Rest-api-in-python/ . Although it deals with Python 2 and I had Python 3 installed.

I really resented the suggestion that I’d have to add another dependency for this simple task and thought that Python had enough “batteries included” to just do it with the standard libs. So I researched a bit more about urllib2 (or urllib.request in Python 3) and it seemed to suffice for my use-case. It can’t send anything else than POST or GET requests. So if you need to use DELETE or other http methods you’ll likely need to use the http.client module.

So I clobbered together some basic script to dispatch GET and POST requests for the GitHub API for use with personal access tokens as authentication method (which you can generate here: https://github.com/settings/tokens for your account). This doesn’t work if you use two-factor authentication but rewriting the authentication part to work with OAUTH tokens instead shouldn’t be hard and work with two factor auth.

Anyway, long story short, here’s the snippet (sorry for the terrible formatting, you’ll be better off viewing it on GitHub):

import urllib.request
import base64
import sys
import os

def GitHubRequest(repository, credentials, url, data=None, datatype=None, useRawURL=False ):
    """ GitHubRequest(repository, credentials, url, data=None, datatype=None, useRawURL=False ) -> response, returncode
    
    This function is thoroughly unpythonic and you should probably just use it
    as a starting point.
    It dispatches a request to the GitHub API (written with v3 in mind).
    A GET request if data is not specified, a POST request with the string in
    data if datatype is not specified and a POST request with the contents of
    the filename in data if datatype is specified.
    
    Arguments:
        repository  - GitHub with in the format 'User/Repository'
        credentials - GitHub username and personal access token in the format
                      'username:accesstoken'
        url         - the GitHub API url, for example 'issues/3' or a complete
                      url if useRawURL is set to True
        data        - string specifying the data to be send, if datatype is
                      None the string is send, otherwise a file with the name
                      is opened and sent
        datatype    - the type of data to be, if it's a str it will be used as
                      MIME type for the POST request, if it't not a str or 
                      NoneType then the MIME type will be 'application/octet-stream'
        useRawURL   - specify whether the url is the full request URL (when
                      True) or just a partial url to append to 'apiurl/repo/'
    """
    if isinstance(credentials,str):
        credentials = bytes(credentials,'UTF-8')
    if useRawURL == True:
        requesturl = url
    else:
        requesturl = "https://api.github.com/repos/%s/%s" % (repository, url)
    
    print("request: %s" % requesturl)
    #GET request
    if data==None:
        req = urllib.request.Request(requesturl)
    else: #POST request
        if datatype==None:#JSON POST request
            req = urllib.request.Request(requesturl, data=bytes(data,'UTF-8'))
        else: #File POST request
            filehandle = open(data,'rb')
            req = urllib.request.Request(requesturl, data=filehandle)
            filesize = os.path.getsize(data)
            req.add_header("Content-Length", "%d" % filesize)
            if isinstance(datatype,str):
                req.add_header("Content-Type", datatype)
            else:
                req.add_header("Content-Type", "application/octet-stream")
    if credentials!=None:
        base64str = base64.b64encode(credentials)
        req.add_header("Authorization", "Basic %s" % base64str.decode("utf-8"))
    
    try:
        handle = urllib.request.urlopen(req)
    except IOError as e:
        code = -1
        if hasattr(e,'code'):
            code = e.code
        message = str()
        if hasattr(e,'fp'):
            message = e.fp.read()
        if 'filehandle' in locals():
            filehandle.close()
        return message, code
    response = handle.read()
    if 'filehandle' in locals():
        filehandle.close()
    return response, handle.getcode()

And here’s my main usage case (similar version better viewable at GitHub):

#!/usr/bin/env python

from GitHubRequest import GitHubRequest
import urllib.request
import base64
import sys
import os
import json

def main():
    credentials = bytes(sys.argv[1],'UTF-8') #in the format 'User:privateaccesstoken'
    repository  = "Bigpet/rpcs3-buildbot-tools"
    filename    = "some.zip"
    releasename = "sometag"
    commitish   = "3d2659fb20061d43a0057830fca30101c329e06a"
    
    #Check if the tag already exists
    response, code = GitHubRequest(repository,credentials,"releases/tags/%s"%releasename)
    print("code: %d"%code)
    if code == 200: #already a release with this tag there
        #expected
        code = 200 #do nothing
    elif code == 404: #no release with this tag yet
        #Create release
        requestdict = {'tag_name': releasename, 'prerelease': True}
        if commitish != None:
            requestdict['target_commitish'] = commitish
        response, code = GitHubRequest(repository,credentials,'releases',json.dumps(requestdict))
        if code != 201:
            print("got unexpected return code %d while creating a release: %s"%(code,response),file=sys.stderr)
            sys.exit(1)
    else:
        print("got unexpected return code %d while looking for release: %s"%(code,response),file=sys.stderr)
        sys.exit(1)
    
    #Get upload_url
    resdict = json.loads(response.decode('utf-8'))
    upload_url = resdict['upload_url']
    upload_url = upload_url.replace('{?name}',"?name=%s"%filename)
    
    assets = resdict['assets']
    for asset in assets:
        if asset["name"]==filename:
            print("File %s already exists in tag %s"%(filename,releasename),file=sys.stderr)
            sys.exit(1)
    
    #consider just using "application/octet-stream" for generic files
    response, code = GitHubRequest(repository,credentials,upload_url,filename,"application/zip",True)
    if code != 201:
        print("got unexpected return code %d while trying to upload asset to release: %s"%(code,response),file=sys.stderr)
        sys.exit(1)

if __name__ == "__main__":
    main()