UCL WIKI

UCL Logo
Will's Space

Blog

git is a distributed version control software package that enables very powerful management of software development projects. But what if you are developing an energy system model, whose inputs are large numbers of Excel spreadsheets?

Although Git works best with text files, Excel spreadsheets are not beyond its capabilities. The standard setup of git means that *.xlsx files are viewed as binary files.  Actually, xlsx files are a zipped archive of XML files, and so can be unzipped and compared relatively easily with the help of a script to parse the XML.

TIMES models consist of a folder containing a nested file structure of Excel spreadsheets. On model compilation, the spreadsheets are read using the VEDA front-end (a proprietary software package) into a locked Access database. But to perform effective version control of a TIMES model, it is necessary only to record the changes to the Excel spreadsheets themselves.

Below, I'll outline the best practice for working with git and Excel spreadsheet files.

But first, here some examples of what is possible. Here I'm using the TIMES_DEMO model. This TIMES model contains four main spreadsheets, one for each of four key sectors.

Here's a picture of one sheet in the VT_DEMOT_ELC_V5.xlsx workbook entitled COM_Attribs.

The python script produces text output of the worksheet in the following format: 

=================================
Sheet: COM_Attribs[ 13 , 6 ]
=================================
    A1: * values associate with commodities
    C3: ~COMEMI
    C4: CommName
    D4: OILDST
    E4: GASNAT
    F4: COAHAR
    C5: ELCCO2N
    D5: 70.0
    E5: 49.0
    F5: 94.0
    C7: <more emittants here>
    C9: ~COMAGG
    C10: CommName
    D10: ELCCO2N
    E10: CH4
    C11: GHG
    D11: 0.001
    E11: 0.21
    F11: <more commodities>
    C13: <more commodities>

Okay, so I'll now make a change and enter the value 72 in cell D5, replacing the existing value of 70. Then I saved the spreadsheet.

Then I type git diff at a command prompt. I obtain the following report.
Sure enough, the change has been picked up. The - sign indicates the removal of a value, and the {+} the addition of a line.
Usefully, the heading in the text file, which corresponds to the sheet name (COM_Attribs in this case) is retained, so that it is easy to identify in which sheet of the workbook a change has been made. 

$ git diff
diff --git a/VT_DEMOT_ELC_V5.xlsx b/VT_DEMOT_ELC_V5.xlsx
index 77ee5f0..8982118 100644
--- a/VT_DEMOT_ELC_V5.xlsx
+++ b/VT_DEMOT_ELC_V5.xlsx
@@ -356,7 +356,7 @@ Sheet: COM_Attribs[ 13 , 6 ]
     E4: GASNAT
     F4: COAHAR
     C5: ELCCO2N
-    D5: 70.0
+    D5: 72.0
     E5: 49.0
     F5: 94.0
     C7: <more emittants here>

Installation and setup

As TIMES works only on Windows systems, other operating system specific installs will not be covered, but the python script linked to later and some of the git setup routines are applicable to those using Unix and OS X systems too. Feel free to adapt these guidelines to your systems.

  1. If you're happy working at a command prompt (i.e. typing in text commands) download and install git for Windows. Otherwise, download and install github client for Windows and setup an account. The github website has a range of very useful help files to follow on how to setup git on your computer. These include entering a username and e-mail address and setting up password caching for signing in to the website.
  2. Next, we need to install Python. There are various distributions of Python out there, but for ease, I'd recommend using Enthought Canopy - which is a free (for academics) distribution of the Python programming language and a whole bunch of useful packages. Request an academic licence and then download and install.
  3. Now, download the Python script from my github repository online here and save the file as git_diff_xlsx.py in a memorable location on your drive.
  4. Add the following lines to the global .gitconfig file - this is normally in your user folder e.g. C:/Users/will/:
    [diff "zip"]
    binary = True
    textconv = python c:/path/to/git_diff_xlsx.py
    Note that in the final line above - your need to include the path to the git_diff_xlsx.py file you downloaded in the previous step.
  5. Add the following line to the repository's .gitattributes
    *.xlsx diff=zip

That's setup over and ready to go. Now, when you navigate to a git repository and type the git diff command, any changes to xlsx spreadsheet files will be recorded as human-readable differences.