(Note to web-viewers - press space to advance a slide. Hit 'S' to see speaker notes)

VOEventDB

and

Sustainable Software

Tim Staley / 4pisky.org

Hotwiring the Transient Universe V

Villanova, PA, Oct 2016

4 Pi Sky: detection of radio-transients, and co-ordination of their follow-up.

Before I get into the main talk, I just wanted to mention for any of you interested in working with transient alerts in real-time - we currently publish a combined feed of GCN, ASASSN and GAIA transient alerts, in VOEvent format - just browse to 4pisky.org and click 'voevents' at the top for more information.

For this talk I was originally just going to present our recently published archiving and query tool VOEventDB, but since I'm in an extended slot, I thought I should try to talk on a slightly broader theme.

So, instead I thought I'd try to use VOEventDB as a sort of window onto the much bigger topic of sustainable software.

Sustainable software?

So: What the hell is sustainable software? It's a little bit buzzwordy - and incidentally there are now entire conferences devoted to software sustainability, so clearly I'll only be scratching the surface - but to me it basically means making software reusable.

Sustainable software?

• can be used by others
• can be reused outside original context
• can be modified by other devs
• is robust to changing dependencies

NB: kind of a Platonic ideal.

So here's my definition.

Sustainable software:

• can be used by someone other than the dev that wrote it (Without the dev sat looking over their shoulder). This includes installation!
• can be reused outside of its original context (perhaps with minor modification)
• can be modified by someone other than the dev that wrote it
• is robust to changing dependencies - i.e. it won't break because some other library releases a new 'latest-and-greatest' version with an altered interface.

It's a set of goals to bear in mind - no project is perfect, and there's a cost/reward trade-off - there's little point writing extensive documentation for your 10 line shell script.

Why is this relevant?

• There have always been large multi-person / long-term software projects in astronomy.
• (I think) we're seeing an explosion in the number of 'smaller' public codes (cf ASCL)

• This is a really good thing, but comes with difficulties of success
• The easier it is to evaluate, re-use, modify and recycle these codes, the better.

• There have always been a few large-multi person software projects in astronomy - these produced documented, more-or-less usable software because it was expected from them.
• (I think) we're seeing an explosion in the number of 'small' codes, (previously, projects that would have been written by a PhD student and then disappeared?)
• This is a really good thing -- we're getting better at sharing code and replicating our science -- but comes with difficulties of success - hard to know what's out there, and whether it's any good.
• The easier it is to evaluate, re-use, modify and recycle these codes, the better.

What I'll try to cover

• VOEventDB - what it's for, what it does
• A few items on the 'sustainable software checklist'
• (Python) Tooling to make your life easier

In this talk, I'll talk a bit about

• VOEventDB - what it's for, what it does.
• Along the way I'll point out some of the ways I've tried to make it suitable for re-use
• and some tools that I found helpful.

... to try to persuade you that opening up your code can be done with minimal time investment. The tooling will be specific to Python. The vague moralising applies to any programming language of choice.

VOEventDB, in brief

Context

• VOEvent is a standardised format for astronomical transient alerts.
• NASA-GCN have been transmitting alerts in this format for over 2 years.

• Previously, there was no public archive for alerts in this format.

• VOEvent standard has always referred to a 'registry' of 'repositories' - clear gap to fill.

This a problem!

• Difficult to plan a follow-up strategy if you can't back-test it
• No way to know what's out there, what the rates are like
• Impossible to check for missed alerts

... also helps with converting web-pages into VOEvent feeds.

This a problem!

• Difficult to plan a follow-up strategy if you can't back-test it with historical data to see what you would have observed previously.
• No way to know what's out there, what the rates are like
• Impossible to check for missed alerts - if your system goes down for half an hour, you want some way to catch up, right?

And anyway, it solved this problem I had with scraping web-pages to generate VOEvent alerts (you need some way to check if you already alerted the world about a given entry on a web-page).

VOEventDB: Spec

• Store raw VOEvent XML, provide XML content at a persistent URL
• Store a common subset of VOEvent metadata in regular database schema
• Make queries based on this common subset
• Including spatial (cone-search) and citation-based queries
• 'RESTful' web-API
• Python client-library for remote-queries

• Store raw XML - same data as you'd receive in real-time
• The VOEvent Schema has some flexibility, but there's a core subset we expect to see in all packets - use that for filtering and searching.

Reusable, decentralized

• Agnostic about inputs and outputs
• Easy for any team to set up their own local repository

• So the idea is that this is a component you can use as part of your larger alerts-handling pipeline.
• It would be neat to see wider adoption of VOEvent as a universal format (it may yet happen).
• This is not so much about VOEvent - it's fine, but maybe it will get replaced, whatever - my point is that it seems foolish for multiple different teams to write the same boring, nuts-and-bolts code independently, rather than re-use a shared codebase.

Schema

• Schema is very simple: Author, date of authoring, whether it's an observation or a test-packet, when it was received, etc.
• Note, there's really nothing specific to the VOEvent standard about this - VOEvent is the data-interchange format, but the database code could be quite easily modified to work with another data-format.

Implementation

• Postgres + SQLAlchemy
• Spatial queries powered by qc3 Postgres extension.
• Flask-powered RESTful interface
• Partially-autogenerated documentation.
• Extensive test-suite using pytest fixtures.

http://voeventdb.4pisky.org/

• The web-interface is very bare-bones - designed to be just usable enough for developers to test it manually, but it's not user friendly.
• I'm expecting that casual users will use the Python client-library, so let me give you a quick teaser demo of that

Getting started: Client installation

!pip install voeventdb.remote --quiet 

So to get the client-library up and running, you just use the standard python package installer to install this 'voeventdb.remote' package, which is a one line command...

import voeventdb.remote.apiv1 as api
api.count()

1271241

And now you can import the api, fetch the number of packets in the remote database

api.map_stream_count()

{u'com.dc3/dc3.broker': 22570,
 u'nasa.gsfc.gcn/AGILE': 6174,
 u'nasa.gsfc.gcn/AMON': 4,
 u'nasa.gsfc.gcn/CALET': 79,
 u'nasa.gsfc.gcn/CAlet': 1,
 u'nasa.gsfc.gcn/COUNTERPART': 113,
 u'nasa.gsfc.gcn/Fermi': 41556,
 u'nasa.gsfc.gcn/GRO': 6569,
 u'nasa.gsfc.gcn/HETE': 6014,
 u'nasa.gsfc.gcn/INTEGRAL': 33120,
 u'nasa.gsfc.gcn/IPN': 486,
 u'nasa.gsfc.gcn/KONUS': 449,
 u'nasa.gsfc.gcn/MAXI': 6369,
 u'nasa.gsfc.gcn/MOA': 1553,
 u'nasa.gsfc.gcn/SNEWS': 44,
 u'nasa.gsfc.gcn/SUZAKU': 17,
 u'nasa.gsfc.gcn/SWIFT': 1117763,
 u'nasa.gsfc.gcn/UNRECOGNIZED_TYPE': 2,
 u'nvo.caltech/voeventnet/catot': 66,
 u'nvo.caltech/voeventnet/mlsot': 147,
 u'svomcgft.naoc/VOEVENTTEST': 3091,
 u'voevent.4pisky.org/ALARRM-OBSTEST': 5780,
 u'voevent.4pisky.org/ALARRM-REQUEST': 42,
 u'voevent.4pisky.org/ASASSN': 1747,
 u'voevent.4pisky.org/GAIA': 1272,
 u'voevent.4pisky.org/TEST': 10,
 u'voevent.4pisky.org/TEST-RESPONSE': 14,
 u'voevent.4pisky.org/TEST-TRIGGER': 14,
 u'voevent.4pisky.org/voevent-broadcast': 7761,
 u'voevent.4pisky.org/voevent-receive': 7724,
 u'voevent.phys.soton.ac.uk/AMI-REQUEST': 84}

Or fetch this mapping showing you the number of packets in each stream, and so on. Except that, actually, a bunch of these streams are just composed of test-packets which verify the network is up and running, so let's filter those out.

filters={api.FilterKeys.role:'observation'}
api.map_stream_count(filters)

{u'nasa.gsfc.gcn/AMON': 4,
 u'nasa.gsfc.gcn/CALET': 79,
 u'nasa.gsfc.gcn/CAlet': 1,
 u'nasa.gsfc.gcn/COUNTERPART': 113,
 u'nasa.gsfc.gcn/Fermi': 8072,
 u'nasa.gsfc.gcn/INTEGRAL': 1111,
 u'nasa.gsfc.gcn/IPN': 486,
 u'nasa.gsfc.gcn/KONUS': 449,
 u'nasa.gsfc.gcn/MAXI': 269,
 u'nasa.gsfc.gcn/MOA': 1553,
 u'nasa.gsfc.gcn/SUZAKU': 17,
 u'nasa.gsfc.gcn/SWIFT': 1042121,
 u'nvo.caltech/voeventnet/catot': 66,
 u'nvo.caltech/voeventnet/mlsot': 147,
 u'voevent.4pisky.org/ASASSN': 1747,
 u'voevent.4pisky.org/GAIA': 1272}

To compose a filter-set, we just create a basic Python dictionary, choosing from a number of pre-defined dictionary keys which relate to the filters. And then we pass that dictionary into the endpoint-fetch function, and get back a neater list of streams.

Extensive examples:

http://voeventdbremote.readthedocs.io/

Back up a moment...

There are extensive examples online, and I can show some of those in the demo session if people are interested. For now, let me back up a bit and focus on that install process.

!pip install voeventdb.remote

Collecting voeventdb.remote
Collecting requests (from voeventdb.remote)
  Using cached requests-2.11.1-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): six in /home/staley/.virtualenvs/jupyterpres/lib/python2.7/site-packages (from voeventdb.remote)
Requirement already satisfied (use --upgrade to upgrade): simplejson in /home/staley/.virtualenvs/jupyterpres/lib/python2.7/site-packages (from voeventdb.remote)
Requirement already satisfied (use --upgrade to upgrade): astropy in /home/staley/.virtualenvs/jupyterpres/lib/python2.7/site-packages (from voeventdb.remote)
Requirement already satisfied (use --upgrade to upgrade): pytz in /home/staley/.virtualenvs/jupyterpres/lib/python2.7/site-packages (from voeventdb.remote)
Requirement already satisfied (use --upgrade to upgrade): iso8601 in /home/staley/.virtualenvs/jupyterpres/lib/python2.7/site-packages (from voeventdb.remote)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in /home/staley/.virtualenvs/jupyterpres/lib/python2.7/site-packages (from astropy->voeventdb.remote)
Installing collected packages: requests, voeventdb.remote
Successfully installed requests-2.11.1 voeventdb.remote-1.0.0

What just happened?

• Pip fetched the relevant source-code package from the Python Package Index
• Read setup.py, parsed the list of dependencies
• Checked what's currently installed, fetched those missing (possibly from the local cache).
• Installs each of those dependencies in turn,
• then installs our package

The reason I want to focus on this is that it's easy to forget when you're developing your own code, how much of a mental barrier it is to actually trying something out, if it's a pain to install.

Packaging

• Encourages re-use as a component
• Removes 'install friction': just add a package to your requirements list
• Adoption has historically been slowed due to fragmented ecosystem, lack of good docs.
• Good, short, up-to-date tutorial on packaging your code: http://python-packaging.readthedocs.io/

One snag...

setup.py:

#!/usr/bin/env python
from setuptools import setup, find_packages

install_requires = [
    'iso8601',
    'pytz',
    'requests',
    'simplejson',
    'astropy',
    'six',
]
packages = find_packages()
setup(
    name="voeventdb.remote",
    version=0.1,
    description="Client-lib for remote queries...",
    author="Tim Staley",
    author_email="[email protected]",
    url="https://github.com/timstaley/voeventdb.remote",
    packages=packages,
    install_requires=install_requires,
)

• This is (a condensed version of) the setup.py for voeventdb.remote.
• It's almost all boilerplate which you could easily modify to any other project- author details, a list of dependencies, and so on.
• Mostly, you can set this up once and then forget about it.
• However, the version number has to manually edited every time you make a new release, which is easy to forget.
• It's also silly - we already have a version control system - Git!

So here's a neat trick...

Package Versioning

Versioneer:

• Adds a standalone Python module to your codebase
• Automatically sets version number according to most recent git-tag
• Git commit-id also available as a string in your library.
• Super convenient, keeps everything in sync

setup.py with Versioneer:

#!/usr/bin/env python
import versioneer

setup(
    name="voeventdb.remote",
    version=versioneer.get_version(),
    cmdclass=versioneer.get_cmdclass(),
    "...",
)

import voeventdb.remote
print("Git tag:", voeventdb.remote.__version__)
print("Git commit-id:", voeventdb.remote.__versiondict__['full-revisionid'])

Git tag: 1.0.0
Git commit-id: 02b727d168797a9ae9bc6835c15b37e384ea1557

Documentation

Minimal docs:

• Description of what your package does (+ links for context!)
• One or two brief usage examples
• One big README is typically fine

Extended docs:

• Maybe also some autogenerated-API docs (see sphinx, sphinx-napoleon).
• Put it on ReadTheDocs

Read The Docs:

• Free hosting for Sphinx-generated documentation
• Links to a Github repository
• Every git-push results in a new documentation build

Documenting examples

• Examples are very useful... until the code changes and they go stale
• Python notebooks are a great format for writing examples - but tricky to publish.

Documenting examples with nbsphinx & RTD

• nbsphinx lets you generate docs from notebooks.
• The notebooks are re-run with every docs-build - so if the examples are broken, you'll notice.
• This is how the voeventdb client-docs are generated.

Note: The cake is a lie. This isn't actually how the voeventdb.remote docs are generated, because nbsphinx was only released around the same time as I was writing those docs. I hacked something together before finding nbsphinx. But if I were writing those docs today, I'd use nbsphinx, and I'll likely migrate the existing docs at some point.

Deployment & Hosting

For multi-component systems, deployment details are crucial.

• If you're writing a basic Python library, you don't need to think about this - document it, package it, test it, you're done.
• However... if you're releasing a package which works with a database, or is served through a web-interface, or talks to other custom code... you are a cruel, cruel person if you don't at least document your install process.
• Example...

• Postgres backend
• Served via Apache + mod_wsgi
• Use Comet to receive latest VOEvents.
• Configuring a server like this is non-trivial.
• The best way to document your install process is to automate it

• Deployments scripted with Ansible

• Deployment scripts are open-source!
• Can test-drive locally using VirtualBox and Vagrant

• All of our deployments are scripted with Ansible
• This is a configuration management tool (you may have heard of Chef or Puppet)
• There's no getting around the fact that there is a setup overhead
• The first time you automate your install it will take days
• But then new server setups take minutes, rather than hours

Related sustainability issues I've not had time to cover

(But you should be aware of it you're new to astronomy, or academia in general. And it's perhaps worth reminding those of you with tenure.)

• To all new grad students I would recommend
  • Wilson et al 2014, Best Practices for Scientific Computing, and
  • Wilson et al 2016, Good Enough Practices....

• If you want a reminder of the basics, or an extended checklist for writing 'good' software - to all new grad students I would recommend

• Lack of software development training for grad-students

  (What do we drop, to replace with software-carpentry?)

Grads get lectures in electromagnetism, GR, stellar evolution, etc. What do we cut-out, or make optional?

• Lack of long-term career-path for 'research software engineers'

(This is changing, slowly, e.g. UCL's RSE team)

There have always been a few employed on large multi-year projects (space missions, telescopes, etc), though this is perhaps a slightly different line of work to software development for general research.

It's notable STSci now employs people to work solely on Astropy, though I don't know if those positions are permanent or time-limited.

A few universities / departments have started to employ 'departmental software specialists' or even 'in-house consultancy teams' (cf UCL).

Summary

VOEventDB

• Provides a 'turn-key' queryable repository for transient alerts.
• Can be used as a remote service
• Or run your own
• Overview paper: arXiv:1606.03735

Packaging

• Make use of your packaging ecosystem
• Think about use of your code as a component
• Keep versioning information in your version control system! - automate package versioning

Documentation

• Minimum: description + example usage + install requirements
• Documentation goes stale - test your examples
• In Python, notebooks are a great format for this - try nbsphinx!

Deployment

• Docs are a start, but easily go out of date
• Automate!

Fin

Thanks!

Links:

• Slides: http://tiny.cc/voeventdb-talk
• Project page: https://4pisky.org/voevents/
• Overview paper: arXiv:1606.03735
• Client-docs and tutorial: http://voeventdbremote.rtfd.io/
• Server-docs: http://voeventdb.rtfd.io/
• NBSphinx: https://nbsphinx.readthedocs.io/
• Versioneer: https://github.com/warner/python-versioneer