A huge step forward has been made: the release-to-TER process has been completely automated and now happens just by updating the version numbers and merging our "staging" branches into their "master" counterparts. Here's how you can use the same setup in your own repositories for your TYPO3 TER extensions.
Those of you using GitHub for developing TYPO3 extensions which you put on TER, most likely use a CLI utility or TYPO3 extension to upload your extensions to TER. There is a way to automate this step now by implementing a GitHub web hook - and yours truly, Claus, have been writing a tiny set of components which combine to create this automation.
This means we will be providing much more frequent TER releases of all extensions - perhaps mere minutes after solving a bug or adding a feature.
Read on if you're interested in setting up such automation yourself
If you are interested in the approach used, take a look at the FluidTYPO3 Gizzle component: <link https: github.com fluidtypo3 fluidtypo3-gizzle external-link-new-window>github.com/FluidTYPO3/fluidtypo3-gizzle. Currently there is no documentation for this component, but there is basic documentation for the dependencies which you can all find on my GitHub repository list: <link https: github.com external-link-new-window>github.com/NamelessCoder. The following is a brief write-up on how to structure repositories and create the web hook receiver as well as mirroring each of your repositories' branches to separate folders on your web server.
It's alpha quality so don't say I didn't warn you. The first successful automated release was made just a few minutes ago.
Structure of repository
Although you only need a single branch to successfully automate TER uploads, it helps a lot to have at least two branches and only push (merge) to the "master" branch when you want to create a release. This saves the web hook receiver from attempting a new release on commits which do not update the version number.
The suggested structure is to have a development, a staging and a master branch. Development receives all commits initially, staging is used to create intermittent "beta" type releases and master of course contains the current production-quality release.
Installation of Gizzle component
To install, simply fork and clone the repository linked above, run composer install and create a virtual host pointing to the "web" directory inside the project folder. This allows you to use that virtual host domain as URL for your web hook in the repository settings and you can now commit any modifications you make to the script (and return pull requests if your modifications should be shared). Then, in the root of the project folder safely outside the virtual host document root called "web", create a new file called .secret and enter as content of the file, a secret phrase or password that only you know. You'll need this secret when you configure the repository or repositories on GitHub.
In order to authenticate against TER you also need to create one more file in the same location, this one called .typo3credentials and containing a special username:password value with the credentials you use when logging in to TER. Since these credentials are not encrypted I recommend creating a new typo3.org user to hold your extensions as owner and keep your "real" user reserved for you yourself. However: this information will never be disclosed in logs or in responses to Github. It does get sent as cleartext to TER when uploading extensions, but this is simply the only way TER accepts authentication in SOAP. If at any point a token-based solution becomes available I'll be the first to adopt it!
To fully enable Gizzle you need to create a personal access token with permissions tailored to your needs (usually, simple status update permissions are sufficient). You can skip this step, but if you choose to add a token, place a .token file in the root of the folder you cloned. This file is already configured as ignored file.
Setting up the GitHub web hook
This is the easy part: in every repository which you want to automatically release to TER, add a new "web hook" (not a service) and enter the URL of your virtual host (must of course be public) and the secret passphrase you put in the .secret file. Then add a parameter indicating which of the included settings files (see the settings directory) you wish to use. The final URL you enter looks along these lines:
http:// ... /github-webhook.php?settings=settings/ExtensionRelease.yml
As you can see, we point to the settings/ExtensionRelease.yml file which is located in the project folder. You can edit this file to change which directory is used as temporary storage will pulling the branch that gets released - and you can change the name of the branch that will be receiving commits when you build new releases.
Also included: development-, staging- and production code checkout
There is another settings file called SiteDeploy.yml which you can reference from the web hook to cause Gizzle to git pull repositories in a directory which contains one marker that can be the name of your branch. For example, we use this to automatically pull new commits to /var/www/staging.fluidtypo3.org and /var/www/development.fluidtypo3.org and of course /var/www/master.fluidtypo3.org which is the virtual host serving this site. These automatic pulls happen whenever commits land in the corresponding branches of the flux, vhs etc. repositories and the path to the directory containing the repository on the server is calculated using dynamic values. As a result, whenever we create a new TER release we immediately deploy it to our own site as well. Eating our own dog food before anyone else tastes it, so to say.
You can do this for your own repositories as well - simply reference another settings file in the web hook:
http:// ... /github-webhook.php?settings=settings/ExtensionRelease.yml
Or reference two files at the same time:
...hook.php?settings[]=settings/ExtensionRelease.yml&settings[]=settings/SiteDeploy.yml
Then sit back and let Gizzle do the boring stuff for you.
Should you need to inspect logs, Gizzle writes a brief summary which you can read by clicking the latest item in the entry of "payloads" when editing the web hook. Click the "response" tab to view a small piece of JSON describing the actions taken. This is also where you can read errors.
Before you go all "someone already made that!" on me
I just want to say: I know. I am completely aware of the many different GitHub web hooks out there. I'm also aware of several deployment applications. None of them really fit right with TYPO3 public extension development using GitHub - but this one does. And it's written in PHP, has a very simple architecture to allow you to create your own plugins and it includes (when personal access token is provided) a way of executing actions on GitHub as response to a payload (new comments, updating commit status, merging branches when Travis replies "good" - you name it) - which you'd normally need to combine and initialise two different libraries to do. Everything this component does has been made before in other variations, all I did was package it neatly to fit a very specific set of use cases for integrating with GitHub.
So yes I'm aware it's not exactly innovative. But it is exactly what we needed in order to automate the whole TER release work flow - and I'm happy to share it with you for free.
Cheers,
Claus