02/10 2014
Automated TER releases

A huge step forward has been made: the release-to-TER process has been completely automated and now happens just by updating the version numbers and merging our "staging" branches into their "master" counterparts. Here's how you can use the same setup in your own repositories for your TYPO3 TER extensions.

Dear developers!

Those of you using GitHub for developing TYPO3 extensions which you put on TER, most likely use a CLI utility or TYPO3 extension to upload your extensions to TER. There is a way to automate this step now by implementing a GitHub web hook - and yours truly, Claus, have been writing a tiny set of components which combine to create this automation.

This means we will be providing much more frequent TER releases of all extensions - perhaps mere minutes after solving a bug or adding a feature.

Read on if you're interested in setting up such automation yourself

If you are interested in the approach used, take a look at the FluidTYPO3 Gizzle component: <link https: github.com fluidtypo3 fluidtypo3-gizzle external-link-new-window>github.com/FluidTYPO3/fluidtypo3-gizzle. Currently there is no documentation for this component, but there is basic documentation for the dependencies which you can all find on my GitHub repository list: <link https: github.com external-link-new-window>github.com/NamelessCoder. The following is a brief write-up on how to structure repositories and create the web hook receiver as well as mirroring each of your repositories' branches to separate folders on your web server.

It's alpha quality so don't say I didn't warn you. The first successful automated release was made just a few minutes ago.

Structure of repository

Although you only need a single branch to successfully automate TER uploads, it helps a lot to have at least two branches and only push (merge) to the "master" branch when you want to create a release. This saves the web hook receiver from attempting a new release on commits which do not update the version number.

The suggested structure is to have a development, a staging and a master branch. Development receives all commits initially, staging is used to create intermittent "beta" type releases and master of course contains the current production-quality release.

Installation of Gizzle component

To install, simply fork and clone the repository linked above, run composer install and create a virtual host pointing to the "web" directory inside the project folder. This allows you to use that virtual host domain as URL for your web hook in the repository settings and you can now commit any modifications you make to the script (and return pull requests if your modifications should be shared). Then, in the root of the project folder safely outside the virtual host document root called "web", create a new file called .secret and enter as content of the file, a secret phrase or password that only you know. You'll need this secret when you configure the repository or repositories on GitHub.

In order to authenticate against TER you also need to create one more file in the same location, this one called .typo3credentials and containing a special username:password value with the credentials you use when logging in to TER. Since these credentials are not encrypted I recommend creating a new typo3.org user to hold your extensions as owner and keep your "real" user reserved for you yourself. However: this information will never be disclosed in logs or in responses to Github. It does get sent as cleartext to TER when uploading extensions, but this is simply the only way TER accepts authentication in SOAP. If at any point a token-based solution becomes available I'll be the first to adopt it!

To fully enable Gizzle you need to create a personal access token with permissions tailored to your needs (usually, simple status update permissions are sufficient). You can skip this step, but if you choose to add a token, place a .token file in the root of the folder you cloned. This file is already configured as ignored file.

Setting up the GitHub web hook

This is the easy part: in every repository which you want to automatically release to TER, add a new "web hook" (not a service) and enter the URL of your virtual host (must of course be public) and the secret passphrase you put in the .secret file. Then add a parameter indicating which of the included settings files (see the settings directory) you wish to use. The final URL you enter looks along these lines:

http:// ... /github-webhook.php?settings=settings/ExtensionRelease.yml

As you can see, we point to the settings/ExtensionRelease.yml file which is located in the project folder. You can edit this file to change which directory is used as temporary storage will pulling the branch that gets released - and you can change the name of the branch that will be receiving commits when you build new releases.

Also included: development-, staging- and production code checkout

There is another settings file called SiteDeploy.yml which you can reference from the web hook to cause Gizzle to git pull repositories in a directory which contains one marker that can be the name of your branch. For example, we use this to automatically pull new commits to /var/www/staging.fluidtypo3.org and /var/www/development.fluidtypo3.org and of course /var/www/master.fluidtypo3.org which is the virtual host serving this site. These automatic pulls happen whenever commits land in the corresponding branches of the flux, vhs etc. repositories and the path to the directory containing the repository on the server is calculated using dynamic values. As a result, whenever we create a new TER release we immediately deploy it to our own site as well. Eating our own dog food before anyone else tastes it, so to say.

You can do this for your own repositories as well - simply reference another settings file in the web hook:

http:// ... /github-webhook.php?settings=settings/ExtensionRelease.yml

Or reference two files at the same time:

...hook.php?settings[]=settings/ExtensionRelease.yml&settings[]=settings/SiteDeploy.yml

Then sit back and let Gizzle do the boring stuff for you.

Should you need to inspect logs, Gizzle writes a brief summary which you can read by clicking the latest item in the entry of "payloads" when editing the web hook. Click the "response" tab to view a small piece of JSON describing the actions taken. This is also where you can read errors.

Before you go all "someone already made that!" on me

I just want to say: I know. I am completely aware of the many different GitHub web hooks out there. I'm also aware of several deployment applications. None of them really fit right with TYPO3 public extension development using GitHub - but this one does. And it's written in PHP, has a very simple architecture to allow you to create your own plugins and it includes (when personal access token is provided) a way of executing actions on GitHub as response to a payload (new comments, updating commit status, merging branches when Travis replies "good" - you name it) - which you'd normally need to combine and initialise two different libraries to do. Everything this component does has been made before in other variations, all I did was package it neatly to fit a very specific set of use cases for integrating with GitHub.

So yes I'm aware it's not exactly innovative. But it is exactly what we needed in order to automate the whole TER release work flow - and I'm happy to share it with you for free.

Cheers,
Claus

 

Archive

09/01 2017
09/11 2016
09/06 2016
21/12 2015
07/12 2015
25/11 2015
25/09 2015
22/09 2015
01/08 2015
10/03 2015
03/03 2015
12/02 2015
25/11 2014
01/11 2014
16/10 2014
02/10 2014
02/10 2014
19/09 2014
18/09 2014
05/09 2014
22/08 2014
02/08 2014
27/06 2014
06/06 2014
13/04 2014
27/03 2014
12/03 2014
11/03 2014
05/02 2014
25/01 2014
17/12 2013
08/12 2013
03/12 2013
04/11 2013
Flux 7.0 Teaser
06/08 2013
21/07 2013
10/06 2013
04/06 2013
01/06 2013
27/05 2013
19/05 2013
19/05 2013
11/05 2013
26/04 2013
30/03 2013
19/03 2013
17/03 2013
13/03 2013
10/03 2013
10/03 2013
05/03 2013
04/03 2013
03/03 2013
02/03 2013
01/03 2013
28/02 2013
27/02 2013
25/02 2013
24/02 2013
24/02 2013
23/02 2013
10/02 2013
03/02 2013
03/02 2013
27/01 2013
Asset management in Fluid
20/01 2013
16/01 2013
13/01 2013
08/01 2013
16/12 2012
25/11 2012
18/11 2012
08/11 2012
07/11 2012
05/11 2012
04/11 2012
28/10 2012
22/10 2012
14/10 2012
13/08 2012
08/08 2012
31/07 2012
30/07 2012
25/07 2012
29/04 2012
29/04 2012
22/04 2012
16/04 2012
21/03 2012
Flux 1.4.0 released
08/03 2012
Flux 1.3.0 released
04/03 2012
03/03 2012
28/02 2012
19/02 2012
A Sneaky Sneak Preview of the next version of Flux
13/02 2012
12/02 2012
06/02 2012
30/01 2012
27/01 2012
15/01 2012
26/12 2011
24/12 2011
11/12 2011
11/12 2011
10/12 2011
04/12 2011
04/12 2011
30/11 2011
26/11 2011
25/11 2011