01/11 2014

Open letter: Call for conventions

The following is an open letter to anyone involved in TYPO3 CMS development. In it, I will attempt to highlight every key decision that FluidTYPO3 has made and why our choices, if any choices, should be adopted by CMS itself.

Dear TYPO3 core members and interested enthusiast!

First of all, thank you for taking the 30-some minutes it takes to read and consider this letter. I know time is precious for most of you so I'll try to keep it short. The motivation for writing this is the reports of goals and intentions for CMS version 7 - there is such a great number of "we do that in FluidTYPO3 already" experiences that now would probably be a good time to write up a concise description of FluidTYPO3 choices and why we feel those could, should, might be adopted by the TYPO3 CMS core.

I will begin with our overall integration strategy since this would be an excellent convention to adopt.

1. Conventions, conventions, conventions! And extensions!

Every integration in FluidTYPO3 - content templates, page templates, TCA generation, etc. - relies on convention over configuration. Where current strategies depend on a large number of configuration options to be defined before a site can even be operated correctly, we've tried to bring those down to a bare minimum and instead require that (extension-)developers follow a few basic rules:

1. Every template and piece of configuration gets shipped in an extension, a concept we call a Provider Extension. The obvious benefit of this is the tighter integration between Extbase/Fluid's controller contexts and extension keys. The extension follows a predefined set of rules (Extbase's) for determining where files must be placed in order to be used.
2. We consider each context in which a template can be used - content, pages, plugins - a separate controller context and we provide options to have such controllers resolved, by naming convention, when rendering a page, content, whatever template from a specific extension context (our Provider Extension). We reserve the obvious names: Content, Page, etc.
3. We automate the traditional registration you would normally have to perform in order to use a Fluid template or an Extbase controller with a Fluid template, as a page template or content element. We simply detect, by naming convention, the files or controllers which can be used by CMS because of our integration layer.

Now, these are by no means innovations - when it comes down to it, FluidTYPO3 extensions simply abstract an already existing API. And this is exactly why I believe these make an inordinate amount of sense to be implemented as features of CMS itself. Not necessarily the whole FluidTYPO3 solution in this area as-is, but in general terms. In having a convention which reserves a key set of controller names and associated Fluid template scopes which must belong to an extension. In relying on all of these conventions to minimise the amount of configuration required.

The convention over configuration strategy is what raises productivity at the base level across all (templating) concerns when using FluidTYPO3. I strongly suggest it, or a similar strategy, be implemented at the core level.

If generating an extension is too complicated, let's automate that instead of waiving the requirement that developers as well as integrators simply must get used to extensions as static asset storage.

2. No special treatment, love.

AKA "eat our own dog food".

This is a continuation of the previous chapter but deserves a dedicated section. The problem at hand: the special privileged treatment that some extensions are receiving from the core simply by being too tightly coupled. These include, but are not limited to, several hooks which are introduced in core code to serve one specific community extension, and most prominently of all system etensions, the "css_styled_content" extension. This is already a priority of the core team members and I am sure is being handled competently, but it still deserves to be put in this spot. The following are examples of couplings that are too tight and give privileges to one specific extension, leaving little or no room for other extensions to serve a similar purpose:

1. CSS Styled Content. This is the ultimate example of an extension that used to be core code but is now a system extension. Although this is technically separated, it is still tightly coupled by the fact that all TCA which is specifically used only by CSS Styled Content is not contained in CSS Styled Content but in the core and the completely preferential treatment of static TS templates loaded by this and only this extension, when generating TS setup. It only works because there is a core entry which identifies CSS Styled Content by extension key (actually, by path to the TS template...) as "to be loaded before everything else". A special privilege which actually makes it impossible for other extensions to provide the features that CSC provides, without requiring additional configuration (see fluidcontent_core).
2. Versions, Workspaces. Only one of these two are technically possible to remove at this point (workspaces), the other is so tightly wound into the TYPO3 core that uninstalling it leads to unrecoverable errors. The wrong way of doing it is, as it is done now, to implement all these features at every code point that requires it, directly. The right way would be to follow the pragma of Extbase and register an implementation for an interface. Although the perfect (but perhaps too far reaching) approach could be something along the lines of giving every feature that TYPO3 CMS provides, its own implementation-for-interface which can then be toggled and replaced as required.

In FluidTYPO3, none of the extensions that deliver features are given any preferential treatment in relation to other extensions which might want to deliver the same feature. There's no equivalent of, for example, the special condition hidden deep in the core which allows CSC to work the way it does, but not other extensions.

This point of preferential treatment is one I have identified as a stopping block for innovation in these areas of TYPO3, but it closely relates to the point about using conventions. Had CSC been an implementation-for-interface and had the core consumed it as such, and had it relied on conventions to place controllers, template files, setup etc. that gets loaded, we would have seen much more innovative ways of actually performing these core features.

My clear advise for the core team members: dedicate some focus to decoupling this (a decision that is already made on an abstract level) but do so using the pragmas learned from Extbase. Program the core's consumption of these features to an interface and ship a default implementation for that interface (if you ship anything at all; often a sane default esp. in development contexts is a NULL implementation!).

Allow us to identify any extension as "this provides core feature XYZ"; a sort of Trait for extensions (speaking in PHP). Example: extension "mysite" is assigned the "Traits" for Content, ContentRenderer, Pages, PageRenderer, Plugins, TypoScript, Language, Versioning, UserInterface etc. and this allows the core to detect PHP classes, template files, asset files etc. from within that extension and allow those detected assets to be selected/configured in the CMS. The same goes for all SignalSlot listeners and hook-type classes: create an expected convention naming for those, analyse and cache a list of detected listeners and use that when/if triggering the signals and hooks. And the same applies to other asset types such as CSS and icons/images which could be loaded and used simply by being present.

Let's get rid of all those ext_localconf.php lines to register this and configure that, load foo and do bar so the extension will even work. Use devlog to communicate when such convention based resolving fails, to give developers traceable debugging information in a sensible way. We can also better encapsulate this process and protect it from failures for example arising during extension upgrades affecting classes with a cached reflection (damn, I should have carved that little tidbit into a pumpkin yesterday...).

3. Honey, I Forgot the LLL Label Name again - episode 3,429 of 9,744

Now that we require an extension for storing our custom assets - our content templates, page templates, configuration, controller classes and so on - we have a rare case of puzzle pieces just coming together naturally to form a new, beautiful picture: every last bit of our extension and our templates can now be expressed as tree values. Because of this and because we enjoy convention-over-configuration, we can predict the expected names of LLL references depending on where in the extension the entity-to-be-named gets used. Such automatic names can be generated for literally everything, including individual sections in each fluid template. All it takes is a solid, collision-free set of expectations.

That's a little vague perhaps. A concrete example: a Flux form has an id and a hierarchy of fields. To identify any given field at any point in this hierarchy, a dotted-path expression can be used. Examples: formX.sheets.displayOptions, formX.fields.specialOption, formX.description and so on. So, rather than requiring a developer to tediously define and then reference this LLL value by adding "label" attributes everywhere and remembering the naming convention - or worse, inventing his own.

So rather than opening up this free-for-all regarding LLL values, FluidTYPO3 introduces an assortment of LLL naming conventions which ensure that all form fields have predictable label names and will display the full path to the expected LLL value if one is not found. It is my suggestion that the core itself adopts this naming convention for everything: 

1. All metadata of extension itself.
2. Properties of domain objects' forms and other TCA-related LLL.
3. Labels of plugins.
4. Any exceptions, regardless of type, thrown from within the extension's scope (passthrough to core's default), identified by exception number.
5. Specifically any form validation error or argument mapping errors, same rule as above with one addition: additional lookups allow even more narrow matches. The following examples would translate the same error message differently depending on where the error happens, checking from the bottom of the list:
   1. exceptions.123456789
   2. controllers.SomeController.exceptions.123456789
   3. controllers.SomeController.someAction.exceptions.123456789
   4. controllers.SomeController.someAction.someArgument.exceptions.123456789
   5. plugins.MyPlugin.exceptions.123456789
   6. plugins.MyPlugin.SomeController.exceptions.123456789
   7. plugins.MyPlugin.SomeController.someAction.exceptions.123456789
   8. plugins.MyPlugin.SomeController.someAction.someArgument.exceptions.123456789

The fewer nodes we as developers are required to use in our Fluid templates, the better those templates will perform. The more flexible the lookups are without causing performance issues, the further we can go without ever having to dispatch a manual translation function.

My suggestion here is: define the basic constituent parts of an extension and reflect those in a tree structure. Identify any narrowing of lookup rules like the example above. Then generate expected LLL references for every part of the extension and all variations and as a last puzzle piece, let "Development" context actually output the raw LLL:EXT:... value in case the actual value is not found.

Bonus info: the extension called FLLL (EXT:flll) hooks into the translation routines of TYPO3 and will actually continuously write any LLL values which are referenced but not found. There's no reason why such roundtripping could not be an integral part of the way extensions integrate with TYPO3 CMS. Some day perhaps we would choose to package such "developer's assistants" together, further increasing the productivity of everyone using TYPO3 CMS.

4. Mine cutting utensil be too grand

Cryptic heading? Perhaps. Very obscure reference? You bet! But there's a point. TYPO3 CMS is much like the Swiss army knife - it tries to bundle everything that you could possibly need in a small and light bundle. Unfortunately, TYPO3 CMS is much like the Swiss army knife's later, bulky iterations: there are three tools you use and 38 you don't, but the shaft still has to hold them all. The result is that it might be a great knife with many applications, but now the ergonomics are all wrong. Rather than attempt to ship something big that does everything 90% of everyone might ever need, we should focus on shipping something excellent that everyone needs. Instead of a Swiss army knife, let's make a katana that you re-smith into a Swiss army knife if you need it.

In concrete terms: it's actually possible to ship TYPO3 CMS without 80% of the system extensions it contains without any negative impact. The only thing lacking is a way to distribute such extensions. Of course, I've got an idea for that as well.

5. It's a jungle up there

So now we have no shipped preferentially treated extensions and we lack some way of distributing these. We already have TER and it works wonderfully (but could use a performance checkup). But what TER doesn't have is an official way to indicate that an extension is worth while; a favourite of the community proven to add value. Twitter has the "verified" badge, Apple's App store has the "Featured Apps". TER has a download count and graph which by the way lacks a time scale.

Why don't we introduce a "recommended" and/or "official" flag on TER? Allow such extensions to be listed specially. Raise them as top priority for suggestions. Ship the core extensions from our Swiss army knife as "official" extensions. And let the community decide which ones will get that distinction; not some arbitrary number hits from an Apache log (sincerely, no offence! I realise exactly why the solution is the way it is and I respect it! I only wish to make it better).

There's another perspective here. It's almost a tradition in TYPO3 that if XYZ does not fit our exact intention, we go ahead and roll our own. If we got around this and were able to pipe efforts into improving existing solutions I am certain we could all benefit. The examples are plenty. I don't mean to appear arrogant or hipster about this, but again: in my experience FluidTYPO3 and before that, FED, were first movers - later on joined by other alternatives like Gridelements and DCE (the former of which is at least planning to adopt the principles of Flux). This is nothing personal against those projects or their teams but they're solving the same problem already solved by FluidTYPO3. But they all make the same mistake of not relying on convention-over-configuration and not fully utilising the powers given to us through Extbase. If these efforts had gone in one direction instead, imagine where we could have been now?

So the second perspective is: having a way to selectively recommend extensions might encourage the pooling of efforts into those extensions. I hope FluidTYPO3 to be among those selectively recommended, but that's up to the community. But I digress into meta - ever so slightly.

6. There's a ViewHelper for that

...and chances are that we're the ones who wrote it for you.

Perhaps the most popular feature of FluidTYPO3 is the ViewHelpers. There's the slimmest, cutest interface to inserting and processing components in templates. We use them for pretty much everything. We rely on their XML nature to define complex form structures (TCEforms API abstraction) inside each template. Structures which can then be parsed out, turned into arrays, rendered as grids and much more.

Like the other features this one is merely an abstraction of another API. Had this been TYPO3 3.8, Flux might instead have parsed out a flexform XML structure from a marker-based template. But since we gained access to Fluid, we now have the perfect base for creating XML-like APIs, which as it happens, can be merged into the template file that is responsible for the rendering.

So this suggestion is in two parts:

1. Adopt Flux's form- and ViewHelper API for all purposes where a (flex-)form or a grid structure must be associated with a template (or with a controller action that is then associated with a template). Flux has both a PHP and Fluid flavour and the Fluid flavour will return objects of the PHP flavour. The PHP flavour can be consistently consumed in any way desired - this is, in so few words, "just" an OOP strategy for modelling what used to be an array. The fact that everything is type-hinted PHP ensures fewer mistakes - and yes, it's possible to build a complete Flux form structure based on a large input array. In all modesty it's the ideal API for how TYPO3 CMS currently specifies forms. If you're going to make Fluid more widely used on a platform that has a TCEforms engine, Flux is just the natural next step on the ladder.
2. Adopt the key aspects of VHS - the ability to render menus, language switches, content elements and other CMS component types with a ViewHelper instead of TypoScript. Every designer can learn to use a special XML tag. Far from every designer can learn the intricacies of creating a custom menu in TypoScript, not to speak of rendering content elements without the assistance of CSC (which we've already sort of evicted in my suggestions above). Do not require f:cObject use - there need not be that barrier between TypoScript and Fluid template; we can and indeed should have all our rendering instructions in the template.

If I had to choose the two most important feature aspects of FluidTYPO3 it would be those two, in that order. I choose that order because I mainly develop plugins - a designer or integrator would likely reverse the priority.

7. Not just for enterprises

The marketing of TYPO3 CMS has been very focused on big enterprise. While TYPO3 CMS certainly is a good choice for that, it's not all that TYPO3 CMS is good for. But sadly, that enthusiast-level prioritisation has been close to neglected. I strongly disagree with this which is also why you won't find a single mention of TYPO3 CMS being an "Enterprise" CMS anywhere in FluidTYPO3. I believe TYPO3 CMS is accessible to anyone, given an appropriate amount of abstractions. TYPO3 CMS clearly has all the power we could want - what we need now is to smack a couple of metaphorical motorcycle handlebars and a gas throttle on it to make actually learning it, an intuitive experience. Not just for developers but also for users.

For example, do give EXT:site - our official "distribution" (I'll explain the quotes later) - a try. It results in the exact type of minimalist setup that a beginner would be able to assimilate in a very short time. It hides away an incredible degree of complexity and provides the most direct way to a "edit and reload, changes are there" experience.

Rather than install a mountain of pages, templates, extensions and content in a distribution package, this distribution's goal is to set you up with the absolute bare necessities for starting a purely Fluid-based web site in the least amount of time possible. It takes around 2-3 minutes from starting the TYPO3 CMS install until you're editing your first Fluid template and seeing the results on every page reload.

Just the top of the pop

The suggestions above don't cover everything that FluidTYPO3 does - I tried to extract the concepts that make most sense as standard conventions. Every suggestion has upwards of five years of experience and trial-and-error behind it and most of them have been proven in the wild. They not only work, they work very well - and they can save you from reinventing the wheel for an umpteenth time. So I urge you to get inspired by this. Look into the extensions mentioned, see how the user experience is and take as much as you want and need with you when you go back to working on CMS version 7.

A bright TYPO3 future to all,

Claus