Transcription

SAM: … a lot of people sneak in, we don’t want people to try and run in front.  So if there is some space over here at the back.  Come on people! You are not moving!

Please?

Okay, now this is the final session before lunch.  So just a few things to mention about lunch.

There is lots of signage about the instructions for lunch so you should be okay but standing on the right if you are in the main queue so people can get in and out.  Be, not as fast as you can, but try and get your food and try and find somewhere to sit.  There is a nice room with tables up.  Some more, if it is not raining there a space outside to eat as well.  It would be good because everybody is just going the rush and go straight to lunch straight away, perhaps take the time to go visit some sponsors maybe pick up your T. shirt, talk to people and then you are not waiting in the queue for a long time.  Any special dietary requirements go past the queue and straight to the table at the end.  That is anybody with a star on their badge.  If you are not vegetarian and not halal and you don’t have a star, go to reception to get one.

I think that is it for now.  This is the last session I will be hosting you have got Wendie MC’ing this afternoon in the room.  We have Alain Schlesser who has travelled all the way from Germany to be with us, he will talk about legacy, modern pp around a legacy WordPress site.

(APPLAUSE).

ALAIN SCHLESSER: I will write a lot of stuff.  I don’t want to stand between a hungry crowd and lunch so I will start immediately with with the actual problem I am trying to describe here.

So I started working on a project for client as a subcontractor, that had a lot of differing sites that all had very different problems but, they wanted, wanted to, to work on them as a career and as a whole.  I tried to find a way to solve this by providing a common architecture that all the sites can use that just provides common ground for all the custom functionality that can be used that would make all the custom code reusable across the sites although each site might still differ in their specific requirements.

So the client is, it is a futures trading company and they, they have several brands.  They have mergers and acquisitions, which causes them to add yet other sites to the bunch.  So this architecture should allow us to, to make a code not only robust but also a future proof for all changing requirements.

So the goals of the web efforts.

First of all get rid of all the different platforms and technologies that were being used and have a common base set of a WordPress site with this architecture.

Refactor everything that is not a good fit in the existing sites to, to use specialised services where this makes sense and to generally get rid of a lot of existing bugs as well that can be eliminated by coding with practices.

All the, all the changing requirements mean that changes should be easy to make on the sites, changes should be easy to build in a robust way so that when the markets change, the sites can simply adapt.

An additional requirement that is more in the enterprise space than that you will not encounter when you are dealing with personal, is the entire regulations stuff.  So for each website change for each content change and for all the e-mails that are sent you need to have audits and locks and you need to be able to prove what did happen and what was changed.  When did we show this content and so forth.

So, the sites, they are pretty different this is just a shot overview of the front page of 3 examples sites.  As you can already see from the front page it is not one common set up.  The individual sites they are rather complex, this is one example site.  It has a lot of custom post types, it has a lot of all sorts of functionality that does not only relate to the database schema, it has the user interfaces for the content editing team to allow them to add content to the content tabs.

Lots of custom content of course means lots of custom code.  So it is very important that code is of high quality, that it is able to enforce some, some basic functionality without needing to run hundreds of tests and all sorts of debugging just to create basic functionality.

So these are the challenges, it is a vast scope with all of the different sites, it amounts to lots and lots of thousands of, of code lines.  It is a moving target.  Whatever we we wanted to build it needs to not only encompass all the requirements of the current site and the current functionality but we also needed to, to just think about, you know, what race this might grow in the next ten years for example.

Maintenance can be quite difficult when you are not only dealing with several sites but they also come in very different states.  So you have site that is are already mature running on several languages and sites not yet finished.  Sites with new architecture, sites with no architecture and sites that run on some other technology.  So the simple maintenance just keeping all this running is a challenge in itself.

So the current development work when I joined the project, was, it was pretty clear to me was not scalable enough to really keep up with the growing scope and be able to deal with this with the rather small team.

So the approach that we used.  I defined some primary goals that the architecture should be able to fulfil and tried not to be too technical about this when I made the proposal for the company.  I tried to identify business goals and relate them to the technological goals I had in mind.

So first of all, I tried to think of the best practices I would need, that I would like to have on these sites and then translated these into such business cards.

First best practice was, with all the sites they, they are, they come in very different shapes and forms so, we have a lot of subsystems that make up such a site and I wanted to have a centralised way to access each of the subsystems because I wanted to, to have the subsystems be reusable and then when one subsystem is used on, on site A and B but not on site C this means that this subsystem will be built as WordPress plugin, so that plug in can be installed on sites where it is needed and not on the sites where it is not needed.  But the WordPress plugin system, it doesn’t come with any dependency management, doesn’t come with any mechanisms to resolve loading all the problems we might face and most of the time you just, you check whether a class exists and then use it in the hopes that everything will work out in the end.  We wanted to have a reliable standardised way to just say, I want this subsystem and it either works or it will throw an exception because I did an assumption for that site that was not fulfilled by the plugins I installed.

So one of the business goals I translated this to the company is that code should be independent of file system late out.  Because the the file system lay out for example is something that WordPress is really causing problems with plugins.  If your plugin is named abc it gets loaded before plugin called bcd.  So it is basically the file system lay out that is decides the loading order.

I will also wanted to get rid of global states I don’t want to have, to have variables from one plugin, to be used in a different plugin.  When this, when this needs to be done, the entire subsystem is ask and the subsystem is requested to give the variable that is needed.

So next best practice I wanted to integrate is that, for, for all the subsystems I wanted to define one interface that is then used across all sites in all environments but, the actual implementation can change.  So, for, for subsystem logger for example, all the sites just use the interface logger and they don’t care what actual logger will be used.  So, this causes the decision about what logger to use to be deferred to some later point in time so I can just build all my sites, use the logger interface and the specific implementation can change depending on site and depending on environment whatever is needed in that area.

This causes the isolating, the code changers to be isolated to a single subsystem at a time which means that whenever we do a code change in one subsystem because we only rely on interfaces, this means that this change will not have side effects across the entire rest of the code change.  Probably know this phenomenon, when you make a change with this file and plugin, suddenly a change in some other file and plugin.  Even in your, how this could happen this is because the changes are not isolated because of global state as in the first goal, because of just using concrete implementations and when you change the implementation you have a break in change and so forth.

The next best practice we, we had a lot of problems with the existing sites that they contained bugs that we didn’t know where to start to locate them.  So we had very complex sites, that were doing remote requests to other servers that were using several different databases at once and when something bad happens, you only get a narrow and some, some file location and it was always very difficult to pin down the exact cause of that error, you only knew when it finally broke but you didn’t know what the cause of this was.

So, that is why from the beginning I wanted to enforce that we have application wide log in across all of the sites that every subsystem, every mechanism that we use has debug logging and also locks warning in case something bad happens so we can trigger notifications for the development team for example for the sys admins.

The lock data should be freely routeable.  That means in our case we have a centralised log in server, working on that so far which can take the log in data of all the sites of all the subsystems of all the database servers of all the web servers and regroup them in one log in server so when something bad happens we have an overview of the entire system as a whole and can for example, also debug instances where there is no concrete bug in some code that we wrote but, the relationship betweens the systems just worked in a way that was not expected and so you can see that when the web server did this and the database server did that, then the application server threw this error.

We also send some of the critical log in data directly to slack, because we work on a slack team and there is several slack channels that everyone can subscribe to the notifications of.  This means that when the system has a critical error it will immediately pick slack and whoever is on-line can immediately look after this.

So we want to be able to retrace errors and that was something that company really agreed with.  We also wanted to monitor for exceptional scenarios like the critical errors.

The fourth best practice, this is something that we currently actively working on which is planned to be completed in during this year, not yet finished to, have a messaging queue, the main problem, with WordPress sites, really great content management system but its goal to deal with web front end requests.  There is some business processes that should be run as background process and not have a direct impact or be coupled to the front end web request.  So when somebody submits a form and at one point the communication with the CRM fails to send the data to the CRM, there shouldn’t be a error thrown for the user at the web front and the other way around, if there was an error on the web front, this shouldn’t cause data to be lost while it is sent to the CRM.  When it is killed, whatever is attached gets killed as well.  If you are still doing remote request and if you are still doing persistence to database, this will be killed and not complete.

So this is added another business goal of maintaining transaction integrity, which is basically means that when the user does an action we want to record the action, at this point either it completely succeeded or fails and leaves traces in log files.  We don’t want anything that causes data that is not very integral to be committed to the database.

Also, our events should be able to cross session boundaries so when the web front end request gets killed the background process that was communicating with the CRM should still continue to run and maybe it should even run on a different server than the web server.  We want also to be able, if the web server has scalability issues, has the ability to remove directly not the responsibility of the web server.

So with the secondary goals of the architecture proposal, I tried to take care of some of, some of my pet peeves as well.  Because the, as we were building this architecture, I also tried to inject some of, some of the goals that were not directly relevant for the business goals but would make life for the developers easier, which in turn makes it easier for the developers to, to better fulfil the next business goals that the company will decide on.

So first challenge, sorry, first challenge was that the, as the number of sites is constantly changing, wanted to have one goal of being able to deploy the architecture incrementally, there is a lot of nice frameworks out there, that provide all sorts of beneficial best practices and all sorts of built in objects and factories and whatnot.  But most of these have the problem that you can’t just say okay, I have an existing WordPress site and this specific part I want to use that framework for.  Most of the frameworks are meant to be used for the entire application.

So in our case, I want to make sure that whatever we were doing could be installed for example, as one single plugin adding one single type of functionality to an otherwise standard legacy site.

Both the new code and legacy code should be able to run properly side by side, they do, they shouldn’t have negative impact on each other and it shouldn’t be requirement to completely eliminate the old legacy code just to be able to use the new code.

Some subsystems are not integrated into WordPress which means that when we are concentrating only on using WordPress plugins we might not be able to cover every scenario so all the architecture code even though it’s built to be used within WordPress is pure PHP code.  It doesn’t use any of the WordPress built in functions, it doesn’t require WordPress to be loaded, it can just be run from the command line.  So sometimes this means that we might have for specific functionality 2 plugins or 2 separate services, one providing the actual functionality and the other one doing the integration with WordPress but the one with the functionality can just be run from the command line can be awe automated in whatever way you think necessary without WordPress being loaded.  This helps a lot with testing because you can run all your tests without loading the entire WordPress deck each time.

Some subsystems cannot directly be changed.  We have some third party plugins.  We have very popular forms plugin for example that’s used everywhere.  We have custom fields plugin that is used on most sites.  So it can’t directly change these third party plugins to just make use of our architecture so whatever we’re building it needs to be able to also work with third party code without forcing you to change that third party code.  So there is some of the mechanisms that I will present now that allow us to even include third party code properly without directly changing its force.

The site stack we’re using for each of these sites now it’s composer site white stick based on starter which means you can basically set up complete WordPress sight with one composer install which will download the correct WordPress version that you want to use, it will download all the plugins that will use theme, it will set up the entire layout as you need it.

Then we have environment configuration provided with env files everything specific for environment like database files for example, it’s not in source code, it will not get committed to our version control system because of security reasons.  It is all saved in env files.  So each server and development environment has its own .env file customised for that environment so on the local machine the env file will contain database credentials for local database where on production server it is connected to the production database server.

This also means that a lot of the differences between the environments can simply be eliminated from your code by putting some logic into .env files so the rest of the code becomes properly re-useable in this instance.

The third party plugins are currently pulled in via SatisPress.  There are other solutions to do this.  This is one we currently use.  It specifically let’s you have one WordPress site where you install all your third party plugins, for example, you can use normal WordPress up-date mechanism to keep this up-to-date and composer can use this reference WordPress site to always pull the plugins in via its composer mechanisms.

The composer JSON file is the file that e-fines what the current state of the file is, defines what version of WordPress code we’re using and what version of plugins and themes we’re using so we can look to specific versions so that when we test a given set up on our development and test servers, we can deploy that exact configuration with the exact same versions to production machine because sometimes simple update, minor update of one plugin might be enough to break the production site, so you always want to be sure that whatever gets up to the production site was already tested in this exact state.  So this is done by defining the site through a composer file where you can define the version requirements and when you run this on your test server you can create a composer look file and this look file looks for all these versions, these are now very general version requirements and the look file will contain the exact same version that you used for testing.

The .env file as I said will contain environment specific data.  This is only very short extract.  In every .env file we have the site as a code, so that we can provide site specific logic if we need it.  We have an environment so developments, staging production, and then below that comes all the special variables, constants that we need to set up the site.

So the main components.  First of all it is an auto-wiring dependency injector for those who don’t know what that is about it is specifically it gives you central control overall instantiations of your site so that you don’t have new this, new that all around your code base when you need to make a change to class you need to go through all your files to see where that class was used because that means that all your source files were directly coupled to that class so the injector makes it possible to just use interfaces and through the configuration of the injector say at run time which specific implementations to use so we have one central location where we can make these changes.

The service locater, it’s what we use to cross boundaries between plugins because there is often talk about the service locater being anti pattern.  It is quite something that you shouldn’t over use, but here with WordPress it makes sense because there is no point in WordPress deck where you can say this is the central point where I want to have my dependency injector instantiate the entire site.  This doesn’t exist with WordPress so we instantiate per plugin and use the service locater to bridge the gap between the plugins.

Then we have a config mapper so most of the objects, most of the services are heavily config based.  This basically means that the code only ever contains re-useable code.  Everything that is specific to one site, to one implementation, to one environment gets put into config files and at run time we can decide what config files to use.  I’ll show an example later on how this works.  It basically means that you can create code that is 100 per cent re-useable that you can just put in a composer package and require in every single site and when you instantiate it you pass it, the config file with the specific stuff for that site for the specific business logic that changes between sites.

Then we have log manager.  It’s basically just the place where the log is configured instantiated and where we can fetch a reference to a logger when we can’t use dependency injection because most of the time we just use dependency injection the logger gets injected through the instructor, sometimes in legacy, procedural code you can’t do that so log manager is responsible to handout logger references and make sure that we still have a control of everything.

Then there is a message bus which is still work in progress which is meant to decouple the processes from background tasks, from the actual web server work.

So auto-wiring dependency injector.  Quite a difficult word.  I tried to explain it bit by bit here.  First of all we have dependency.  Well, dependency is for a given piece of code dependencies are all the bits that that code needs to work as it is expected to work.  : The injector.  Injecting something means that the code will not fetch it in itself.  The code doesn’t call the function to get some valued code, doesn’t create an object to work with it.  The code just declares what dependency it has and it will just trust that the overarching system will provide the correct instance of it.  This means that whenever you use injection, you decouple your code from its dependencies so it can change dependencies without needing to modify the consuming code.

Then we have the bit auto-wiring which I don’t know if I pronounce it correctly, which means that this all is done recursively so basically when you have an object that needs dependency A and that dependency A needs dependency B and dependency B needs dependency C and D, when you instantiate the object at the top through this dependency injector it will check all of the dependencies instantiate all of them and inject them where needed.  That’s the auto rowing pot.

So this injector should the sole source of instantiations with a special qualification for all types that are not considered to be newable.  There are some newable types.  I print a bit of explanation below.  Whenever you have value objects like the daytime class for example it’s a value object these are meant to be newable so you wouldn’t use one shared date time instance across all of your projects.  You always create new instants whenever you need it and it should even be immutable so for all the types that are not considered to be newable like this date time object the injector should the sole source of instantiation so in this way you can for example get rid of the well-known singleton pattern where every class tries to make sure that it can’t be instantiated several times so when you use an injector you simply say that this class whenever it is used it is shared.  So, instead of always instantiating a new instance and passing that reference out it is instantiated once and every reference is to the same object.

It allows coding against interfaces so consuming code never needs to know about the specific implementations to use.  It only requires any object fulfilling the interface it needs and this means that you can rely on your code to just work when it interfaces – interfaces are contracts.

Then the default implementations can always be overridden.  This means that as long as the code just use the interface it doesn’t care about what implementation is used and we can through dependency injector at any point in time give it a different implementation and use that one instead.

So an injection example, I pass through this quickly because I am already running much later than I expected, we have an interface customer database and we have implementation work press customer database.

You see that this implements the interface and it uses as dependency WPDB which is not a class so the injector, we tell it that customer database is implemented by WordPress customer database and we say that when ever we need WPDB class we get it from the globals so whenever we now say injector should return customer database class it will return instance of WordPress customer database with injection WPDB already properly injected so it can use these customers because interface says it has get customer method, I mean just use that without needing to care about what implementation it is.

The customer dashboard here in this example it now takes the customer database so in this case when we make a customer dashboard it will inject the WordPress customer database inside of this database here.

The service locater each plugin has one or more service providers and the service providers they can provide services and I use this concept to include dependency resolution into this so that each service provider can have dependencies and it only gets activated and will register its services when all of its dependencies are met.  This allows us to just code all of our logic as if everything was available and working and the dependencies just avoid even activating code that is not meant to be run.

A quick service locater example.  Here we have again this customer dashboard.  The service provider it will have a name, it will have dependencies and provide a service.  So when the dependencies are met, the service customer dashboard will be registered and this means that we can just in whatever code tell our service locater, static facade to get service locater it should just get customer dashboard and we know that the customer dashboard in its interface has method render and we can just use it and we don’t care what instance what exact object we were getting there.

So with this dependency resolution, we have all of the service providers always come with dependencies as you can see here and whenever dependencies are not met they will just stay enqueued and not be active if all of the dependencies are properly met so this also allows us – for example, here we have a virtual service; it’s front-end, so this entire service provider will only get loaded on the front-end just by simply stating its dependency is front-end.

The config mapper is for the reusable business specific data that gets injected into re-useable code.  The config mapper takes the config files from several locations that might be within the plugin as default, config files, it might be site specific files, environment specific files, and it assembles everything it finds and will inject central config into plugin so we have same code but different configuration per site.

We can also configure environments for example on my local development I always make sure whatever could make changes to the database or send emails the implementation gets replaced by a {inaudible} object so instead of my Amazon email provider sender I have a null email sender replaced instead on my local environment.

A quick example of the config mapper when we fetch the config for plugins so-and-so we – on the site DT, in the environment development, it will first load the default, load the site specific configs, load environment specific configs and even load custom config if it finds it, and then moves everything into coherent whole and injects that into re-useable classes.  This means that our class can just use the code and assume that everything is already dealt with that might be site specific.  So in this case for our notifier here on 2 different environments, we might have 2 different config files so here is production config file, here is the development config file where the email sender interface is implemented by different classes and this means that whenever I use the notifier service I can just send a message and when I happen to be on the local, on the development machine, this message will be sent to the null email sender so it won’t be sent it all.

The look manager it’s not that interesting.  It basically appears PSR 3 user monologue which we inject everywhere to have logging accessible to all systems.

I quickly browse over this now to the example.

Basically I’ve reused the name space as the identifier for the logger and when it happens to have a configuration for specific name space it uses that specific configuration otherwise it uses default logger.  Here we have an example where we log in 2 separate classes we log the back logging and one in logging and the configuration has general configuration for deeply nested name space which both classes are part of that looks at the info level and deeply nested log space B looks at the back logger so when we run the method from class A it only creates the warning logging because it only logs at level info and for class B it will log the declass {inaudible} space to use a more granular logger.

The message bus is still work in progress.  It’s meant to properly decouple all business and background processes from the web front end.  It simplifies transactional integrity, so you can, you can create one event or one command and keep this command active until you know that whatever has been done, should have been done successfully or we try it until it is done.  You can use it as reach between processor and server, you can have a web request, call the service into an action and it doesn’t even notice that the message queue in the background has contacted the different server and run the process on a different server and returned to resolve it.

This is a short example of the Message Bus on the web server, we created a command on the business process server, we created a handler, we let it know to handle a specific command, it will split the process and the business process server will then be able to handle that and that command in whatever way it thinks necessary whereas, on the web server, it just does its normal rendering and is done with all the work.

The Message Bus separated between command and presents, event can have 0 or more handlers.

The results all the sides are still very different but all share the same fundamental code base, you can see in the admin bar we have the small back plugin, shows the service provider and active, means there are 64 service active and one enqueued.  Here are the simpler sites, they all share the same architecture, the benefits fast site on boarding, the new site can be raised upon this new architecture in the matter of weeks.  Sometimes we have been able to do this at a very short notice, we have very robust codes because it contains typing, hinting interfaces everywhere, which eliminate a lot of entire back family that is are not possible.  We have flexible code because of the interfaces we can even after months decide that, ah, that implementation that was not good, so replace it by a different implementation altogether, don’t need to make changes to any of the other code.

We have reusable code which means that when we fix one bug on one side, we fixed it for all the sites.

We have fast error detection because of all the logs, often times when we notice an error, it is a matter of minutes to pinpoint the exact location of where it happens and and fix the bug.

I will jump over this, it is basically, it is a summary of how we managed to meet our business goals.  I will give you a link to the slides, so you can read through this at your leisure.  I expressly built this, I am running late, you can read it on your own.

Yes take aways, use config files it is very important that you are able to make a difference between reusable code and business specific code, for business specific logic.

Use dependency injection it completely changes the way you build, launch the code constructs and mean that is this, there is a an initial investment you need to make and the learning hurdle.  After that the code is easier, much better to deal with and much easier to maintain, you can do it faster and cheaper also which is good for your clients.

Use WordPress as a web server and application framework that can communicate with other specialised server to do the work.  Never attach too much to the web front end that gets rendered to the browser.  That is not the way it is meant to be used.

If you want to try it yourself, I wrote a small blog post here, where you can see an overview of what I have put into Opensource, I want to Opensource the end to end architecture, to provide a scaffolding tool, so it can use architecture you can use on your own sites by simply using the plugin.

It is not finished yet, some components already Opensourceed, I am working oen the rest, on this site, overview and subscribe works for when you want news for when it is released or scaffolding tool is ready, there will be no spam on there, but just about specific architecture.

Thank you.  (APPLAUSE)..

SAM: Thank you Alain, it is lunch time, no time for questions, head straight to lunch.

Is this on?  Yes, there we go.

So yes, if you remember for lunch, stand on the right, queue on the right.  If you don’t like standing in queues, go and see some sponsors and go, there is lots of them over in the other building as well.  But lunch is in the Graduate Centre which is back through this way and then upstairs.  I am sure you will see the queue! Great.  Have a good lunch.

NEW SPEAKER: Thank you Sam!