Let me start with the TL;DR:

If production breaks, it is not the fault of a single person but a faulty process.

The story, all names, characters, and incidents portrayed in this blog post are fictitious. No identification with actual persons (living or deceased), places, buildings, and products is intended or should be inferred.

We are professionals, nothing of this would ever happen to us.

Question 1

One of our customers using Shopware called me and said, they would like to have a trailing / on every URL. I thought that is no problem, opened an SSH terminal and added a few lines to the .htaccess on production. Tested it and worked. Two hours later the customer called and told us, that it is not possible to put anything into the cart anymore.

What happened?

Shopware uses its API to add items to the cart and the API doesn’t like a trailing /.

How could that have been mitigated?

On our “safety” checklist we have a few things:

• git
• deployment pipeline

We don’t have tests though.

But tests wouldn’t have helped. Why you ask? Because some dumb idiot (me – this is a fictional story!) accessed the production system, ignored all processes and broke the system.

So the simple answer to “How could this have been mitigated” is: Don’t allow access to the production system. (But it is important to state the downside of this rule: You can’t fix anything fast).

Question 2

Customer calls: “No secure connection to the server is possible”. Ok, obviously something with the SSL certificate is not working. Yesterday I read an article about an expired R3 certificate by Let’s Encrypt, which breaks one or another connection. And checking the certificate shows: Yes, it uses the expired R3 intermediate certificate.

Easy fix: Create a new certificate. Connect via SSH on the server (no, that was not the problem! :D) and run certbot for a new certificate. Testing, done.

A little time later the telephone rings again, this time the admin asks, why all websites are down. I told him, I updated the certificate and tested the site after it, it is online – still.

What happened?

The certificate contained 12 domains before I updated it. After the update only 1 was in it.

How could that have been mitigated?

The bad thing: If I hadn’t access to production I couldn’t have tried to fix it in the first place.

The easiest to avoid this are two things:

1. Know what you are doing (I’m a developer not an admin).
2. Use a TLS monitoring service. The one we are using is oh dear.
3. Automate your certification updates (ok, this was a special case)

Question 3

Customer calls, their e-commerce site is down. Investigation starts. Login with SSH is already full of errors – hard disk full.

We already had a discussion with the admin, that the old Shopware version we use has a cache leak and we need to update(!!), but it is end of November and the customer prefers to do it next year. Ok. The cache writes about 90GB a week, we have 160GB disk, about 120GB free, so it is no problem, if we clean the cache each week once.

But now it happened, the cache flooded the disk and we don’t now why. First thing: Clean the cache and the store is online again. And now check what happened. We only have 40GB disk free. Why? Good question. With each deployment we create a new cache, which means we “archive” the old cache and don’t delete it. So with each deployment our disk space decreased and we didn’t know this.

So we installed a cronjob which not only deletes the cache ones a week, but once each night – just to be sure – and not only the current cache but all caches of all releases. And warm the current cache up.

How could that have been mitigated?

Simple answer: Hard disk monitoring. You can monitor a lot, but free disk space and used inodes* is a good start point. Used memory and CPU utilisation can be the next – but I’m not expert no this topic.

• inodes are the entries of files in the file system, so you can find each file on the hard disk. It can happen (e.g. if you write TONs of small session or cache data) that you have a free disk, but you are out of inodes, then you still can’t create new files, because the file system can’t remember the position anymore. Like a database when you run out of AUTO_INCREMENT ids.

Things you want to have (and are not always easy to sell – I know)

Software, tipps and services we use(d) to mitigate our risk:

• git, GitHub, bitbucket, gitlab (VCS, pipelines)
• Tideways (Performance and Error monitoring)
• koality.io (HTTP/Ecommerce Monitoring)
• ohdear.app (HTTP/SSL Monitoring)
• Zabbix (Hardware Monitoring)
• Deployer (Deployment with PHP)
• PHPUnit – (readonly) backup (this is possible with
  B2)

And some ending notes:

1. Think hard about what you want to offer your customers for free, because great services cost constant money and customers should pay for it, not you. Yea I know, obvious, isn’t it?
2. Developers want to play. If they can’t they use production. Talk with your developers about ways to harden your system. With logging, monitoring, tests, better processes.

And the most important thing you always need to keep in mind, when shit hits the fan:

If production breaks, it is not the fault of a single person but a faulty process.

It is not the responsibility of a developer to make up bad business decisions. But it is our all responsibility to tell people the weak points of our processes if you spot them.

A lot has been written about the PHP Community. How it is an integral part of why PHP is not dead. About who is part of the community. About how it helped people become better developers. About how people improved because of the community.

And most of that is about the PHP language. Almost all of us came for the language. But what makes the PHP-Community so special for me is that a lot of the people stayed for the friends.

Friends that are beyond “everything is fine”. Friends you can count on and friends that actually take care for each other.

Whether that is people enquiring about the health of others just because they were suddenly not that active on Twitter anymore. Or people making plans to more closely engage with someone to help them through a rough time. Up to realizing that there are people talking about their health issues openly in chat-rooms or in blog posts.

And I think it is great that we as a community stand together in good times as well as in bad times.

We for sure have our differences. And we can fight! Over almost nothing!

It starts with Tabs vs Spaces or VSCode vs. PHPStorm, and doesn’t end at deprecating dynamic properties.

We are also capable of toxic behaviour. There are some people that are challenging the community in every way. And there are even some that we had to say “no” to. To be able to become more welcoming and more empathic.

And especially in bad times, it doesn’t matter whether we have a background in WordPress, Laravel, Symfony, Drupal, PHP, or whatever else we started out with. What matters is that we are not afraid to say that not everything is fine. And that there are people that take care of each other beyond mentoring through the next development challenge. People that help each other out. No matter what!

And it gives me a tingling feeling that I can be a part of that.

Thank you all for providing such an overall safe space.

I spent my whole career writing code. I consider myself as a Jack-of-all-trades – I know a bit of everything when it comes to development, CI/CD, SEO, UX, project management, and probably many other branches. While this is very useful in some situations, I saw my limits much quicker when it come to learning some things in-depth.

I even started to search for a job thanks to which I could move from writing code all day to something where I could use my broad knowledge. I imagined myself as a project manager with a technical background. Luckily, out of the blue, I got an offer to work as a WordPress Ambassador at Buddy. I’ll be honest – back then I didn’t have a clue what a person like this does, also I was a bit skeptical because I was sure that it will require talking with people (and I learned PHP so could limit those contacts).

It’s been almost a year since I started working in Developer Relations and you know what? This is a true dream job for people like me. First of all, I don’t have to code all day, but I still get enough chances to do so. I learned that I love talking with other people – this was the biggest surprise for me. I also discovered that I’m a pretty solid webinar host and organizer.

What is most funny about this is the fact that all of this started with an “echo ‘hello world’” many years ago. So if you feel, that you are getting tired of coding all day, but you still want to use your technical skills, maybe DevRel is also for you?

Or, how I learned to treat myself with kindness.

Last year, a call for entries to 24 Days of December came out, and I wanted to write something discussing PHP documentation to encourage more contributors, but … the words just weren’t coming to me.

Last year, I had started a new job a couple of months prior, after being unemployed for a year and a half. I was adjusting and acclimating to my first full-time dev job. I was assigned a project that had a deadline of end-of-January, and I had to quickly scale up learning the code I was working with and how to solve the overarching problem. I felt the pressure because I wanted to succeed. I wanted to prove myself, both to my new employer and to myself, that I could do this.

You see, in my previous role, while development was part of my job duties, it wasn’t a focus, and it wasn’t the only duty. I tended to pick up programming related tasks, push them until I was nearly complete, and then… become stuck. And I had very few options to unstuck myself. I wasn’t an experienced developer, I had taken a handful of traditional, introductory courses, and had some basic development experience, but I’d regularly become baffled why something wasn’t working, struggled to make examples that I could share with others online to ask for help, and I didn’t work with any other developers who I could ask questions. I regularly felt stuck, and regularly felt incapable. Impostor syndrome was borne from the weight of several unfinished programming projects over the years.

I had to complete this deadline to show myself that I could figure out programming problems and write code. And I did.

But the cost was burnout and the inability to work on other things for a while.

A common indicator of burnout is feeling lack of control over one’s life. The feeling of ambition and motivation turns into pressure and stress. “I want to do XYZ” becomes “I have to do XYZ.” Sometimes, this stress is necessary to complete a task, but there’s the subconscious promise of reprieve, a break, afterwards. That this pressure is temporary, and that I will have time later to recover and care for myself. Except, at the time, I didn’t understand this. I wasn’t able to identify I was already burnt out. I had become so accustomed to feeling overwhelmed in everyday life, I thought it was “normal,” or “normal for me.”

To give a short summary: I was bad at keeping my apartment clean. There are underlying reasons that are out-of-scope for this topic, but I struggled to motivate myself to keep my apartment in a state that provided me comfort. It was common for me to wash individual dishes because my cupboards had no clean dishes. It was common for me to take a shower and have no clean, dry towels ready. It was common for me to wait until my trash bags had built up before I wheeled out my trash can for waste services to pick up. You get the picture. Cleanliness in my living area was an overwhelming struggle, and another piece in my burnout puzzle that had to be resolved before I could begin the journey of caring for myself.

Year after year, I failed to clean my apartment. I forgot what “clean” looked like for me. It was the project that I would always procrastinate, avoid, or push off to next month, next season, next year. Like mental health, I realized this was something I couldn’t accomplish alone. I needed help. I contacted a cleaning company to do it for me. And it was worth every penny. I hired them to clean once a week thereafter, to build a stable foundation of comfort, of safety, in my mind.

There were objects in my apartment that I incorrectly assumed I wanted because other people had these items in their homes. I came to accept that what worked for others, may not work for me, and that’s okay. Having these items in my apartment occupied space, in both my mind and my living area, and increased my stress because they were additional things to clean or account for while cleaning. So, I got rid of them.

With the weekly cleaning, I adjusted to the feeling of having a clean apartment, the happiness that the feeling of a clean apartment brought me and built an image in my head of what my apartment looked like cleaned, and what was necessary for me to maintain it. I canceled the cleaning service and started doing it myself. Around this time, I came to learn the definition of self-care. I knew the word from therapy, but my idea of self-care actually turned out to be coping behaviors.

Self-care is taking the steps necessary to ensure my state of feeling happy, which happened to include a clean apartment. I still struggle with motivation to clean, but reframing cleaning as self-care was an epiphany. I started to realize and understand what I needed to do to feel happy, to attempt to prevent burnout from overtaking me again.

One of the hardest challenges has been acknowledging what I’m able to complete in one 24-hour day. I make a to-do list that I want to complete, but sometimes I overestimate my energy level for that day. I’ve learned to accept that I may not complete everything I intended, and I’ve learned how to forgive myself if I’m unable to finish, to not let that further weigh on my mind. Learning the act of kindness towards myself.

I tend to view two-to-four days as one really long 48–96-hour day with breaks for sleeping in between. I struggle with the flow of time, and often think I’m able to accomplish more than I can. But working towards following an internal day-to-day schedule is helping. Journaling out my hopes, dreams, and aspirations so that I can form a rough plan towards achieving those goals, is helping.

I also have days where I don’t feel up to working on anything, and that’s also okay. I spend those days taking care of myself, coping with life, and recharging in the hope that tomorrow is a better day. I remind myself that what I’m feeling in the present is temporary, and that it will pass. Another day will come that I will feel ready to take on the world, or at least the items on my to-do list. 🙂

In a recent discussion about the PHP ecosystem, I was told that most Open Source maintainers and contributors are paid for the OS work that they do. Nothing could be further from the truth! While a very few contributors are paid for their work, a few are able to build businesses around consultancy for their projects, and some accept the occasional donation toward their work. But the vast majority of maintainers and contributors work on these projects in their own time, and don’t receive any form of recompense for all the hours that OS takes up in their lives.

The discussion itself was all about the pace of change of PHP, and the extra work that deprecations and backward-compatible breaks (documented or otherwise) create for OS maintainers that want to keep their projects working with the latest versions of PHP. Maintainers are pressured to ensure that their projects work without issue with every new release of PHP, sometimes even before the official release date, while being pressured by other users to maintain backward-compatibility with older versions of the language. It’s a situation that can create a lot of stress, and a lot of extra work for them as they try to find a balance between the two demands, even without the general maintenance work of support and bugfixes, and the requests to add new features to a library. And there are so many Open Source tools and libraries that we use every day in our work that are maintained by unpaid, unthanked individuals.

At the end of last month (and long overdue) the PHP Foundation was announced, a non-profit organization whose mission is to ensure the long life and prosperity of the PHP language. Backed by funding from some of the biggest companies involved within the PHP ecosystem, and from individual donations and pledges, it plans to pay developers who contribute to the PHP core for their work. It’s a very positive step to ensure that the language continues to evolve and improve; but the core of the PHP language is just one of the many Open Source project that we rely on in our daily work. But the PHP language itself isn’t the only Open Source project that we use every day in our work.

We use tools like XDebug for debugging. Our CI pipelines might include PHPUnit or Pest for testing, PHPStan or Psalm for static analysis, PHP CodeSniffer for ensuring that our work conforms with coding standards. We build our applications on frameworks like Symfony or Laravel or CakePHP. If we’re doing an upgrade, then we could use Rector. We use libraries such as Doctrine or Monolog. And many of these tools and libraries are dependent on others that we might not even realise we’re using. Just take a look in your composer.lock file to see all the libraries and tools that we’re dependent on for our work.

But even behind all those, there are also the tools that make up our development ecosystem. We run composer (and connect to packagist) for managing our dependencies. If we have a CI pipeline, we’re probably using Shivam Mathur’s setups to prepare PHP for that pipeline; while others like Remi Collet maintain RPM libraries for PHP. If we want to test small snippets of PHP code against different versions of PHP (or even a new RFC branch), then we might use a site like 3v4l.org.

And don’t forget the PHP documentation. Even outside of the software tools that we use, there are also blogs posts and tutorials that we read, podcasts that we listen to, or streaming events that help us learn how to work with those tools, or to improve the way that we work.

All of these make up the PHP ecosystem that allows us to make our living as developers. So this December, when we’re celebrating the holiday season and the turning of the year, it can be good to remember all those individuals that look after the tools and libraries that we use in our working lives, and perhaps to give them some show of appreciation for all their efforts. Just spend some time thinking about all the work that Open Source contributors do each year to ensure that this ecosystem runs smoothly, and perhaps (if you’re feeling generous) make a small donation toward some of these projects; or hunt down a contributors wishlist, and send them a little “thank you” for their efforts.

Given that I don’t live in a US-centric country, but I’ve watched enough US television over the years, I know about the concept of thanksgiving. In South Africa, Christmas day is usually a time to share gifts, spend time with family, and be thankful for all the good things in the year that has passed.

This year, I am thankful that my family has made it through another year of living with the pandemic.

I am thankful that the career I stumbled into 16 years ago continues to grow and thrive and present new opportunities.

I am thankful that I have a job that allows me the ability to work from wherever, whenever, and schedule my work so that it suits my life, and not the other way around.

I am thankful that 26 years ago a Danish gentleman created a scripting language for the web, and then spent the next few years putting work into the server stack around that scripting language, a key piece in making it the dominant platform of the web still to this day.

And I am thankful for all of you, my extended PHPamily, some of who I’ve met in person, some who I only interact with on social media or in chat channels. Every day I learn from you and I’m challenged by you to become a better programmer, writer, and person.

If the last two years have taught me anything, it’s that we need those connections, especially when working in remote environments, with little social interaction.

Without the love and support of my IRL family and the interactions with my PHPamily, I probably wouldn’t be here today to be thankful.

Once upon a time, the web was a space where only the geeks could play, but because of the geeks choosing to share, PHP has created an environment that gives almost everyone the ability to share stories and information, regardless of our technical ability.

Once upon a time, to create a website folks needed to understand HTML and hosting and a lot of technical bits (which are wonderful and important and shouldn’t be trivialized) but what we have now is the ability to put our thoughts onto a screen and share them across the world without having to consider the infrastructure that makes it possible.

And that is something that fills me with joy and gratitude.

And so, to all who have done anything with this language that began as a way to program a personal home page, well done, and thank you.

December is a time of waiting for many people. Children await the arrival of Santa Claus and gifts. The world waits for the arrival of winter and a return to the daylight we all crave. We’re anticipating the beginning of a new year. For me and my family personally, we’re awaiting the arrival of a new baby. December brings about anticipation, and with it comes waiting.

Software developers are used to waiting. We’re often asking for time to improve the code our employer charges us to maintain, only to hear, “we need to wait on that.” We beg desperately for upgrades to systems, servers, and technology, only to be told, “that’s not in the budget this year.” We seek to upgrade to a newer version of PHP, only to be told, “that will have to wait.”

Even the most energetic and positive software developer can look at the wait as a time of distress, where waiting ticks off the seconds, days, and months of useful life our software still has. We know that waiting to upgrade will only push us farther behind until upgrading may no longer be an option.

What is a developer to do?

The answer is to embrace the wait. So many things about software development fall outside our control: when we’ll have time to refactor our code, when we can upgrade our servers, when we can buy new technology. But that doesn’t stop us from doing the tasks necessary to prepare for the day when those things are possible.

Children do not sit idly by and wait for Santa Claus. They write letters, bake cookies, and decorate their homes. We do not sit idly by waiting for winter to come; we prepare for festivals and parties and spend time with loved ones. We prepare for a new year with new year’s resolutions. My wife and I are preparing for our new arrival by buying clothes and setting up a nursery.

Just because you are waiting does not mean you are powerless. Waiting gives you the time you need to prepare: bringing your code up to modern standards slowly over time and anticipating the arrival of new technology or new hardware. You need no one’s permission to upgrade your development environment, to stand up Docker, or to write tests. These things are part of your job and well within the realm of your responsibility and power.

As we enter this time of waiting and anticipation, I ask you: what are you waiting for? And as a companion to that question, how will you prepare while you wait? Will you be ready when the time comes? Will you be prepared to act?

Sometimes people wonder how I got into PHP. Well … it was almost by accident as I never really learned PHP. It has already been three years… !

Not everyone loves this language, but I quickly realized that PHP is much more than a language, it is a community.

Like many languages, PHP has a mascot, the elephpant, and there is a culture around it. People buy them, win them in contests or conferences, exchange them, import them from the other side of the planet … There is even a register of all known elephpants

I have to admit that it is not a cheap collection, but a very funny one.

In my case, as I love handicrafts, the elephpant has been a source of inspiration.

Some years ago I made my first elephpants in crochet which I am giving away this Christmas. It is a way for me to share what I love doing and get in touch with other developers of this beautiful community.

In France, we are lucky to have the AFUP (french PHP users association) that brings together PHP developers to share experiences. Last year, we did our first secret santa, and the idea was to offer a gift that was blue or had an elephant, in tribute to the mascot, and I did this lovely multicolor elephpant in cross stitch

At Forum PHP, the biggest event organized by the AFUP, I had the opportunity to talk about the need to combine features and technical vision. In the two days I’ve been in Paris, I’ve seen many elephpant exchanges ! Conferences are also the moments where the community comes together.

On this occasion, I even won a huge collector elephpant by doing some improvisation on the Karaoke Slideshow.

Elephpants come from all over the world and give visibility to versions, frameworks, companies, events, and even women ! A PHP women elephpant was given to me recently which has an especial meaning for me as there are not many women in tech.

And you, when will you start your elephpant collection ? Feel free to make your own creations, imagination is the only limit !

In the early 2000s, I was studying at the University of Cape Town. My undergraduate degree was initially computer science, then information systems, and during that time I learned multiple programming languages, Java and C++ were the main ones. While I enjoyed the structured nature of C and the libraries of Java, to me it was a frustrating experience, in particular, the (at the time) long compile times on KDE, and the endless debugging after compile time. While I enjoyed creating something from nothing in code, I didn’t find it by any means to be “fun”.

Eventually, I moved over to C# during my postgraduate degree on advice from one of my lecturers, and there I found that C# balanced fun and structure really well, and to this day it’s still my favourite language – but it’s not the most fun I’ve had 🙂

Towards the end of my postgraduate degree, I encountered PHP. A few classmates were developing their year-end project in PHP and they showed me how quickly you could get up and running, and with NO COMPILE TIME, I was sold.

PHP quickly became my choice of language, and while some folks mock it for not being a “real” language – whatever that means – it can be strongly typed and structured if you choose – but more importantly, PHP has enabled the innovation and explosion of the web, largely through projects like WordPress, CakePHP, CodeIgniter, and Laravel. These projects all had a low barrier to entry and made writing software accessible to the masses.

Hacking away through a PHP file, mixing classes and inline code, functional and object orientation, and discovering the joys of MVC frameworks – you could really move fast and break things.

I still love PHP, it’s going to be around for a long long time 🙂 It’s just so.much.fun!