My job title is officially backend engineer but this is pretty vague. I wanted to explain a bit what I do on a daily basis. First, it is a good reflective exercise for myself. Then, if readers like what I do, maybe some of you will want to join us!
If you’re in a hurry, this is a quick recap of the main points of the article:
Of course, all this only reflects my own beliefs and perceptions. I’m happy to discuss anything you read here and encourage you to say Hi! on Twitter.
Disclaimer: Sometimes I’ll say I, sometimes I’ll say we. I’m not doing anything completely by myself. All of what I do greatly relies on and involves the work of the team.
I think the easiest way to share what my job is like would be to tell the story of the main areas I work on. I’m in what we call the finance squad so my work is mostly focused on how the company accepts money from our customers, how we keep track of it, and how we dispatch it to our various partners. The scope is a bit broader than this but it is a good starting point…
There is no particular order here.
We support credit cards via Stripe. We also accept Paypal in some countries. We used to use other providers, so our integration aims to be provider-agnostic. This integration style works quite well and I can’t thank enough the people that set it up.
I’ve been busy keeping up with regulations. For instance, I worked on being PCI-DSS-compliant. Thanks to Stripe, becoming compliant was a smooth process. We use Stripe Elements and Sources so we don’t have to handle sensitive credit card information.
You may have noticed, the European Commission released the PSD2, a directive aiming at securing online payments with two-factor authentication. All the payment providers operating in the European Commission had to work really hard to be ready for this. Account managers at Stripe did their best to help us, they made themselves very available, invited us to workshops, and so on. On my side, the main challenges were:
The new (Payment Intent and Payment Method) APIs introduced some pressure on our existing abstractions. For instance, the system was designed in such a way that a call to the payment provider should either end up in a successful transaction or a failed one. This change introduced another state: pending_action when the bank requires a two-factor authentication to complete the transaction. How to handle that? What’s the impact on our apps, on our data, etc?
In both situations, when moving to Sources and then to Payment Methods, my role was to adapt and extend our existing abstractions to isolate those new concepts, specific to Stripe, as much as possible from the rest of the system to finally plug the new APIs in. I attended Stripe’s workshop, worked with other squads to clarify what the impact was going to be on our product, did a lot of Q&A, monitored the releases, challenged our testing strategy for Stripe’s integration, tuned the integration during the releases, fixed bugs and so on.
As I mentioned, in addition to receiving money, we also dispatch it to our partners. Each payment must match certain transactions between actors.
We need to pay or debit each of those actors. The squad maintains and scales the payout system. We do batches of bank transfers to the owners almost every working day. The system has to decide what we should pay to whom and what need to be reviewed by the finance department. With the company’s growth, this part of the system was often subject to technical and operational scaling issues.
In order to be able to expand to other countries easily we integrated Stripe Connect. With that came a lot of work around KYC (Know Your Customer) compliance. We needed to collect identity documents, billing information, legal information, to add rules in the payouts, etc. Introducing Connect triggered the need to introduce new concepts and internal processes. The impact of such a thing is far-reaching: we needed to make changes to our on-boarding flow to communicate with our partners, ensure that customer support were ready, etc.
It is better to identify the consequences of the new constraints as soon as possible. Fortunately, this is a team effort. Most of this effort is done by the product owner; coworkers will help through kick-offs and reviews too. I need to be involved in the product in order to help to spot those consequences. When we miss something, it is no big deal, we find out and adapt.
Our application and its accounting system was centered around a single concept: the rental. The assumption that we’re dealing with a rental had firmly taken root throughout the app and coupled things together.
With Drivy Open, we were actually selling something else entirely: subscriptions to a service. Whilst Open was still a startup inside the startup, we operated a separate system for everything. Because it was a great success, we incrementally merged Drivy Open to the main application. Everyone worked hard to make Drivy Open the future of Drivy. On my side, it was a challenge to untangle all the rental-coupled logic that was everywhere in the application in order to build more flexible sub-systems.
To accomplish this, we needed to establish a long-term vision. Refactoring a whole system in one go would be a very big investment. Also, by doing it slowly, we can learn along the way. Each feature becomes a good opportunity to advance toward the vision. If a feature doesn’t fit that vision well, both the vision and the feature get the opportunity to get reworked.
Not rushing through the vision could get frustrating. It would solve most of our current issues, be more efficient, easier to maintain and so on. I said frustrating because moving on to that vision is definitely subject to the needs of the company. As long as the users are happy, and we can move fast enough, we don’t need to rush into building it. Thus we need to be patient. We wait for the opportunities to make progress on it. When planning and discussing with the squad, we can elaborate that vision together. We also have to make trade-offs between short-term and long-term investments.
We’re dealing with a majestic monolith. A given functional scope matches a certain area in the code. Many scopes require interaction with the payments, or with the accounting. Team organization evolved quickly. It evolved quicker than the code itself. We’re organized in cross-functional teams with a given scope (inspired by Spotify’s squads).
Many squads will at some point need something from the finance squad. It could be some assistance for a given task, or it could be a feature that isn’t yet supported and that we’ll need to add. This brings its fair share of effort in term of planning prioritisation. It also encourages interaction and cross-squad collaboration which I enjoy. So in addition to working on the system itself, I’m often in a support role to other squads. It’s very rewarding, but sometimes this dependency becomes a bottleneck for the team’s outcomes. Fortunately, the long-term vision solves it!
To avoid that bottleneck, we aim for a modular monolith. We need to expose clear boundaries of what the subsystems are that the finance squad must provide, in term of scope and APIs. I, obviously, would like to reduce our scope. But wait, it’s not that simple… Features are built on the scope I would like to get rid of. It means than if we move a boundary between A and B, the scope of subsystem A will shrink and leave a hole between A and B. So my job would be to deprecate that scope; to prevent further features to rely on it. Then, thinking about how to fill the void for the existing features. Maybe by adding stuff to B? Maybe by filling the void with another subsystem? Maybe the void belongs to subsystem A after all?
To achieve that, I must stay alert, keep the vision in mind, say no, offer alternatives, make it clear that it is a long-term investment, … Deprecating scope provides no short-term business value. But, the more the team grows, the greater the value of this investment when it pays off.
One of the most important responsibilities of the squad is to produce reporting for the finance department to use. Seems boring, doesn’t it? Not to me it doesn’t! The needs of a finance department can create a lot of constraints that are fun to play around with.
Over time, this reporting pushed the system to be more robust. At some point in the past, the finance department was using a single report with all the figures for each rental the company did, since forever. In order to get a monthly vision, the technique was to calculate the difference between the report from month N and N-1. This clever way was found in order to cope with the fact that some changes could happen to past data, for instance an adjustment on an old rental. Because of the volumes, such a technique wasn’t working anymore. The extract was too big to work with, and too long to generate.
It took months of work to take the system to a point where it could generate many small monthly extracts with only the data of what happened that month. We needed to find out a clear way of cutting exports, to detect modification of the past, to find out ways of avoiding those modifications, to be confident that we all we did was equivalent to the previous technique, and to do the migration correctly. We went from rental-based extracts to invoice-based ones. Maybe this doesn’t seem much to you but I’ve faced lots of technical implications while trying to figure out reporting.
.png "Invoice-based approach")
A good thing is that I mostly care about keeping the accounting system sound. The reports are, in the end, a dozen of - relatively simple - SQL queries that don’t move too much.
Alongside the big projects and the long-term vision, there is a lot of other features that I do. Those are smaller and can be done in less than a few days. Usually those are features that improve the team efficiency. Here are a few examples: new administration tools, financial processes that get automated, or updates on existing facilities.
What I like about those is that, in contrast to the other features, they provide predictable, easy, and quick satisfaction. A feature is usually done quickly, shipped, and instantly useful. Many of those features are internal tools. This context is pretty easy to work with:
I also like having those small features because they have a direct impact on my coworkers workload so I can be a hero :-)
We are dealing with a database that has been holding the company’s data for almost a decade. Still more challenging, is the fact that we collect data from our providers, unstructured jsons fields, and even others companies’ data that we imported. Given the timescale and the diversity of sources, data isn’t always consistent with today’s happy path. We’ve got some safety nets in place to help us gain confidence that the data stay as we expect over time.
Still, before adding a feature, we often need to dig in the data a little bit. It helps understand the volumes we’re dealing with, to be sure we have no holes in our thinking, … We don’t have a data-analyst like other squads could have thus we need to do that digging ourselves. This routine is very useful. It allows us to know the data well, and to keep SQL skills sharp.
We’re also blessed with a product owner that understands, tweaks, and writes SQL requests. When I have an unexpected result, she can proofread my requests and spot missing bits!
Writing SQL is a must as we sometimes have CPU-expensive logic that we need to translate from Ruby to SQL for efficiency purposes. With our growth and with the diversity of our customers, the performance truly matters. We’re constantly trying to find better ways than duplicating logic from our application code to SQL. To do that, many solutions are used such as introducing immutability, storing more information in the database, building specialized tables, caching, and more.
We also have a dedicated data department that makes available hundreds of tables with all the information you can imagine. We’re in the process of gaining the ability to maintain our own ETL pipelines in order to suit our specific needs. I’m really excited by that prospect!
I didn’t mention the way we manage bugs, how we deliver software, how we support the rest of the team when they have a question about the system, … There are many aspects of my job that I left out here but I think I pictured the biggest part.
If you find all this interesting, if you want to know more or dig into specific points, I encourage you to reach out to me. It would be my pleasure to discuss all this even more!