It’s incredibly hard to work with someone new to the company, it’s even harder when a decision maker joins the company.

Back in January Maksymillian Majer joined the RLS CTO Coffee and asked about Tech Debt. It quickly became clear this wasn’t a run of the mill tech debt situation so I DM’d him and offered to spend an hour digging into the unique situation.

We spoke two weeks later. He’d been at Nursa for almost five years and helped build the engineering org, but he was having trouble in the latest reorg. Maks had trouble understanding what his VP was trying to accomplish or how to collaboratively bring his years in the company to the table to accomplish it. He was facing an information asymmetry challenge that was driving a lack of meaningful communication. They were stuck in Differing Realities.

After realizing the problem I asked Maks how he’d approached working with his new VP and he explained that he was bringing project proposals to the table, seeking out a green light and then pursuing the projects. This kind of approach is difficult when there is a high information asymmetry between the decider and the proposer; one party is always unable to provide a critical lens.

Until all parties can look at the same information and agree that it says the same thing it’ll be impossible to make the best decisions. To overcome this issue we need to build bridges between the two people’s perception of reality.

Getting Practical

From Maks’ perspective he’d need to stop telling and start asking. I coached him through a couple of ways he could do this:

1. Vulnerability & Curiosity - “I’m looking at a handful of possible paths forward and realizing that I don’t understand what I should be optimizing for or what information you’ll need to make the necessary tradeoff decisions. Can you help me understand the objectives you’re trying to achieve?”

2. Curiosity & Support “If I were to bring you three different possible paths for tackling our technical debt, what data would you need for each path so that you could make an informed decision?”

3. Curiosity & Support “I’d like to help you with KPI X and Y, but I’m uncertain how to help. I’ve been here five years and built the core technology. Can you share a few of the challenges you’re currently facing so I can propose a couple of possible ways I can help?”

The goal behind these questions was to give Maks a way to get the information he needs to find projects that would move the needle the exec team is watching.

During the drafting of this post Maks told me:

I’ve started working from exec goal down to technical implementation instead of taking a guess at technical solution and asking my VP to evaluate whether it’ll suite their goals. This change has been so profound that it was positively remarked upon during my performance review.

I like asking questions, if you’d like help navigating a similar situation, send me an email at luke + substack at luke mercado dot com.

Subscribe now

Share

You ever wonder what makes a good refactor? Ensuring better problems.

Last week Evan Hu and I were out for a run talking about an in-flight refactor. His company (Shortbread) is an independent, AI scaled, romance comic publisher. Their business depends on delivering hundreds of new pages a week to their customers. Evan built an AI boosted comic creation system that allows their artists to scale far beyond the industry norm. This system has placed inference in the critical path for their artists and any downtime dramatically impacts their ability to deliver and risks leaving their customers unsatisfied.

As the Co-Founder CTO and sole engineer on the team, every time something breaks he’s forced to fix the problem and temporarily abandon growing the business. The routing layer powering the comic creation system had been reliable for the last four months, but during a trip to Spain it snapped as it encountered hidden infra limits.

So he embarked on a refactor.

As we talked through our run it became clear that this refactor had the potential to ensure the business scaled through its next growth phase. Unfortunately, as it was currently planned, it would still have the same failure mode: A surprising and frustrating crash at the most inopportune time. So I asked him if he’d think the refactor was worthwhile if it blew up out of the blue four months from now. “No.”.

One of the most difficult things about scaling any business is knowing how to build things such that you can stay proactive and rarely, if ever, are forced into reacting to a critical fire. This is the fabled art of making complex problems disappear. One way to do this is to detect their deformations and respond before they snap. I like to end my runs with coffee, so as we sat down at the cafe I asked if he’d like to explore a better system.

Deformations & Hidden Requirements

With the goal of stability and better failure modes in mind I pushed the conversation through the following questions:

1. What is the amount of downtime per-week of the current system?

Like most early startups, this information wasn’t readily available, so we decided the new system wouldn’t launch without tracking in place, likely using their existing DataDog instance. We’ll talk more about metrics further down.

2. What is an acceptable amount of downtime in this system per-week for the business today?

In every business this is ultimately a function of the business’ sales cycle. Shortbread sells weekly subscriptions paired with a weekly publishing cadence. If they miss a single publishing deadline they fail their customer’s (rightful) expectations. This means the system is unusually sensitive to downtime for a company of this size.

We pegged this number at thirty minutes/week.

3. Assuming the deployment goes well, when should this system be re-examined so that you can stay ahead of any impending failure?

To answer this we explored what prompted Evan to rewrite the system rather than simply kick it until it turned back on. The core driver here is that the bulk of their artists are in China and kicking the system wasn’t a matter of a slack message and a shell script. Any outage in the system meant, at best, lost sleep and sleep deprivation for the rest of the week as he tried to recover. At worst it would mean the artist team lost the day’s work and new comics wouldn’t go out on time, risking customer attrition. A while back I’d heard a great question about working across timezones: Where are you going to put the ocean? It looked like the ocean was threatening to drown Evan.

The realization that the goal wasn’t really to improve system resilience but to preserve Evan’s productivity, revealed a host of hidden requirements. We codified these as:

1. Self Healing: To minimize interrupting Evan’s sleep and the artist’s work the system would need to be more resilient. It should be able to self-recover from a host of likely issues such as network failures, dead inference workers and stuck jobs.

2. Incident & Downtime Tracking: Simply tracking downtime wouldn’t be enough to know if the system is starting to [deform or about to snap](https://www.aframeworkforthat.com/p/when-and-how-will-it-break). We also need to track incident rates since each incident risks waking Evan and ruining his week.

3. Action Driving Alerts: Evan shouldn’t have to wait for his team in China to tell him there’s a problem, the system should be able to alert when the day’s work is at risk or when the system appears to be deforming under load. We decided on setting three distinct types of alerts.

   1. An intervention alert when the system slips above our target incident (2) or downtime (30 minutes) in a rolling seven day window.

   2. A deformation alert when the system is running near (50%) our target incident or downtime per rolling seven days thresholds for longer than fourteen consecutive days.

   3. A snap alert when the system is running 3x above our target incident or downtime in a rolling seven day window.

Each alert type prompts different action. Intervention gets Evan out of bed to quickly paper over the issue to avoid business impact. Deformation requires him to look at the errors and figure out what’s increasing the failure rate and consider planning resolution work. Snap requires him to stop what he’s working on and fix the problem for good. This combination of three distinct thresholds should allow him to stay much more proactive and ultimately more productive.

Here’s what it looks like when you put them all together.

4. How can we ensure the deployment goes well?

To answer this question we explored the existing deployment pipelines he’s got in place and, fortunately, they’re relatively mature. Sadly, given the criticality of the system and the stakes at play, the cold hard truth is that you can’t fully trust your infrastructure. Losing a day of artist time is just too damaging to the business, so you’ll have to personally do post-deployment acceptance testing. We discussed the pros and cons of different deployment times and landed on deploying and testing during US hours then staying on call during Chinese hours. Eventually Evan might be able to move to a more robust set of End to End tests capable of running safely in production to prove this critical system’s function, but we decided that wasn’t worth exploring at this time.

5. Is it worth delaying the release to add all these new features?

This conversation was somewhat more interesting; what is the value of one’s time in this situation? You’ve already sunk three days into this, it’s going to cost another three (between sleep deprivation and so on) to get this thing out the door safely. So it’s a week in. We discussed estimation for the new features and ultimately Evan felt the added cost (a couple more days) would be worth the added visibility and ease of maintenance.

The core of this discussion was the criticality of the system and the cost of even minor outages. If we were talking about something outside the critical path for the business or operated by a team in US timezones this whole refactor might not have been worth doing, let alone the features we’d scoped out over coffee.

I like asking questions, if you’d like to go for a run or have a conversation like this, send me an email at luke + substack at luke mercado dot com.

Subscribe now

Share

One of the hardest parts of building a business is understanding how to balance the trade-offs between up-front cost (Capital Expenditure, aka CapEx in finance terms) and ongoing costs (Operational Expenditures, aka OpEx in finance terms) when trying to add something to the business. Often times getting this right is the difference between improving the business in some material way or drowning in work while your customers suffer.

Over the last few years, first at Daybreak Health and now at The Commons I’ve come to realize that you can make better decisions if you can predict When it will break and How it will break. The exact implementation of the “it” doesn’t really seem to matter.

The following is my current model for predicting When and How it will break.

When Will It Break?

The when seems to follow a simple rule → Orders of Magnitude.

A system that works at 1 won’t work at 10, 10 won’t work at 100, 100 won’t work at 1000, and so on.

To put this differently, a system that delivers a high quality service well at a given throughput will not deliver the same service at the same quality at an order of magnitude higher throughput for the same fully loaded cost. It “won’t work”.

• Fully Loaded Cost doesn’t just mean CapEx or OpEx in dollars, it’s the combination of dollars, time and energy to deliver the service across both CapEx and OpEx.

How Will It Break?

Ok, so we know, roughly when it’ll break, but knowing when something might break isn’t enough information to make a decision. We need to know how it’ll break so we can respond appropriately.

Complex Systems are more like steel and less like glass. Glass is perfectly fine under load until all of a sudden it explodes in a shower of shards. Steel slowly yields, deforming, until it snaps in half. Both break, but if you watch steel carefully you’ll notice it deforming before it snaps.

Systems under load experience deformation in these areas:

• Quality - The quality of the service provided by the system

• Operational Cost - The cost in time, energy and dollars to provide the service

  • Time - The time, per instance, required to offer the service

  • Staff Emotional Energy - The amount of emotional energy required by the operators of the service to provide an instance of the service.

  • Dollars - The volume of dollars, per instance, required to offer the service

If load isn’t removed (throughput lowered) then the system might snap. There are two common ways systems snap:

1. A failure cascade is created. This occurs when a failure causes the capacity of the system to be exceeded, thus causing more strain, thus causing more failures, and on and on.

   1. e.g. A customer support team at an e-commerce site at capacity during Black Friday / Cyber Monday has a critical employee quit. The capacity is now lower and quality drops which leads to employees feeling they aren’t doing their job. Lack of satisfaction in their work results in another employee quitting, further reducing capacity.

2. Demand exceeds the capacity of the infrastructure, requiring new infrastructure.

   1. e.g. A startup crosses the magical “Product Market Fit” line and is inundated with new customers. The tooling and systems that onboard new customers was originally built for a handful of customers (<10) per month and the sales team is now trying to onboard 100 or more a month. No amount of all-hands-on-deck or new-hires will make it possible to achieve the previous quality of service until new infrastructure is built.

Relationships between Quality, Time, Dollars & Staff Emotional Energy

What is Staff Emotional Energy?

The concept of Emotional Energy is something I stole from Ruby K. Payne’s a framework for Understanding Poverty. In the book, “Emotional Energy” is a battery and that battery is charged and discharged throughout the day at different rates by different tasks. Thus Staff Emotional Energy is a subcycle within the broader cycle we’re discussing.

Dollars (Labor Cost) by Available Work Hours

More Available Work Hours can be purchased for Dollars (labor cost). It’s generally a sawtooth relationship though where one spends more money to buy hours in a batch.

Available Staff Emotional Energy by Quality

Quality and Staff Emotional Energy can be traded off to a point. This relationship is generally fraught since Emotional Energy is one of the few things that cannot generally be purchased; it must be inspired. This is often what people refer to when they say “Discretionary Effort”.

Demanded Work Pace by Available Staff Emotional Energy

One can drain Staff Emotional Energy at a lower rate with a lower Demanded Pace of Work or drain it faster with a higher Demanded Pace of Work.

Non-Labor Cost by Quality

As Quality decreases the Non-Labor Cost per transaction will trend up due to mistakes driving customer attrition or additional work. As Quality increases, to a point, Non-Labor Costs per transaction will go down.

Time / Transaction by Quality

More time can be spent per-transaction to increase Quality, though there are diminishing returns. Conversely, less time can be spent and Quality will go down.

Stability

The relationship between Quality, Time, Dollars & Staff Emotional Energy is only stable within a narrow range at the center of the graph. Beyond this point the system becomes unstable and a Failure Cascade is likely.

Well spent capital, in the form of process or automation (aka infrastructure), can shift the point of equilibrium.

All Together Now

Systems often break at orders of magnitude. They appear to break catastrophically when they encounter hidden infrastructure limits or are mismanaged into a failure cascade. Fortunately systems tend to deform before they snap; careful leaders can observe the Quality, Time, Dollars and Staff Emotional Energy to see this deformation.

Systems should be built such that either they ship with this observational ability or they’re able to build these observational tools before the next transaction magnitude is reached. By doing so leaders will be able to determine when and how to make capital investments to avoid a catastrophic failure. Systems that do not have these observational tools will suffer extreme deformation leading to catastrophic failure and that failure will be a surprise.

Hopefully this helps you avoid the mistakes I’ve made.

Subscribe now

Share

A year or so back I got fired from my startup. I was the first co-founder. I’ve been struggling with my sense of identity and purpose since. Tonight I hosted a solo-founder dinner at my place; six seats, half gals, half guys, all people I’ve either gotten to know or really connected with in the time since.

The rules were simple; everyone here has committed to founding a company and everyone here is without a co-founder. In South Park Commons terminology, we aren’t quite in negative one and we aren’t quite at zero. We’re somewhere in the messy middle. Only one pair had met each other outside of tonight; it was a cold open and I flubbed it. One of the gals gave us a push, Rose & Thorn, and jumped in seeking romantic relationship advice. Vulnerability and grit in equal measures. Our resident romantic, a man I’ve known for over a year but haven’t every truly connected with, weighed in with stark wisdom and a tone that said more about how he’d earned it than any story could’ve.

Another of the gals, she’d just moved into the city, explicitly to build her startup. She’d flown in from Dallas after hoofing it at an industry conference for half a week. She’s not LARPing; none of us are, but we’re all alone. We welcomed her to the city, and though she was last to answer the prompt, and the night was taking its due, the group stayed to hear her tale.

We talked about why we do this; why do we build companies. Is it that our needs were unmet as children? Is it an immortality project? Is it a chip on our shoulder? Are we broken? Hands go up. Debates are had. We dive down rabbit holes and offer advice. We challenge each other. Our resident philosopher walks us through the five defining fears: insecurity, stagnation, failure; I’ve forgotten the others. Maybe I wasn’t afraid of those? We stack rank them, we caveat them, we drink wine and smoke vapes. We talk about addiction. We talk about psychosis, family trauma and boxing. There was 7.5lbs of flat iron steak and somehow it’s dwindling.

Frameworks are exchanged, debated, and discarded. The Learning Engine is debated alongside the cost of failure. Those of us with expectations of performance in our upbringing cringe and buck under the idea that failure is inevitable, healthy, and required. Someone asserts that people are not kind and our boxer says you need to fight and defend yourself. I say they’re wrong, I take the table and I fight the finality, the binary nature of it. They let me rant, they sooth me, they point out the nuance, I thank them and cede the topic and the mic.

We talk about the co-founder search. The petty fights. The relationships we’ve lost in the hopes that friends or family could be that ally. We lament the cost of it all. Some of us think we work better with a partner; others are silent. The cost of a co-founder; the distraction of searching, the value of success, the phases of leadership, what it means to be CEO. It flows alongside the wine, the nicotine, the lights dimming because I forgot to turn off the timer that usually sends me to bed on time.

It’s interesting to reflect on what we didn’t talk about. Runway. Traction. Risk. Go To Market. AI. Some simply weren’t mentioned, others were a footnote or an asterisk at best. I’ve heard about SF founder dinners. I’ve hosted a couple. I’ve never achieved this level of intimacy; of openness, of tolerance and space.

We’re all motivated by something; and some of us don’t yet know what it is. If there’s one thing tonight has made clear to me it’s that the old adage “Powerful people have command over others. Great people have command over themselves.” is true.

Thanks friends.

A good estimate comes with Specificity, Precision, Assumptions, Confidence and Exactness. It is Specific in its units of measure, Precise in its timeline, detailed in its Assumptions, articulates its Confidence, and Exact in what is being estimated.

Getting to a SPACE estimate isn’t cheap or easy, but once one has been delivered it becomes possible to build atop the estimate and incorporate it into larger plans.

Not all estimation techniques deliver all of these and getting all of these in an estimate can be quite expensive. When we leave out parts of the S.P.A.C.E. framework we take on risk while cutting costs, but not all of these risk are easily apparent.

Leaving out a Letter (or two)

I was once in a meeting where we agreed to spend roughly $2MM of the engineering org’s bandwidth over the next year on a major replatforming. I learned, only later, that half the room thought we weren’t going to need support from two critical teams. This added another ~$250,000 to the cost of the project and, had the decision makers known, it might’ve been enough to scuttle the whole thing before we’d committed more than a few tens of thousands. Had I communicated, Exactly, what was going to be done, we might’ve had a better outcome for the company.

Early in my career I watched a Product Manager (PM) lean on a Sr. Engineer pretty hard for an estimate. The engineer was trying her damndest to say she couldn’t give a good estimate, but she was a pretty conflict adverse person and the PM was pretty influential around the company. I was too new to even recognize how bad this situation was at the time, so I sat there as he wore her down and she caved. She told him something to the effect of six days. The PM told the rest of the company it’d be done in six days and an Account Manager told a major customer the feature would be live on the 7th. As so often happens the engineer was asked to work nights and weekends and management got more people involved and despite it all the team didn’t quite make the date and quality was lacking. Had she shared her Confidence in the estimate the whole team might’ve had a better outcome.

Somewhere around my third year as a software engineer I was estimating a multi-month project in my capacity as a Tech Lead. I was rather pleased with myself, I’d done the research, talked with the team, pressured the PM on the requirements, written up a technical proposal, gotten input from the other engineers and slapped an estimate together. I told the PM we’d get it done in four weeks. The PM took my word for it and said we were on track for our quarterly goals to upper management. Two weeks later half my team started putting in for PTO during ski season. We missed our deadline by three weeks. I was young and a workaholic and forgotten to even ask my team about their holiday plans. Had I done that, or included no PTO as an Assumption the outcome might’ve been different for the team.

At one point I was in a quarterly planning meeting at a company that operated under a SAFE model. We were all sitting around the table throwing things on team’s plans and trying to figure out our capacity for the quarter. Half the teams were talking about how many points they had, another group was talking about T-Shirt sizes, one of the architects was throwing out Stupid Wild Ass Guesses (SWAGs) in real time. It was impossible for anyone to understand, with any Specificity and Precision how much of any given team’s bandwidth had already been allocated. It was also impossible to compare a team’s velocity, success rate or capacity quarter over quarter because sometimes the unit would change!

A good S.P.A.C.E Estimate

My team and I believe we can build the password reset flow defined in Jessica’s V2 Password Reset Flow document. We think we have a 30% chance of delivering in five business days, a 60% of chance of delivering in eight business days and a 90% chance of delivering in fifteen business days. We’re assuming that the work can be split amongst three engineers relatively easily, that our frontend engineer can work with mocked data for a few days, that our mocked data isn’t incorrect, that Jessica can deliver the templates for the emails by the end of day two and that no one has any unplanned PTO for the duration of the project.

We know

1. Exactly what is being estimated: Jessica’s V2 Password Reset Flow.

2. How Confident the team is in their estimations.

3. The Assumptions the team has built into their estimations.

4. In Specific units and with Precision how long it’ll take: Between five and fifteen business days.

Paying for S.P.A.C.E.

1. Specific units is usually just a matter of setting up the culture and repeatedly asking “What unit is that?” when the units are missing.

2. Precise numbers are generally pretty expensive as it requires the estimator to have an understanding of the current system, how it might be evolved to solve the problems being discussed, what the team is capable of, and everyone’s velocities involved. This is usually the most expensive part of the process and can take many days and in some cases weeks. It also isn’t often done within the bounds of Engineering. Don’t be surprised if your engineers are trekking all across the company to get questions answered!

3. Assumption lists are usually pretty cheap once your team has built the muscles. A good checklist can help to prompt engineers about common assumptions until they’ve learned to think through them on their own.

4. Confidence intervals are a bit wonky. It’s pretty fast (minutes) to give low confidence estimates, but getting higher confidence estimates that aren’t ridiculous is usually pretty expensive. Don’t be surprised if your engineers want to spend a week figuring out a plan before they’ll give you a 90% confidence estimate less than a full calendar year or an outright “I don’t know”.

5. Exact definitions of what problems are being solved are really dependent on the quality of your PRDs. If you’ve got mini-ceos who are very exact in their problem definition and user stories your engineers should be able to lean pretty heavily on that and this will be cheap. If you’ve got less talented PMs you might need to rebuild your product org. Fortunately that’ll pay dividends in other ways!

Working Theories are semi-polished concepts and ideas that I’m screwing around with.

I’ve been doing some co-founder dating over the last couple months as part of figuring out what I want to do after having left Daybreak Health. Over this time I've noticed that the SF startup scene has two main groups, the first understands that they aren’t operating in a vacuum and the latter doesn’t, but how do you quickly spot the difference?

I was at a meetup the other day and a founder was talking to me about how they're going to take over Revenue Cycle Management (RCM) with AI agents. For those of you not knee deep in healthcare, RCM is the term of art for a suite of tools that allows care providers to make sense of, track, and nurture an insurance claim from creation through to being paid (or not!). It is, broadly, the primary weapon wielded by care providers in the hot war between them and health insurers over who gets paid and who doesn’t. This person's plan was to create an AI agent that would replace the user of the RCM and save on the medical biller’s salary. It sounds great on the surface, and it'll probably make a few million dollars for someone, but it isn’t going to shift the dynamics at all, merely accelerate them. When I asked what their plan was to deal with the fact that insurers would just do the same thing on their end, they got mad at me and ended the conversation. Not only did this founder detest feedback, they were only interested in playing “half-court tennis” and I think that’s the tell worth exploring.

I first heard of "half court tennis" from Sarah "Sally" C.M. Paine over at the US Naval War College. Just imagine, you're playing tennis but you can't see beyond the net. You can't position yourself to counter your adversary, you can only react once the ball has soared over the net.

For those of you looking at joining an early startup, here's a few questions for the interview process that might help you figure out whether a founder is only interested in playing half-court tennis.

1. What three things must go right in the next six months to achieve your vision?

   1. The goal with this one is to figure out the way they think about the road ahead. A good answer here is going to demonstrate prior deep work and include both tactical execution and competitive action (or inaction). A bad answer here will be reactive and off the cuff, it will include their own actions and maybe their organization’s actions, but likely won’t include anything from outside their own control.

2. What three events would require the company to pivot?

   1. The goal here is to explore the founder’s downside planning. Do they have plans A through D? Or is this their one shot and they have no idea what to do if things don’t go to plan? You want a founder who can easily figure out what to do once the enemy gate is no longer down.

3. Which trends are currently making this business viable?

   1. The goal here is to get the founder to explain how the business fits into the broader tapestry of current events. A founder who isn’t aware of the world around them is going to make the wrong decisions at the wrong time. A good answer here will be an explanation of the investor zeitgeist that drove or will drive their funding and the consumer or business tailwinds that will drive adoption of their product(s) for years to come.

I think founders who aren’t interested in feedback and only want to play half-court tennis are pretty common and aren’t worth following. Hopefully these questions will help you quickly avoid folks like this in your search for a co-founder or a job at a startup.

Subscribe now

Share

Working Theories are semi-polished concepts and ideas that I’m screwing around with.

Over the last few months I've been reflecting on my time at Daybreak Health and thinking through what's different in the tech ecosystem. Back then (2020) I chose to use a Ruby on Rails + React stack. My reasoning was that it would simplify the initial prototyping phase (I'm a RoR dev) by minimizing the learning curve and allowing me to easily hire in expertise in my known weaknesses in the next proximate objective (e.g. get a frontend dev).

That first year or so went really well on the technical front because I had what Rumelt would call Good Strategy. My objective was to get us through the prototyping phase to PMF. My guiding policy was to minimize my own learning curve and ease my hiring burden. My actions were to work within my own core competency (RoR), select technologies for immediate use that would be easy to hire for (React > Angular), and leverage as many consumer off the shelf technologies as I could. It wasn't perfect, but it enabled the business in a big way in that first year and a half, and I'm pretty proud of that.

As I've been debating next steps I've found myself repeating portions of this exercise and I'm starting to realize that if I want similar success in the prototyping phase and more success in the PMF and post-PMF phase it's probably not going to be the same tech stack. The world's moved on, and while RoR is still a solid API backend, I don't think the technical org you end up with is flexible enough for the modern startup world.

I think a full TS/JS stack is ideal now for early CRUD app prototyping because it means your team will be massively more flexible. The syntax barrier is lower across the frontend and backend and core concepts (async, promises, callbacks, etc) are similar across all elements of the stack. Mix in the latest co-pilots and I suspect a team of Senior devs would be damn near full stack by default. Thinking a little deeper, the real question is whether or not you can survive as a pure-play TS shop. You can delegate out to vendors via API for your AI/ML (and let's face it, everyone's going to need AI/ML), but I don't think you can escape the slither that is python wrapping your data stack. With that in mind a Django or Flask stack might be competitive post-PMF, but would probably suffer pre-PMF.

Do the engineers amongst you have any thoughts?

Subscribe now

A couple of days ago a founder I admire, Wes Winler, posted a series of tips on things to look out for to help quickly assess hiring efficacy1. One of these tips was to use onboarding surveys, and wow. Despite having spent dozens of hours building Success in Role docs and Onboarding Plans for each new hire on my team at Daybreak I hadn’t put any thought at all into how I would determine whether these plans were working or not. Oops…

So… what would I want in an onboarding survey?

What do I want feedback on?

As a hiring manager I would want signal on whether the employee’s expectations are being met and whether I’m doing a good job onboarding them or not.

As a manager of hiring managers I want to know whether onboarding as a whole is working at our company and whether new hires are being setup for success or not.

As an employee I don’t really want to do this and if I have to do it I want to spend as little time as possible on it.

I’ve distilled this down into the following core things I want signal on from my onboarding survey:

1. Operational Excellence - Did the initial logistics go smoothly?

2. Foundation Setting - Did the initial logistics set the new hire up for success?

3. Feedback Loops - Is the manager setup to assess the candidate over the next thirty, sixty and ninety days for fit and performance?

4. Feedback Loops - Is the employee setup to build a relationship with their manager?

Amusingly, these things also say a lot about what I think good onboarding is. Let’s flesh that out a bit more though.

The initial logistics of hiring someone are all about earning trust by demonstrating competence through using their time wisely. This work includes, the legal paperwork, necessary physical tools (aka laptops), credentials to any digital tools, confirmation of access to those digital tools, and access to any supporting infrastructure. By my eye this should take no more than an hour of the employee’s time on day one.

Setting the new hire up for success is much harder, in my mind the new hire will need:

1. Scheduled training for each bespoke tool or process.

2. Weekly recurring Manager Tools style 1:1s scheduled and a brief explanation from the manager on the process since it might be unusual.

3. Thirty, sixty and ninety day expectations in an actionable checklist (building this is beyond the scope of this post)

4. Intro’s scheduled for each person they depend on and each person who depends on them.

Without these things it’ll be very difficult to evaluate the new hire’s performance in these critical first days. Put another way, the risk of losing the team’s trust by retaining a bad hire for longer than necessary goes way up if you don’t have these things.

Speaking of evaluating performance, we need some feedback loops here. I believe the most critical at this stage are end of week “Onboarding Progress” business meetings with their manager where the manager has come prepared to talk about what is and isn’t going well. Doing these well can be hard, but the core of it is probably about an hour of prep for the manager.

With this strawman, what should my onboarding survey look like?

Gathering Feedback from the New Hire

While the science of surveying is very much outside the scope of this post, we need to keep in mind that this survey fundamentally asks a new employee to judge their boss. That’s going to make a lot of people uncomfortable and so we need to navigate around this challenge. To overcome this I prefer to use what I think of as “lateral questions”.

Below are the questions I would put in the onboarding survey, a bit about why I chose those questions, and how I would evaluate answers. Here’s the survey itself.

Operational Excellence

Question 1: Is there anything we could’ve done to speed up getting you everything you needed to start work? → open text field

• Why this question and format?

  • I want signal on whether or not the manager mismanaged the initial provisioning of the employee. I’m opting for an open text field because I don’t want to lead the new hire down a certain path. I opted for words and phrases like “anything”, “speed up”, “everything” and “start work” because I wanted to anchor the question as a positive and collaborative one with the new hire.

• Answers that imply the manager did well:

  • e.g. Nope, I got my laptop the day before and my company email had invites to every service waiting for me when I logged in.

  • e.g. Went great! No notes!

• Answers that imply the manager did poorly:

  • e.g. My laptop showed up late, but everything went well after.

  • e.g. I can’t get into slack… plz send halp.

Question 2: How long did it take for you to get everything you needed to start work? → pick one minutes to hours, hours, days

• Why this question and format?

  • I want a concrete signal on how long it took to get setup that I can then cross reference with the manager’s input.

  • By placing this question after the more abstract I get a calibration point on what the employee considers to be acceptable. If they give a positive answer in Question 1 but answer Hours here I know that they have low expectations for this portion of the work and can weight the response accordingly.

• Answers that imply the manager did well:

  • e.g. Minutes to Hours

• Answers that imply the manager did poorly:

  • e.g. Hours

  • e.g. Days

Foundation Setting

Question 1: When do you expect to have completed your training on our tools and processes? → open text field

• Why this question and format?

  • I want signal on whether or not the manager scheduled the training they should’ve. I’m opting for an open text field because I don’t want to lead the candidate to the correct answer and I want to capture lack of understanding if that’s the case. I opted for this phrasing around completion because it makes the question difficult to answer if the training was not scheduled but very trivial to answer if it was scheduled.

• Answers that imply the manager did well:

  • e.g. My last training day is August 22nd.

  • e.g. I’ve got a bunch of scheduled training meetings, so whenever those are done?

• Answers that imply the manager did poorly:

  • e.g. I don’t know

  • e.g. There’s training?

Question 2: How many meetings do you expect to have with just your manager during your onboarding period? → text field

• Why this question and format?

  • I want signal on whether or not the manager scheduled both the 1:1s and the onboarding progress meetings. I opted for this phrasing because it’s explicit in what it’s asking while not leading the new employee too much. I opted for a text field because google forms doesn’t have a number field and I don’t want to prime the new employee too much.

• Answers that imply the manager did well:

  • e.g. 20 to 30 (90 days, 12 weeks, 2 meetings/week)

• Answers that imply the manager did poorly:

  • e.g. <20

  • e.g. >30

Question 3: Who do you expect your work to enable the most? → open text field

• Why this question and format?

  • I want signal on whether or not the intro meetings with those that depend on them have been scheduled. I opted to frame this with “expect”, “enable” and “most” because I want them to internalize that these are the people they’re helping and expect those words to signal that without explicitly prompting them that these are the people they have downstream dependent intros with. I opted for an open text field because I don’t want to have to keep a list of employees in the survey tool and in smaller orgs I don’t want to lead them with the names.

• Answers that imply the manager did well:

  • e.g. thirty or forty percent of the names of the people they’ve got as dependencies.

  • e.g. The velocity team and the badger monkey team.

• Answers that imply the manager did poorly:

  • e.g. I don’t know

  • e.g. Florence? (<30% of dependencies)

  • e.g. The San Francisco Giants

Question 4: Who do you expect for your strongest stakeholders to be? → open text field

• Why this question and format?

  • I want signal on whether or not the intro meetings with those they depend on have been scheduled. I opted to frame this with “expect”, “strongest stakeholders” to avoid prompting them that these are the people that they depend on to do their job.

• Answers that imply the manager did well:

  • e.g. thirty or forty percent of the names of the people they depend on.

  • e.g. The customer onboarding team and the engineering team.

• Answers that imply the manager did poorly:

  • e.g. I don’t know

  • e.g. Jonathan? (<30% of dependencies)

  • e.g. The San Francisco Giants

Question 5: How are you evaluating your performance in this new role? → open text field

• Why this question and format?

  • I want signal on whether or not the employee has understood how the onboarding plan is going to be used. I opted for this phrasing because I associate completing the plan with “good performance” and hope they will too. I opted for open text field because I don’t want to prompt the employee directly.

• Answers that imply the manager did well:

  • e.g. Whether I’m on or off track from my onboarding plan

  • e.g. Whether I finish the plan or not

  • e.g. Whether my manager tells me I’m doing well or not.

• Answers that imply the manager did poorly:

  • e.g. I don’t know

  • e.g. If you fire me or not

  • e.g. Whether my coworkers tell me or not

Feedback Loops

Question 1: If things aren’t going well, when and where would you expect your new manager to provide that input? → text field

• Why this question and format?

  • I want signal on whether or not the manager’s onboarding progress meetings purpose was communicated effectively. I want this because I’m wary that employees might not understand the difference between 1:1s and onboarding progress meetings. I opted for the negative tone of “aren’t going well” because I want to see if they associate the onboarding progress meetings as feedback meetings.

• Answers that imply the manager did well:

  • e.g. the end of week meetings

  • e.g. the onboarding progress meetings

• Answers that imply the manager did poorly:

  • e.g. my 1:1s with my manager

  • e.g. I don’t know

  • e.g. standups

Question 2: What input, if any, have you received from your manager this week? → text field

• Why this question and format?

  • I want signal on whether or not the manager has scheduled and run at least one onboarding progress meeting.

  • I want signal on whether or not the employee has understood how the onboarding plan is going to be used. I opted for this phrasing because I associate completing the plan with “good performance” and hope they will too. I chose to include “if any” because I wanted to imply that “None” is a perfectly valid answer. I opted for open text field because I don’t want to prompt the employee directly.

• Answers that imply the manager did well:

  • e.g. My manager said I’m doing well.

  • e.g. My manager said I’m meeting expectations but I took longer than expected to attach the widget to the flabberjam

  • e.g. My onboarding progress meeting hasn’t happened yet, but it’s scheduled for end of day.

• Answers that imply the manager did poorly:

  • e.g. None

  • e.g. I don’t think they’ve said anything?

Gathering Feedback from the Hiring Manager

Below are the questions I would put in the hiring manager’s onboarding survey, a bit about why I chose those questions, and how I would evaluate answers.

Operational Excellence

Question 1: Is there anything we could’ve done to speed up getting your new hire everything they needed to start work?

• Why this question and format?

  • I want signal on whether or not the new hire provisioning process is breaking down. I opted for a positive framing because I want to encourage the manager to ask for help.

• Answers that imply the system is working:

  • e.g. None, all went swimmingly

  • e.g. Nope, finished in forty five minutes

• Answers that imply the system isn’t working:

  • e.g. IT delivered the laptop late, Xuan was tooling around on her phone for the whole day!

  • e.g. We couldn’t get Jeff’s account setup until end of day because the email wasn’t in their inbox like usual.

Question 2: How long did it take for you to get the new hire setup? → pick one minutes to hours, hours, days

• Why this question and format?

  • I want a concrete signal on how long it took to get setup that I can then cross reference with the new hire’s input.

• Answers that imply the manager did well:

  • e.g. Minutes to Hours

• Answers that imply the manager did poorly:

  • e.g. Hours

  • e.g. Days

Getting Value out of our Onboarding Survey

To get value out of these surveys I want the following to happen:

1. The employee has ample time to experience the onboarding process prior to being prompted to weigh in on it.

2. The hiring manager’s manager is notified when there is work for them to do, either in interpreting the survey’s results or in nudging the employee to complete the survey.

To accomplish this I would do the following:

1. Use my HRITS to send the new hire survey via email to new hires Tuesday morning on week two of their employment with the subject [Action Required] - New Hire Onboarding Survey

2. Use my HRITS to send the new hire onboarding survey to the hiring manager on Tuesday morning on week two of the new hire’s employment with the subject [Action Required] - New Hire Manager Onboarding Survey

3. Have each survey notify the Hiring Manager’s manager via email when a new result has been received for a new hire in their org.

4. Have the HRITS notify the Hiring Manager’s manager via email when a new hire in their org, or their manager, hasn’t yet completed the survey.

5. Have the results display in a spreadsheet only accessible to the hiring manager’s manager and those above.

6. Ideally have employees start on a Monday.

This system would empower the hiring manager’s manager to follow up when need be and understand how their directs are doing on the first stages of the onboarding process with a quick look at a single email.

Like the rigor? Running a startup or business of your own? Want input on how to tackle your problems? I’ve opened a few advisory and consulting slots and can bring this kind of rigor to your challenges → luke[at]lukemercado.com.

Subscribe now

Working Theories are semi-polished concepts and ideas that I’m screwing around with.

I was talking with my partner the other day and she mentioned that three of her team’s engineers were out sick and I off handedly remarked that it seemed like people were getting sick more often after COVID. Turns out I was right.

According to the U.S. Bureau of Labor Statistics, every January from 2012 through 2021 roughly three million people would miss work at least once due to illness. In Jan of ‘22 that number increased to 7.8 million, 2.6 times as many people.1 Fred Schott has been keeping an eye on this for a while using the BLS’ Annualized Absence Rate data2 and looking at his snapshots it seems like we’re experiencing a 10% enduring increase in sick days post pandemic.

This got me thinking; when and how should project planners think about sick days?

In the USA, the CDC claims that adults average one common cold a year3, meanwhile the Mayo Clinic says two to three colds a year4 and John Hopkins says two to four per year5. That gives us a range of between one and four colds/year per person. The same sources provide us a range of duration as well, Mayo Clinic says seven to ten days while John Hopkins says “several days to several weeks”. In my head this says seven days to twenty one days out. I couldn’t find good statistics for the distribution curve within that range though. So I made up my own curve, 25% are sick for a week, 50% for two weeks and 35% for three weeks. These numbers are intentionally pessimistic because, well, project planning. When you roll all this together you get a pretty neat thing: on average, you can expect an 8% drag on productivity from sickness. Sadly, averages hide a lot of things.

Let’s start with the impact of sickness on other employee’s productivity. Odgerel Chimed-Ochir et-al out of Japan published a study in August of 2019 that states: “Overall, employees lost 7.65% of their total working days, or an average of 17.92 days were lost per employee per annum, due to sickness leave and sickness presence combined.”.6 Only 1.1% of that was people actually leaving the office, the remaining 6.55% was what they call “sickness presence”, aka your coworker giving you a cold and dragging your productivity down. This study was done in Japan though and their work norms are a bit different, that said I’d wager that folks in the USA may have different reasons for working while sick, but that the rate is probably pretty similar. That means we can reasonably expect to pay this 6.55% productivity tax whenever someone gets sick. When you add this in, the average loss of productivity from sickness jumps to 12%.

These averages are hiding a few other things though, like what happens when someone on the critical path gets sick. That can have cascading effects, but the big questions are figuring out how likely that is, whether it’s going to happen, and what to do if it does happen. Here’s a few quick rules to help you out:

1. Sickness scales with Calendar Days and People, the longer your project goes in calendar time and the more people you have involved the more likely you’ll have an illness.

2. For every 183 Calendar-People-Days, you’ve got a statistically guaranteed chance of illness when you consider the average statistics.

3. The average illness will take your critical resource out for two weeks, but might be as long as three.

Another fun question is when should we start thinking about the rate of illness in our planning process? Here’s a few things that came to mind for me:

1. If you’re a remote company you should expect that any and all gatherings you run will trigger a wave of sick days. Anecdotally from my time at Daybreak, we would see roughly 25 to 30% of our team take sick time in the following two weeks. Admittedly, this was during ‘22 and ‘23.

2. Illness rates double in January according to the Bureau of labor Statistics, so you should expect to see impact in that month for any in-flight projects.

3. On any project with more than ten people lasting more than ten calendar days you’re flirting with a greater than 50% chance of an illness and if Odgerel Chimed-Ochir et-al are to be believed, one illness will cost you ~6 calendar-person days or about twelve percent of your total budget!

Amusingly, startups probably don’t need to care too much about these numbers because the impacts are relatively small compared to other frictions in the business.

As part of writing this I threw together a calculator, which you’re welcome to snag a copy of in case it’s helpful to you.

My partner came over for date night (I made stew) and, as usual, we ended up talking about her career. This is a pretty normal date night conversation for us, my friends jokingly say we’re a “Power Couple”. She’s at a bit of an inflection point in her startup and is trying to figure out whether she should push for more responsibility or leave and seek that elsewhere. Somewhere around 9pm I was trying to figure out how to explain to her that I thought one of her assertions didn’t contribute to her stated goals. Over the last few conversations we had built up the concept of “Career Capital” with the notion that one earns career capital through known accomplishments. I in turn had built an expected value model in my head for a number of possible scenarios she might encounter in the coming years. My initial impulse was to tell her directly what I thought, but one of my major goals these days is to ask, not tell. So I asked her if she was up for playing a quick game with me to help me understand how she’s thinking about the different scenarios. She said yes and I grabbed my notebook and scribbled out something like the below table.1

I filled in the top left most cell with an arbitrary number of 10 to provide an anchor point and asked her to fill in each cell with how much additional career capital she expected to earn in the scenario. My original intention was to simply get her to tell me how she valued each of these possible outcomes, but as she worked I realized this plan was creating something much greater. As we added different scenarios, we began to build out a projection of potential earnings. When the doc was finished, we’d built her a map AND her desire for Career Capital had become a compass!2

I pointed this out to her and the “Aha!” moment was magical. We discussed a few additional scenarios and I pointed out that she could calibrate this compass / map by talking to a handful of people. If she cold contacted ~30 people on LinkedIn with a spread of CEOs, VPs of Eng, Investors, Early Founders and Series B Founders she could probably get thirty minutes with one of each. In that time she could ask them for their perception of the relative weightings between these scenarios and that would help her build confidence in the map.

Once she had confidence in the map, she could start to make really meaningful tradeoff decisions about her career using it. This compass should last years and allow her to quickly make really big decisions.

Needless to say, I’ll be doing this for myself at my next major inflection point.

Thanks to Cherry Yang for her input on this piece.

Subscribe now

The Testing Pyramid is a representation of the types of software testing and how they relate to each other. The further up the pyramid you are, the more certain you are that the thing under test is performing as expected, in exchange your tests are also more expensive to design, run and maintain.

This representation is particularly useful because:

1. It captures the fact that generally it isn’t worth running the higher level tests if a lower level test has failed since you can usually conclude that one or more higher level test will fail.

2. It captures the fact that generally it isn’t possible to build or economically run the higher level tests without having built the lower level tests.

Unit Testing

A Unit Test tests that a singular function has the desired behavior in a given scenario.

Unit Tests are cheap to write, cheap to run, and quickly point out what is and isn’t working at a purely technical level. Unfortunately, they’re also incredibly granular and so it can be hard to determine, based on a failing unit test alone, what exactly is broken from a feature or product perspective.

Unit Tests, on a per-test basis, typically take seconds to minutes to write, run in microseconds and often have no meaningful maintenance costs.

Component Testing

A Component Test tests multiple functions together at once and proves that they have the desired behavior in a given scenario.

Component Tests add breadth on top of Unit Tests and are often written and run in the same tooling. Typically Component Tests are only slightly more expensive to write, run and debug than Unit Tests. While they’re less granular than Unit Tests, they do not generally explain what is or isn’t working at a feature or product level.

Component Tests, on a per-test basis, typically take minutes to write, run in microseconds to seconds and often have trivial maintenance costs.

System Testing (aka Integration Testing)

A System Test takes entire components, orchestrates them together under a given scenario and proves that together, they have a desired behavior.

Typically a System Test will test not just the business logic, but integrate together multiple discrete pieces of infrastructure such as the application, its database, and its server. This allows a System Test to prove that not only does the business logic do as expected, but all the pieces required to actually run that business logic in production all work together to accomplish the desired behavior as expected.

System Tests generally cannot explain what is or isn’t working at a feature or product level, but can be thought of as the last “Technical Testing” in the Testing Pyramid.

System Tests are typically much more complex to write than Component Tests as they require multiple tools working together to setup and tear down.

End to End Testing

An End to End Test tests a singular user journey from start to finish (end to end) across an entire fully integrated system.

End to End Tests often require that entire environments be created specifically for testing and as such cost substantially more than System Tests in both dollars and clock time to design, build and maintain.

End to End Tests are capable of answering feature and product level questions such as:

1. Does Login work?

2. Can new users belonging to client Omega successfully register from the email we send?

To make End to End tests economical, your team will need to build out the same infrastructure required to spin up and provision an entire new environment. You can expect this to take dozens or hundreds of hours prior to being able to build the first scalable test. Once this infrastructure is in place you can expect each individual system test to take minutes to hours to create and seconds to minutes to execute. The reliability and maintainability of your tests will be determined by the developer experience of the infrastructure used to build and execute them.

Acceptance Testing

An Acceptance Test is the exact same as an End to End Test, except it isn’t automatable. The entire point of an Acceptance Test is to have a human experience, and quality audit, the user journey as if they were the user themselves.

When the Product Manager responsible for a given Product runs an Acceptance Test against a user journey in their Product it is the only time you can truly say whether the given user journey is “Correct”. As such, having your PM write and execute a handful of acceptance tests per feature is incredibly valuable.

Given the reliance on a human you can think of Acceptance Testing as the single most expensive type of testing. Typically Acceptance Tests can leverage non-production environments and be executed manually by the PM. You can expect each Acceptance Test to take minutes to hours to write and minutes to hours to execute. Depending on the maturity of your systems a PM may require support from an Engineer to provision or otherwise execute the test which will double the execution costs.

Note: Developers can and should run acceptance tests on their work as part of contributing it to the code base.

Subscribe now