Why Estimating Tasks is So Hard and Why Developers Hate It

Almost all developers (even those starting their careers) sometimes could be asked the famous question: “When do you think this task will be done?” and we just choke for a moment until we realize that we don’t know.

We, humans, are bad at predicting the future so we often fail in task estimation, some developers might see this as micromanagement, but the reality is that the PM (Product Manager) needs to know this, not because they want to micro-manage (I mean, some people micro-manage, but we are excluding them) but because they need to prioritize the roadmap of features so the chain of the leadership from the PM to the CEO and investors can see if the objectives of the teams are aligned with the objectives of the company and what could be left behind now for the good good.

But task estimating is a common frustration among developers, because in reality sometimes we don't know exactly when something will be finished, maybe because it's something completely new (new domain, new technology) or even there are a lot of external connections that we can't measure the complexity.

To be honest, from my career of more than 10 years of software development I've never seen a project that was estimated to be delivered in the defined target date without removing scope, over hours, etc...

Let's explore this :)

The Challenges of Task Estimation

Uncertainty and Complexity: Even the most experienced developers fail to estimate because software development by nature is complex and hides a lot of uncertainty, imagine that you want to implement an integration with an external partner, you take a look at their documentation on details and then estimate the task duration, but when you start implementing you discover that this API sometimes gives timeouts and even their response for a specific payload is different from what you were expecting so you'll need to build a new mechanism to handle those timeouts and gracefully map some of those error scenarios.
Changing Requirements: You start a task with all requirements defined, but in the middle of it when someone starts testing they realize that instead of building a list they want to build now a table, and this table is not present in the current design systems. What was planned could double time easily since you'll need now to build and publish the component.
Interdependencies: We can try to break as many possible tasks into small units, but sometimes some tasks are interdependent on another one, and delaying one task could affect the entire estimation of others.
Lack of Historical Data: When a team was just formed we didn't have historical data on metrics like lead time and even metrics on the company about specific parts of the system. This gets worse when we think about innovative or experimental projects (Hey A.I. 👋🏾).

Why Developers Dislike Estimation

Pressure and Stress: I know some people who like to work under pressure but the majority just don't like it, it's stressful to provide accurate estimations when you're under pressure. Missed estimates can lead us to feelings like failure or frustration.
Distraction from Development: Sometimes estimations take a long time, especially for large projects and this could be seen as a distraction from actual coding... I've seen many times a developer's preference for focusing on tangible progress rather than just speculative timelines.
Misalignment with Reality: Estimates can often be misaligned with the reality of complex tasks, imagine that you need to build a completely new feature to be shipped in 2 months for an event that the company will present. There's lots of frustration when those external expectations based on estimates don't match the actual progress (for both sides, developers and managers).

Strategies to Improve Estimation

Agile Methodologies: I don't want to get into details about agile methodologies, since there are multiple ones and each one will work differently for different companies, but thinking about good things that those methodologies bring is trusting data and the role of iterative development in refining estimates. When I was working as a software consultant we tested multiple types of measurements to predict to our customers when something might be delivered and one that had a great return was the lead time, where we gathered the data month by month to understand the team's velocity to try to predict the delivery date.
Break Down Tasks: Another thing that I've seen working well is breaking down larger tasks into smaller ones, this is one of the more effective strategies that a team can use because when breaking tasks you can have faster code reviews, faster deployments, people working in parallel, less chances of introducing bugs.
Regular Review and Adjustment: Periodically review estimates to ensure they are still on track and revise them as needed. Teams can iterate and make adjustments to their estimates as they deliver, accounting for these dynamics. This also allows for identifying potential roadblocks on time and making changes to work timelines for better estimation that is more flexible.

Conclusion

Task estimation is hard, and there's no right way to do it—only the way that works for you and your team. We can follow the strategies that I've outlined here, but in the end, it's up to you to test and decide if they work and if it's worth continuing that way. Remember, like almost all things in life, we get better only by doing it multiple times, so try, experiment, and iterate on the strategies.