Root Cause analysis question: 

‍Order cancellation on Nykaa is up by 20%. The order cancellation window is 24 hours. Can you help us diagnose the issue?

From the format of the question it is easy to identify that this is a Root-cause analysis question which tests one's ability of simplifying a problem, narrowing its scope and communicating it structurally to the team members. I also see it as a way in which Product Managers can do a lot of the ground work before engaging precious Engineering resources. It is critical to not make the mistake of interrogating the interviewer to find the answer. The candidate is evaluated on how they strike a delicate balance between asking the right questions and inferring from the information provided to ultimately converge to a solution.

Now with that background let’s take a stab in answering the above question.

Nykaa has catapulted the beauty industry in India which is embracing the world of makeup and skincare, earlier just a Western concept. Nykaa has expanded to selling clothes as well as creating their own skincare and makeup products.

Here are a few clarifying questions that I’d like to get answers for in order to scope out the problem.

Me: Nykaa has fashion and beauty as e-commerce services. Are we observing this for the fashion platform or beauty or both?‍

Interviewer: We are observing this for Nykaa beauty.

Me: Are we concerned about order cancellations, both within and beyond the 24 hour cancellation window or either of them?

Interviewer: No, the problem is for orders that get cancelled after the 24 hour cancellation window as the items are packed and are ready to ship.

I want to narrow the scope of the problem with some segmentation:

Me: Is this happening for a specific type of product like makeup or skincare?

Interviewer: Yes, mostly makeup products.

Me: Is this on the app or the website or both?

Interviewer: Just the app.

Me: Is this happening for a particular type of order, like Cash on Delivery or EMI based payments?

Interviewer: No

Me: Since when are we observing this drop, is it sudden or gradual? Has this ever happened before?

Interviewer: This has happened in a span of a week and we have never seen this before.

Okay. So in real life situations we most likely know if we launched a new feature or changed the user experience and realise that it is probably why the metric has been affected. I could be tempted to start exploring that option. However, before going there let’s take a step back for a moment because there may be other reasons as to WHY this metric went down and I want to consider them even if we rule them out quickly.

Me: If it’s okay I want to start broad and explore all the possible reasons and then dive deeper into some of them to answer the question. Is that okay?

Interviewer: Sure

Pro tip: You want to lay out all the possible causes that are mutually exclusive but collectively exhaustive i.e. reasons that don’t overlap but cover all possible reasons for why a metric moves.

System level causes: These are the easiest to figure / rule out.

Me: Was there a change in the formula of how this metric was calculated?

Interviewer: No‍

Me: Is there a possibility that our analytics tool is not working well or is not getting the right data?‍

Interviewer: No‍

Me: Are there any tech experiments or A/B tests going on?‍

Interviewer: No‍

‍

External factors: These are the ones that are usually outside our control.

Me: Have we seen Sephora or other competitors offer newer / more popular brands that are not available on our platform? Or are they running some seasonal campaigns?

Interviewer: No‍

Me: Are products available at a better price on other websites?‍

Interviewer: We have competitive prices on our platform.‍

Me: Is this pattern happening across a particular geography, as some parts of the country have been experiencing storms and there is a new wave of the virus? Such things affect consumer behavior / desire like maybe anticipation of another lockdown?‍

Interviewer: No, this dip is worse than what we experienced during the pandemic.‍

Me: Is there any negative PR about us on social media ? Something around product/brand authenticity? Maybe an influencer has criticized us and suddenly their followers are cancelling their orders?‍

Interviewer: No, we haven’t heard anything of this sort from the brand and PR teams.‍

Me: I also have not come across any new regulation that the government has passed around this industry.‍

Interviewer: Yeah, there has been no such regulation.‍

Internal factors: It looks like it’s probably something internal that has led to this dip now that we have not been able to point this to a cause in the other two categories. Let’s quickly take a look at the user journey so that I can examine each step carefully to see which ones may have caused the dip.

If I look at the user journey, the user first likes an item, looks at the details of the product, selects the shade or size, adds it to the cart and then places the order. After this step, the order is confirmed and ideally after the 24 hour period of cancellation it is packed, shipped and then is delivered to the user.
‍

Me: Have we recently shipped a feature that has impacted any of these steps in the flow?‍

Interviewer: Yes we have recently released a few features.‍

In this journey there are a few things that can cause the users to cancel the order:

Me: We do have a form asking users to input the reason for cancellation, do we have any data around it, to identify patterns or themes?‍

Interviewer: Yes that the product was not what they wanted. Sure, we cannot completely rely on this because the users sometimes just want to get through the process and may not always select the right reasons.‍

Me:The users may have ordered the wrong shade or size of the product. Were there any changes made to the detail page of the product? ‍

Interviewer: Yes, we actually just introduced the virtual try-on feature where users can see how the shade looks on their skin tone. It seems like this feature is helpful to users in selecting the right shade, therefore this shouldn’t lead to a drop unless users had already placed an order and then later came across this virtual try-on which made them realize they had ordered a wrong shade. ‍

Me: Is there an upcoming sale? The users may have heard of the promotion a bit late and decided to cancel their current orders to get a better price for their purchase

‍Interviewer: No, in fact we have a sale running now and we usually promote any campaigns way in advance.‍

Me: Was there a change in the checkout flow, the way in which a user inputs their address? Maybe the customer ordered it at the wrong address? ‍

Interviewer: But this shouldn’t be a lot of people to cause such a massive dip hence I'd discard this reason. Similarly product not reaching in time shouldn’t bother such a large percentage of customers hence it is safe to rule this out as well.‍

Me: Maybe they read a bad review for the product. Have we changed anything in the way we show ratings and reviews? ‍

Interviewer: Yeah we are prioritising to show good reviews over bad reviews through pagination. This could be another reason for users to start cancelling orders once they read bad reviews of the product elsewhereIt is important to show a healthy mix of positive and negative reviews as we come off as a trustworthy brand.‍

Me: Maybe users don’t think the products they have ordered are from authentic brands. Have we changed showing the stamp of authenticity or some changes along those lines?‍

Interviewer: So we thought having too many images is causing the product detail page to load slower. So we removed the image which showed the stamp. Instead we just added that as text in the product description.

So people shouldn’t have placed the order if they didn’t find the product authentic. It can happen though that they later realised that the stamp wasn’t there. Factoring this in with the small amount of population that orders luxury products it may not be too big a contributor. But since these are high ticket items, they definitely impact the revenue and therefore I will like to consider it.‍

‍

So it looks like the cancellations can be attributed to three changes:‍

1. Not showing a healthy mix of positive and negative reviews

2. Users realising they might have ordered the wrong shade after using the virtual try-on feature

3. The missing stamp of authenticity.

Now that I have found the probable causes I’d like to understand the motivations behind these changes, the metrics they mapped to and if they have had a positive impact on those. It is important to prioritise the features / changes that impact the north star and make the right trade-offs.

Post the RCA question, you can have multiple follow-up questions around resolution from the interviewer. However, we have only focussed the script here pertaining to figuring the root causes. To further learn about PM Interview questions and problem statements, join the PM School program.