Question: You are a PM at Flipkart who looks at cancellation rate as a metric. Cancellation rate goes up by 20% on Day 1 of the sale. Why do you think this is happening? Do a Root Cause Analysis.

Clarifying questions

Me: I believe that the cancellations we are talking about are only the items cancelled after placing an order and not abandoned checkouts/failed payments. Am I correct?

Interviewer: Yes, that’s a fair assumption.

Me: Flipkart also hosts various other features other than eCommerce like Flight bookings, Mobile Recharge, Voucher purchase. For the purpose of this question, are we only talking about Flipkart as an ecommerce website or shall I consider all the Flipkart services?

Interviewer: For this question, limit your purview to only Flipkart acting as an eCommerce aggregator platform.

Me: How do we define cancellation rate at Flipkart?

Interviewer: We define the cancellation rate as:

Cancellation rate = Number of orders cancelled divided by Numbers of unfulfilled orders in the last 7 days.

The numerator is a subset of the denominator.

Me: What do you mean by unfilled orders?

Interviewer: The orders which have been placed but haven’t been delivered yet.

Self Notes: The orders being cancelled here are still under transit and haven’t been delivered yet. The order statuses could be: Placed, Packed, Shipped, Out for Delivery.

Me: What do you mean by a “Sale”? Is it fair to assume that a sale is something like “Big Billion Days Sale”?

Interviewer: Yes, a sale is usually a publicised period where users are lured with attractive offers and BBD is an example of it.

Me: What's the competitive landscape during the period of the sale? Are our competitors also hosting a similar sale?

Interviewer: Yes, all the eCommerce platforms keep these occasional sales nearly around the same period only.

Me: What's the average change in the price during a sale?

Interviewer: Price change varies across categories and the percentage varies at different times of the year, but on an average, the final checkout value of an item is lower by 10-15%.

Self Notes: Average price of the products is reducing across all platforms around that time.

Defining Scope

Me: You said that the cancellation rate goes up by 20% on the Day 1 of the sale. How long have we been observing this trend or did this occur in the last sale only?

Interviewer: We have data about the last 5-6 sales in the last 1 year and it has been observed to go up on the Day 1 of the sale every time.

Self notes: Since it has been occuring from the last 1 year, it is highly unlikely that it is related to a recent app release or change in the cancellation flow. The reasons must be related to user behaviour.

Me: Have we observed any trend around order cancellations in a particular category of products?

Interviewer: We couldn’t find any trend there. Orders are being cancelled across categories.

Me: Are the order cancellations happening from a certain age group or geographical location?

Interviewer: No, we couldn’t establish a trend there.

Me: Is the order cancellation happening majorly through a certain channel like app, website, chatbot or calling the customer care?

Interviewer: Order cancellation is majorly happening via the app but that percentage is similar to the order placement channel as well.

Self Notes: Higher number of orders are being placed from the app and similarly higher number of orders are being cancelled via the app.

Me: Once the order has been placed, it can be majorly classified into “placed”, “packed”, “shipped” and “out for delivery”. Is the cancellation happening majorly at a certain stage?

Interviewer: Cancellations are happening across stages, except “out for delivery” as cancellation option is not available on that stage.

Me: I couldn’t spot any trends around channels, categories or stages. I would like to talk about the user journey here and go deeper on that. Shall I proceed?

Interviewer: Yes, go ahead.

Self notes:

User stage: User has placed an order in the Day 1 to Day -6 of the sale and their order has not been delivered yet.

User journey: User is aware that they have an order under transit. They:

1. Go to the app
2. Visit the order section
3. Click on the product under transit
4. Select the option to cancel the product
5. Select the cancellation reason
6. Place a request for cancellation.

Building a hypothesis

Me: Do we have any cancellation survey where we ask the user the reason for cancellation? What are the list of reasons in that survey? Is there any trend there?

Interviewer: Yes, we ask the user for a cancellation reason. The options include “delivery delayed”, “don’t want this product anymore”, etc. The trend was normal here.

Me: Do we levy any cancellation fee upon cancelling an item after it has been shipped?

Interviewer: No, currently we don’t levy any cancellation fee and give the user a full refund.

Me: While trying to view the status of their order, users land on the homepage of the app. How early before the sale is a banner put up on the homepage? Do we reveal sale prices of all the products before the sale?

Interviewer: We put up the sale banner as the main banner 3 days before the sale and no, sale prices of all the products aren’t revealed before the sale.

Self notes: Prices are dropping during the sale and they get revealed only when the sale starts ie. Day 1. Users get aware about an upcoming sale when they visit the app while tracking the order. A reason why the cancellation rate might be going up is the change in prices of the previously ordered product.

Hypothesis: Users are incentivized to cancel their order as the prices might have reduced and there is no cancellation fee. They would get the same item for a lower price.

Validating the hypothesis

Me: I have built a hypothesis and would like to validate it. Do we have any data on the price difference between the day the item was ordered vs the day it was cancelled?

Interviewer: On an average, the prices of the cancelled products had reduced by 15-20%.

Self notes: 15-20% average price reduction is a good enough incentive to cancel the product.

Me: How many of the users who cancelled an item ordered the same item again during the sale from Flipkart or any competitor platform?

Interviewer: We don’t have any data of the orders being placed at competitor platforms but 25% of the items cancelled were ordered again by the same user during the sale.

Me: Did the placement of order for the same item happen before cancelling or after cancelling the order?

Interviewer: It happened mostly after the in transit item was cancelled successfully.

Self notes: Users still wanted the product and they were comfortable getting it late. 25% is a big number to move ahead in this direction. Hypothesis seems validated.

Conclusion

Me: I have a hypothesis to present. The change in user behaviour can be highly accredited to the change in price of the products. In the user journey to view the current status of their order, they are being made aware of an upcoming sale or upcoming reduction in price.

Interviewer: Go ahead, complete your thought.

Me: This is my final analysis as an RCA of why this would be happening:

1. While trying to know the latest status of my order, the user is going through various touch points where they are motivated to check the latest reduced price of their in-transit item. When the prices are revealed on the day 1, they have a call to take. If the difference is huge and they are comfortable with a delayed delivery, they are incentivised to cancel the order and place it again. Loss aversion principle is the motivator here as they can get the same item at a lower price without paying any cancellation fee.
2. Also, since competitor platforms are organising a sale during the same period, it can be concluded with a high probability that a good percentage of users might be cancelling on Flipkart and ordering on other websites.

Interviewer: Thanks for helping us with the RCA. I believe this is a great starting point for us to build a user story and make changes to the product, in an attempt to control the cancellation rate on the Day 1 of the sale or cancellation rate in general.

________________________________________________________________________________________________________________________________________________________________________________________________________

Strategy to solve RCA questions:

1. Understand and don’t assume. Make sure every keyword in the question is clear to you and you are not making any assumptions.
2. Clarifying questions. Ask clarifying questions to understand the problem better.
3. Define scope. Limit scope by talking about internal and external factors and finding trends.
4. Build a hypothesis. Observe the user journey and build a hypothesis.
5. Validate the hypothesis. Repeat above step if hypothesis is not validated or is parked for the time being.
6. Conclude

Question: Why do you think this is a worthy problem to solve? Who are the stakeholders getting affected in this situation?

The primary stakeholders in this situation are:

1. Customers
2. Flipkart, the listing platform
3. Vendors/Sellers

The secondary stakeholders in this situation are delivery partners/agencies.

Here are the pain points of every stakeholder:

1. Customers:some text
   1. Delayed delivery times for getting the same product
   2. Loss of trust on the platform due to fluctuations in the prices
2. eCommerce Platform:some text
   1. Loss of business to competitive platforms
   2. Loss of business due to lower revenue/profit on the same item
   3. Increased spends on pickup and delivery leading to higher operational/logistical costs
3. Vendors/Sellers:some text
   1. Losses due to damage of goods during transit
   2. Loss of margins on selling the same product again at a lower price
   3. Difficult to optimize for stock keeping before and during the sale
4. Delivery partners:some text
   1. Higher number of return order handling during the peak businesses days of the year
   2. Delayed delivery timelines due to increased workload, which could have been prevented

According to me, we should solve this problem for the vendors who list their products on Flipkart as they are getting affected the most due to this problem. High return rate creates an illusion of fake stock out of products, where the website shows that the product is sold out and sale is denied to users who might actually want to buy it. It causes loss of business for the vendor, while they still had stock available which wasn’t sold after the sale. The user who placed an order twice(second time at a lower price) reserved 2 units for themselves, out of which they are only going to keep 1 or return that one as well.

Also, huge losses occur during transit. The products which are returned get damaged and don’t remain in a condition to be sold again.