Q1: If we wanted to evaluate the performance of an arm app vs. a drm app, how would you define performance?
A1: The goal of customers is to maximize the likelihood that their messages are accepted by their target recipients. Therefore the app type with the higher percent of messages accepted out of all messages accepted and messages expired is the better performing app.
I chose to omit queued, and failed messages from the performance calculations because I would assume that they are not a result of the type of messsage sent, but rather Crowdriff's delivery software. I also omitted sent (pending) messages because we do not yet know whether they will be accepted or not.
It should also be noted that each type of app likely gives different permissions to our customers which may be of differing values, but given that there is no way for us to quantify this value with the data we have, we will treat an accepted message as being of equal value in both cases.
Q2: Based on your definition, how did the two app types perform?
A2: After removing for outliers and accounting for discrepancies in the database, the two app types performed relatively evenly. DRM messages had a 66% acceptance rate compared to a 62% ARM acceptance rate.
It should be noted that the sample size for DRM messages was 48,000 messages, compared to only 1,100 ARM messages. We may want to wait for a larger sample size before assuming that 62% is an accurate ARM acceptance rate.
Q3: Do you spot interesting patterns on how our customers (collections) use the drm and arm apps? Tell us what you think!
A3: An overview of all my findings and action items to follow up on can be found in the cell below.
Even of those 14% of active app users, they only use 61% of the apps that they have created.
In order of importance:
First let's import and clean our raw data
Let's create a locally hosted PostGres server and a 'Crowdriff' database so we can store our data and manipulate it more easily.
Next let's create tables in our database and insert rows into the tables from our dataframes
Now that we have a database to work with, let's start by getting an overview of how many customers have access to ARM vs. DRM vs. Both
Almost no customers have only ARM access, most have both. Let's see a breakdown of the success statuses of ARM vs. DRM messages
Ok so it looks like a lot of ARM messages are still waiting on responses. Let's filter those out since we don't know whether they will be accepted yet. Let's check only those that have been accepted vs those that have expired.
Wow ARM messages have a very high acceptance rate! There are also a lot more DRM messages than ARM - was that because the ARM feature was added more recently? Let's check the oldest dates we have on record for each...
Nope they started at the same time, so people are just blasting out way more DRMs than ARMs. Maybe that's because they only send out ARMs selectively after relationships are establish through DRMs... Let's check if there are apps with an ARM message sent that have also sent at least one DRM message
Not a single app with an ARM message also has a DRM message. So apparently apps are either ARM or DRM but cannot send both message types. Let's take a look more broadly at the behaviour of the apps sending out the most messages
The vast majority of our ARM messages (80%) are coming from two users who are spamming messages. One of them only has an 8% acceptance rate, which is dismal compared to what they could be achieving by just sending DRM messages. The other has a sub 50% acceptance rate, also below what one would expect from DRM messages. This could mean that the other people sending ARM messages only send them after first establishing contact via a DRM message.
Maybe the customer success team should give these customers a quick call to let them know about DRM messages, since they could have a much higher success rate. Let's quickly check those collection_id's so we can let CS know.
Let's circle back now and check the success rates for ARM messages when omitting those two anomalistic use cases
Ok it looks like a very high rate of ARM messages are being accepted after accounting for the outliers. I wonder how long each of the sent messages has been pending for, when compared against DRM sent messages. I suspect that ARMs will be pending for less time because of my theory that they are only being sent after having a pre-established relationship with the message recipient.
According to the most recent entries in the database it is now at least 2018-12-05 at the time of this case. Let's see how many days between message creation and that date have elapsed on average for DRM and ARM 'sent' messages.
What?! I thought messages were marked as expired after 30 days, but apparently that is only DRM messages that are marked as expired... Let's mark ARM messages older than 30 days as expired and then compare pending 'sent' times again.
Because of that expiry difference, that means that we haven't been comparing apples to apples yet when looking at success rates of ARM vs. DRM. Let's circle back and check our acceptance rates again.
Suddenly, ARM seems to have terrible acceptance rates compared to DRM, but although the ARM acceptance rate looks terrible now, we should check it without the two ARM spamming outliers that we identified above...
Looks like there isn't much difference between the two after all when accounting for the spammers. This seems too inconclusive for an interview case... Suspicious. Note our ARM sample size is very small
Let's try a totally different approach and see what else we can discover. Let's bucket our customers by how many apps they have created.
The majority of customers have only created one app. I wonder if those customers have even used this feature of crowdriff. Let's check to see how many customers have sent a rights message.
Hold up. We have 2486 customers with access to apps, and only 359 of them have actually sent rights messages. Something is going on here. Perhaps many customers haven't been educated well enough on how to use this feature?
This also makes me wonder if those 359 customers are power users who account for the bulk of the apps created... Let's break down those 359 users by number of apps created
As expected, 880 out of 1168 apps created have been made by our 359 power users (Out of 2486 total users). In other words 75% of apps created are made by 14% of our users. The 80/20 rule is in full effect here...
Let's check to see what fraction those 880 apps make out of the total number of distinct apps used to send rights messages.
Very interesting. Our power users have created 880 apps but only 540 apps have been used to send rights messages. Perhaps our power users are intentionally choosing not to use ARM or DRM apps for some reason.
Unfortunately, we cannot check what type of app these unused apps are because we can only see app type after a message has been sent. It is very plausible that power users have identified one type (DRM or ARM) that is superior or easier to use and gravitate towards using only that app type.