Q1: If we wanted to evaluate the performance of an arm app vs. a drm app, how would you define performance?

A1: The goal of customers is to maximize the likelihood that their messages are accepted by their target recipients. Therefore the app type with the higher percent of messages accepted out of all messages accepted and messages expired is the better performing app.

I chose to omit queued, and failed messages from the performance calculations because I would assume that they are not a result of the type of messsage sent, but rather Crowdriff's delivery software. I also omitted sent (pending) messages because we do not yet know whether they will be accepted or not.

It should also be noted that each type of app likely gives different permissions to our customers which may be of differing values, but given that there is no way for us to quantify this value with the data we have, we will treat an accepted message as being of equal value in both cases.

Q2: Based on your definition, how did the two app types perform?

A2: After removing for outliers and accounting for discrepancies in the database, the two app types performed relatively evenly. DRM messages had a 66% acceptance rate compared to a 62% ARM acceptance rate.

It should be noted that the sample size for DRM messages was 48,000 messages, compared to only 1,100 ARM messages. We may want to wait for a larger sample size before assuming that 62% is an accurate ARM acceptance rate.

Q3: Do you spot interesting patterns on how our customers (collections) use the drm and arm apps? Tell us what you think!

A3: An overview of all my findings and action items to follow up on can be found in the cell below.

Summary of findings

  • Customers tend not to have ARM rights on their own. They almost always have them in conjunction with DRM rights.
  • There were two factors skewing ARM message success rates that had to be accounted for before comparing message acceptance rates between ARM and DRM messages:
    • ARM messages were not being correctly marked as 'expired', thereby artificially decreasing the number of 'expired' messages in the pool
    • There were two ARM message spammers that accounted for 80% of the ARM messages in our database
  • After accounting for both of those factors, DRM messages had a slightly higher acceptance rate of 66% compared to an ARM acceptance rate of 62%.
  • The vast majority of our clients (88%) have only created one app (or their account started with an app and they never made another).
  • Only 14% of our customer base has ever sent a rights message at all. Those 14% of customers account for 75% of the total apps created.
  • Even of those 14% of active app users, they only use 61% of the apps that they have created.

Action Items


In order of importance:

  • Why do 85% of our customer not use an ARM or DRM app? Do they not understand how it works or is it not providing them any value? CS can speak with customers to find out.
  • Find out why our power users are not using many of the apps they have created. Is it because they tend to discard ARM or DRM apps without trying them?
  • Follow up with the ARM spammers. Are they not educated on using DRM messages in bulk over ARM, if so do we need more prominently featured educational tools for these apps?
  • Why are ARM messages in the database not being marked as expired? Fix if it's a bug.

First let's import and clean our raw data

Let's create a locally hosted PostGres server and a 'Crowdriff' database so we can store our data and manipulate it more easily.

Setting up server

Setting up server

Next let's create tables in our database and insert rows into the tables from our dataframes

Now that we have a database to work with, let's start by getting an overview of how many customers have access to ARM vs. DRM vs. Both

Loading output library...

Almost no customers have only ARM access, most have both. Let's see a breakdown of the success statuses of ARM vs. DRM messages

Loading output library...

stacked bar chart

Ok so it looks like a lot of ARM messages are still waiting on responses. Let's filter those out since we don't know whether they will be accepted yet. Let's check only those that have been accepted vs those that have expired.

Loading output library...

Wow ARM messages have a very high acceptance rate! There are also a lot more DRM messages than ARM - was that because the ARM feature was added more recently? Let's check the oldest dates we have on record for each...

Loading output library...

Nope they started at the same time, so people are just blasting out way more DRMs than ARMs. Maybe that's because they only send out ARMs selectively after relationships are establish through DRMs... Let's check if there are apps with an ARM message sent that have also sent at least one DRM message

Loading output library...

Not a single app with an ARM message also has a DRM message. So apparently apps are either ARM or DRM but cannot send both message types. Let's take a look more broadly at the behaviour of the apps sending out the most messages

Loading output library...

DRM vs. ARM messages by the 20 most active apps in each category

The vast majority of our ARM messages (80%) are coming from two users who are spamming messages. One of them only has an 8% acceptance rate, which is dismal compared to what they could be achieving by just sending DRM messages. The other has a sub 50% acceptance rate, also below what one would expect from DRM messages. This could mean that the other people sending ARM messages only send them after first establishing contact via a DRM message.

Maybe the customer success team should give these customers a quick call to let them know about DRM messages, since they could have a much higher success rate. Let's quickly check those collection_id's so we can let CS know.

Loading output library...

Let's circle back now and check the success rates for ARM messages when omitting those two anomalistic use cases

Loading output library...

Ok it looks like a very high rate of ARM messages are being accepted after accounting for the outliers. I wonder how long each of the sent messages has been pending for, when compared against DRM sent messages. I suspect that ARMs will be pending for less time because of my theory that they are only being sent after having a pre-established relationship with the message recipient.

According to the most recent entries in the database it is now at least 2018-12-05 at the time of this case. Let's see how many days between message creation and that date have elapsed on average for DRM and ARM 'sent' messages.

Loading output library...

What?! I thought messages were marked as expired after 30 days, but apparently that is only DRM messages that are marked as expired... Let's mark ARM messages older than 30 days as expired and then compare pending 'sent' times again.

Loading output library...

Because of that expiry difference, that means that we haven't been comparing apples to apples yet when looking at success rates of ARM vs. DRM. Let's circle back and check our acceptance rates again.

Loading output library...

Suddenly, ARM seems to have terrible acceptance rates compared to DRM, but although the ARM acceptance rate looks terrible now, we should check it without the two ARM spamming outliers that we identified above...

Loading output library...

Looks like there isn't much difference between the two after all when accounting for the spammers. This seems too inconclusive for an interview case... Suspicious. Note our ARM sample size is very small

Let's try a totally different approach and see what else we can discover. Let's bucket our customers by how many apps they have created.

Loading output library...

Customers bucketed by client

The majority of customers have only created one app. I wonder if those customers have even used this feature of crowdriff. Let's check to see how many customers have sent a rights message.

Loading output library...

Hold up. We have 2486 customers with access to apps, and only 359 of them have actually sent rights messages. Something is going on here. Perhaps many customers haven't been educated well enough on how to use this feature?

This also makes me wonder if those 359 customers are power users who account for the bulk of the apps created... Let's break down those 359 users by number of apps created

Loading output library...

As expected, 880 out of 1168 apps created have been made by our 359 power users (Out of 2486 total users). In other words 75% of apps created are made by 14% of our users. The 80/20 rule is in full effect here...

Let's check to see what fraction those 880 apps make out of the total number of distinct apps used to send rights messages.

Loading output library...

Very interesting. Our power users have created 880 apps but only 540 apps have been used to send rights messages. Perhaps our power users are intentionally choosing not to use ARM or DRM apps for some reason.

Unfortunately, we cannot check what type of app these unused apps are because we can only see app type after a message has been sent. It is very plausible that power users have identified one type (DRM or ARM) that is superior or easier to use and gravitate towards using only that app type.