Tim Rowe:
Cory Gill, you did what is really, really hard to do. You went viral on DTC Twitter and that's usually it's not, it's not for a good reason, but you went viral on DTC Twitter for really good reason. You released an incredible piece of research about the Shopify app store. And I'm excited to break down some of what you've learned through that and then how specifically you're applying that to your own startup. Cory, thanks so much for being here.

Cory Gill: Thanks for being here, Jim. Thank you for inviting me.

Tim Rowe: Absolutely. Absolutely. I think a good place to start. You've got a great professor. We got a shout out. Shout out your professor along the way for letting you pick the Shopify app store. as the subject for this this project, but what was maybe what was the project and why did you pick the Shopify app app store?

Cory Gill: Yeah, so it was for my econ capstone class in my last semester, which I just graduated from at Northeastern in Boston. And there's you basically just pick a topic within economics. And as people that say economics, that's a pretty broad category. And there was people in the class that did everything from like looking at the health care system to But there was not many people that chose the Shopify app store ecosystem.

Tim Rowe: But you got to pick a subject in class for the project. There wasn't a big line for the Shopify app store. You weren't competing with anyone else.

Cory Gill: All right. Even when I told my professor, there's so many times throughout the semester where he'd accidentally say Spotify instead of Shopify. So funny. Yeah, but the project was really just creating an economics thesis paper or a research paper looking at some topic and testing some hypothesis. But it was the entire purpose of the class, because at that point you've taken all the base requirements. And yeah, he was the best professor I probably could have gotten. And even at the beginning, it wasn't initially my idea. But when I told him that I had a startup, which is why I wanted to do the paper, and because I had a Shopify app, he said, you have to do this. There's no way that you're not going to do something related to it, which was cool to get that support and have him be interested in me and my startup within it.

Tim Rowe: So the chicken came before the egg in this case, you already had founded the company and you were interested really in kind of like competitive research. This was an opportunity for you to do a deep dive into the ecosystem that you specifically play in. And we'll talk about that. Well, what did you learn when you started to go back and kind of study historically, you know, app stores like Google and Apple, there was some really interesting data. And I think it'll be good for us to partition between those are consumer app stores. Shopify is unique in that it's a B2B app store. I don't know if there was an analysis before this, but it would seem like these are kind of first learnings, no? Yeah, they, they were for the most part.

Cory Gill: It was definitely the first. analysis of the Shopify app store that I could find, there was one that was in a separate field, kind of more like computer science, looking at AI review techniques and things like that, but not the actual algorithm or the factors that impact rankings, which there's quite a bit of research, even still not a ton, but quite a bit of research on like the Apple App Store or the Google Play Store, where people do look at those factors. But very little. And just the B2B app store market in general is not very investigated just because I mean, Shopify is probably the largest B2B app store ecosystem that I can think of. There's some research that was dabbled in for Salesforce as well. But even still, no real economic analysis, I would say, if you can still call it that, what I did. But yeah, it was definitely interesting to see that just because it is kind of a new domain. And I think there's a lot of data that we can talk about later that's untapped in terms of what you could do and the research you could do within the Shopify ecosystem. So it's cool to be able to hopefully start a little something where people might become more interested in it like they are in the B2C markets.

Tim Rowe: 27 pages of deep research, so I'm sure you've got a lot of people's wheels turning. And even as someone who isn't participating in the, in the Shopify ecosystem at all, I found it to be interesting. What did you learn looking back historically at some of those B2C app stores and marketplaces? Like what, what were the, the learnings that came from kind of laying the groundwork to this?

Cory Gill: Yeah, so there was a few papers that were really interesting. I just have it up next to me right now as well. But a lot of it was figuring out the factors that determine what makes apps successful and what helps them survive, which was the main paper I think that I took inspiration from. One of the big ones, though, was by Carrere in 2012. And what he found was how much the willingness to pay for a top-ranked app compared to an unranked app. And he found that a top-ranked app, there was about $4.5 greater willingness to pay than for an unranked app, which was pretty remarkable to see just how important the rankings play in that data. And there was a lot of research as well around trying to determine the install amounts that apps have. And those were also very mathematical and Even my professor didn't really understand some of the math that was going in behind that, so I didn't exactly dabble in it, which was one of the limitations of my paper, I would say, because that's not public data. But those were a lot of what I was doing. I was looking at different factors, how it impacted ranking, the willingness to pay, install data. So like that big paper for the install data was trying to back into how many installs an app is getting most likely, given the other factors that they have on their app, like reviews and whatnot.

Tim Rowe: So, and I guess if we distill it down the element of social proof, reviews, ranked versus unranked, some sort of credibility given to the app, that app is going to perform better if we're comparing the same type of app. Is that about a fair summation?

Cory Gill: Yeah, definitely. And it both, both in terms of. consumer's perception and just the actual algorithm that boosts their rating. Oh, interesting. It's kind of self-reinforcing as well, which is a big takeaway, is that the best keep on getting better within an app store because of the added impact of being higher ranked. They get more installs, they get better reviews, most likely most of the time. And I talk about that which is one of the reasons why my model is called a lag-dependent variable model, which essentially means that one of my variables on the right-hand side of the equation that impacts the app's ranking on the other side of the equation, like what I'm looking at, is the previous week's ranking. And it's basically just saying that yesterday's or last week's ranking impacts my current week's ranking, which is the case because apps do get better and it results in the other factors getting better as well.

Tim Rowe: So like a song on the billboard charts, it was number 10 last week. Doesn't necessarily mean it's going to number eight this week, but what your research found was that if it's an app, if it performed well last week, we're using that as some sort of indicator for how the algorithm is going to reward the app this week.

Cory Gill: Yeah, exactly. It's kind of just like momentum in general, like a way to describe her, like for even a stock that gets invested in like a meme stock, once a few people start to get in better and there's The curve isn't linear. It's a lot more exponential in terms of the factors that are on that app in their growth as well.

Tim Rowe: Was there any evidence or any, any learnings around how those early or how those apps got early traction? Was there any evidence of what they had done to get themselves into that position?

Cory Gill: In terms of actual strategies, not anything super notable within the actual research, like from my experience with my own startup and talking to founders in the space, there's stuff that I've learned from that. But one notable thing, which I guess ties back to the last point as well, is that if you look at the data and the number of reviews that apps do get, it is pretty much always exponential in terms of how they grow. The time that it takes to get from 0 to 30 reviews or 0 to 100 reviews is much, much longer than it takes to get from 100 to 1,000 or exponentially higher. And that is because you're getting to that top 10 position within your category or within a keyword, and you start to experience that exponential growth. And that applies to pretty much every app that I looked at because the apps I looked at were currently within the top 50 of a particular category as well.

Tim Rowe: Makes a lot of sense. Critical mass momentum. We'll call it the snowball effect. What were some of the learnings that you found from other founders around strategy? Any, anything that, that you can share with us or if it's top secret, you don't have to share, but curious if there's anything that you can share that led to early success.

Cory Gill: Yeah, definitely not top secret because we're still figuring it out and we're hoping to hit that crux inflection point where you really start to see the benefits of that. There's a lot of stuff around just app store optimization and your keywords that you're using, your copy that impacts your ranking as well. Because those are other factors that I couldn't account for, for the most part. You probably could in theory, if you scraped different app listings and saw. But that has a big impact on how high you're going to peer for. If you're looking for upsell apps, how many times you mention upsell, is it in your app's name that you say upsell? You're probably going to rank higher. And those are kind of just not really tricks. I guess they're somewhat tricks of the trade in terms of just apps or founders talking about them. But even Shopify talks about them a little bit on their website and what makes an impact. Other big things, the competitiveness of your category obviously makes an impact. And the competitiveness of the keyword. If you're in the pop-up space like us, there's a lot of apps with a lot of reviews that are often free. So a lot of people try them out, making the pop-up keyword difficult. But if you're in a… Another keyword that we used to try was education. Okay, we basically shot up to the top first. But that's just because not many people look at it. Not many apps are focused on customer education. So it's a less dominated area. That's in terms of the app, the app store. But then there's also a lot of these companies are very large businesses as well that are on the app store, which I talked about at the beginning as well, like Klaviyo is listed on the app store, and they generate billion dollar company now. Yeah, yeah, exactly. And I think 70% of their revenue that they announced when they went public was from Shopify, or close to that. Crazy. Obviously, not all of that's from the App Store. And they also have a partnership with Shopify. But It kind of speaks to the scale of people that are in there and their marketing strategies stretch far beyond. Right.

Tim Rowe: You're also in the category with the proverbial $8 billion gorilla. Yeah, exactly. Yeah. It makes sense. And kids, this is, this is why we study SEO. I think it's this recurring theme that. They all end up being, it's always some sort of search engine function. The same thing with podcasting, the show notes, the title, all of that. It's about being discoverable, which I think is, is an important theme for us to, to carry throughout any conversation. Did you have an initial hypothesis about what you might find, or did you kind of just go in looking to, to discover unique things about the Shopify store?

Cory Gill: It was both. I had some idea of what factors I knew would make an impact on the app ranking. I knew reviews. I knew installs would. I knew the built for Shopify badge would as well. I knew the average view rating would. And then there was other factors I couldn't really measure, which was unfortunate that I also had a hypothesis that do impact it. But even as Shopify talks about it a little bit in terms of what factors go into the algorithm. And there's previous research as well for other app stories like we talked about that looked at what factors impact it. It just wasn't yet looked at for Shopify, but yeah, I definitely had some hypothesis going into it about what might happen.

Tim Rowe: What was your first, I'm sure there was a moment when you're looking at the numbers and it came to life and you were like, oh my gosh, what was the first, oh my gosh moment as you were working through the research?

Cory Gill: Probably. Well, like more in a bad way, probably when I was taking in the data from sassy, which is the big, um, app data provider that I use at least for the paper. And it was probably the most notable one in the ecosystem. Um, but it's just how many data points I would have to gather. And then when I realized that I couldn't scrape it automatically, so I had to manually and put it, that was the, Oh shit.

Tim Rowe: Hey professors, is it cool if I changed my topic? Yeah. Yeah.

Cory Gill: Well, I actually like I realized then that I couldn't unfortunately do more than 50 apps probably because I had the deadline and More than that, it was already the top 50, which is another factor to consider when thinking of the results of the paper. But yeah, that was probably the big moment. And just being able to see all the data and all these apps where they started from, because there was someone there that was pretty cool to see where they only started a year ago or a year and a half ago. And now they're in the top 50 within their category and the growth of the number of reviews they have was pretty cool to see.

Tim Rowe: Oh, that's really cool. What were, what were some of the findings that you, you discovered throughout that you mentioned reviews, the number of installs, the badge. I think that there's, there's some interesting things that maybe the audience would think were more important that end up not being as important. And maybe some things that you would think are helping, but maybe aren't entirely helping. What did you find with all of those factors combined?

Cory Gill: Yeah, so the big thing to keep in mind, which I tried to preface a lot in my tweet was that, and I just said is it's the top 50 apps within that particular category. And the particular category doesn't matter too, too much. But the fact that it's in the top 50, means that it's not necessarily translatable to how an app in the bottom 50 would perform. And that's important to keep in mind if you are a new company. And if I were to do the research again, it'd be nice to just have a random scattering of apps to be able to do that analysis. Okay, a more mixed sample. Yeah, exactly. But in terms of the results, so there was four factors I looked at, the number of reviews, the average review rating, which is just the out of five score for the app, the built for Shopify badge and whether or not they had it, and then the last week category rank that I talked about. And the interesting thing, so I ran four models with it, and they were all pretty consistent, which was a good sign. But the number of reviews had a basically negligible impact on the app ranking or how high they appear in the category, which was my dependent variable that I was looking at.

Tim Rowe: So the number of reviews didn't have a discernible impact to the overall performance?

Cory Gill: Not big, no. Okay. Average review rating did have a big impact. So if the average review rating went from a four to a five or it maxed out from four to five, the impact on your ranking position would go up by almost nine spots, um, which for an app within the top 50 is huge. 20% jump. Yeah, exactly.

Tim Rowe: And wherever you are, that's incredible.

Cory Gill: Yeah, so that had a big impact in the direction that I expected. The number of reviews was somewhat what I expected, but not really. I did expect it to have an impact where it would boost your ranking pretty significantly. And then the bill for Shopify badge was the biggest surprise. It said that my results said that if you did get the badge, your ranking would become worse by about 14 positions, which I'll talk about kind of the contingencies with that result later because it's, as I'll say, it's definitely not fully accurate because my app has to build for Shopify badge and our app ranking went up the day after getting it. So there was some shortcomings also with like how I collected that data and then for the category rank. less impactful somewhat, just because I just used it in my model. So there's not really even a point to focus on that. But those were the three main variables and how it performed. The other big takeaway is that my R squared value, which just tells how much how much of the relationship between the variables that I just said and the ranking can be described by those variables? Or in other words, how much do those variables matter in the ranking was about 58.58 or like 58%. So it had good explanatory power for what your app ranking could be.

Tim Rowe: So by choosing those factors, this is a good test of what it takes given the sample size of top 50, top 50 ranking. That's, that's what that number means. Yeah. Okay.

Cory Gill: It's just saying that it's good at explaining what your ranking is. Even just, even though it's only three variables, it's pretty good at explaining what you're ranking and how it will change.

Tim Rowe: Right. The outcome is not due to random chance. It's, it's on a, it's on a confidence that these are factors that are important to grow. What does it all mean? You're, you're a Shopify app store founder yourself. How are you applying the things that you learned? And I'm sure you've met with a bunch of founders along the way, especially since releasing the research. What are you, what are you distilling it down to? What's, what's the big takeaway?

Cory Gill: Yeah, the, the big things I would break it up into two. One, if I'm a large. I found it was already fairly high in the rankings. It's important to bear in mind your rating considerably because even though an improvement in your rating by one means nine better positions, it also means if it goes down by one, that's nine worse positions. Great point. That's an important thing to keep in mind. And it's also important to keep in mind that for those large apps, they have so many reviews already that another review is not very incremental. It's not going to move the needle. But over time, ensuring that you're still getting good ratings on those reviews will matter a lot. So that's important to bear in mind. And then the bill for Shopify badge. So the contingency there is that there was only four apps within the top 50 that had it. And I don't know exactly when they got the badge as well. And I think just candidly, as an app founder, it does make a difference. And especially for smaller apps, where it's less uplift to make the changes to apply for it, you should still apply for it. And I think that's common belief now in the ecosystem. And for smaller app founders, how you can apply this research, it's still less so. And I wouldn't say like, take it for like, that you shouldn't focus on reviews because you should. It's that you need to bear in mind your rating still try to get as many reviews. And the important takeaway is that for large apps, their reviews start to matter less, but I'm sure if you go on the smaller apps, their reviews would matter considerably more to the position.

Tim Rowe: getting lots of great reviews. I think that's my key takeaway is go get more great reviews in whatever you do. What are you working on? You've got it. You've got a, an app that we were talking in the lead up and it was an aha moment for me, but it's a new take on pop-ups.

Cory Gill: Yeah, so we're a pop-up alternative that lets you reward shoppers for learning about your brand or products. So if I'm a shopper and I go on to one of the sites that uses our app Alia, I could go on and in one or two minutes, through engaging content, learn about what makes the brand special, what makes the products unique. answer a few questions and then get a reward at the end or 15% off. So the idea behind it and the benefit to stores is it improves their conversion rate considerably more than traditional pop-ups. And you're able to ask a lot more questions to collect data on your merchant. Oh, interesting. Or on your shoppers as well.

Tim Rowe: And it would seem like that's a more valuable customer, I would say. We've all signed up for a discount on a pop-up for a thing that we have no intention of buying just to make the pop-up go away because the X was too hard to find. Whereas this is, it reminds me more of like, for anyone listening, Coinbase, how when you learn about a new cryptocurrency, you can earn fractions of a share of it. For me, just as a user of Coinbase, that was always fun and engaging. Are you collecting a lot of data for brands around that engagement piece? Like what is, what is the engagement with this style of discount? What are you finding with that?

Cory Gill: Yeah, so not only do they get the email or the phone number that you would get from a traditional pop-up when you sign up, but you also get to ask more survey questions that are often more insightful. So, for example, if you're on a brand, let's just say Nike, for example, and you see the pop-up, and you start clicking through, it'll might ask you what type of product you're most interested in. You say running shoes, you might say, are you shopping for men or women? Sure. We're not sure yet. And you're able to collect that data or maybe asking what makes them most hesitant to make a purchase and using that data plus the email and phone number. Not only are you going to convert them better because they just learned about the brand and products, whether that be in a follow up or in the same session, but You're able to segment their targeting later and personalize it more. So it makes it easier to sell to them now in that moment and afterwards as well.

Tim Rowe: More efficient on the backend because I'm building those buckets of customers based on it's really, really smart. It's really, really smart. How'd you get the idea for that? Obviously you were working on that before all of this research. Where did, where did this originate from?

Cory Gill: Yeah. So my co-founder, his mom previously ran a Shopify store. And him being a dev and then our other co-founder being a dev as well. We came together because her brand and her products had a lot of story behind them. And he wanted to get her story across better and get it to more shoppers that came to the site. So he wanted to build a tool to help her with that problem. And we came together to do it. Unfortunately, our store is not open anymore. But now other stores get to take advantage of it, which is cool.

Tim Rowe: Yeah, I would say that's probably a problem that most folks have that are, that are on Shopify, right? We're selling something that is personal to us and that has a story and that has a founder behind it. How do you, how do you convey all that? And oftentimes, yeah, pop-up doesn't seem like it's doing justice to the story. So. Kudos to you guys for figuring that out. The research is incredible. If it's okay with you, Cory, I'd love to link to, I can host it right on the Out of Home Insider blog page. If it's okay, we can link to that research so folks can get it there. Is that okay? Or do you want to send to your site? I know you had a landing page set up for this. Where should folks go? if they want to find the research.

Cory Gill: Yeah, the landing page is probably the best spot. Cool, that's what we'll do, we'll link there. Free to download as well, you don't need to put in your email. I had no intention to try to capture leads from this, it was more just to get it in the hands of as many people that found it interesting. So yeah, you can just go to the landing page, download the paper, Um, and have a, have a read.

Tim Rowe: Perfect. That'll be linked in the show notes. Where else should we go? Cory, you're active on Twitter. Where do we go to learn more about you? What you're up to stay in the know, learn more about the app. Give us the lat long. That's what we use here for billboards. Give us the lat long on where to find you.

Cory Gill: Yeah, the website's a great place to learn more about the app. And then Twitter, it's just at CoryLGill on Twitter. And then on LinkedIn as well, just CoryGill. And you can follow us on there. See, I post a lot on Twitter primarily, just about how the app is doing and other stuff related to Shopify and e-commerce. So that's probably the best place as well. If you want to reach me, shoot me a DM whenever. Beautiful.

Tim Rowe: That's how we got in touch. I can't thank you enough for not only doing the research, but then being willing to engage in this follow up conversation. I'm sure there's a lot of folks out there who were taking something away and applying it to their business, whether they've got a Shopify app or not. So Cory, thank you. Thank you so much, Tim. It was awesome.

Cory Gill: Absolutely.

Tim Rowe: If you found this episode to be helpful, please, I'm going to redo that. If you found this episode to be helpful, please share it with someone who could benefit as always make sure to smash that subscribe button and wherever you're listening. More importantly than ever before. Leave the podcast review. That's how you help us grow. You heard it from Cory first. We'll see y'all next time.