We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 – 28. Join AI and data leaders for insightful talks and exciting networking opportunities. Register today!
Using algorithms to make purchasing suggestions is big business. Netflix reported that its recommendation engine contributes $1 billion to its bottom line every year. However, sometimes the suggestions are way off.
Take, for example, an ad I received to apply for a job as a van driver. I have never been a professional driver, I don’t even like driving and I have never owned a van. It’s clear that this recommendation engine knows nothing about me.
There are several different ways recommendation algorithms can reach the wrong conclusions. Here are just a few examples for each type of recommendation engine.
1. Collaborative filtering
This filtering method is based on collecting and analyzing information about user preferences. The assumption is that if two users have one common interest, they will have other interests in common, so product recommendations will be a match for both. The benefit of this type of analysis is that the algorithm doesn’t need to use inferences from deep learning to understand the item that’s being recommended, it just needs to identify users that have similar interests.
However, one downside of collaborative filtering is that it needs a large dataset with active users who have rated or purchased a product in order to make accurate predictions. If you have little user activity, it is much harder to generate good-quality recommendations. The number of items sold on major e-commerce sites is extremely large. Therefore, even the most popular items could have very few ratings. This is considered the long tail, or scarcity of data problem.
There is also no way to handle new items that haven’t been rated before.
In addition, there are millions of users and products in many of the environments in which these systems make recommendations. Thus, a large amount of computation power is often necessary to make the required calculations, which means many companies are forced to limit the amount of data their models ingest, which can negatively impact accuracy.
2. Content-based filtering
Content-based filtering methods use keywords that describe an item to make a match between recommendations and people. For example, when recommending jobs, keywords of the job description can be matched with the keywords in the user’s resume.
The biggest downside of this model is that it can only make recommendations based on the existing characteristics of the user. It also requires text analysis, which can introduce mistakes when the algorithm needs to identify keywords that are written differently; for example: instructor, trainer, teacher or facilitator.
This type of recommendation engine is also challenged when the solution is multilingual and requires translating and comparing words and phrases in different languages.
3. Hybrid recommendation engines
Hybrid recommendation systems use collaborative filtering and product-based filtering in tandem to recommend a broader range of products to customers with more accurate precision.
Hybrid recommendation systems can generate predictions separately and then combine them, or the capabilities of collaborative-based methods can be added to a content-based approach (and vice versa). In addition, many hybrid recommendation engines include analysis based on demographics and include knowledge-based algorithms, which make inferences about users’ needs and preferences based on deep learning.
However, even if hybrid recommendation engines can improve accuracy, they can suffer from longer compute times. The importance of speed differs based on the application. For example, movie and ecommerce recommendation systems can learn at a slower pace while an application that recommends who to follow on Twitter is bound to change frequently, forcing a recommendation engine to make predictions in close-to-real-time based on fresh data.
In addition, personal interests have different levels of time sensitivity. For example, individual sports like running or swimming are long term, while following sporting events like championships for favorite professional teams can change all the time. Recommendations based on real-time matches need to be more frequently updated.
Improving accuracy for all types of recommendation engines
In all cases, to be more reliable, recommendations should be varied, adapt quickly to new trends, and have the ability to scale up quickly to process more data. One way for developers to improve the accuracy of their recommendation engines is to use off-the-shelf pretrained models and invest in MLops tools that can help speed up the process of putting models into production and regularly monitor models to check for drift.
I am personally always happy to see recommendations for restaurants, bars, books and music performances. Even if the predictions are way out there, I can be convinced to try new things. But using more complex models that are pretrained with more data will reduce the likelihood that I will be prompted to apply for a job as a van driver.
Michael Galarnyk is an AI evangelist at cnvrg.io.
Welcome to the VentureBeat community!
DataDecisionMakers is where experts, including the technical people doing data work, can share data-related insights and innovation.
If you want to read about cutting-edge ideas and up-to-date information, best practices, and the future of data and data tech, join us at DataDecisionMakers.
You might even consider contributing an article of your own!
Nvidia GeForce RTX 3090 vs. AMD Radeon 6950 XT: Which GPU should you buy?
Receive near instant feedback on logos, images, text, and more with Helpfull
Confessions of an in-house creative strategist on feeling unfulfilled, difficulty in returning to agencies as the ‘pay is less’
The war for talent between agencies and brands’ in-house agencies has cooled. Even so, for adland talent who’ve made the move in-house, some say they are looking to go back to agencies after feeling creatively stifled. It’s not the easiest strategy to execute.
In the latest edition of our Confessions series, in which we trade anonymity for candor, we hear from an in-house creative strategist about their experience, why they want to go agency-side now and how pay is keeping them from doing so.
This conversation has been lightly edited and condensed for clarity.
What’s the in-house experience like?
I’ve been in-house for about a year. It’s very one-sided. The difference between agency and in-house is that with agencies, there [are] a lot of opinions and ideas [outside of the brand message] that go into creative. With in-house, you have the brand’s message and all creative is reflective of the brand’s message. With in-house, regardless of trends in the market, it’s a lot of ‘we’re going to stick to this one way of doing things’ mentality. It’s a lot of opinions about what the creative should be based on what it has been before. It makes it hard to introduce something fresh. It makes it hard to hire or be a new hire. If you’re not actually going to adhere to advice from new hires, what’s the point in getting new people? Are you just bringing people on board for a second opinion? That’s what it feels like.
Sounds like you don’t have the creative control you desire.
It feels like more of a second opinion role than to get something to manage or control. [Where I am now] it feels like we’re leaning more into what [our strategy] used to be than thinking about what we could be. That’s a big issue with in-house. With agencies, like I said, there’s a lot more trial and error. With in-house, a lot more of this is what we’re doing, these are the funds we have and this is what has worked in the past. In reality, a lot of what worked in the past, when you put it back into the market, it’s not going to work anymore.
Why do you think it’s more challenging to get to a new creative strategy in-house?
With agencies, you have multiple perspectives. You’re working on multiple brands. You can see something working for another brand and talk to your client about it. You can pivot. You have the background and perspective to [pitch that pivot]. When you’re in-house, you only have the knowledge of your brand and what’s working for you.
Are you looking to go back to agencies?
Personally, I am looking to go from in-house to agency but I get paid a lot more being in-house than what I’ve been offered at agencies. I’ve been in interviews with agencies where they’re telling me that I’ll be learning [programs I already know how to use] so that’s why the pay is less than what it should be. There are agencies I’ve interviewed with who ask me to move to New York for less than what I make now and make that work. [With inflation,] there’s no reason why salaries aren’t also increasing.
So you’d like to make the jump creatively but it’s hard when the compensation isn’t up to what in-house offers?
It’s hard. I’ve been lowballed, too. They’ll post a salary for a position, go through the interviews and then offer less than what’s listed on the salary description. What was the point of putting the salary range there? I feel like people are putting salary ranges on job descriptions just to attract people with the experience that they are looking for but by the time they make the offer, it’s not what they said it would be. It’s offensive.
When Will Bitcoin Bottom Out? Pi Cycle Bottom Says It Will Happen on July 9
Sky Mavis to Reimburse Axie Infinity Hack Victims and Restart the Ronin Bridge as Early as June 28th
Celsius Considers Bankruptcy, Hires More Advisors
Kyber Network Admits A Portion of Its Treasury Was Held By 3AC, But Not Significant to Affect KNC Operations
Sam Bankman-Fried’s FTX is Reportedly Planning to Acquire a Stake in BlockFi
‘Continue to ebb and flow over time’: Denny’s chief brand officer on how consumers’ moods inform brand messaging
Bitcoin hits $45K ahead of July inflation report, but one fractal hints at looming correction
Smart Marketing Token (SMT) Is on a Mission to Help Blockchain Projects Reach Their Goals
Identity management org Sailpoint unveils no-code tool
Japan crypto exchange bitbank upgrades performance of its matching engine by 4x
Ethereum3 months ago
Michael Saylor’s MacroStrategy Secures $205 Million Loan to Buy More Bitcoin
Bit Coin3 months ago
Nifty Gateway Partners With Samsung to Develop ‘First-Ever Smart TV NFT Platform’
Tech3 months ago
How ‘eQuad’ Electric Bikes Could Change UPS Delivery
Bit Coin3 months ago
Bitcoin suddenly dives to $46K as attention focuses on large CME futures gaps
Tech3 months ago
Instagram’s latest test makes it easier to support social causes
Bit Coin3 months ago
Binance launches Binance Bridge 2.0 to integrate CeFi and DeFi
Cryptocurrency3 months ago
Layer-2 aggregator platform Coinweb receives crypto exchange license from Lithuanian regulator
Bit Coin3 months ago
Axie Infinity hacked for $612M, OpenSea expands support to Solana, EU’s unhosted wallet regulations cause a stir: Hodler’s Digest, March 27-April 2