Dubizzle eliminates competitor scraping, improves user experience by removing spam, and increases advertiser ROI using Imperva Bot Management (formerly Distil Networks)
Overview
Dubizzle.com is the leading classifieds website in the UAE. It’s the number one platform for users to buy, sell, or find anything in their community. As with any two-sided marketplace, it operates within a balance between buyers and sellers; every shared interaction must be helpful. Dubizzle is an OLX company—the global classifieds brand leader in more than 50 growth markets throughout Asia, Africa, the Middle East, Latin America, and Europe. More than 240 million unique visitors generate over 17 billion page views per month.
Challenges
Competitors used bots to scrape Dubizzle’s listings
As a classifieds website, Dubizzle’s product is its information. Nefarious competitors used bots to steal listing data and monetize it on copycat sites. An increasing number of new sites were using Dubizzle listings to ‘seed’ their networks, making them appear larger than they actually were. Such copycat sites could steal customers away and decrease SEO performance, as similar data posted in multiple places hurts page Rankings.
Bots competed with legitimate users, causing Dubizzle site slowdowns and downtime
Floods of bot traffic taxed Dubizzle’s infrastructure and overloaded its applications. At times a service disruption— caused by bad bots—rendered the site unavailable to legitimate users.
Skewed lead conversion ratios weakened advertiser relationships
Bots hitting a page result in an ad view, but they generally don’t complete a desirable action, such as filling out a lead form. Featured Ads (premium paid ads) had a much higher number of views relative to lead actions (i.e., direct seller contact). This caused an artificial decrease in key advertiser performance metrics, resulting in partners believing their ads performed weaker than they actually did.
Fake listings weakened consumer trust
Malicious users exploited Dubizzle’s marketplace by posting spam/fake ads in related categories. For example, a moving company would promote itself within the Property for Rent category. This diminished KPIs for Featured Ads while also damaging the user experience— wasting their time as they sorted through the junk.
Fake listing hurt paid ad performance
In addition to causing a poor user experiences, phony listings hurt performance of real paid listings. Forced further down a page, users were seeing them less often. Dubizzle’s previous solution wasn’t proactively stopping such spam posts, instead only flagging them after the fact. The damage had already been done.
SMS spam caused a poor user experience
On the Dubizzle site, each Detailed Page View provides an option to directly contact an advertiser via SMS or a phone call. Competitors and ancillary rogue businesses created bots to scrape advertiser phone numbers and spam them using SMS messages. This misled advertisers, wasted their time, and weakened the trust Dubizzle had established with important advertising partners. Initially, Dubizzle combatted scraping by embedding phone numbers within an image. But using machine learning, competitors soon developed more sophisticated bots to extract the numbers, rendering this defense no longer effective.
Results
After many attempts to solve the aforementioned challenges with both homegrown and commercial solutions, Dubizzle partnered with Imperva Bot Management to solve its bot problem once and for all.
Eliminated most spam ads seen by users
“Before implementing Imperva Bot Management users were seeing spam ads on the first or second page of listings, which hurt their experience, ”says Mudasir Mirza, Dubizzle’s DevOps manager. “Now spam ads don’t appear until page seven, which means almost no one sees them. Their impact is negligible.”
Better bot insights enable user-level rate limiting
Imperva’s reporting lets Dubizzle examine bot behavior at the user level, enabling it to implement user-level (fingerprint) rate limiting. This lets its team identify specific users who abuse the system. They’re able to block just their activity, without impacting that of valid users. This approach has all but eliminated phony listings.
Increased protection on highly targeted pages
Dubizzle’s Property for Rent, Property for Sale, and Motors sections were high-value targets for competitors’ scraper bots. Now with Imperva’s path-specific rules, it has been able to add increased protections to defend its revenue on these important paths. And this has the added benefit of decreasing costs by only bolstering such protection on those pages that required it most.
Improved ROI and purified KPIs for advertising clients
By removing fake listings, Dubizzle has been able to increase KPIs for its advertiser partners, helping it improve important relationships.
“We have shown advertisers what we’ve put in place, and it’s a huge win,” says Mirza. In several cases, we’ve won business away from a competitor because we are able to show better performance. One advertiser upped their spend with us since we’ve been able to prove we’re a more efficient channel.”
Improved user experience and a reduction in support calls
Previous attempted bot mitigation methods only led to customers complaining to Dubizzles support team. By preventing bots from affecting listings, Imperval prevents phone number scraping and eliminates SMS spam to Dubizzle’s customers. In turn this has reduced the number of complaints to its support team, freeing them to address other customer concerns.
Eliminated bad bot traffic across the entire network of eCommerce sites
“Before deploying Imperva Bot Management, I estimate that 20% of our traffic was bots, but it varied tremendously. One of our sites was almost 80% bots, while some of the premium domains were around 10%. Now we don’t have to deal them at all,” says the CIO.
Imperva Bot Management saves hundreds of hours per quarter and makes everyone’s lives so much easier
“Our infrastructure team is small, so it’s important to engage with services that intelligently augment the team. I really like the reporting capabilities the Imperva Bot Management Portal offers. It’s so intuitive that we don’t have to hire additional people to support all that it offers us. Being so utilitarian, Imperva has made our lives so much easier.”
Clean analytics yield a better testing environment for eCommerce funnel optimization
“And the mere fact that there are legitimate customers visiting our site—versus bad bots—should yield a better testing environment and better analytical results. The marketing and metrics teams are just now coming up to speed with these additional benefits that Imperva offers, but I know they’re going to like what they see once they do, he adds.
Imperva Click Fraud Report provides evidence for discounts and credits with ad networks
“One report that especially interests me is the click fraud report in relation to pay-per-click advertisers. Previously, bots were clicking the ads and we had to pay for that bogus activity. Imperva providing that data really opened my eyes regarding all that it can do for us. Marketing was able to use those reports to turn off bot-ridden ad Networks.
Tuning up for eCommerce holiday season
The CIO says they are probably using 85 or 90 percent of Imperva Bot Management’s capabilities. They plan to tune the solution before the busy holiday shopping season to get even more out of it. “Every website in the world has a bot problem,” he explained. “I heartily recommend the Imperva solution to solve the problem.
