Bad Bots

Kindus has discussed the role of bots on the Internet and how webmasters can use ‘robots.txt’ to control them.  Unfortunately many bots do not obey the rules.  They do more than scrape freely available data and do their best to avoid any restrictions put in place by ‘robots.txt’.  These are the bad bots.  According to a report by Imperva these could have accounted for 32% of Internet activity in 2023, good bots making up 17.6% and humans the remaining 50.4%.

Bad bots may act in a way to mimic human activity and hence bypass ‘robots.txt’.  This could include simulated mouse movements and clicks.  They could also pretend to be coming from a reputable browser when in reality this is automated traffic.  Imperva reported that Chrome was the most commonly impersonated browser (40% of requests) but the mobile browsers Mobile Safari (18.5%) and Mobile Chrome (14.4%) were significant vectors.  Mobile browsers are attractive because their privacy settings mean that they pass less information to target sites. Another approach is to work from within malware installed in genuine browsers and run silently without the user’s knowledge.  These activities slow down the performance of host computers and the response time of target sites.  They can also cause a direct financial loss as a host may be paying an API to connect to and retrieve the data that it uses but the bot is directing that data to a competitor.   For example the professional ‘X’ API is $5,000 per month, the bot scraper accessing someone else’s registered API will be getting this data at a significantly reduced cost.

Kindus has already discussed how junk news is prevalent over the Internet; this and SPAM comments are generally the result of bot activity.  Other consequences can be significantly more harmful.  Some of the bots spread malware or attempt to steal confidential data, others run for commercial advantage.  Attacks could aim to take over Internet accounts through credential stuffing leading to identity theft or loss of confidential data.  In a legally-grey area bots might be exploiting the restrictions of supply and demand by hoovering up goods or services and then selling them on when the market rate goes up.  Concert or sports tickets have a restricted supply and the bots try to exploit rules on how many an individual can buy.  Restaurant reservations are another field that can be snapped up and re-sold; often at zero risk as the initial reservation would be free.  The restaurant will lose out as it is unable to make use of the ‘reserved’ tables to serve real customers until a slot has clearly expired.  A similar tactic works for the travel industry especially with airline seats.  Seat spinning has been reported as prevalent in the Asia Pacific region.  Online travel agencies will reserve seats with no financial obligation for up to 24 hours.  They can then sell these on but if they do not the airlines see the flights as almost full and increase the seat price.  This deters genuine sales and could lead to apparently full flights departing with empty seats.

Another key target is gaming platforms.  An account can benefit from a bot trading or performing in-game activities much more quickly than a human.  This will earn in-game credit or goods that can be sold on.  It reduces the potential income from sales or promotions to the game host.  It also makes the game less attractive to the genuine player as goods become artificially scarce or hard to obtain.  This in turn reduces the attraction for new players to sign up; further lowering game engine income.

Kindus has described the use of ‘robots.txt’ but reducing the impact of bad bots requires further precautions to identify and block the source of bad bots.

• Monitor API traffic (and fees).
• Block access from older browsers; these are more likely to have been compromised by bots.
• Restrict access from known harmful bulk IP services.
• Analyse and monitor traffic; look for unexpected spikes in activity.
• Look for multiple failures on cart checkout or discount code entry pages.