User-agent: * Disallow: /wp-admin/ Disallow: /online-development/ Allow: /wp-admin/admin-ajax.php # https://ahrefs.com/robot/ crawls the site frequently User-agent: AhrefsBot Disallow: / # Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) User-agent: MJ12bot Disallow: /