ip-rotator

One of the toughest hurdles in web scraping is getting blocked by rate-limiters, but there are ways around it. You can tweak headers, swap out your User-Agent string or even downgrade the protocol. Another approach that’s worked flawlessly for me is setting up an IP rotator.

The core idea is simple: periodically switch the IP address your scraper uses. Here’s what you’ll need to get started:
A VPN provider (I recommend NordVPN) and Tunnelblick (or any OpenVPN-compatible client)

1. Provision multiple IP configurations

Create a handful of VPN configuration files, each one tied to a different exit-node (i.e. IP address). For NordVPN you can download .ovpn files for various locations from their dashboard.

Comprehensive Guide

2. Automate IP rotation

  • List VPN Configurations from Tunnelblick (a)
    1
    2
    3
    4
    tell application "Tunnelblick"
        set vpnList to name of configurations
    end tell
    return vpnList

vpn-list.scpt

  • Usage

osascript /path/to/your/vpn-list.scpt

osascript (Apple Script command)

  • Auto-Switch VPN Every 5 Minutes
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    property vpnList : {"_.nordvpn.com.tcp", "_.nordvpn.com.tcp"}
    property currentIndex : 1

    on run
        repeat
            set vpnName to item currentIndex of vpnList

            tell application "Tunnelblick"
                disconnect all
                delay 5 -- short pause to allow disconnect
                connect vpnName
            end tell

            display notification "Switched VPN to " & vpnName with title "VPN Auto Switch"

            -- Move to next VPN in the list
            if currentIndex = (count of vpnList) then
                set currentIndex to 1
            else
                set currentIndex to currentIndex + 1
            end if

            delay 300 -- wait 5 minutes (300 seconds)
        end repeat
    end run

ip-rotator.scpt

property vpnList : {“.nordvpn.com.tcp”, “.nordvpn.com.tcp”} // vpnList from the (a) script.

  • The comprehensive script
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    property vpnList : {"_.nordvpn.com.tcp", "_.nordvpn.com.tcp"}
    property currentIndex : 1
    property logFilePath : ((path to desktop folder as text) & "VPN_Switch_Log.txt")

    on run
        my writeLog("VPN Auto Switcher Started")
        
        repeat
            set vpnName to item currentIndex of vpnList
            
            tell application "Tunnelblick"
                disconnect all
                delay 5 -- wait for disconnection
                connect vpnName
            end tell

            set logMessage to "Switched VPN to " & vpnName
            display notification logMessage with title "VPN Auto Switch"
            my writeLog(logMessage)
            log logMessage -- show in Script Editor log pane

            -- Advance to next VPN
            if currentIndex = (count of vpnList) then
                set currentIndex to 1
            else
                set currentIndex to currentIndex + 1
            end if

            delay 300 -- wait 5 minutes
        end repeat
    end run

    -- Write log to file
    on writeLog(msg)
        set timeStamp to do shell script "date '+%Y-%m-%d %H:%M:%S'"
        set fullMsg to "[" & timeStamp & "] " & msg & linefeed
        try
            set fileRef to open for access file logFilePath with write permission
            write fullMsg to fileRef starting at eof
            close access fileRef
        on error
            try
                close access file logFilePath
            end try
        end try
    end writeLog