Worker pool

This example shows how to use a pool of workers to fan out work to multiple actors and then collect the result using get_all().

The example perform DNS lookups concurrently. It creates a pool of 4 workers and distributes the work of resolving a list of IP addresses among them. Each worker performs the DNS lookup and returns the result to the main thread, which collects and prints the results.

Example

examples/worker_pool.py

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "pykka",
# ]
# ///

import pprint
import socket
from ipaddress import IPv4Address

import pykka


class Resolver(pykka.ThreadingActor):
    def resolve(self, ip: IPv4Address) -> str | None:
        try:
            info = socket.gethostbyaddr(str(ip))
            print(f"Finished resolving {ip}")
            return info[0]
        except Exception:
            print(f"Failed resolving {ip}")
            return None


if __name__ == "__main__":
    ip_addresses = [IPv4Address(f"193.35.52.{i}") for i in range(1, 20)]
    pool_size = 4

    # Start resolvers
    pool = [Resolver.start().proxy() for _ in range(pool_size)]

    # Distribute work by mapping IPs to resolvers (not blocking)
    hosts: list[pykka.Future[str]] = []
    for i, ip in enumerate(ip_addresses):
        resolver = pool[i % len(pool)]
        hosts.append(resolver.resolve(ip))

    # Gather results (blocking)
    result = list(zip(ip_addresses, pykka.get_all(hosts), strict=True))
    pprint.pprint(result)

    # Stop all actors
    pykka.ActorRegistry.stop_all()

Output

$ uv run examples/worker_pool.py
Finished resolving 193.35.52.3
Finished resolving 193.35.52.2
Finished resolving 193.35.52.1
Finished resolving 193.35.52.4
Failed resolving 193.35.52.8
Failed resolving 193.35.52.6
Failed resolving 193.35.52.5
Failed resolving 193.35.52.7
Failed resolving 193.35.52.12
Failed resolving 193.35.52.9
Failed resolving 193.35.52.11
Failed resolving 193.35.52.10
Failed resolving 193.35.52.14
Failed resolving 193.35.52.13
Failed resolving 193.35.52.15
Failed resolving 193.35.52.16
Finished resolving 193.35.52.18
Finished resolving 193.35.52.19
Failed resolving 193.35.52.17
[(IPv4Address('193.35.52.1'), 'merete.samfundet.no'),
 (IPv4Address('193.35.52.2'), 'osl.samfundet.no'),
 (IPv4Address('193.35.52.3'), 'fnis.samfundet.no'),
 (IPv4Address('193.35.52.4'), 'trd.samfundet.no.52.35.193.in-addr.arpa'),
 (IPv4Address('193.35.52.5'), None),
 (IPv4Address('193.35.52.6'), None),
 (IPv4Address('193.35.52.7'), None),
 (IPv4Address('193.35.52.8'), None),
 (IPv4Address('193.35.52.9'), None),
 (IPv4Address('193.35.52.10'), None),
 (IPv4Address('193.35.52.11'), None),
 (IPv4Address('193.35.52.12'), None),
 (IPv4Address('193.35.52.13'), None),
 (IPv4Address('193.35.52.14'), None),
 (IPv4Address('193.35.52.15'), None),
 (IPv4Address('193.35.52.16'), None),
 (IPv4Address('193.35.52.17'), None),
 (IPv4Address('193.35.52.18'), 'bablefisk.samfundet.no'),
 (IPv4Address('193.35.52.19'), 'altostratus.samfundet.no')]