(For context, I’m basically referring to Python 3.12 “multiprocessing.Pool Vs. concurrent.futures.ThreadPoolExecutor”…)

Today I read that multiple cores (parallelism) help in CPU bound operations. Meanwhile, multiple threads (concurrency) is due when the tasks are I/O bound.

Is this correct? Anyone cares to elaborate for me?

At least from a theorethical standpoint. Of course, many real work has a mix of both, and I’d better start with profiling where the bottlenecks really are.

If serves of anything having a concrete “algorithm”. Let’s say, I have a function that applies a map-reduce strategy reading data chunks from a file on disk, and I’m computing some averages from these data, and saving to a new file.

  • xia@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 months ago

    I recall reading a white paper on how multi-processing is pretty easy to debug and get right, but that multi-threading was actually impossible due to cartesian explosion of possible states and multiple writers to the same memory space.