Category "parallel-processing"

Running parallel jobs in slurm

I was wondering if I could ask something about running slurm jobs in parallel.(Please note that I am new to slurm and linux and have only started using it 2 day

Python Multiprocessing concurrency using Manager, Pool and a shared list not working

I am learning python multiprocessing, and I am trying to use this feature to populate a list with all the files present in an os. However, the code that I wrote

how to see which process owns a python thread

My understanding from multi-threading has been that one process (cpu core) can have multiple threads running with it. So far in python when I want to check whi

CUDA - Implementing Device Hash Map?

Does anyone have any experience implementing a hash map on a CUDA Device? Specifically, I'm wondering how one might go about allocating memory on the Device an

Running shell script in parallel

I have a shell script which shuffles a large text file (6 million rows and 6 columns) sorts the file based the first column outputs 1000 files So the pseu

Running two threads at the same time

I want to know if a program can run two threads at the same time (that is basically what it is used for correct?). But if I were to do a system call in one func

How to abort a Task like aborting a Thread (Thread.Abort method)?

We could abort a Thread like this: Thread thread = new Thread(SomeMethod); . . . thread.Abort(); But can I abort a Task (in .Net 4.0) in the same way not by

Is it better to start multiple erlang nodes per machine, or just one per machine?

Preface: When I say "machine" below, I mean either a physical dedicated server, or a virtual private server. When I say "node" I mean, an instance of the erlan

Can Powershell Run Commands in Parallel?

I have a powershell script to do some batch processing on a bunch of images and I'd like to do some parallel processing. Powershell seems to have some backgrou

Parallelize Python Dictionary Comprehension

I'm trying to parallelize the subsetting of a Python dictionary. The code below creates a new dictionary, positions_sub, based on if the keys in positions dicti

Tensorflow and Multiprocessing: Passing Sessions

I have recently been working on a project that uses a neural network for virtual robot control. I used tensorflow to code it up and it runs smoothly. So far, I

spacy with joblib library generates _pickle.PicklingError: Could not pickle the task to send it to the workers

I have a large list of sentences (~7 millions), and I want to extract the nouns from them. I used joblib library to parallelize the extracting process, like in

How to execute pytest paralelly through xdist plugin?

How can I run my pytests parallely on multiple custom environments ? I have pytest-xdist as well. Not sure if this plugin helps test_test.py @pytest.mark.env(