Python writing files in parallel crashes computer randomly -
i writing piece of code takes input user code , each code creates file , writes something. have write 10k files, code wrote computer randomly crashes. after 40 files written, after 2500.
def createuseritemfinalarray(user): stringfile = "/home/alvin/pycharmprojects/recsys/files/tagssimilaritycb/userfinalarrayscosine/" stringfile += str(user) + ".csv" interacteditems = userinteractiondict[user] interactedjobsmatrix = itemattributematrix[interacteditems, :] open(stringfile, "w") csvfile: writer = csv.writer(csvfile, delimiter="\t", quoting=csv.quote_none) writer.writerow(['nonzerocolumns', 'nonzerovalues']) if (len(interacteditems) == 0): writer.writerow(['[]', '[]']) else: simmatrix = sp.sparse.csc_matrix( cosine_similarity(interactedjobsmatrix, comparationmatrix, dense_output=false)) simmatrix = simmatrix.toarray() simmatrix = sp.sparse.csc_matrix(bn.nanmean(simmatrix, axis=0)) nonzerocolumn = simmatrix.nonzero()[1] similarityvalues = simmatrix.data nonzerocolumn = list(map(str, nonzerocolumn)) similarityvalues = list(map(str, similarityvalues)) nonzerocolumn, similarityvalues = formatarraystowrite(nonzerocolumn, similarityvalues) writer.writerow([nonzerocolumn, similarityvalues]) stop = time.time() print("time taken: ", stop - start, "calculated user: ", user)
this how called function. went n_jobs = 4
n_jobs = 2
. recall, computer crashed without launching in parallel. targetusers
list of users need create file
parallel(n_jobs=2)(delayed(createuseritemfinalarray)(user) user in targetusers)
at first thought maybe had problems closing of files. after adding close manually nothing changed. interactedjobsmatrix
, comparationmatrix
both sparse matrices. big sparse well.
Comments
Post a Comment