区块链技术博客
www.b2bchain.cn

与joblib库的空余会生成_pickle.PicklingError:无法腌制任务以将其发送给工作人员 – python程序员分享

本文介绍了与joblib库的空余会生成_pickle.PicklingError:无法腌制任务以将其发送给工作人员 – python程序员分享,有助于帮助完成毕业设计以及求职,是一篇很好的资料。

对技术面试,学习经验等有一些体会,在此分享。

我有很多句子(约700万个),我想从中提取名词。

我使用了joblib库来并行化提取过程,如下所示:

import spacy from tqdm import tqdm from joblib import Parallel, delayed nlp = spacy.load('en_core_web_sm')  class nouns:      def get_nouns(self, text):         doc = nlp(u"{}".format(text))         return [token.text for token in doc if token.tag_ in ['NN', 'NNP', 'NNS', 'NNPS']]      def parallelize(self, sentences):         results = Parallel(n_jobs=1)(delayed(self.get_nouns)(sent) for sent in tqdm(sentences))         return results  if __name__ == '__main__':     sentences = ['we went to the school yesterday',                  'The weather is really cold',                  'Can we catch the dog?',                  'How old are you John?',                  'I like diving and swimming',                  'Can the world become united?']     obj = nouns()     print(obj.parallelize(sentences)) 

当并行化函数中的n_jobs大于1时,我得到这个长错误:

100%|██████████| 6/6 [00:00<00:00, 200.00it/s] joblib.externals.loky.process_executor._RemoteTraceback:  """ Traceback (most recent call last):   File "C:Python35libsite-packagesjoblibexternalslokybackendqueues.py", line 150, in _feed     obj_ = dumps(obj, reducers=reducers)   File "C:Python35libsite-packagesjoblibexternalslokybackendreduction.py", line 243, in dumps     dump(obj, buf, reducers=reducers, protocol=protocol)   File "C:Python35libsite-packagesjoblibexternalslokybackendreduction.py", line 236, in dump     _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)   File "C:Python35libsite-packagesjoblibexternalscloudpicklecloudpickle.py", line 267, in dump     return Pickler.dump(self, obj)   File "C:Python35libpickle.py", line 408, in dump     self.save(obj)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 836, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 841, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 836, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 770, in save_list     self._batch_appends(obj)   File "C:Python35libpickle.py", line 797, in _batch_appends     save(tmp[0])   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 725, in save_tuple     save(element)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libsite-packagesjoblibexternalscloudpicklecloudpickle.py", line 718, in save_instancemethod     self.save_reduce(types.MethodType, (obj.__func__, obj.__self__), obj=obj)   File "C:Python35libpickle.py", line 599, in save_reduce     save(args)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 725, in save_tuple     save(element)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libsite-packagesjoblibexternalscloudpicklecloudpickle.py", line 395, in save_function     self.save_function_tuple(obj)   File "C:Python35libsite-packagesjoblibexternalscloudpicklecloudpickle.py", line 594, in save_function_tuple     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 836, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 841, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 836, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 599, in save_reduce     save(args)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 740, in save_tuple     save(element)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 740, in save_tuple     save(element)   File "C:Python35libpickle.py", line 495, in save     rv = reduce(self.proto)   File "stringsource", line 2, in preshed.maps.PreshMap.__reduce_cython__ TypeError: self.c_map cannot be converted to a Python object for pickling """Exception in thread QueueFeederThread: Traceback (most recent call last):   File "C:Python35libsite-packagesjoblibexternalslokybackendqueues.py", line 150, in _feed     obj_ = dumps(obj, reducers=reducers)   File "C:Python35libsite-packagesjoblibexternalslokybackendreduction.py", line 243, in dumps     dump(obj, buf, reducers=reducers, protocol=protocol)   File "C:Python35libsite-packagesjoblibexternalslokybackendreduction.py", line 236, in dump     _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)   File "C:Python35libsite-packagesjoblibexternalscloudpicklecloudpickle.py", line 267, in dump     return Pickler.dump(self, obj)   File "C:Python35libpickle.py", line 408, in dump     self.save(obj)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 836, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 841, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 836, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 770, in save_list     self._batch_appends(obj)   File "C:Python35libpickle.py", line 797, in _batch_appends     save(tmp[0])   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 725, in save_tuple     save(element)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libsite-packagesjoblibexternalscloudpicklecloudpickle.py", line 718, in save_instancemethod     self.save_reduce(types.MethodType, (obj.__func__, obj.__self__), obj=obj)   File "C:Python35libpickle.py", line 599, in save_reduce     save(args)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 725, in save_tuple     save(element)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libsite-packagesjoblibexternalscloudpicklecloudpickle.py", line 395, in save_function     self.save_function_tuple(obj)   File "C:Python35libsite-packagesjoblibexternalscloudpicklecloudpickle.py", line 594, in save_function_tuple     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 836, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 841, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 810, in save_dict     self._batch_setitems(obj.items())   File "C:Python35libpickle.py", line 836, in _batch_setitems     save(v)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 599, in save_reduce     save(args)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 740, in save_tuple     save(element)   File "C:Python35libpickle.py", line 520, in save     self.save_reduce(obj=obj, *rv)   File "C:Python35libpickle.py", line 623, in save_reduce     save(state)   File "C:Python35libpickle.py", line 475, in save     f(self, obj) # Call unbound method with explicit self   File "C:Python35libpickle.py", line 740, in save_tuple     save(element)   File "C:Python35libpickle.py", line 495, in save     rv = reduce(self.proto)   File "stringsource", line 2, in preshed.maps.PreshMap.__reduce_cython__ TypeError: self.c_map cannot be converted to a Python object for pickling  During handling of the above exception, another exception occurred:  Traceback (most recent call last):   File "C:Python35libthreading.py", line 914, in _bootstrap_inner     self.run()   File "C:Python35libthreading.py", line 862, in run     self._target(*self._args, **self._kwargs)   File "C:Python35libsite-packagesjoblibexternalslokybackendqueues.py", line 175, in _feed     onerror(e, obj)   File "C:Python35libsite-packagesjoblibexternalslokyprocess_executor.py", line 310, in _on_queue_feeder_error     self.thread_wakeup.wakeup()   File "C:Python35libsite-packagesjoblibexternalslokyprocess_executor.py", line 155, in wakeup     self._writer.send_bytes(b"")   File "C:Python35libmultiprocessingconnection.py", line 183, in send_bytes     self._check_closed()   File "C:Python35libmultiprocessingconnection.py", line 136, in _check_closed     raise OSError("handle is closed") OSError: handle is closed    The above exception was the direct cause of the following exception:  Traceback (most recent call last):   File ".../playground.py", line 43, in <module>     print(obj.Paralize(sentences))   File ".../playground.py", line 32, in Paralize     results = Parallel(n_jobs=2)(delayed(self.get_nouns)(sent) for sent in tqdm(sentences))   File "C:Python35libsite-packagesjoblibparallel.py", line 934, in __call__     self.retrieve()   File "C:Python35libsite-packagesjoblibparallel.py", line 833, in retrieve     self._output.extend(job.get(timeout=self.timeout))   File "C:Python35libsite-packagesjoblib_parallel_backends.py", line 521, in wrap_future_result     return future.result(timeout=timeout)   File "C:Python35libconcurrentfutures_base.py", line 405, in result     return self.__get_result()   File "C:Python35libconcurrentfutures_base.py", line 357, in __get_result     raise self._exception _pickle.PicklingError: Could not pickle the task to send it to the workers. 

我的代码有什么问题?

参考方案

问:我的代码有什么问题?

好吧,问题很可能不是来自代码,而是来自出现的“隐藏”处理,一旦n_jobs指示(和joblib内部编排)准备好主过程的许多精确副本,以便让他们彼此独立工作(有效地逃避了GIL锁定,并将多个流程流映射到物理硬件资源上)

此步骤负责制作所有pythonic对象的副本,并且已知使用Pickle来执行此操作。 Pickle模块以其在可以腌制和不能腌制的历史上的主要限制而闻名。

错误消息确认了这一点:

TypeError: self.c_map cannot be converted to a Python object for pickling

可以尝试一种技巧来提供Mike McKearns的dill模块而不是Pickle并进行测试,如果您“有问题的” python对象将被该模块腌制而不会引发此错误。

dill具有相同的API签名,因此纯import dill as pickle可能有助于使所有其他代码保持相同。

我遇到了同样的问题,因为要使用大型模型将其分配到多个进程中或从多个进程中分配回来,因此dill是可行的方法。性能也有所提高。

奖励:dill允许保存/恢复完整的python解释器状态!

这是找到dill的一个很酷的副作用,一旦完成import dill as picklepickle.dump_session( <aFile> )将保存一个完整的python解释器会话的全状态副本。可以根据需要进行恢复(崩溃后的恢复,受过训练的经过训练和优化的ML模型的状态保存/恢复,增量学习ML模型的状态保存和重新分发,以用于已部署用户库的远程还原,等)

我正在使用Retrofit来获取JSON答复。这是我实施的一部分("/api/report/list") Observable<Bills> listBill(@Query("employee_id") String employeeID); 而条例草案类是-public static class…

我知道PHP中的__FILE__魔术常数将变成当前执行文件的完整路径和文件名。但是,有什么方法可以使函数的调用文件获得相同的信息吗?例如://foo.php: include "bar.php"; call_it(); //bar.php function call_it() { echo "Calling file: &#03…

我正在尝试使用Retrofit和Gson解析一些JSON。但是,我得到的响应机构是空的。当我尝试从对象中打印信息时,出现NullPointerException。我确保URL正确,并且我也确保POJO也正确。我正在使用jsonschema2pojo来帮助创建POJO类。这是我要解析的JSON{ "?xml": { "@versi…

我有很多类似以下内容的JSON文件:例如。1.json{"name": "one", "description": "testDescription…", "comment": ""} test.json{"name&#034…

我正在使用Phone Gaps(Cordova 2.1)文件传输API将用户照片库中的图像发布到我的服务器上。文件传输API似乎运行良好。我只是对在服务器上检索此信息感到困惑。理想情况下,我需要做的是检索图像,然后将其上传到我的服务器。但是,我似乎无法从文件传输中检索任何信息?我的JavaScript代码(发布图像数据)为:function onDevice…

赞(0) 打赏
部分文章转自网络,侵权联系删除b2bchain区块链学习技术社区 » 与joblib库的空余会生成_pickle.PicklingError:无法腌制任务以将其发送给工作人员 – python程序员分享
分享到: 更多 (0)

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址

b2b链

联系我们联系我们