feature(xjx): multiprocess tblogger, fix circular reference problem #156
Conversation
Codecov Report
@@ Coverage Diff @@
## main #156 +/- ##
==========================================
+ Coverage 84.90% 85.22% +0.32%
==========================================
Files 427 429 +2
Lines 32751 33567 +816
==========================================
+ Hits 27806 28609 +803
- Misses 4945 4958 +13
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
| if self._thread_pool: | ||
| self._thread_pool.shutdown() | ||
| self.middleware.clear() |
There was a problem hiding this comment.
add some comments for the sequence of calling clear method
| @@ -0,0 +1,97 @@ | |||
| from tensorboardX import SummaryWriter | |||
| from typing import TYPE_CHECKING | |||
| if TYPE_CHECKING: | |||
There was a problem hiding this comment.
add some comments for why use this TYPE_CHECKING, I guess it's because of circle import
| and event components of the task (see ``writer.plugin``). | ||
| """ | ||
|
|
||
| def __init__(self, *args, **kwargs): |
There was a problem hiding this comment.
add some comments for additional kwargs
ding/utils/log_writer_helper.py
Outdated
| Plugin ``task``, so when using this writer in the task pipeline, it will automatically send requests\ | ||
| to the main writer instead of writing it to the disk. So we can collect data from multiple processes\ | ||
| and write them into one file. | ||
| Usage: |
There was a problem hiding this comment.
You can use Example comments like this
| self._get_file_writer() | ||
| self._lazy_initialized = True | ||
|
|
||
| def __del__(self): |
There was a problem hiding this comment.
whether need to check some flag first to avoid calling close method repeatedly
There was a problem hiding this comment.
Double close check is done in the super method, so we don't need to check this
ding/utils/log_writer_helper.py
Outdated
| self._is_writer = False | ||
| self._lazy_initialized = False | ||
|
|
||
| def plugin(self, task: "Task", is_writer: False) -> "DistributedWriter": |
|
|
||
|
|
||
| ready_to_parallel_fns = [ | ||
| 'add_audio', |
There was a problem hiding this comment.
maybe you can wrapper all the attributes of SummaryWriter by this way
There was a problem hiding this comment.
I have wrapped all the attributes starting with add_ in summarywriter, but I am not sure whether certain methods will succeed through rpc, such as add_image.
…pendilab#156) * Fix recur reference in task and parallel, add distributed logger * Update logger * Clear ref list when exit task/parallel * Put task in with statment * Fix test * FFix test * Test is hard * More comments
Description
Related Issue
TODO
Check List