You are not logged in.
- Topics: Active | Unanswered | Last 2 weeks
#11 2017-12-13 07:11
- B00ze
- Member
- From: Montreal, Canada
- Registered: 2016-03-11
- Posts: 13
Re: Rename 500,000 files - very slow clearing
Good day.
Wow, ReNamer needs a Tree? Binary trees are good for fast "random" insert/delete/searches, but I would not have guessed that you needed that for manipulating a list of filenames; I thought all ReNamer needed was fast sequential operations (i.e. a linked list or an array). AVL Trees look amazing! Literally everything is O(log n) and balancing stuff is all O(1) on average; they've even got documented algorithms for Union, Intersection and Difference.
If you remember, let me know once you have a beta.
Is there a way to get this forum to email when someone replies? I have to come back here regularly right now just to check.
I rarely use ReNamer so the "Free" option was enough for me (but I do pitch it to anyone who has rename issues, I tell them it's like a prototyping tool only you don't have to code it once you're done) but I'll definitively purchase a license if it does as well as you say.
Thank you.
Best Regards,
Offline
#12 2017-12-13 07:14
- B00ze
- Member
- From: Montreal, Canada
- Registered: 2016-03-11
- Posts: 13
Re: Rename 500,000 files - very slow clearing
Ok, I found the "Set your subscription options" in profile, activated both options; should get emails now...
Offline
#13 2017-12-13 17:32
- den4b
- Administrator
- From: den4b.com
- Registered: 2006-04-06
- Posts: 3,370
Re: Rename 500,000 files - very slow clearing
I thought all ReNamer needed was fast sequential operations
Not exactly. The objects which represent files are abstracted away from the visual files table, which contains only virtual nodes. Linked file objects and their properties are retrieved on demand as needed. The table can be shuffled around without ever affecting the underlying file objects or their order. There are multiple different structures in play to make most operations fast, with a smallest possible memory footprint.
However, as you have noticed, disorderly and partial deletion operations suffer a severe inefficiency. The use of AVL tree should overcome this inefficiency.
If you remember, let me know once you have a beta. Is there a way to get this forum to email when someone replies?
Updates will be posted in this thread, but you don't seem to be subscribed to this topic.
Click on the "Subscribe to this topic" link at the bottom of the page.
Offline
#14 2018-01-22 14:24
- den4b
- Administrator
- From: den4b.com
- Registered: 2006-04-06
- Posts: 3,370
Re: Rename 500,000 files - very slow clearing
After implementing various optimizations for the file object management operations, they now take about 8% of the total time taken by the clearing operation of 500K files. Using AVL tree or not, the majority of time is still consumed by the clearing of the tables nodes. To be specific, 916.571 seconds in total, or 837.449 seconds to clear the table nodes only.
All additional optimizations have been implemented in v6.8.0.2 Beta, including a more efficient selective clearing when all files match the clearing criteria.
No further optimizations can be applied at this stage, because it is a very complex 3rd party visual component that is the bottleneck now.
Offline
#15 2018-07-03 07:23
- B00ze
- Member
- From: Montreal, Canada
- Registered: 2016-03-11
- Posts: 13
Re: Rename 500,000 files - very slow clearing
Good day.
Somehow, I missed your reply; maybe it got filtered by Hotmail? Anyway, so you've improved it, but it still takes 15 minutes to clear a table of 500k entries? And if it's not O(n) then more than double that for 1M entries - this is still so long, considering that it takes just a few minutes to rename everything. I had to do this again recently, but I used a simple FOR loop in the command prompt; it was a bit slower than using reNamer, but I didn't have to wait an hour for the window to close afterwards.
Alright, I'll probably be using reNamer for something more than just changing file extensions soon, where the number of files is in the thousands rather than hundreds of thousands; I'll see about purchasing it then.
Thanks.
Offline
#16 2018-07-08 20:38
- den4b
- Administrator
- From: den4b.com
- Registered: 2006-04-06
- Posts: 3,370
Re: Rename 500,000 files - very slow clearing
but it still takes 15 minutes to clear a table of 500k entries?
If you are clearing only a portion of files (any number between 1 and N-1 out of N files), then a selective deletion would be involved and that is a slow process, for reasons explained earlier.
However, if you are clearing all files, then it should be significantly faster, more-less instantaneous. If it is still slow in your case, then please provide more specifics so that it can be investigated, e.g. version number, exact number of files, method of clearing, etc.
Offline