At the risk of derailing the thread:
We do a lot more in the script than is mentioned here: We pull out parts of the path and mangle them (for example turn them into a UNC path for users to use, or pull out a client name or job number using a known folder structure). As for deleted files, here's how the script works in totality:
- Script runs first time, finds every file and puts into formerly empty Solr DB. For every file found, set date_last_seen = current_time. Writes out last_begin_time file and ends.
- Secondary function of script runs sometime after, looks for any file with a date_last_seen < last_begin_time. Nothing is found this time around.
- Script runs next time, see's that there is a last_begin_time file, reads in that time. Script then runs in a "delta" mode, looking for all files modified later than last_begin_time. If it finds them, it re-indexs them and their contents. All other files that have mod_time less than last_run_time merely have their date_last_seen updated. Script ends, writes out last_begin_time file.
At this point, any files that were deleted between the first and second run were not updated , so their date_last_seen is different from all the others. This gives me something to look for.
- Secondary function of script runs sometime after, looks for any file with a date_last_seen < last_begin_time. This time around, some files are found. These files have their isDeleted field in solr set to 1.
Hopefully that makes a bit more sense.
-----Original Message-----
From: Walter Underwood [mailto:***@wunderwood.org]
Sent: Thursday, 13 February 2014 5:26 PM
To: solr-***@lucene.apache.org
Subject: Re: Solr delta indexing approach
Why write a Perl script for that?
touch new_timestamp
find . -newer timestamp | script-to-submit && mv new_timestamp timestamp
Neither approach deals with deleted files.
To do this correctly, you need lists of all the files in the index with their timestamps, and of all the files in the repository. Then you need to difference them to find deleted ones, new ones, and ones that have changed. You might even want to track links and symlinks to get dupes and canonical paths.
wunder
Post by Sadler, Anthony- Scan the whole filesystem and create an XML that is submitted into Solr for indexing. As this might be some 600,000 files, I break it down into chunks of N files (N = 200 currently).
- At the end of a successful scan, write out the time it started to a file.
= If it has a mod_time greater than the begin_time, it has changed since we last updated it, so reindex it.
= If it doesn't, just update the last_seen timestamp in Solr (a field we created) so we know its still there.
We're doing that and its indexing just fine.
-----Original Message-----
Sent: Thursday, 13 February 2014 4:45 PM
Subject: Re: Solr delta indexing approach
Thanks Alex,
Yes my source system maintains the crettion & last modificaiton system of each document.
As per your inputs, can i assume that next time when solr starts indexing, it scans all the prsent in source but only picks those for indexing which are either new or have been updated since last successful indexing.
How solr does this or in short what is solr strategy for indexing? I would definitely like to know more about it & if you can share your thoughts on same, it would be great.
Regards,
Lalit.
--
http://lucene.472066.n3.nabble.com/Solr-delta-indexing-approach-tp4117
068p4117077.html Sent from the Solr - User mailing list archive at
Nabble.com.
________________________________
==========================================
Privileged/Confidential Information may be contained in this message. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to email for messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of Burson-Marsteller shall be understood as neither given nor endorsed by it.
==========================================
--
Walter Underwood
***@wunderwood.org