Serialization of link/file state
Posted: 2013 Mar 16, 02:44
Problem:
For every search, links that have been previously determined to be invalid/not available are checked again. When dealing with thousands of links this has a big performance impact.
Solution:
Serialize the link/file state (availability) into a database (e.g. sqlite). If development resources are constrained the existing "blacklist" infrastructure can be (ab)used: Store every link for which the status is determined to be "not available" into the blacklist.
Thank you.
For every search, links that have been previously determined to be invalid/not available are checked again. When dealing with thousands of links this has a big performance impact.
Solution:
Serialize the link/file state (availability) into a database (e.g. sqlite). If development resources are constrained the existing "blacklist" infrastructure can be (ab)used: Store every link for which the status is determined to be "not available" into the blacklist.
Thank you.