<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">FYI.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><b><font face="monospace, monospace">Hilton Gibson</font></b></div><div><span style="font-family:monospace,monospace;font-size:12.8000001907349px">Stellenbosch University Library</span><br></div><div><font face="monospace, monospace"><font color="#0000ee"><u><a href="http://staff.lib.sun.ac.za/~hgibson/about" target="_blank">http://staff.lib.sun.ac.za/~hgibson/about</a></u></font><br></font></div><div><br></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Tim Donohue</b> <span dir="ltr"><<a href="mailto:tdonohue@duraspace.org">tdonohue@duraspace.org</a>></span><br>Date: 8 September 2015 at 22:02<br>Subject: Re: [dspace-tech] Re: Problem with solr reindex command in the 13 upgrading step to 5.3<br>To: Hilton Gibson <<a href="mailto:hilton.gibson@gmail.com">hilton.gibson@gmail.com</a>><br>Cc: Andrea Schweer <<a href="mailto:schweer@waikato.ac.nz">schweer@waikato.ac.nz</a>>, "Pottinger, Hardy J." <<a href="mailto:PottingerHJ@missouri.edu">PottingerHJ@missouri.edu</a>>, DSpace Technical Support <<a href="mailto:dspace-tech@googlegroups.com">dspace-tech@googlegroups.com</a>><br><br><br>
<div bgcolor="#FFFFFF" text="#000000">
Hi Hilton,<br>
<br>
With regards to your release testing brainstorms on your own wiki,
we honestly would appreciate institutions stepping forward and
offering resources (technical infrastructure, staff, etc) during our
yearly Test-a-thons. As an established open source project, we have
a broad community of users, but our developer core is still very
volunteer oriented. There literally is no one who works full time on
DSpace (not even myself). We are reliant on the kindness of
individuals (and oftentimes their bosses!) to help us to build,
support, improve and test DSpace. It's amazing what we have been
able to get done entirely by volunteer work (with a little bit of
coordination).<br>
<br>
We do hold yearly Testathons where we encourage the broad community
to take part, bang on the software and help us to make the next
release as "battle tested" as we possibly can. Also, the DSpace
Community Advisory Team (DCAT) has begun an initiative (just this
week) to help our community to develop a more extensive "Test Plan"
(which we can use to ensure each piece of the system has received
testing attention from our volunteers). <br>
<br>
Their work has begun here, and I'm sure they'd love to have
additional contributors to the work overall<br>
<a href="https://wiki.duraspace.org/display/cmtygp/DSpace+6+Testathon+Testplan+Working+group" target="_blank">https://wiki.duraspace.org/display/cmtygp/DSpace+6+Testathon+Testplan+Working+group</a><br>
<br>
If you or your institution is willing to help out in any way, we'd
appreciate the support. (This is the same for anyone else reading
this thread!) We'd honestly love to have institutions with larger
production environments help us test early versions of DSpace. But,
as of yet, we've never been able to find those volunteers (or a
large corpus of test data). So, we tend to rely on a more "crowd
sourced" testing model (where we put up a couple of test instances
and ask folks to help us bang on them). This crowd-sourced model
tends to find most software stability bugs, but it admittedly may
not always catch all of the scalability bugs.<br>
<br>
In all honesty, we want and need more testers to get involved during
Testathons and just before major releases. If anyone else is
interested and willing to help, get in touch, or join up with the
DCAT initiative! We'd love to have your help.<br>
<br>
- Tim<br>
<br>
<div>On 9/2/2015 5:20 PM, Hilton Gibson
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif">Hi Andrea,</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">There are many
who contribute to this community project. I do not wish to
single out any persons for fear of offending others.</div>
<div style="font-family:arial,helvetica,sans-serif">I think I do my
bit with a wiki that I update as often as I can.</div>
<div style="font-family:arial,helvetica,sans-serif">I am upset, we
had some violence on campus today, and perhaps it spilled over
into my reply.</div>
<div style="font-family:arial,helvetica,sans-serif">I apologise.</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">However
introducing new features to DSpace that are not "battle
tested" seems to be a common theme lately.</div>
<div style="font-family:arial,helvetica,sans-serif">I am trying my
best with my limited programming skills and extensive
production system experience to maintain a production ready
repository and help others in developing countries to do the
same. (This is what I meant by the global south - most
developing countries are in the south)</div>
<div style="font-family:arial,helvetica,sans-serif">So the issue
with re-indexing solr came at a very bad time.</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">If you have
time please read and consider the following:</div>
<div style="font-family:arial,helvetica,sans-serif">1. <a href="http://wiki.lib.sun.ac.za/index.php/SUNScholar/Reference_Architecture" target="_blank"></a><a href="http://wiki.lib.sun.ac.za/index.php/SUNScholar/Reference_Architecture" target="_blank">http://wiki.lib.sun.ac.za/index.php/SUNScholar/Reference_Architecture</a></div>
<div style="font-family:arial,helvetica,sans-serif">2. <a href="http://wiki.lib.sun.ac.za/index.php/SUNScholar/Software_Release_Cadence" target="_blank"></a><a href="http://wiki.lib.sun.ac.za/index.php/SUNScholar/Software_Release_Cadence" target="_blank">http://wiki.lib.sun.ac.za/index.php/SUNScholar/Software_Release_Cadence</a></div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">Cheers</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">hg</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b><font face="monospace, monospace">Hilton
Gibson</font></b></div>
<div><font face="monospace, monospace">Ubuntu
Linux Systems Administrator</font></div>
<div><font face="monospace, monospace">Stellenbosch
University Library<br>
</font></div>
<div><font face="monospace, monospace"><a href="http://staff.lib.sun.ac.za/%7Ehgibson/docs/cv/cv.html" target="_blank"></a><a href="http://staff.lib.sun.ac.za/~hgibson/docs/cv/cv.html" target="_blank">http://staff.lib.sun.ac.za/~hgibson/docs/cv/cv.html</a><br>
</font></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On 2 September 2015 at 23:46, Andrea
Schweer <span dir="ltr"><<a href="mailto:schweer@waikato.ac.nz" target="_blank">schweer@waikato.ac.nz</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi Hilton,<br>
<br>
I'm sure you're aware that even those of us not in the
global south have limited resources. I put in a lot of
work to fix a problem that was introduced by people other
than myself. I then took the time to share my solution
with the community, even continuing to improve my code
based on the feedback of several testers to make the
reindex work for people without solr knowledge at I time
when I had already upgraded all my own solr cores and I
could just have easily declared this to be someone else's
problem. The reindex script I wrote may not work in all
situations, but had I not volunteered my time (and had my
employer not let me do so) then there might not be a fix
at all. Perhaps there are cultural differences at play
here, but your e-mail below reads to me as quite
aggressive and as if you'd rather have no code at all than
code that worked fine for everyone who helped me test when
I developed it and worked fine for all "my" DSpace
instances. I'm sure that's not what you actually mean.<br>
<br>
A more constructive reaction than the e-mail I'm quoting
might be to try out the suggestions made in this thread
(increasing the heap space does not require you to put the
server into debug mode) and/or to share more details of
what you tried and what happened when it failed so that
the volunteers (!) on this list can try and figure out
what's going on. <br>
<br>
I've given you the technical reasons why an incremental
reindex is not really doable. You say verbose log output
would be nice -- what types of things would you like to
see logged? The reindex does log to dspace.log at INFO
level; it will tell you every time it <a href="https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/util/SolrImportExport.java#L590" target="_blank">writes</a> an export file and every time
it <a href="https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/util/SolrImportExport.java#L431" target="_blank">reads</a> an import file. Export happens
before import -- even without programming skills, the code
comments in the reindex method might let you determine
this much (<a href="https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/util/SolrImportExport.java#L306" target="_blank">https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/util/SolrImportExport.java#L306</a>).
So you should be able to use the log output to determine
during which part of the process this fails. All access to
Solr is also logged in the solr.log file, so again there
you will be able to see whether solr is busy doing exports
(/select) or imports (/update). <br>
<br>
cheers,<br>
Andrea<br>
<br>
<div>On 03/09/15 02:55, Hilton Gibson wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div style="font-family:arial,helvetica,sans-serif">Hi
Hardy,</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">I do
not have the time to experiment on a production
server.</div>
<div style="font-family:arial,helvetica,sans-serif">If
this could be done incrementally as I asked before,
then perhaps.</div>
<div style="font-family:arial,helvetica,sans-serif">When
there is no verbose output and no log details, it is
very difficult to debug.</div>
<div style="font-family:arial,helvetica,sans-serif">I am
not going to put a production server into debug
mode.</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">So we
will live with faulty geo stats for now.</div>
<div style="font-family:arial,helvetica,sans-serif">Remember
not all institutions have expert java
programmers/system personnel at their disposal to
fix these things.</div>
<div style="font-family:arial,helvetica,sans-serif">In
the global south we have to make do.</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">Cheers</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
<div style="font-family:arial,helvetica,sans-serif">hg</div>
<div style="font-family:arial,helvetica,sans-serif"><br>
</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div><b><font face="monospace,
monospace">Hilton Gibson</font></b></div>
<div><font face="monospace, monospace">Ubuntu
Linux Systems Administrator</font></div>
<div><font face="monospace, monospace">Stellenbosch
University Library<br>
</font></div>
<div><font face="monospace, monospace"><a href="http://staff.lib.sun.ac.za/%7Ehgibson/docs/cv/cv.html" target="_blank"></a><a href="http://staff.lib.sun.ac.za/~hgibson/docs/cv/cv.html" target="_blank">http://staff.lib.sun.ac.za/~hgibson/docs/cv/cv.html</a><br>
</font></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<span><font color="#888888"> </font></span></div>
<span><font color="#888888"> </font></span></blockquote>
<span><font color="#888888"> <br>
<pre cols="72">--
Dr Andrea Schweer
IRR Technical Specialist, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
<a href="tel:%2B64-7-837%209120" value="+6478379120" target="_blank">+64-7-837 9120</a></pre>
</font></span></div>
</blockquote>
</div>
<br><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
-- <br>
You received this message because you are subscribed to the Google
Groups "DSpace Technical Support" group.<br>
To unsubscribe from this group and stop receiving emails from it,
send an email to <a href="mailto:dspace-tech+unsubscribe@googlegroups.com" target="_blank">dspace-tech+unsubscribe@googlegroups.com</a>.<br>
To post to this group, send email to <a href="mailto:dspace-tech@googlegroups.com" target="_blank">dspace-tech@googlegroups.com</a>.<br>
Visit this group at <a href="http://groups.google.com/group/dspace-tech" target="_blank">http://groups.google.com/group/dspace-tech</a>.<br>
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank">https://groups.google.com/d/optout</a>.<br>
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<pre cols="72">--
Tim Donohue
Technical Lead for DSpace & DSpaceDirect
DuraSpace.org | DSpace.org | DSpaceDirect.org</pre>
</font></span></div>
</div><br></div>