[Irtalk] Fwd: [dspace-tech] Re: Problem with solr reindex command in the 13 upgrading step to 5.3

Tue Sep 8 23:34:23 SAST 2015

FYI.

*Hilton Gibson*
Stellenbosch University Library
*http://staff.lib.sun.ac.za/~hgibson/about
<http://staff.lib.sun.ac.za/~hgibson/about>*

---------- Forwarded message ----------
From: Tim Donohue <tdonohue at duraspace.org>
Date: 8 September 2015 at 22:02
Subject: Re: [dspace-tech] Re: Problem with solr reindex command in the 13
upgrading step to 5.3
To: Hilton Gibson <hilton.gibson at gmail.com>
Cc: Andrea Schweer <schweer at waikato.ac.nz>, "Pottinger, Hardy J." <
PottingerHJ at missouri.edu>, DSpace Technical Support <
dspace-tech at googlegroups.com>

Hi Hilton,

With regards to your release testing brainstorms on your own wiki, we
honestly would appreciate institutions stepping forward and offering
resources (technical infrastructure, staff, etc) during our yearly
Test-a-thons. As an established open source project, we have a broad
community of users, but our developer core is still very volunteer
oriented. There literally is no one who works full time on DSpace (not even
myself). We are reliant on the kindness of individuals (and oftentimes
their bosses!) to help us to build, support, improve and test DSpace.  It's
amazing what we have been able to get done entirely by volunteer work (with
a little bit of coordination).

We do hold yearly Testathons where we encourage the broad community to take
part, bang on the software and help us to make the next release as "battle
tested" as we possibly can. Also, the DSpace Community Advisory Team (DCAT)
has begun an initiative (just this week) to help our community to develop a
more extensive "Test Plan" (which we can use to ensure each piece of the
system has received testing attention from our volunteers).

Their work has begun here, and I'm sure they'd love to have additional
contributors to the work overall
https://wiki.duraspace.org/display/cmtygp/DSpace+6+Testathon+Testplan+Working+group

If you or your institution is willing to help out in any way, we'd
appreciate the support. (This is the same for anyone else reading this
thread!) We'd honestly love to have institutions with larger production
environments help us test early versions of DSpace. But, as of yet, we've
never been able to find those volunteers (or a large corpus of test data).
So, we tend to rely on a more "crowd sourced" testing model (where we put
up a couple of test instances and ask folks to help us bang on them). This
crowd-sourced model tends to find most software stability bugs, but it
admittedly may not always catch all of the scalability bugs.

In all honesty, we want and need more testers to get involved during
Testathons and just before major releases. If anyone else is interested and
willing to help, get in touch, or join up with the DCAT initiative! We'd
love to have your help.

- Tim

On 9/2/2015 5:20 PM, Hilton Gibson wrote:

Hi Andrea,

There are many who contribute to this community project. I do not wish to
single out any persons for fear of offending others.
I think I do my bit with a wiki that I update as often as I can.
I am upset, we had some violence on campus today, and perhaps it spilled
over into my reply.
I apologise.

However introducing new features to DSpace that are not "battle tested"
seems to be a common theme lately.
I am trying my best with my limited programming skills and extensive
production system experience to maintain a production ready repository and
help others in developing countries to do the same. (This is what I meant
by the global south - most developing countries are in the south)
So the issue with re-indexing solr came at a very bad time.

If you have time please read and consider the following:
1.  <http://wiki.lib.sun.ac.za/index.php/SUNScholar/Reference_Architecture>
http://wiki.lib.sun.ac.za/index.php/SUNScholar/Reference_Architecture
2.
<http://wiki.lib.sun.ac.za/index.php/SUNScholar/Software_Release_Cadence>
http://wiki.lib.sun.ac.za/index.php/SUNScholar/Software_Release_Cadence

Cheers

hg

*Hilton Gibson*
Ubuntu Linux Systems Administrator
Stellenbosch University Library
<http://staff.lib.sun.ac.za/%7Ehgibson/docs/cv/cv.html>
http://staff.lib.sun.ac.za/~hgibson/docs/cv/cv.html

On 2 September 2015 at 23:46, Andrea Schweer <schweer at waikato.ac.nz> wrote:

> Hi Hilton,
>
> I'm sure you're aware that even those of us not in the global south have
> limited resources. I put in a lot of work to fix a problem that was
> introduced by people other than myself. I then took the time to share my
> solution with the community, even continuing to improve my code based on
> the feedback of several testers to make the reindex work for people without
> solr knowledge at I time when I had already upgraded all my own solr cores
> and I could just have easily declared this to be someone else's problem.
> The reindex script I wrote may not work in all situations, but had I not
> volunteered my time (and had my employer not let me do so) then there might
> not be a fix at all. Perhaps there are cultural differences at play here,
> but your e-mail below reads to me as quite aggressive and as if you'd
> rather have no code at all than code that worked fine for everyone who
> helped me test when I developed it and worked fine for all "my" DSpace
> instances. I'm sure that's not what you actually mean.
>
> A more constructive reaction than the e-mail I'm quoting might be to try
> out the suggestions made in this thread (increasing the heap space does not
> require you to put the server into debug mode) and/or to share more details
> of what you tried and what happened when it failed so that the volunteers
> (!) on this list can try and figure out what's going on.
>
> I've given you the technical reasons why an incremental reindex is not
> really doable. You say verbose log output would be nice -- what types of
> things would you like to see logged? The reindex does log to dspace.log at
> INFO level; it will tell you every time it writes
> <https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/util/SolrImportExport.java#L590>
> an export file and every time it reads
> <https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/util/SolrImportExport.java#L431>
> an import file. Export happens before import -- even without programming
> skills, the code comments in the reindex method might let you determine
> this much (
> https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/util/SolrImportExport.java#L306).
> So you should be able to use the log output to determine during which part
> of the process this fails. All access to Solr is also logged in the
> solr.log file, so again there you will be able to see whether solr is busy
> doing exports (/select) or imports (/update).
>
> cheers,
> Andrea
>
> On 03/09/15 02:55, Hilton Gibson wrote:
>
> Hi Hardy,
>
> I do not have the time to experiment on a production server.
> If this could be done incrementally as I asked before, then perhaps.
> When there is no verbose output and no log details, it is very difficult
> to debug.
> I am not going to put a production server into debug mode.
>
> So we will live with faulty geo stats for now.
> Remember not all institutions have expert java programmers/system
> personnel at their disposal to fix these things.
> In the global south we have to make do.
>
> Cheers
>
> hg
>
>
> *Hilton Gibson*
> Ubuntu Linux Systems Administrator
> Stellenbosch University Library
> <http://staff.lib.sun.ac.za/%7Ehgibson/docs/cv/cv.html>
> http://staff.lib.sun.ac.za/~hgibson/docs/cv/cv.html
>
>
>
> --
> Dr Andrea Schweer
> IRR Technical Specialist, ITS Information Systems
> The University of Waikato, Hamilton, New Zealand+64-7-837 9120
>
>
-- 
You received this message because you are subscribed to the Google Groups
"DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to dspace-tech+unsubscribe at googlegroups.com.
To post to this group, send email to dspace-tech at googlegroups.com.
Visit this group at http://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.

-- 
Tim Donohue
Technical Lead for DSpace & DSpaceDirect
DuraSpace.org | DSpace.org | DSpaceDirect.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lib.sun.ac.za/pipermail/irtalk/attachments/20150908/940dded1/attachment-0001.html>