During testing for the S452 experiment yesterday we were running through the steps of a restart.
The FEEs were not powercycled at this stage but the MIDAS server was restarted.
Initial setup went smoothly and no issues were encountered during setup.
NewMerger and TapeServer were both setup and set to going.
Upon ''Going'' the run control the data transfer for aida09 dropped out.
Going to the NewMerger it could be seen that data was making it through and data items were being merged.
The tapeserver however was seeing no data and neither was the MBS dataspy.
During this time the FEEs became unresponsive likely as their buffers filled. Trying to reset gave the attached errors.
Checking the folder manually it could be seen that the Options file was there.
At this point a powercycle was pewrformed but upong 'Going' the DAQ the same issue was again encountered.
This time we were able to recover control access to the FEEs by running multiple NewMerger sessions which helped clear the buffers.
During our search for the problem we looked at the Options from within MIDAS and could not see anything out of the ordinary.
We also tried restoring the Options from within MIDAS.
We again reached the same issue with the tapeserver.
At this point we restored the Options folder from by copying manually within terminal.
Following this we were able to start the DAQ normally. |