Please find the updates for the Lent term at the Summer LUVLE Update page

Friday, 25 March 2011

Questionmark Perception (QMP) experienced a problem this morning at around 5am. You may recall that QMP is the data engine behind both LUMES and Computer Based Assessment.  The back end database took about 4.5 hours to run a routine maintenance process which normally takes half an hour, making it unavailable for use. This process finished at about 9:30am and the service is back to normal without a re-start. During the period, although the server did not go down, there were approximately 25 unsuccessful visitors to the site. Investigations into the reasons for the prolonged process are still ongoing.

Wednesday, 23 March 2011

At 04:28 of 22 Mar one LUVLE server (domino.lancs) had an outage which lasted 131 minutes.  Because it was shortly after the 04:10 re-start, fewer than 29 were likely affected.  Crash data were sent to IBM for analysis.  Although our analysis is still not complete, we suspect it was caused by a logon to an old template LUVLE Space.  We have denied anonymous access to that and other sites.  

Wednesday, 16 March 2011 - 18:00 PM

Announcement: LTG and Learning Zone Extra LUVLE Cover for 16-27 March 2011

We have entered a critical time in the academic calendar as we approach the end of term.  Thus ISS are providing a special set of services from 16 through 27 March.  Often we see an increased use of Coursework Submission during this period right before the end of term.  Below is a listing of proactive changes we are making to assure that this period of increased demand on LUVLE 10 will be met:

1. We are testing the latest Hotfix from IBM to assure it does not cause any problems when subjected to load.  Once we have completed the testing regimen, we will determine whether and when we will install it.
2. We have increased the number of CPUs on the server that provides the largest number of LUVLE Spaces, as required to meet the expected increase in demand. 
3. LTG technical staff are available on weekdays afterhours and the weekend to respond to potential outages during Learning Zone hours (see below).
4. We are increasing the number of student employees in the Learning Zone for the next two weeks (until 27 March) who can both test the LUVLE servers and respond to student problems if one does occur.  In addition ISS Service Desk and Learning Zone student staff will be completing logons to LUVLE at all hours to test server responsiveness.
5. We are engaging both the Library and Learning Zone staff in a plan to keep LTG informed without delays of serious problems with LUVLE if they arise until 27 March. 
6. We have changed the parameters on our server monitoring software to notify us earlier if a problem appears with decreased server responsiveness.
7. We are working closely with other-ISS staff to assure there are no changes to systems that might affect the access to and availability of LUVLE until the end of term (27 March).

Learning Zone (LZ) hours (staffed)

•  M-F 09:00 to 22:00
•  Sat  10:00 to 14:00
•  Sun  10:00 to 18:00

Wednesday, 16 March 2011

Because of the work we did to stabilise the one LUVLE server yesterday (see below), we were unable to complete full testing of the IBM-provided hotfix before the scheduled installation time of 15:00 today.  We felt it was best to not take the chance of disturbing service at this critical time of the academic year, so we are now planning to fully test the hotfix this week and, if the results are favorable, we will consider installing the hotfix next Wednesday.   Thanks for your understanding.

For those of you who plan ahead, we are planning another upgrade of the LUVLE servers during the Easter break time.  Here's the scheduled dates:

• 4-5 April - MyPlace (and MyPGR)
• 6-7 April - luvle and ltg-luvle
• 11-12 Apr - domino and ltg-domino
• 13-14 Apr - ktru-main

Later we will more fully describe what we are doing and who these server changes will affect.

Tuesday, 15 March 2011 (18:45)

Full LUVLE service from all of our servers has been restored.  The recap is as follows:

• The LUVLE server hosting MyModules and numerous LUVLE Spaces (ltg-domino) started to run slowly today at about 12:30.  The other servers were unaffected.  While the response time was often slower than our standard, the server did not stop running.
• There were about 1,300 simultaneous users logged on to the service, higher than our normal usage pattern.
• To mitigate this problem, we brought forward an approved plan to upgrade the hardware by adding another CPU to ltg-domino.  This necessitated a re-start of the server at about 14:40.  However the hardware upgrade did not bring the expected improvement to server responsiveness. 
• We moved MyModules to another server (domino) and re-started it at about 16:50.    The service was restored about 17:00 and appears to be running with improved responsiveness.
• We are working to establish a conclusive cause for the slowdown in service, and are monitoring the situation closely so that we can continue to provide you a responsive service. 

Tuesday, 15 March 2011

The ISS server monitoring tools indicated a slow responsiveness in one server of LUVLE starting about 12:30.  This was intermittently decreasing the responsiveness to as slow as 15 seconds.  To remedy this problem, we are adding another CPU.  This requires that the server be shut down and restarted.

We apologise for any inconvenience but feel this will improve responsiveness for the foreseeable future.

Monday, 14 March 2011

On Saturday 12th March, ltg-domino suffered an unplanned re-start and was unavailable for 13 minutes. The cause was the same as all the recent re-starts on this server. We expect the hot fix (a software patch) that we received on Friday 11th March will prevent further recurrence of this problem after it is applied on Wednesday 16th March afternoon. LUVLE is scheduled to be unavailable for application of the hotfix at 3 pm, it should not take more than 30 minutes.

The MyPlace server suffered a unplanned re-start later on Saturday for 5 minutes, this is also a known but rarely occurring issue and one that should disappear when MyPlace is moved to a 64 bit platform at the end of term. MyPlace is scheduled to be unavailable to allow this move on the 4th and 5th of April.

Friday, 11 March 2011

Just before noon we received the latest Hotfix from IBM.  It is intended to clear up the problem we have been having as demonstrated by 11 of the last 14 outages in six weeks.  We have started our testing routine today and, if all goes well, plan to apply it to our LUVLE servers on Wednesday, 16 March starting at 15:00.  Unfortunately we will have to shut down the servers before we can apply this fix, so you should plan to be off LUVLE at 15:00!

We have announced this plan in MOTD today, and will remove that notice as soon as the full service is restored.  Thanks for your patience.

Thursday, 10 March 2011

One of the LUVLE servers, ltg-domino, had an unexpected outage this morning between 08:13 and 08:17 (5 minutes).  Anyone trying to logon to LUVLE through MyModules would have found it unavailable for those five minutes.  We estimate that about 73 users who were online at that time may have lost their connection.  We are working closely with IBM to determine the cause of this intermittent problem, and have escallated it to the vice-president level.

Wednesday, 9 March 2011

One of the LUVLE servers (ltg-domino) had a short problem this morning at 09:10 for three minutes.  It appears to be the same problem as listed for 4 March.  IBM are being contacted.

Friday, 4 March 2011 (PM)

The two instances of one of the LUVLE servers (ltg-domino) being re-started were both caused by calls for information to LUSI. This is the same very intermittent problem that caused six other server problems last month. We have sent the server log files to IBM and expect them to send us another Hotfix soon.  Our request is at the highest priority in the IBM queue.

Friday, 4 March 2011 (AM)

We have had two instances of one of the LUVLE servers (ltg-domino) being re-started at 08:42 and 08:59 this morning.  They lasted 8 and 6 minutes respectively.  Both re-starts were triggered by slow response times.  We are investigating the cause, and will update this site with the results.  

Thursday 24th February 2011

There have been occasional, unscheduled breaks of service since the previous update in December but over the last few days the server hosting MyModules and a large proportion of module spaces has suffered a short break in service each day. We know the problem is in IBM software and they have been on the case since this sequence began, we expect to receive a system hotfix from them soon. At the same time we are working on other ways to avoid the problem recurring.