|
|
Post by Dstock7080 on Jan 23, 2023 17:48:36 GMT
District Line, Circle and H&C suspended, Met part suspended Wembley Park - Aldgate due to a signalling system failure. Is this CBTC related? Yes, total VCC failure at Hammersmith. |
|
|
|
Post by A60stock on Jan 23, 2023 18:01:01 GMT
Assuming this will not be resolved for the remainder of the day given how serious this is?
|
|
|
|
Post by Dstock7080 on Jan 23, 2023 19:10:29 GMT
Assuming this will not be resolved for the remainder of the day given how serious this is? Everything rebooted, service should resume 19.20 |
|
|
|
Post by david123 on Jan 23, 2023 20:49:07 GMT
Its gone again
|
|
|
|
Post by xtmw on Jan 23, 2023 21:36:30 GMT
A shuttle service is operating Parsons Green - Wimbledon. Severe delays Upminster - Dagenham East. H&C and Circle are suspended. Met part suspended, severe delays rest
|
|
|
|
Post by roman80 on Jan 23, 2023 22:00:28 GMT
Has a root cause been identified yet? Extraordinary to see a whole line go down like this.
|
|
Tom
Administrator
Signalfel?
Posts: 4,196 
|
Post by Tom on Jan 23, 2023 22:05:47 GMT
Not yet. I expect there will be a lot of poring over log files tomorrow.
This is, of course, the risk with a highly centralised infrastructure.
|
|
|
|
Post by A60stock on Jan 23, 2023 22:05:55 GMT
Appreciate there is probably a reason for how things operate but the message on the met line was completely poor and out of sync. TFL say the met is running from wembley park and ofcourse staff can only reiterate what is said there.
Get to Wembley park and guess what.......no met line and people being advised to take local buses from Queensbury on the jubilee. Strangely enough a train was parked on platform 1 not in service whilst another came out of the depo and passed out of service on what was a very busy platform 2. Why could either of these not have been run in service if they were clearly departing in the direction towards harrow?
This is quite frankly ridiculous for customers who need stations harrow and onwards as they would have been better advised to have gotten Chiltern from Marylebone to harrow.
The message should really say the met is suspended from harrow to alagate.
|
|
|
|
Post by plunet on Jan 24, 2023 16:02:04 GMT
Appreciate there is probably a reason for how things operate but the message on the met line was completely poor and out of sync. TFL say the met is running from wembley park and ofcourse staff can only reiterate what is said there. Get to Wembley park and guess what.......no met line and people being advised to take local buses from Queensbury on the jubilee. Strangely enough a train was parked on platform 1 not in service whilst another came out of the depo and passed out of service on what was a very busy platform I think the regulars on the Met Line will be used to the smoke and mirrors customer information that is fed to the passengers. It seems to me that TfL towers conjure up passenger information processes based upon the simplest of service patterns, but fail to cope with more complicated services like the Met. |
|
|
|
Post by roman80 on Jan 27, 2023 0:00:19 GMT
Was there any prognosis as to the cause for the outage across the line? Not yet. I expect there will be a lot of poring over log files tomorrow. This is, of course, the risk with a highly centralised infrastructure. |
|
|
|
Post by piccboy on Jan 27, 2023 11:44:38 GMT
Was there any prognosis as to the cause for the outage across the line? Yes. |
|
Colin
Advisor
My preserved fire engine!
Posts: 11,346
|
Post by Colin on Jan 27, 2023 12:03:43 GMT
Was there any prognosis as to the cause for the outage across the line? From what I've heard, it was caused by a timetable edit that set up a never ending run of route reservations which overwhelmed the system causing it to give up and shut down. Once the cause was identified, a system reboot successfully cleared the error. As mentioned above, this is the downside of using centralised computer control. |
|
|
|
Post by Dmitri on Jan 27, 2023 14:47:19 GMT
it was caused by a timetable edit that set up a never ending run of route reservations which overwhelmed the system causing it to give up and shut down Note to programmer: watch for the possible infinite loops  . |
|
|
|
Post by spsmiler on Jan 27, 2023 15:02:40 GMT
Note to programmer: watch for the possible infinite loops . When I was a single-digit aged schoolboy and we did computer programming (on punch cards!) the system was designed to look out for infinite loops and halt the routine before these caused problems. This would have been in the late 1960s! |
|
|
|
Post by ijmad on Jan 27, 2023 23:10:23 GMT
I'm somewhat surprised that a timetable change could bring down the entire signalling system, is it because the system would be unable to route trains to the correct destinations at junctions?
|
|
Tom
Administrator
Signalfel?
Posts: 4,196
|
Post by Tom on Jan 28, 2023 10:02:36 GMT
It wasn't a timetable change, it was an edit to a trip. The two are different.
The Baker Street era systems were similar that the computers could only handle a certain number of trip entries and edits. When services were severely disrupted it became necessary to reboot and begin again - a process known as giving a TT Reset. I'm not sure there is a CBTC equivalent or if indeed the number of trip edits actually was the problem - the reports will reveal all, when they are available.
|
|
|
|
Post by ijmad on Jan 28, 2023 16:08:45 GMT
It wasn't a timetable change, it was an edit to a trip. The two are different. Thanks very interesting. As a software engineer myself (who is not involved in physical or safety critical systems) I find this sort of thing intriguing in the detail. |
|
|
|
Post by d7666 on Jan 29, 2023 14:22:42 GMT
It wasn't a timetable change, it was an edit to a trip. The two are different. The Baker Street era systems were similar that the computers could only handle a certain number of trip entries and edits. When services were severely disrupted it became necessary to reboot and begin again - a process known as giving a TT Reset. I'm not sure there is a CBTC equivalent or if indeed the number of trip edits actually was the problem - the reports will reveal all, when they are available. I got trained on - but subsequently job content changed - on this part of HSCC ATS (automatic train supervision) servers. There is an equivalent HSCC reload process, but I myself have only theoretical contact with it on a formal training course; all this this is outside my area now. AIUI, on Monday a timetable / server reboot was tried first, unsucessfully. But that is third hand info; I was not on site; don't quote me. I have earlier been involved with timetable server issues both on the pre-TBTC JLSCS control system at Neasden, and on CLSCS at Wood Lane. The latter, before my time, did experience a significant outage when the number of trips in a special timetable exceeded the limit. That limit was increased. That event was what? 20 years? ago now. Later we did have one repeat CLSCS failure NOT on the live system, but found on test rig before loading live system where one abnormally complexicated weekend special timetable for engineering works exceeded the (increased) trip count; it was not used; different version generated, tested off line, and used. Then me directly involved with a further increases for 2012 Olympics and Night Tube. Also involved with one potential JLSCS timetable failure again headed off by loading on test rig first. In this case it was not a trip count issue - one of the data fields contained a typo - a number "0" (zero) had in error a letter "O" (capital o) - but the timetable file would not load as a result. Again different version generated, tested, and used. None of those events I was involved with were linked to trip edits; they were issues in the timetable itself. PS the test rigs used for this at Wood Lane CLSCS and the old Neasden JLSCS were the operator training simulators; we had no luxury like the PC based CTBC 'op-sim' to do all this. I'll point out ALL timetables including EVERY special timetable, not just the base timetable, are tested before installing and activating on the live railway; and for every line. |
|
|
|
Post by d7666 on Jan 29, 2023 14:30:45 GMT
Note to programmer: watch for the possible infinite loops . When I was a single-digit aged schoolboy and we did computer programming (on punch cards!) the system was designed to look out for infinite loops and halt the routine before these caused problems. This would have been in the late 1960s! Compare with Swiss cheese model. No matter what tests are in place, something some time will get through every test and cause a failure. IMHO that is almost certainly what happened here. Eventually someone will do a genuine valid edit and the precise exact details at a certain time on a certain day on a certain date with a certain timetable will fall through every hole in the model. |
|
|
|
Post by d7666 on Jan 31, 2023 23:08:43 GMT
And the cause is within the central CBTC kit at HSCC before any data gets to the central CIS kit (also at HSCC). Underlying issue identified yesterday; it needs a Canada fix. Fix installed engineering hours last night (30/31Jan) was looking good today but obviously time will tell |
|
|
|
Post by jimbo on Feb 1, 2023 3:55:15 GMT
Since SMA6 started, the Circle and Hammersmith trains no longer pass train-stop fitted signals. Does this mean that they can operate using S7 stock with defective tripcocks? Although they could not be used to reform into a District Line train during the day. This would exclude those workings starting or finishing at depots off the line, but would not be a problem for around two-thirds of their trains. Once the new signalling reaches Upminster and Neasden in the next two area conversions, only workings to and from Ealing Depot will use a tripcock. Are we possibly going to end up with a sub-group of the fleet that is not suited to working on the District Line?
The original resignalling plans would have covered the whole District Line, allowing the removal of tripcocks from S7 trains. With western branches of the District Line being dropped from the programme, it was once reported that consideration was being given to fitting Network Rail compatible equipment on S7 trains to enable trainstops to be removed even there. I have seen nothing more about this, but believe it could only be a long-term solution. Trainstops are to remain for Piccadilly Line trains around Acton Town and Ealing Common for some time, even with the new fleet arriving, since that resignalling remains unfunded. It would also require the Network Rail equipment to be fitted west of Stamford Brook to Gunnersbury and to Ealing Broadway. The removal of trainstops and tripcocks also allows the removal of lineside air mains and related compressors, etc. saving maintenance costs.
|
|
|
|
Post by Dstock7080 on Feb 1, 2023 7:52:05 GMT
The removal of trainstops and tripcocks also allows the removal of lineside air mains and related compressors, etc. saving maintenance costs. There are still legacy pneumatic point installations remaining in certain areas; Edgware Road, High Street Kensington, West Kensington, Aldgate East etc. |
|
|
|
Post by gigabit on Feb 1, 2023 9:43:17 GMT
They're now showing a notice at Earl's Court that says the information for Edgware Road trains won't show up until 5 minutes from Earls's Court (assume this is moving into the CBTC zone and in time will be earlier and earlier?). This is a good change as people often are bemused where the Edgware Road trains are.
|
|
|
|
Post by Dstock7080 on Feb 1, 2023 9:59:01 GMT
They're now showing a notice at Earl's Court that says the information for Edgware Road trains won't show up until 5 minutes from Earls's Court A pre-recorded announcement has been playing for a while now, giving the same information. Data picked up from Fulham Broadway. |
|
|
|
Post by Dstock7080 on Feb 2, 2023 14:56:12 GMT
It may have already been mentioned that 17's points between Barons Court and Hammersmith are being removed next month (or so) so it will no longer be possible for trains from the westbound fast (Piccadilly) to carry out a move to the westbound local (District). Confirmed for decommissioning night of 16 February, final removal 18 March. |
|
|
|
Post by PiccNT on Feb 2, 2023 17:53:55 GMT
It may have already been mentioned that 17's points between Barons Court and Hammersmith are being removed next month (or so) so it will no longer be possible for trains from the westbound fast (Piccadilly) to carry out a move to the westbound local (District). Confirmed for decommissioning night of 16 February, final removal 18 March. And that's our get out of jail card gone. Might request one last trip down there. |
|
|
|
Post by t697 on Feb 2, 2023 19:02:38 GMT
A bit like the 'emergency' reverse done in Mansion House p2 on its last evening, eh!
|
|
|
|
Post by joshua on Feb 4, 2023 11:08:47 GMT
As part of the resignalling are they straightening the track in certain area's?
|
|
|
|
Post by Dstock7080 on Feb 4, 2023 12:14:53 GMT
As part of the resignalling are they straightening the track in certain area's? Apart from removing some paintwork and plain-lining, there isn't any track straightening. Where are you thinking of? |
|
|
|
Post by joshua on Feb 4, 2023 13:13:46 GMT
As part of the resignalling are they straightening the track in certain area's? Apart from removing some paintwork and plain-lining, there isn't any track straightening. Where are you thinking of? I was thinking where the old Hammersmith Link Line viaduct was. I was not sure if they would demolish the rest of the viaduct to permit straighten of the track for higher line speed on both Piccadilly and District Line. |
|
.
