Use this thread for reporting outages or requesting support.
Current DDoS attack update
A permanent fix is very close, we've peered up with Micron21 now and as soon as they can take our routes traffic will pass through their scrubbers and we should be back to normal.
Once we have a resolution there'll be an email sent out to all customers explaining the events and the solution we have implemented.
Thank you for your patience and apologies again.
Can currently register but not make calls on sip.telecube.net.au
2 days now that we can't make or receive calls this is terrible. I'd like to think we'd be given some re-assurance that these issue would not occur again as this is business crippling stuff.
we'd be given some re-assurance that these issue would not occur again
Per John M's post:
A permanent fix is very close, we've peered up with Micron21 now and as soon as they can take our routes traffic will pass through their scrubbers and we should be back to norm
Given no calls work at the moment... why not null route the sip servers on transit links, and allow peering traffic (assuming it's clean)... at least then some portion of the customer base would have calls working and the others wouldn't have the misleading "registration successful" status.
Can currently register but not make calls on sip.telecube.net.au
Sort of similar:
Two accounts plus a sip trunk on sip.telecube.net.au.
Can receive inbound calls but not outbound.
Tried changing to .com.au, but it wouldn't register.
Not a biggie. I can operate as long as I have inbound, a mobile and email.
As I see it, things like this are just part and parcel of "modern" life and we all live and learn as we jump each hurdle.
Pretty sure this has been mentioned before but in case it helps someone this is what I have done
I have 6 yealink handsets and a cordless phone through an ATA
I have a Pennytel DiD that I haven't used in a while that allows multiple registrations
1. Registered all devices to the Pennytel DID
2. Set up a line hunt in the TC portal to ring the Pennytel DID
3. Associate my TC DID to the new line hunt
Now when anyone rings my business number it forwards to the Pennytel DID and all phones ring
At least I am able to keep taking calls until the situation is sorted
A mobile also works but restricts you to 1 incoming call
It would be good if you could add mobile/landline numbers to a ring group as well, this might also work but not sure
Pepe
Hi everyone. People have mentioned a line hunt would have allowed incoming DID/1300/1800 to divert to a mobile or PSTN line during this attack. I had a line hunt set up for a number of clients accounts and the line hunt also failed to divert to mobiles and PSTN lines.
I am a big supporter of Telecube and John.
I need a fail safe solution for incoming calls for my clients moving forward.
Does anyone have any recommendations?
I am thinking of moving DID's to a dedicated DID host like DIDWW and routing calls to Telecube as priory 1 with other another provides trunk as a fail safe.
Does anyone have thoughts on this or other solutions?
Cheers.
Does anyone have any recommendations?
This is the first major issue. Don't jump ship yet.
x2 at least 75% of my incoming calls to mobile failed or rang for maybe 2 seconds before terminating.
Leads me to believe it rings longer on TC's side but the network issues can't get the call out fast enough.
Edit: I'm sticking with TC, just consider it an upvote for such a feature
I had a line hunt set up for a number of clients accounts and the line hunt also failed to divert to mobiles and PSTN lines.
I did this also. It worked for one or two calls but that was it.
When I tried diverting to a Telstra line, it didn't work but some reason a TPG line worked okay.
I'm no expert by any means. But could DID's perhaps be put on a separate layer to the rest of the SIP services? In the event of catastrophic failure they would remain alive and routable elsewhere?
We've got routes out through Micron21 now and seeing some attack traffic coming across them and being dropped .. won't be long now.
I'll update again here shortly.
I am not thinking of jumping ship. My clients spend 100s of dollars a month on outgoing calls each which would remain with Telecube. With the Whirlpool offer each client spends 0.50c per month on a DID.
I was only trying to think of a more redundant solution with backups for incoming calls.
2 days now that we can't make or receive calls this is terrible. I'd like to think we'd be given some re-assurance that these issue would not occur again as this is business crippling stuff.
Do you have a 1300 number with them? They offer an SLA and so will credit you according to time down.
If not, well this service doesn't claim to be a business grade bulletproof service so if you needed that you should have paid for it with a VoIP provider that does offer it.
I resell services with another provider for this exact reason � I tell my clients if they need guaranteed uptime and guaranteed QoS they have to pay a higher price per month � simple as that.
Even then nothing is completely bulletproof � Telstra's NBN voice services were down on Friday for half the day, and two of their main DNS servers were down for several hours on top of that!
Also waiting for resolution. My lines at my office and my home office are down. If it wasn't for my backup service we'd not be able to operate.
SIP registration not working for hours now, hope this is resolved very quickly.
A bit disappointed that the only option when calling for support is to leave a message.
I am sure you're very busy but it would be helpful to know when you expect services to be fully restored.
right now
Congratulations and sincere thanks to you and your team John
We're not fully protected yet but I think we're on the home stretch
I have full service again at this point, John
Spoke a little too soon. Can ring out, but not in. No problem it's quitting time here anyway.
registration and outgoing calls working again here too (on Amnet). I don't have a DID, so can't test that.
If the extra protection means and increase in tariff, you will get no complaints from me. Happy to pay extra for a safer service.
If the extra protection means and increase in tariff
Nope, I know there's people saying you can't expect the stability from a discount provider but I plan to prove them wrong. :-)
Ok.....incoming to DID is working too
Can ping sip.telecube.net.au
Can't ping sip.telecube.com.au
Just advising...
No outgoing calls possible on IP-authenticated trunk.
Incoming calls on URL-authenticated trunk working with the usual delay.
Password-authenticated trunk failing to register. (Only used for testing, not urgent.)
Cannot ping 'sip.telecube.com.au' but I can trace route to it.
Thanks for all the hard work.
Incoming and Outgoing now working for us here on .net.au
Reaching Telecube over WAIX
So the IP for sip.telecube.net.au changed? From .89 to .95???
Why not tell customers!
Was expecting to go via a scrubbing service but still going via Equinix from Telstra DSL:
traceroute to sip.telecube.net.au (103.193.167.95), 30 hops max, 38 byte packets
1 gw130.ken10.sydney.telstra.net (203.45.254.1) 28.170 ms 28.321 ms 28.517 ms
2 bundle-ether7-100.chw-core10.sydney.telstra.net (203.50.20.129) 29.807 ms 30.837 ms 29.912 ms
3 bundle-ether8.exi-core10.melbourne.telstra.net (203.50.11.125) 43.966 ms 44.692 ms 44.760 ms
4 bundle-ether1.lon-edge901.melbourne.telstra.net (203.50.11.108) 43.474 ms 43.153 ms 43.494 ms
5 bundle-ether2.win-edge901.melbourne.telstra.net (203.50.11.114) 41.741 ms 42.180 ms 41.652 ms
6 equ1651503.lnk.telstra.net (165.228.53.66) 43.446 ms 41.198 ms 41.779 ms
7 183.177.59.26 (183.177.59.26) 41.488 ms 40.823 ms 41.650 ms
8 103.193.167.95 (103.193.167.95) 42.055 ms 41.807 ms 41.146 ms
My Gigaset is registered not home to see if it working though.
Why not tell customers!
It's hopped lots of times, as I'm assuming they've been moving around blackholed /32s.
Telling the attackers exactly when to jump to the next ip doesn't sound like a great plan.
Telling the attackers exactly when to jump to the next ip doesn't sound like a great plan.
Bit late now :-)
So the IP for sip.telecube.net.au changed?
I'm still routing around issues created by the attacks. Once it settles I'l explain everything to everyone.
sip.telecube.net.au
mtr showing high loss (10%) from syd to melb
bundle-ether12.win-core10.melbou 10.5%
Was expecting to go via a scrubbing service but still going via Equinix from Telstra DSL:
Local carriers haven't taken up the new routes yet .. hopefully overnight.
At the moment though international traffic is routing through the scrubbers and it's looking like the attacks are all international originated.
No inbound yet for me. But heck Perth is a long way away :)
sip.telecube.com.au registration now ok.
b/w calls ok.
hope it holds.
John when is acl being restored?
There, there, don't feel left out... no inbound for me either in Melbourne.
Outbound OK, inbound still broken, but can completely understand the circumstances.
John, you're in better humor that I would be after however many hours!
Thank you for the extra mile today it always helps enormously to be able to explain in some amount of detail what is going on � calming the natives somewhat.
Softphone clients still seem to be offline. Landline however works.
Still no registration.
Can't ping sip.telecube.net.au � Destination Host Unreachable
What to tell to management team tomorrow?
What to tell to management team tomorrow?
It's working :P coz the IP routes should update overnight :)
yeah, just noted that the IP of the sip.telecube has changed.
Rego is OK now. DID runs OK
My 1800 number can be forwarded to another landline or mobile, but doesn't go to an extension :(
What to tell to management team tomorrow?
I'm guessing you are at a small office as otherwise you'd already have fail-over in place and you'd have asked for SLAs and other things that you have to pay a pretty penny for.
If so tell them that you had no fail-safe plan due to previous excellent performance, and that this event is likely an one-off but no-one can be sure. Ask them if they want a fail-safe plan, and if there is budget for one.
If they keep on complaining also note all the pieces of current infrastructure and personnel that you may not have fail-safe plans for. Backup ISP, modem, router, back-up office in case of fire, maybe a back-up CEO in case of death... (ok, maybe don't mention the last part..).
maybe a back-up CEO in case of death
I think I will be buried without backup first :)
Lost registrations on two devices. Tried to change passwords on them but no luck -server not answer. Did not know about outages and started to check my OBi. And two providers are OK .One connection at Blackburn's OBi with TeleCube is also OK (can see it Online)
Only two connections at Carnegie (Melbourne) can not register. Changed passwords and now don't know what to do. Registrations on my home phone and mobile are failed.
Dania, what I found on my devices is the old I for sip.telecube.net.au
The new one looks like this 103.193.167.95.
You may try to check what is the IP for sip.telecube.net.au in your DNS-server' cache
Can confirm again
Simple routing for 1800XXXXXX
- Works with landline and mobile numbers
- Doesn't work if an endpoint is an extension. Packet monitoring of the extension doesn't show any traffic
1800XXXXXX -> queue � works.
Thank you Grussy. I will try everything tomorrow. May be it will be fixed soon
I'm not sure if this applies to everyone, but I found a configuration change that worked:
My inbounds were not working and it turns out it's related to "Queues".
I changed the DID to "LineHunt" (ok, so not all extensions ringing at once, but it's ok for now) and now inbounds work.
My inbounds were not working and it turns out it's related to "Queues".
I changed the DID to "LineHunt" (ok, so not all extensions ringing at once, but it's ok for now) and now inbounds work.
+1
I can also say that using a line hunt instead of a queue (when queues were working before) seems to have better performance. Before, I would call my DID and it would ring once and maybe sometimes even twice before my desk phone actually rang, but now with a line hunt the desk phone rings immediately upon the caller hearing the "ringring" noise. :)
Here in SYD on Belong/ Telstra.
Both .com and .net registrations working.
Internal and inbound calls working. Outbound calls still an issue
Yay, now working for me as well.
what country are these attacks coming from?
Do we know how many carriers are yet to switch to the new routes?
I think the old routes have been dropped off now without the new routes being loaded (at least for Telstra). Traceroute just shows no progress past Telstra core router in Sydney.
Same with routing for AWS hosts, it now goes nowhere.
We've now got people complaining that the line is crackly and people can't hear what's being said.
I agree with very_itchy although it doesn't really matter how many carriers have switched.. All the routes you advertise should now only be those that go through scrubbers. All other routes are basically ineffective so why hold onto them?
We've now got people complaining that the line is crackly and people can't hear what's being said.
I also got that on one extension that received a call earlier this morning. That is just another symptom of the problem that will be resolved (and is expected)
I have flipped my incoming calls back to my PSTN line, so no need for me to throw the baby out with the bath water.
Some sites that had outbound working half an hour ago now can't dial out. ARGH!
And it's all toast again.
John,
If the routes haven't propagated locally yet, I would suggest that they are not being advertised correctly.
I have changed BGP routing locally and I can tell you it normally only takes a couple of hours locally worst case.
Is there a way to divert my incoming calls to my mobile, please?
Inbound and outbound calls are working again for me in NSW (on net.au).
The only inbound test I can make is from my Telstra pre-paid mobile. Perhaps that is my problem?
Hi Arismac � you probably should set up call forwarding till this is all over.
Call Forwarding
Call forwarding is a telephony feature available to local numbers, which allows redirects the telephone call to another destination. It allows you to automatically divert incoming calls to another number. It can be redirected to a mobile telephone or another telephone number where the desired called party is able to answer.
To enable 'Call Forwarding', log into your portal, access your 'Services'. Go to the 'Manage' tab next to the DID.
Set the profile type to 'Call Forward'.
Input the phone number you'd like to forward the calls. If you would like to forward the call to a VOIP extension, simply input the full extension number. Make sure there are no spaces before or after the extension number
The only inbound test I can make is from my Telstra pre-paid mobile
I can ring in from my ALDI mobile (Telstra network) OK.
We've dropped (are dropping) the routes locally and are now seeing some traffic routing in through the US and Singapore and through the scrubbers so we're expecting services to start coming back online.
Hi Arismac � you probably should set up call forwarding till this is all over.
Thank you so much. I now have call forwarding on both my incoming lines. I did not have a clue how to do that.
I can ring in from my ALDI mobile (Telstra network) OK.
I am in Mandurah in the west. Perhaps this changes area by area?
I now have call forwarding on both my incoming lines
Excellent! That's the good thing about VOIP � you have immediate control over all your own phone arrangements (well, except in the case of a DDoS attack!)
We've dropped (are dropping) the routes locally and are now seeing some traffic routing in through the US and Singapore and through the scrubbers so we're expecting services to start coming back online.
John,
Looks to me like you are trying to advertise /32 routes.
If my memory serves me correct I believe Telstra do not accept anything smaller than /24
if you are trying to advertise /32 routes, Telstra drops them and DOES NOT add them to the routing table.
It is internet best practice not advertise anything smaller than /24
edit Telstra is currently black holing your traffic (no route!)
Regards.
My Skymesh (satellite RSP) extension is working but not Telstra's ... yet.
Looks to me like you are trying to advertise /32 routes
We should only be advertising the /24 .. checking ..
Have a full working service on .net.au again
Is it safe to change back the forwarding?
Currently overseas and can't get Telecube to register through bria app on iPhone. Home phone in Australia is showing as registered on portal. Am using iphone via PIA VPN (Melbourne server) due to being in UAE. Has always worked previously. Could this be due its IP range?
I hope who ever is responsible gets caught out.
Currently overseas and can't get Telecube to register through bria app on iPhone. Home phone in Australia is showing as registered on portal. Am using iphone via PIA VPN (Melbourne server) due to being in UAE. Has always worked previously. Could this be due its IP range?
I just tried an extension using Zoiper.
Worked fine.
Closed Zoiper and brought up PIA via Sydney server.
Zoiper no longer registers.
:(
S
We are getting really bad call quality on the line... cant even hear the person on the other end. Might be lucky to hear 1 word out of 10.
Something still going on here?
We've got calls cutting out as well. It's been really bad in the last 10 minutes.
We've got calls cutting out as well. It's been really bad in the last 10 minutes.
Yeah, I just had that also. Was cutting in and out every few seconds. This was a customer who called in from a landline
We're working with Micron21 now to tune the scrubbers, they are discarding some valid traffic.
Shouldn't be too much longer.
Just getting hit right now with a new attack too .. in the 4 7gbps range
It amazes me who has the resources to push that kind of data at another resource without taking down other national or international links. There must be a forum or chat room where these people gang up and cause this chaos.
This could appear to be the issue currently then. Unfortunately I can't test here in the UAE as Voip won't work without a VPN.
touching 10gbps now
It amazes me who has the resources to push that kind of data
Nah, it doesn't require much resources.
There's all kinds and hacks that allow you to launch attacks with very little expense (who remembers the NTP amplification attack? https://blog.cloudflare.com/u
I also don't know how much zombied PCs go for but it's probably under $100 for a few thousand boxes for a few hours � probably a lot less for a lot more.
It does require a determined attacker however � who or why is anyone's guess, but maybe John knows (they'll usually make demands or can reasonably be speculated based on people you pissed off for some reason).
they're persistent little buggers .. still going .. peaked at 12gbps and around 7 now
the good news is service are uninterrupted
they'll usually make demands or can reasonably be speculated based on people you pissed off for some reason
we've had no demands .. seems like I've pissed someone off
who or why is anyone's guess,
They can come with extortion attempts.
See details of a similar attack on Australian email provider Fastmail... 20 Bitcoin were demanded.
https://blog.fastma
FWIW some fascinating reading on the 'Micron21' website re these DDOS attacks.
Some incredible figures quoted, that make it pretty mind boggling.
http://www.micron21.com/ddos.php
Some links go to other incidents of DDOS on the page...
(Hopefully no-one objects to the link being posted)
Thanks to John and all those involved for a mighty fine effort of stopping these ba$tards....
If only one could get revenge on them. (Legally of course)
One wonders what satisfaction they get from it.
Edit: Also thanks for the very informative email John.
There's all kinds and hacks that allow you to launch attacks with very little expense (who remembers the NTP amplification attack?
Not me but it was an interesting read, it's amazing how easy it is to exploit some things.
CALLER ID PLEASE
The attack finished a few minutes ago .. zero casualties :-)
Very please about that.
CALLER ID PLEASE
Ok .. yep, looking at it now.
Still can't make outgoing calls, can this be fixed?
Al � Pushie Enterprises
It would probably be best to close the thread until you get things straightened out. They probably get a lot of pleasure in reading the turmoil they have created. Use email for important announcements .
Lets hope they are not a customer, otherwise they will know all that we do anyway.
--Double Post--
Had an issue with another VoIP provider; it's a telephony service, therefore they have jurisdiction.
The way I see it, TC has had a massive disruption to service, which it can demonstrate to the TIO. The disruption is somewhat out of its control because of actions of others.
So, what is it the TIO can do? It is no different to a mass disruption on the Telstra network. All the respective parties need to do is demonstrate to the TIO what has happened and provide some evidence it is being attended to.
Some people seem to this there is some magic wand that can be waved and the issue is resolved. fact is there is no magic wand and those affected will need to either sit it out or move on.
I also remind everyone of section 2.2 of the terms and conditions they agree to when they signed up.
2.2 We do not warrant that we will be able to supply Services and we are not liable for any failure to provide all or part of any of the Services, but, to the extent and to the standard that Carriers provide Services to us, those Services will be provided by us to you. When your connection is disrupted, we will do our best to reinstate our Services to you as soon as we can.
Both iiNet and Telstra have had major outages this week. Shit happens. Everyday any operator is closer to their next failure. We just don't know when it will be. The more centralised these operations become, the bigger the impact there will be on the end users. Time to get used to the fact that things will fail and it will have an impact. Huffing and puffing about the fact isn't going to resolve the issue.
Most VoIP providers have a CSG waiver with their VoIP services. This doesn't seem to be explicit in the TC terms and conditions, which in hind sight should be.
IMO most of the current VoIP providers have a limited life, mainly due to the mobile industry, with which they will find it hard to compete with.
In the last 12 months or so, most mobile plans had included national calls including 13/1300 numbers. (Vaya for instance now offers unlimited national calls and 1.5GB of data for $20.00 and a promise of price matching for anyone who can beat them).
In recent times the market has now moved to providing international calls in some of the plans being offered. As more and more people take up these options, VSPs are going to find it pretty tough to remain in the market. Most seem to operate on low margins and high volume. Once the volume dries up, they will find it pretty hard to remain in business.
The woes of TC this week would have cost them dearly and still John is still seeing sustained attacks. Will they be able to ride out the storm and survive? (Obviously someone has it in for TC, and yet no one seems to know why).
John should be applauded for his efforts this week. It must get a bit demoralising when I think he has done everything that is humanly possible in the last few days to mitigate the issue and all some people can think of is the TIO. But that's the way some are, the let emotion get in the way of the cold hard facts.
The way I see it, TC has had a massive disruption to service, which it can demonstrate to the TIO. The disruption is somewhat out of its control because of actions of others.
Yes, not disputing that at all, the complaint would likely be thrown out given the circumstances � but the question was whether or not they had jurisdiction over VoIP providers.
Inbound 1300/1800 failing. They just ring out.
Please email your 1300 numbers to support I have a workaround
I'm a small business that relies on Telecube for incomining call services. My 1300 number just rings out instead of going to voicemail and trying to redirect to a mobile makes no difference. My two DID numbers to voicemail don't even ring but the weird thing is the Fax line DID is working (but who gets faxes these days:-)
Anyhow this is very frustrating. Is there no way to determine who is flooding the service? The traffic must be coming from somewhere. What can law-enforcement do? If you tried to stop people from going into a Wollies I'm sure the police would do something about that!
It would be nice to at least know how long it will take this to fix as our business will go out of business if people can't call us.
BTW � email a fault report an hour ago with our 1300 number......
I'm a small business that relies on Telecube for incomining call services. My 1300 number just rings out
Please email support I have a fix available to bring the 1300 numbers back up.
We've several thousands of 1300 numbers to work through and will be processing emailed requests first
Anyhow this is very frustrating. Is there no way to determine who is flooding the service?
Your 1300 number not working has nothing to do with the current DDoS attacks which have been occurring.
As John has said the DDoS attacks have been mitigated by Micron21 since Wednesday the other issues your experiencing is something else.
John has said the DDoS attacks have been mitigated by Micron21 since Wednesday the other issues your experiencing is something else.
Yes the optus router failure last night. No doubt because things got shuffled around due to the attack, things were not back to normal. Then the optus router failed and took out services in a different way. Because of all the recent changes things were not quite right / redundant yet. You can't re engineer a fool proof solution in a few days. Has just been an unfortunate series of events.
John should be applauded for his efforts this week.
I agree.
It's pretty obvious that John and his staff have spent any money necessary and spared no effort to regain control of his network during this interruption.
For the most part voicemail has worked so those precious calls have been answered and the voicemail delivered.
I've lost incoming and outgoing services again. And a 603 declined response
Dear Random J. Hacker,
you know if you asked John nicely he might be inclined to set up a test bed where you two can play cat and mouse together at a more leisurely pace and a cheaper cost to you both. Yes, you probably want to try your at scale attacks on the production system too, but how about doing that at a regularly scheduled time? What I'm saying is you can have your fun while minimising the casualties to innocent civilians.
Obviously, I don't know your motivations but I hope you realise you are doing harm to one of the good guys of the industry and the little people. Who do you use for your paid ;-) calls? Now, I plan to stick with Telecube unless you can prove that other VSPs are intrinsically more robust against DDOS attacks.
BTW, thank you for helping Telecube improve their defences against DDoS attacks, it was not something I considered when choosing a new VSP. Care to recommend anyone? ;-)
Please email your 1300 numbers to support I have a workaround
Already sent 2 emails.
Incoming IP-auth DIDs failing again.
I emailed a list of the main ones to the support desk at about 12.30pm. They did come back online during the afternoon but no joy now. Outbound IP-auth not working either but inbound is the priority.
Incoming calls working with linehunt but not with queue. Outbound calls OK (NSW).
Hmmm...something different this time around.
Now my inbound IP trunk appears to have issues.
Showing as registered in 3CX but nothing is happening when people try to phone in. No ring tone, nothing, zilch, zero.
It doesn't appear in 3CX as an incoming call.
I have 2 IP trunks and neither is working.
I do have another DiD which is call forwarded to an extension and that works.
So the issues are with my trunks... bother!
Hmmm...something different this time around.
This time is a hardware failure which has caused an outage on our Optus link which is a private network but bound to a specific device .. which failed last night. We've been trying to restore the service with Optus but it requires a work order and will take days or weeks so we are re-routing numbers now.
Only inbound services are affected and not everyones.
We have been getting DDoS attacks all day but voip extensions haven't been affected.
We have been getting DDoS attacks all day but voip extensions haven't been affected.
Telecube a local VSP / SIP provider with DDoS protection that actually works.
John this should be a selling point and a another reason NOT to use Pennytel, Faktortel or Mynetphone who i doubt have a service as good as you.
The issues that make you stronger will make you better than the competition.
So the 1300 numbers that are down are hard down for a few days?
Không có nhận xét nào:
Đăng nhận xét