hate these ads?, log in or register to hide them
Page 1 of 2 12 LastLast
Results 1 to 20 of 29

Thread: downtime 09/25/15 to 09/26/15 (update #2)

  1. #1
    Administrator EntroX's Avatar
    Join Date
    April 9, 2011
    Location
    Miami, FL
    Posts
    10,982

    downtime 09/25/15 to 09/26/15 (update #2)

    yesterday we experienced a downtime and today as well, initially i thought it was caused by some tweaks i did to the database but the second one was far more severe and it didnt resolve itself before i figured out what it was

    the problem was caused by our provider (digital ocean) and i had to go trough quite an ordeal to get them to even look at the issue, i found a workaround by myself and frankly the delay i went trough was far beyond what i can consider acceptable

    i will be evaluating the possibility of switching providers because this is probably the biggest pile of horseshit i've gone trough for a while

    for your amusement, here's the conversation i had with them, im gonna go to bed now

    http://p.desu.es/34weyrsh5.png

    oh, also this

    https://twitter.com/EntroX/status/647719119048056832

    EDIT01: in case someone is actually curious on how this impacted us, its pretty simple, since i use round robin for load balancing (1 load balancer distributes the load between 3 web servers), when the servers tried to return the traffic to the balancer (which has to play around with DNS because if it didn't then sessions wouldn't carry over between the different web servers if you log in after 6 hours and you end up on a different web server, there are more sophisticated ways of handling this but i honestly didn't expect this to ever be an issue) it would end up timing out because of lacking DNS resolution which is why at the start it dished out gateway errors and eventually database errors because of the vast amount of open sessions still trying to resolve.

    this made the database slow to a crawl and die, not that it mattered since traffic wasn't going out anyways. DO somewhat forces what DNS servers you use so i had to do some tweaks in order to stop them from doing that and then i got the site back up, there are still however a lot of issues since there are a lot of things under the hood that use google related services so you can expect some things to be sorta wonky until DO stops being shit and fixes it or i get angry enough to switch back to linode or something.

    EDIT02: and the second downtime was caused by leftover problems from the DNS resolution issue, i have a few things under the hood running that were still having problems due to the original issue and caused two of the nodes to pretty much hang up at 100% cpu usage about 4 hours after i went to bed, i just spent the last 2 hours or so modifying said jobs and re-creating all 3 webservers and re-adding them to the load balancer so hopefully things will be running a bit smoother now, sorry for the downtime
    Last edited by EntroX; September 26 2015 at 10:33:33 PM.
    need me?, contact me on IRC or via PM

  2. #2
    GiDiYi's Avatar
    Join Date
    April 11, 2011
    Posts
    3,348
    You can't stop the signal!

  3. #3
    XenosisReaper
    Guest
    hooray!

  4. #4
    Cosmin's Avatar
    Join Date
    March 14, 2012
    Location
    UK
    Posts
    5,163
    It kinda still feels "dodgy" - pages loading a bit slow and stuff.
    Guns make the news, science doesn't.
    Six shooters ruined PvP.
    What are you doing with your life?ęDoomchinchilla 2015

  5. #5

    Join Date
    April 14, 2011
    Posts
    5,516
    Nawwh, it's called haffrage.

  6. #6
    Donor Sparq's Avatar
    Join Date
    April 11, 2011
    Location
    Strayastan
    Posts
    8,769
    Quote Originally Posted by EntroX View Post
    in case someone is actually curious on how this impacted us,
    NERD!

    (Thank you for continuing your service as our benevolent overlord)

  7. #7
    Super Baderator DonorGlobal Moderator cullnean's Avatar
    Join Date
    April 10, 2011
    Location
    The tower of power, too sweet to be sour, ohhhh yeahh!
    Posts
    16,054
    Farking hell
    Quote Originally Posted by Elriche Oshego View Post
    Cullneshi the god of shitposting.

  8. #8
    Steph's Avatar
    Join Date
    April 10, 2011
    Location
    Canadia
    Posts
    9,249
    HOLY SHIT IT'S AN ENTROX POAST
    Quote Originally Posted by QuackBot View Post
    Canadians are usually cooler.
    Thermonuclear Banana Split - A not-really-weekly Eclipse Phase campaign journal

  9. #9
    Daneel Trevize's Avatar
    Join Date
    April 10, 2011
    Location
    T L A
    Posts
    12,022
    This is a cool yoyo you have m8.
    Quote Originally Posted by QuackBot View Post
    Idk about that, and i'm fucking stupid.

  10. #10
    Daneel Trevize's Avatar
    Join Date
    April 10, 2011
    Location
    T L A
    Posts
    12,022
    The following, older-yet-future post appears below this one of mine (posted at ~September 26 2015, 09:40:38 PM):
    September 27 2015, 01:26:56 AM
    Quote Originally Posted by GiDiYi View Post
    I was getting the shakes!
    Post order/timezone usage is borked?!

    Additionaly the first edit to this post didn't show immediately, even on full page refresh, but the data's there in the edit... and appears after a 2nd refresh.
    Last edited by Daneel Trevize; September 26 2015 at 09:47:15 PM.
    Quote Originally Posted by QuackBot View Post
    Idk about that, and i'm fucking stupid.

  11. #11
    Donor Verite Rendition's Avatar
    Join Date
    April 9, 2011
    Posts
    3,907
    Well that's quite the incident.

    Thanks as always for working on the site for us, Entrox.
    Quote Originally Posted by Daneel Trevize View Post
    Post order/timezone usage is borked?!
    It would seem so.
    Tranquility (EVE) Influence Map
    Life is absurd, but with a little effort we can make it completely ridiculous

  12. #12
    Steph's Avatar
    Join Date
    April 10, 2011
    Location
    Canadia
    Posts
    9,249
    TIME SLIPPAGE



    Posting this at 8:50 UTC 9/26, let's see where we end up!
    Quote Originally Posted by QuackBot View Post
    Canadians are usually cooler.
    Thermonuclear Banana Split - A not-really-weekly Eclipse Phase campaign journal

  13. #13
    Administrator EntroX's Avatar
    Join Date
    April 9, 2011
    Location
    Miami, FL
    Posts
    10,982
    updated the OP
    need me?, contact me on IRC or via PM

  14. #14
    GiDiYi's Avatar
    Join Date
    April 11, 2011
    Posts
    3,348
    Dear diary,

    today I mastered time travel.

    Great success!

  15. #15
    Movember 2012 Stoffl's Avatar
    Join Date
    April 10, 2011
    Location
    The original viennese waffle
    Posts
    20,736
    Quote Originally Posted by GiDiYi View Post
    You can't stop the signal!
    2/10/17 Greatposthellpurge never forget
    23/10/17 The Greatreposteninging ?

  16. #16
    Quote Originally Posted by GiDiYi View Post
    Dear diary,

    today I mastered time travel.

    Great success!
    Instead of using the tesseract to prevent the downtime, you used it to increase your post count. Truly a posting legend.

  17. #17

    Join Date
    April 11, 2011
    Posts
    450
    Thank you for the description of the problem. Was fun to read

  18. #18
    Donor Nu11u5's Avatar
    Join Date
    June 8, 2011
    Location
    Houston, TX
    Posts
    5,405
    Where do we send the beer?

  19. #19
    HEY LOOK AT ME I HAVE A TITAN LordsServant's Avatar
    Join Date
    May 3, 2011
    Location
    [VENIO]
    Posts
    4,639
    Quote Originally Posted by Nu11u5 View Post
    Where do we send the beer?
    Somewhere in Florida last I checked.

    <3 ya Entrox. I saw you post on the mod forum about spasm being awful, then 24h later we had a DT. Was worried you'd pulled the plug on us.

    I am a Lamborghini tractor.

  20. #20
    Super Moderator Global Moderator QuackBot's Avatar
    Join Date
    March 7, 2012
    Posts
    20,878
    Quote Originally Posted by Sparq View Post
    NERD!

    (Thank you for continuing your service as our benevolent overlord)
    I, for one, welcome our digital overlord.

Bookmarks

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •