Site Issues - 4/5 - 4/13

Grand Poo Bah said:

@WesL you gotta look into a different hosting service, brotha. The site hasn't been working properly for weeks, and it goes down almost everyday. It's super sluggish again today, and for the 4th time in the last 30 days image uploads are broken. Really don't want to sound entitled or ungrateful, but you've got a very active and growing community here, and these problems are significantly handicapping it.

Click to expand...

I can not second this hardly enough. The site is extremely slow and is working at least half of the day slow. I don't mean this harsh and it isn't your fault but I feel like this forum has been growing a lot the past year and maybe it's time to utilise a stronger hosting service.

Wednesday and Friday mustn't be ignored, there's decent potential for both days.

Thanks to you both. TalkWeather is a special site in that one day it needs very few resources and a the next it need all the resources. Our current ISP is HostSDG which is owned by me. We have two datacenters that we work out of, one in NW Arkansas and the other in Kansas City. When we first migrated TW out of AWS about 2 years ago, we never did any changes to the file sizes of the machines. That is biting us in the rear at the moment because when we have a failure and the system goes to the standby configuration it takes entirely too long to move the site and databases. Giving us prolonged outages. What's worse is if it is something that fixes 2 minutes after the fail over occured, we can't stop the transfer and go back to the orginal configuration. Additionally the primary ISP at the NW Arkansas has become increasingly unstable with little work being done by the ISP. Doesn't help that they are being acquired by is worse. Not to name names... but it isn't too hard to figure out.

So as of last night we moved the site and database to the Kansas City location, which yes actually costs a bit more, but we multiple connections to Tier 1 ISPs that we switch to automatically. The hardware there isn't my favorite but we have on-site HA so if something does happen it should move nonetheless.

While we have everything parked at KC, we are working making our secondary ISP the primary at NW Arkansas and also will do a few off-line practice runs and adjusting the size of the servers so that like the 200+ other sites we host, they transfer in seconds not minutes.

As for the thumbnails, the developer who created that plugin has ghosted it after 5 years. I've got one of my devs looking at it to see if we can work a solution that would be a long term fix. We will use the offline environment to test but I don't see that as a fast turnaorund.

Update on attachments. We are working a fix in our dev environment. If it works we will take a 1 hour downtime this evening when we don’t see any active conversation.

Hey WesL,

Any update as to when we would be able to post graphics again?

Thank you

Over the next two days we will be taking early morning outages to restore attachments. We will be downloading and then uploading our S3 buckets from AWS and moving them to Cloudflare storage. Naturally, we've had a lot of attachments over the years so this does take a bit and there needs to be some config changes as well. This will also complete our move away from AWS as well.

@WesL

cniemczura said:

This might not be the appropriate place, but probably has the most visibility - if the owner of the site wants any help with technical issues going on, I'm a full-time software consultant and have been running websites on my own for decades. Would love to help any way I can.

Click to expand...

WesL said:

Over the next two days we will be taking early morning outages to restore attachments. We will be downloading and then uploading our S3 buckets from AWS and moving them to Cloudflare storage. Naturally, we've had a lot of attachments over the years so this does take a bit and there needs to be some config changes as well. This will also complete our move away from AWS as well.

Click to expand...

Excellant! Thank you

This is an incredibly minor gripe, and maybe I shouldn't be posting it to this thread, but there is a mildly obnoxious button next to the alerts button in the top-right with no logo on it that "turns on" the light (aka going back to original TalkWeather white background) but it isn't there when you turn on the light, so you need to go to preferences to change it back. I find myself pressing it accidentally a lot, and then have to go to my preferences to change it back.

Again, incredibly minor and small thing, but I can't help but get annoyed when it happens, lol.

slenker said:

This is an incredibly minor gripe, and maybe I shouldn't be posting it to this thread, but there is a mildly obnoxious button next to the alerts button in the top-right with no logo on it that "turns on" the light (aka going back to original TalkWeather white background) but it isn't there when you turn on the light, so you need to go to preferences to change it back. I find myself pressing it accidentally a lot, and then have to go to my preferences to change it back.

Again, incredibly minor and small thing, but I can't help but get annoyed when it happens, lol.

Click to expand...

I've noticed this for a while, and frequently flashbang myself with the light theme when I go to look at notifications. I'm guessing it's a leftover web element - it's just large enough to click on accident. It's not a big deal of course, but I've definitely had my fair share of wincing at getting blasted with the power of the sun

Any updates @WesL ?

With respect to @WesL @JayF and all other TalkWeather staff...

I don't think I'm alone in my opinion that the image and attachment upload feature needs to be fixed ASAP. I feel I've been very patient on the matter (I don't think I've mentioned it even once since Wes' last post on the issue over two weeks ago)

As a sustaining member of the forum, I want to feel like I'm getting my money's worth helping to keep the site running - though it's certainly been a rocky road recently from the database outage in February to all the issues last month, and I'm starting to feel like 'running' is a bit of an overstatement with how buggy the forum has been...

I say this because this site means a lot to me. That's why I'm a sustaining member, and will continue to be, but may I respectfully ask: is there at least an estimate to when the attachment feature will be functional again?

Thanks,
TH2002

Good morning everyone,

Thanks to the help from a few other admins at sites that were running the same attachments plugin, we have perfected the process to migrate our current attachments and allow for new attachments to be uploaded. I just kicked off a migration script for our 52k attachments. So far, it is looking good, and I'll let it run overnight. Once we confirm the hashes, we will use a small outage tomorrow night to update the configuration.

I'll post an update on Wednesday to let you know how it is progressing.

Thanks for your patience.

Wes

TH2002 said:

With respect to @WesL @JayF and all other TalkWeather staff...

I don't think I'm alone in my opinion that the image and attachment upload feature needs to be fixed ASAP. I feel I've been very patient on the matter (I don't think I've mentioned it even once since Wes' last post on the issue over two weeks ago)

As a sustaining member of the forum, I want to feel like I'm getting my money's worth helping to keep the site running - though it's certainly been a rocky road recently from the database outage in February to all the issues last month, and I'm starting to feel like 'running' is a bit of an overstatement with how buggy the forum has been...

I say this because this site means a lot to me. That's why I'm a sustaining member, and will continue to be, but may I respectfully ask: is there at least an estimate to when the attachment feature will be functional again?

Thanks,
TH2002

Click to expand...

I appreciate your patience, but to be clear, the last few weeks haven’t been a period of 'waiting', they’ve been a period of deep-dive engineering. We’re currently resolving a mess caused by a paid developer who ghosted hundreds of sites after a decade of service.

While our tested backout plan was sidelined by recent core updates, we’ve been busy in a non-live environment reverse-engineering a custom fix from the ground up. We’ve not only secured our own platform, but because we actually have the resources to do this right, in part to you, we’re releasing the fix to other struggling forums for free.

So, while the road has been 'rocky,' we’ve been the ones out there paving it. We're happy to get everyone back to where they should be, and we're doing it on our own terms.

WesL said:

Good morning everyone,

Thanks to the help from a few other admins at sites that were running the same attachments plugin, we have perfected the process to migrate our current attachments and allow for new attachments to be uploaded. I just kicked off a migration script for our 52k attachments. So far, it is looking good, and I'll let it run overnight. Once we confirm the hashes, we will use a small outage tomorrow night to update the configuration.

I'll post an update on Wednesday to let you know how it is progressing.

Thanks for your patience.

Wes

Click to expand...

there's a 10% hatched for tornadoes tonight, so maybe wait until tomorrow night for the outage?

Grand Poo Bah said:

there's a 10% hatched for tornadoes tonight, so maybe wait until tomorrow night for the outage?

Click to expand...

Agreed entirely.

WesL said:

I appreciate your patience, but to be clear, the last few weeks haven’t been a period of 'waiting', they’ve been a period of deep-dive engineering. We’re currently resolving a mess caused by a paid developer who ghosted hundreds of sites after a decade of service.

While our tested backout plan was sidelined by recent core updates, we’ve been busy in a non-live environment reverse-engineering a custom fix from the ground up. We’ve not only secured our own platform, but because we actually have the resources to do this right, in part to you, we’re releasing the fix to other struggling forums for free.

So, while the road has been 'rocky,' we’ve been the ones out there paving it. We're happy to get everyone back to where they should be, and we're doing it on our own terms.

Click to expand...

Thanks for the detailed explanation. I appreciate the transparency.

I noted I may have come off as a bit rude in my open letter. That wasn't my intention. Sorry if it came off that way.

And now you deserve a bit of transparency... Started writing this on my home computer, got distracted, went to work, and am now completing this message on my flip phone.

TH2002 said:

Thanks for the detailed explanation. I appreciate the transparency.

I noted I may have come off as a bit rude in my open letter. That wasn't my intention. Sorry if it came off that way.

And now you deserve a bit of transparency... Started writing this on my home computer, got distracted, went to work, and am now completing this message on my flip phone.

Click to expand...

No harm no foul... Flip... Phone.....

Why am I seeing ads for Starlink as a sustaining member, and why is the ad blocking the message box to type this message? I think these words are being considered a link too. This is extremely unusual and it took me a while to even write this message in the first place - I had to navigate around this dumb gigantic starlink ad.

slenker said:

Why am I seeing ads for Starlink as a sustaining member, and why is the ad blocking the message box to type this message? I think these words are being considered a link too. This is extremely unusual and it took me a while to even write this message in the first place - I had to navigate around this dumb gigantic starlink ad.

Click to expand...

Just another example proving that the internet as a whole is literally unusable without an ad blocker, and why I have no shame in using one.

While being a Sustaining Member is supposed to give an ad-free experience, the occasional intrusive advertisement still tries to slip through. Thank goodness for uBlock Origin!