Cloudflare (Please Read)

[Slim]

A bit of site news - you may recall that guest access was disabled in January in an effort to dissuade bots and spiders from consuming site bandwidth.

This hasn't really worked, as the following log analysis captured just now shows:

Highslide Image Viewer

So I'm going to set up a service called Cloudflare. This sits between the user's browser and the site, to determine whether visitors are human or not.

In practice what this means is that to visit the site, you'll get a "verify you are human" tickbox. I hope this will prove only to be a minor inconvenience. Once you've ticked the box, you should get at least half an hour of uninterrupted browsing before the front-end tick box challenges you again.

One advantage of this is that it should allow the site to be configured to allow (human) guest access again, so that non-member users can browse threads.

I hope to do this some time next week, need to sketch out an implementation plan first. It will involve minimal downtime (no more than half an hour I hope) and I'll give advance warning of this in this section.

To get an idea of how the front end tickbox works, please visit this site: https://forum.tz-uk.com/

It's a forum for wristwatch enthusiasts, I have no association with it except that I post there occasionally.

I regret that this is going to be a slight irritation but I'm not tolerating thousands of hits chewing bandwidth and CPU cycles every day with zero value to the site. On the upside: there might be a bit of a performance boost for end users, because Cloudflare caches static content and optimises it - so you might see pages load a bit more quickly.

Any comments or questions welcome.

[Slim]

OK this is pretty simple to do as it turns out, so it's happening in the next 20 mins.

I'm hoping this will be intuitive enough for anyone who didn't see the above notice. Fingers crossed!

[Slim]

OK I've made the change. There should be no downtime. It hinges on a DNS change as below:


Highslide Image Viewer

So: the site will still work, but the change may not take effect for up to 24 hours (more like 4 or 5 probably).

[Slim]

So: this broke the site for a few hours. Apologies if you tried to connect, and couldn't. The error occurred because Cloudflare temporarily couldn't connect to the server due to an SSL misconfiguration which I've resolved now. It's been like being at work again! If I was still at Morgan Stanley I'd have lost some of my bonus.

Cloudflare should now be protecting the site from bots.

You may find (I did) that you don't get the tickbox challenge - possibly based on IP reputation (it likes your ISP). But I've put rules in place to defeat the usual bots and I believe these should be working. I'll have a look at the log analysis later.

I've enabled guest access again, so the site can be browsed by users who aren't logged in.

[Thenop]

On my phone the header is now gone. And no tickbox login yet.

Will check once I am back home in a normal sized browser.

[Slim]

You mean the site banner? That's likely because Cloudflare is a bit more fussy about header elements being from a different site (which the site banner is) - but I can fix that quickly, thanks for the feedback

[Thenop]

back to normal now
Thanks

[Slim]

Unfortunately I had to switch Cloudflare off. It looks like it doesn't play nicely with sites (like this one) that aren't set up with HTTPS.

So: later on today or possibly tomorrow, I will snapshot the site, then set it up to use a free SSL certificate (for https:// access). Then I'll try switching Cloudflare on again.

I'll keep this thread updated.

[Slim]

Right - I've set up an SSL certificate on the site. That went very smoothly. You can still visit the site using the http:// links if you prefer but https:// URLs will work as well now.

Now that that's done - I'm going to switch Cloudflare on again, to resist the bots and crawlers. I'll do this at 4:15pm UK time.

There's a good chance that it will make the site inaccessible or unstable, so if it does I'll switch it off again after a couple of hours maximum, during which time I'll be attempting to fix it. I'll test from three different browsers and a phone.

Apologies in advance for any inconvenience or irritation.

[Slim]

Cloudflare is back on again and it looks like it hasn't kicked the site over this time. I'll keep an eye on it but at the moment it looks good. I haven't been subjected to the tickbox challenge so far but I can see that bots are being blocked from the Cloudflare event logs. Good times.

Guest access to the site is switched on now by the way.

[Thenop]

No issues here, thanks!

[pxr5]

Working fine here to on https 

[Slim]

I have seen a few glitches at this end, with my DNS flipping back to the old IP addresses (instead of the Cloudflare IPs intended to intercept browser traffic):

glower ~> dig +short betweenthewheels.net
172.67.182.235
104.21.59.191
glower ~> dig +short betweenthewheels.net
52.1.80.75
glower ~>

I believe this is due to stale data in my router's DNS cache, or possibly the desktop OS.

If you get an "invalid certificate" error or the site doesn't respond over the next few hours, this is likely to be the cause for you as well. Unfortunately it's just a case of sitting it out; the stale data will expire eventually. Nothing I can do at the server end. You could reboot your router or PC if you're particularly bothered but I'd just wait it out.

[Slim]

This is what the problem is for me:

glower ~> dig +short betweenthewheels.net @81.139.57.100
104.21.59.191
172.67.182.235
glower ~> dig +short betweenthewheels.net @81.139.56.100
52.1.80.75
glower ~>

Those are two BT nameservers, used by my router. One of them has the correct Cloudflare IPs, the other has the old (now incorrect) direct IP.

So when my PC requests a DNS resolution for betweenthewheels.net, sometimes it gets the answer from the nameserver that has the updated info, sometimes it retrieves the stale info from the other one.

It's just a question of sitting it out. If you're seeing similar problems, I recommend the same approach: a provisional interim strategy of calculated inaction. It often served me well in my career.

[The Picnic Wasp]

I had a bit of a busy week with a lot of distractions and missed all of your hard work reports, James. I wondered a couple of times why I was able to access the threads when I had timed out. Thanks for your efforts.