Why did HA go offline?

[HappyGuy]

This time it was because our server space ran out of room. I'm not sure what's taking up so much space in the server. There should be 20 percent of the server space free. Apparently something is taking up that space now. I can't find out what it is. I looked in the file manager and found nothing suspicious. Odd.

Besides investing in more drives why don't you investigate who is lurking on your site? Bots will take down Happier Abroad if you don't do anything. And is there a way to allow Google and other search engine bots to access this site only every few days instead of 24 hours a day?

In total there are 106 users online :: 6 registered, 0 hidden and 100 guests (based on users active over the past 5 minutes)
Most users ever online was 683 on Tue Sep 03, 2019 12:06 am

[Yohan]

You should always have at least 20 percent free disc space. - Unfortunately the forum was again offline for a while - happens now frequently.
Something has to be done about that, during the last weeks the forum was more offline than online...

[Winston]

You should always have at least 20 percent free disc space. - Unfortunately the forum was again offline for a while - happens now frequently.
Something has to be done about that, during the last weeks the forum was more offline than online...

I already fixed that. I found an extra backup on my server and also saw that the server log files were taking up a lot of space too. So I deleted them. There is about 30 percent free space in the server now, which is plenty. So this downtime shouldn't happen anymore. Sorry for being away for a while.

[Winston]

This time it was because our server space ran out of room. I'm not sure what's taking up so much space in the server. There should be 20 percent of the server space free. Apparently something is taking up that space now. I can't find out what it is. I looked in the file manager and found nothing suspicious. Odd.

Besides investing in more drives why don't you investigate who is lurking on your site? Bots will take down Happier Abroad if you don't do anything. And is there a way to allow Google and other search engine bots to access this site only every few days instead of 24 hours a day?

In total there are 106 users online :: 6 registered, 0 hidden and 100 guests (based on users active over the past 5 minutes)
Most users ever online was 683 on Tue Sep 03, 2019 12:06 am

How would I find out who is lurking? Google Analytics will tell you where they are from and what browser they are using. But not who they are. Besides, a hacker will usually use a proxy to hide their identity. A bot can't take down a site unless there are thousands of them. Yes I think you can limit Google bot, but why would you want to do that? It's noninvasive. There are about 20 legit bots from search engines. They don't crash the forum or server memory unless they number over a thousand.

[Winston]

Announcement:

The forum was offline a couple of hours yesterday. But this time it was deliberate, because I was upgrading the forum from 3.2.2 to 3.2.8. It was a pain in the butt and stressful and resulted in unexplained errors, which took a while to fix and get help on. Man computers are such a pain in the butt. Why does everything always have to updated? Why can't they just leave it at a fixed stable version and not update everything all the time? Sheesh.

Anyway, it looks like the update is finally successful. Let me know if there are any bugs or issues with the new upgrade. See the announcement here:

http://www.happierabroad.com/forum/view ... 32#p332032

[Winston]

Announcement:

The forum is now back online. It was down because the database crashed and was too corrupted to fix using the cpanel. So I had to create a ticket for my webhost upper level admins to fix it. It seems to work now. So try posting again.

Thanks,
Admin

[Yohan]

@Winston
Test to see if the forum is OK and ready for posting.
It seems to work, nothing missing. Thank you, Yohan

[Winston]

ATTENTION: It looks like all the posts made after July 7 are gone, because the forum was restored from the latest backup from that day. So please repost any post you made on July 7 or afterward. Thanks.

[Winston]

Email I received from Bluehost about restoring the forum:

Hello Winston,

Thank you for your reply. My name is Joshua and I am a Web Administrator with Bluehost. I am following up on this case regarding repairing your crashed table happier2_phpbbbackup/phpbb_posts.

Typically a crashed database table can be easily repaired, however, in this instance the repair is failing because phpbb_posts.MYD is missing from inside /var/lib/mysql/happier2_phpbbbackup. There may be non-traditional ways to repair the database from this state, however, this is not supported by Bluehost We would recommend you reach out to your server administrator or a MySQL developer for further assistance.

At this time the only supported resolution for this would be to restore your database from a backup. Since we do not have a valid backup for this database you would need to restore it from a local backup (one that is not generated by cPanel backups). I did check /home/happier2/public_html/forum/store and located a database backup from 7/7 backup_1657195374_zlbsrfly0orpog5l.sql.gz. I was able to create a new database happier2_phpbbbackup_restore and import the backup. It looks to be complete and imported successfully, however, since we did not create the backup we cannot guarantee the content of the backup.

After I imported the database backup to the new database I did confirm it generated the missing phpbb_posts.MYD file.

root@server.happierabroad.com [store]# ls -lah /var/lib/mysql/happier2_phpbbbackup_restore/phpbb_posts.MYD
-rw-rw---- 1 mysql mysql 434M Jul 12 15:59 /var/lib/mysql/happier2_phpbbbackup_restore/phpbb_posts.MYD

After importing the backup I also went ahead and updated your site configuration file located in forum/config.php with the new database information (I made a backup of the file prior to updating the database information). I did check the site and it looks to be loading on my end. For instance: viewtopic.php?style=11&p=371305#p371305. Can you please confirm your site is now loading on your end?

As for downgrading your MySQL version, we would not be able to confirm that this would be able to repair the original database. Since phpbb_posts.MYD is missing I highly doubt it would. cPanel does not support the downgrading of your MySQL version, however, it is possible to do. Essentially you would back up your databases, remove the current install of MySQL and then install what version you would like. Since it is not supported by cPanel we also would not be able to offer assistance with this. More details can be found here: https://bit.ly/3uL7fB5

Regarding backups, this is a VPS server and is considered self-managed. Backups are taken based on your backup configuration within WHM > Backup Configuration. Any of your backups are stored on the server unless you have them configured to be stored externally. Right now it looks like backups are enabled, however, you have them disabled for your cPanel user. If you would like server backups to run then you would need to enable them for your cPanel user within WHM > Backup User Selection. You would want to make sure you have enough room for these backups to run.

Lastly, it looks like your server is very limited on storage and we would recommend you reduce your usage as if it reaches 100% usage services may go offline. You should be able to log in to your account and manage your content. You can click this link below for a full breakdown of your account:

https://privatebin.support-tools.com/?a ... Yqk9jjgUg=

Please let us know if you have any further questions by replying to this email.

Thank you,
Joshua C

[Winston]

If anyone wants to know why this forum keeps crashing, see the exchange below I had with Bluehost higher admin about it.

Dear phpbb support,
My forum keeps crashing every week or two, because the TMP file in MYSQL keeps getting full and maxing out. When that happens I get this error message on the forum:

General Error
SQL ERROR [ mysqli ]

Disk full (/tmp/#sql_1d9a_1.MAI); waiting for someone to free some space... (errno: 28 "No space left on device") [1021]

An SQL error occurred while fetching this page. Please contact the Board Administrator if this problem persists.

My webhost said that I can't clear out the TMP file and that only their higher admins can. What is this? I created a ticket for this issue and today my webhost said that the issue is connected to the phpbb_sessions table. See their email to me below. What can I do about this?

Thank you for contacting support. My name is Joshua and I am a Web Administrator with Bluehost. I am following up on this case regarding /tmp being 100% full. Looking into this further it appears that /tmp is filling up due to checks done on your MySQL database happier2_phpbbbackup_restore. Currently, it looks like happier2_phpbbbackup_restore is 3.3G and your /tmp directory is only 1.1G. It does appear that the largest table is phpbb_sessions. Moving forward you can either:

1) Work with your site developer in order to reduce the size of the phpbb_sessions table

2) Have us increase the size of /tmp. Currently, /tmp is 1.1G. You have 2.1G available on your main drive. If you were looking to have us increase the size of /tmp you may want to reduce the current usage of your account first. Here is a breakdown of the account:

https://privatebin.support-tools.com/?a ... iWCB9PjAMq

If you would like us to increase the size of /tmp, please let us know how large you would like to increase it to. If you would like us to remove any content prior, please reply to this email with "I GIVE BLUEHOST PERMISSION TO REMOVE <insert FULL file paths here>". Please note that content removal is irrecoverable.

If you have any further questions, please do not hesitate to reply to this email.

My reply to them:

So you mean the phpbb sessions table is bloating up and causing the forum to crash? What if I just delete that table? That table is expendable and doesn't contain any posts or data and can be recreated automatically by the forum. Why would a sessions table become 1 GB? It's just a temporary table. Could spam bots be filling it up?

Yes you can just delete the TMP folder on my SQL server. You did that 2 or 3 times before right? Check my records. Removing it doesn't erase anything important. It's just a cache folder right? Why didn't you remove it yet? Last time your customer service staff just removed it without any further questions. It looks like your security protocols are more strict than before?

I have little space left on my server. Increasing the size of the TMP file allowance would only be a temporary solution until it fills up again. It's better to prevent it from filling up. Would enabling cloudflare help?

Their next reply to me:

My name is Elissa S. of escalated support,

Generally we should ask before removing client data. Since the sessions table is generated by your site we should be asking before removing the files. For files in /tmp we're generally more willing to go ahead wit those removals without permission given those files are often automatically generated by system services. For example I see we removed the /tmp/wpinstaller zip files as these were generated by wpinstaller and the failure of wpinstaller to manage these files in the past has been an issue. Since those do not contain sites or unique user generated data there's little harm in removing them.

On the sessions table: In my understand removing that table would logout all users on your site. They can just log back in. Generally we do not delete or drop tables automatically because we are not your system administrators nor do we specialized in phpbb. In most cases it would be fine but since we did not build, design or maintain your site I don't know if you've modified what's kept that or if this data was purposeful retained by you for a specific reason.

I do understand that the phpbb administrator tools do have a clear sessions option.

Why is it large? Probably because there are either a large amount of logins or login attempts or your site isn't cleaning up the sessions after they expire or they're set to not expire. Spambots could be at fault. That said we are not database administrators and I would suggest consulting with a phpbb administrator for better insight into this issue Using cloudflare is generally a good idea as their cdn usually filters out known problem bots and attackers that have hit other sites automatically.

As to dropping the sessions table, you can do so in phpmyadmin or via the mysql command line. There might be an option in phpbb to clear the sessions table too, if that's available I would suggest that as it would do it in the way the software expects. If you want me to do it I would login as root, and use the mysql command line to execute the following

USE happier2_phpbbbackup_restore;
TRUNCATE TABLE phpbb_sessions;


Again this would just completely wipe the table and I'm doing this on the inference that the following posts contain accurate information. Again we do not have phpbb administration experience here and I would strongly recommend retaining someone with the appropriate experience or having a working backup that you can restore.

https://www.phpbb.com/community/viewtopic.php?t=1314325
https://www.phpbb.com/community/viewtopic.php?t=1638695
https://www.phpbb.com/community/viewtopic.php?t=2375091

I hope this information helps, but if you require further assistance or have any additional questions, then please do not hesitate to reply to this email so that we may continue to help as needed.

Any advice? What does it mean to truncate the phpbb_sesssions table? How do I do that? Would that help? Is that a permanent fix?

Also, what's the connection between the TMP file in MYSQL and the phpbb_sessions table? I searched this forum's archives and there's no info about that that I could find.

Thanks.

[Winston]

Perhaps we should enable Cloudflare again on this site? It may slow down the forum a little, but it provides protection from spam bots and DDOS attacks that may overload the forum server and cause it to crash?

[Winston]

Update: Apparently the sessions table kept crashing the forum and overloading the SQL server here because I had the systems cron set to yes in the admin panel. It was supposed to be set to no so that the session table would be purged automatically by the forum. See the support thread that I started on the phpbb support forum here if you want more details.

https://www.phpbb.com/community/viewtopic.php?t=2657932

I already turned it off, so hopefully it doesn't happen again. I may also turn on Cloudflare soon too to block the spam bots. Maybe it will prevent the Mercer trolls from coming here using a proxy too.

[Voyager1]

Winston, why is it when you fix it, it's fast but then it starts going slow again? Why is it that you have to contact Bluehost to fix the problem all the time and open a "ticket"? Isn't there a control panel or something where you can fix the problem yourself? What is it that you're paying for?

Isn't there a better host? Surely some of the guys in IT on the forum would know

[Winston]

Winston, why is it when you fix it, it's fast but then it starts going slow again? Why is it that you have to contact Bluehost to fix the problem all the time and open a "ticket"? Isn't there a control panel or something where you can fix the problem yourself? What is it that you're paying for?

Isn't there a better host? Surely some of the guys in IT on the forum would know

Because the hosting space keeps maxing out. I gotta change hosting and go back to Godaddy. Under shared hosting we had unlimited hosting space for a lot lower price. I have to contact Bluehost to fix it becaused the table of the posts crashed and could not be repaired. When I try to repair it from the hosting panel or phpmyadmin it just hangs and crashes, then I have to reboot the server. Only higher admins have the tools that can repair it. They use some type of DOS Shell or Command Prompt to fix it.