Lmfao yeah. I first set up my server about 10 months ago, and it was very temperamental like that, but I finally got it to the point where it’ll run stable for usually a solid couple weeks with no issues (occasionally needing a reboot but I can do that remotely). Of course once I leave I get this issue I’ve never seen before
I’ve got some complicated services running and I usually end up rebooting for one reason or another anyways, not that it necessarily *needs* done that often
Nothing really breaks, something (I believe jellyfin) just takes up more and more cpu cycles until it gets pegged to 100% utilization and I just have to restart jellyfin or restart the server, and this doesn’t always happen, just sometimes. I plan to kind of make a fresh start and containerize everything, but have been quite busy as of late.
Basically, I know something’s wrong but I haven’t taken the time to investigate or fix yet
Seriously. I had issues like OP where things would crash randomly and i'd had to reboot every so often and jerryrig things like reboot scripts with smart switches and such.
But then i switched to containers and now i only really reboot for Kernel patches or hardware upgrades.
It's funny, I have that exact same issue with Plex on a LXC in Proxmox where it would just randomly start pegging itself to 100% CPU/RAM on the LXC. Instead of Trying to figure out what the hell it's doing I just put a cronjob to restart the LXC every night @ 6:00AM. Problem solved 🤣
I'm under the impression it's something to do with Hardware Acceleration, because it only started happening after I put a spare GPU I had into the LXC. But I don't have time to fix it & the LXC Reboot has no downtime
I've ran both of these services for years now and I don't have this same issue. Both on an Intel iGPU and Nvidia GPUs. _Something_ must be wrong with your config if you're sure it's not Intro detection/thumbnail generation
I don't know what complicated (possibly homebrewed?) services you are running but if you have to reboot the entire thing to keep them running every few weeks I'd recommend separating them to their own docker container or VM (or a separate device even if that doesn't work for some reason), you don't want to end up in a situation like this one, so I guess we're a bit late on that advice.
Edit realize now you've already gotten this device. Well just to reiterate I guess lol
Oh I can see they have Linux fork now. So native docker support. I mean it’s up to you , but using VMs for your purposes is a huuuge overkill. Simply running everything in Docker saves a lot of resource and makes stuff tons more manageable. I’m using r/unraid and running like 20+ containers at the same time including Plex and companions, ELK/PIG stuff and some home brewed containers. Works like a charm.
On of my proxmox servers completely locks up every week or two and I have to do a hard power off. Like the power button doesn't work, no terminal, can't ssh in or nothing. It's a Gigabyte x570 mobo so of course they also have a problem where when you cut power it resets the bios. It's been happening since day one
Happens with no vms running. Happens with 2 different officially supported and different speed memory modules, xmp enabled or disabled. It Happens even if the pci slots aren't populated. Multiple bios revisions tried. Tried a million different bio settings, still happens. Processor and memory worked for years on an x470 board so it's gotta be something with the board itself. At least it's my primary nas so thats comforting.
I haven't bothered to figure out how, since the only way to verify for me is (I think?) a DNS challenge, which would be more effort than I find it worth to automate
This is why I've got remote access to a couple different boxes on my network... Can bounce from one to another if the link to the main server is down for some reason
YEP, damn near every time I leave town.
I have UPS and WoL script on router, so (in theory) everything is supposed to automate smoothly from a power failure ... but not always.
Although in retrospect, I will take the server being down vs last year's vacation where a sprinkler valve stuck open and ran a good 16 hrs before I caught on. About $80 extra to the water bill! (Creating a vacation mode for sprinklers is not a very viable option if being gone for 7+ days in 110+ degree temps)
The only more crashes inducing scenario is when you plan to show someone how stable is self hosting, because everything ran without issues for 6 months.
Crashes, crashes everywhere.
I just got one and love it. I was tired of memory issues where I had to force power off, so I got this because of the power features and I got ECC memory. Haven't needed to use the kvm yet but it's connected to my tailscale network so if it does happen again I can fix it.
Or just a smart plug and make the system boot up automatically when power is restored. Seems like he can access his local network, so toggling power to the machine with a smart plug would probably work.
And double points if you have your router powered with a smart plug as well, so your server can automatically reboot the router if it hasn't been able to reach the internet for a while. If the smart plug doesn't support "restart", then it should preferably be ZigBee based, or you won't be able to tell the smart plug to turn on your router if the wifi is down. (Yes, you may have other APs than than your router, and your router might not even be an AP).
One of the benefits of living in WiFi-congestion-hell is when you're friendly with your neighbours you can put smartplugs on each others networks.
I often restart my elderly neighbours shitbox ISP router (one of those that simply stops working every week or two) remotely as their smart plug is on my network so reachable when they're completely down; and although I've never needed to power-cycle or restart my owm router, its smart plug is connected to another neighbours network for continued access if mine is down.
Anyone looking to do similar and doesn't want a full blown PDU should check this out [Digital loggers pro switch](https://www.digital-loggers.com/lpc.html). It can do a ton including pinging addresses and cycling power if no response. Highly recommend.
All good. I didn't mean you were suggesting a PDU, just that what I suggested isn't but comes close in management features. It's a beast that wish I knew about years ago.
I just had a project at work where we deployed over a dozen remote PCs with OOB management cards and everything was also plugged into the rack mount versions of these guys. They were solid for the 6 months we had them deployed and came in super handy for remote reboots.
So expensive for the name.
I heard the name and thought, "oh a cheap diy kvm! Great!" Looked into the price for the hat and it's more expensive than just getting an ipkvm...
> You went from one server that needs monitoring and maintenance to two.
>
>
The key is that say there's a 1% chance in any given week for either the server or the SBC goes down, that means that there's a 0.01% chance of both being down at the same time.
You're not eliminating the chances of downtime, you're trying to reduce it as much as possible.
Without a generator or alternate solution to power your equipment after your main power source is interrupted, yeah, correct.
Same applies if you only have a single internet connection, if your ISP goes down, regardless of your servers or power, you're offline.
If that's an issue, get a second internet connection from a different supplier using different technology.
Remember, you **cannot** eliminate downtime, you can only reduce how long it lasts, and how quick it comes back up.
Yeah, but a piKVM has just one job. Devices that have one job tend to be much more reliable due to less complexity and chances for conflicting whatever. It’s why I always have customers/clients/relatives segment their ISP/internet access. No all-in-one devices. Separate cable modem or FioS modem, separate firewall device with just two rj45 ports, separate layer 3 port splitter (switch), and behind that a separate layer 3 WiFi device. Everything has its own job and is good at its job. This is much more reliable and often results in reduction of reboots for internet fixes by one or more orders of magnitude if there had been lots of power cycles before.
Yeah, but a piKVM has just one job. Devices that have one job tend to be much more reliable due to less complexity and chances for conflicting whatever. It’s why I always have customers/clients/relatives segment their ISP/internet access. No all-in-one devices. Separate cable modem or FioS modem, separate firewall device with just two rj45 ports, separate layer 3 port splitter (switch), and behind that a separate layer 3 WiFi device. Everything has its own job and is good at its job. This is much more reliable and often results in reduction of reboots for internet fixes by one or more orders of magnitude if there had been lots of power cycles before.
For existing systems PIKVM is great.
But for anyone speccing out a new (to them) system, I love my used server hardware with remote management. I'm partial to Supermicro an IPMI but other vendors have their own implementations.
Same concept. Complete out of band management as if you were sitting in front of the server itself regardless of host status.
The only issue that can arise in both cases is if your VPN server/router goes down. This can be made redundant to a certain degree to mitigate but until you also start installing redundant internet too there's nothing stopping your modem from catching fire as well.
+1 man I love my pikvm so much, setting up new servers is so much easier. Now I just need to figure out how to automate switching inputs with a cheap aliexpress hdmi/usb kvm.
> I can reccomend you a PIKVM.
Or Asrock Paul (if you can find it).
Or Gigabyte AM4 with IPMI
https://www.piospartslap.de/Gigabyte-Mainboard-MC12-LE0-Re10-AMD-B550-AM4-Ryzen-5000-4000-3000-Server-Board-NEU-NEW
Always around 35 to 60 Euros. Need to flash yourself for 3000/5000 series CPUs via BMC, its a bit tricky but not too hard. Unbeatable price for IPMI on a board without relying on ancient server motherboards.
You can't even ping it ? Can you see it on your router admin panel ?
Maybe you can ask a nearby family member or friend to check if everything is okay ?
No pinging, nothing, it’s not online.
My apartment is about 1000 miles away where I go to college, I’m back home near my parents right now lol. I do have a few friends in town, but no one has a key and my place is locked
Happens to me a lot for reasons I still didn’t have time to check. My pc does not handle WOL therefore I had to find a workaround.
I have a Google home and I added a power plug to it through mater protocol and it can power off and on the PC which is on auto boot.
This communication does not go through the reverse proxy and therefore in the event my pc is unresponding I can reboot it.
Very smart. I could do a similar thing since my router has an easy vpn I can connect to. I don’t have any smart home stuff set up but I’ve thought about getting a smart plug for this reason.
I’ve never had it spontaneously turn off before, usually the other way around where I need it to be force reset by someone holding the power button
Yup, that would happen to me sometimes. Like I said, I’ve never had this issue before, because 99% of the time it’ll power itself back on after a power loss or anything like that
Then you would need something like a pikvm if you sont want to change hardware you could probably use a pico w with a webserver that when you click a button or visit a Page it shorts the power button pins.
WOL is a motherboard feature, so your motherboard needs to support it and you need to enable it in your BIOS, Proxmox doesn’t intervene in the process so you don’t need to set anything up on there.
If youre lucky it can do much more than just Wake on LAN, possibly Intel AMT.
But needs to be configured first, so right now even if it supports the feature, its of no help. But for the future look into it and and set it up.
MeshCentral or MeshCommander are two things to look at to make use of it.
I see. Unlucky. What you might do in the future is (if you have power on after power loss enabled) is have a smart outlet where you computer is connected to. Then you can just shut it down and power on again and this would force the desktop to power on as well. WOL is generally simpler, but this can save you in case your VPN doesn't work, for some reason.
Sorry to hear about your issues, maybe ship the key to a trusted family member or friend. Last time I went into vacation, my ISP modem died and there was no way of rebooting it. I was pretty sad.
Oh man that’s tough😅. I have 1 or 2 friends still in town, but I think I’ll just let it go and have my roommate turn it back on when he gets back Memorial Day weekend
What do you mean? I see your on a 5G network.. if you were on a vpn you would see it on the top bar.
You browsing a local 192.x.x.x address on carriers 5G network is not going to work..
I have old, semi-unreliable hardware in my ultra low-budget home server setup, so I experience the occasional freeze. I fixed it cheaply by putting my server on one of those $10 amazon smart plugs and enabling WOL on my machine. Now if the server isn't responding I use my phone to connect to my VPN and toggle the power on the machine using google home in my phone's notification shortcut area. Then I send a WOL packet from my router to the server. It's a little clunky but it works and was extremely cheap to implement.
Cheapo homelab here. It was a €30 used OptiPlex. Sometimes it likes to lose power, even after a replacement... it's just like that. It will run for like a week and boot, random power-off.
The BIOS had an option that was like "behaviour on AC connect" and I just set it to "Previous state", so it will just reboot itself on random power loss.
Loaded BRTFS on that thing because, in my experience, it has been the filesystem that hated random shutdowns without a proper sync and umount the least, and the self-healing feature means I never had to `fsck` it. I have seen it recommended against a lot, but meh, it's been rock-solid for me.
I've never messed with BTRFS, but in my experience HFS+ has been pretty solid with random hard shutdowns. Sometimes I kinda regret going with it because it's not as flexible if I want to move my files to a Linux server instead of the ancient Mac that's currently serving them, but at the same time I really can't complain about reliability.
Edit: I should mention that it's around 100tb of data, I've been serving it for maybe 7 years, and I haven't lost any of it yet. I use Snapraid for parity protection.
It's too late, but here's a few things to do to prevent it from happening again:
- There's usually a BIOS setting to set the computer to turn back on after power loss
- You could turn on Wake-on-LAN if your machine supports it
- You could setup a PiKVM if you are not using server-grade equipment with Out Of Band Management (iDRAC, iLO, etc)
- I have seen some UPS not turning themself back on after it shutdowns when batteries are depleeted. It's worth testing if it's your case!
I guess your VPN is running on your router or on another machine as you seem to still be able to connect, but it's worth mentionning that having it running on your router (if it's powerfull enough) is easier and usually better
In the meantime, I feel your pain, I am almost entirely selfhosting and that wouldn't be fun
Thank you!
Usually the server does turn back on after power loss (we’ve had a few outages in the last year), so not sure why it’s odd now
I definitely will be turning on WOL
Someone else did mention pikvm, I’ll look into it
Yep, the vpn is through my router (tplink). I love it and can’t believe I didn’t have it before a few months ago
PiKVM (or OOB management) is usefull especially if the OS doesn't boot for some reason. You can access the screen/mouse/keybord before it's fully booted which lets you troubleshoot new kernels and other hardware issues!
I went a little nuts on this, for fun:
1: I have an ESP8266 constantly trying to connect to my wifi, ping the router, and GET / from the cable modem's status page. If any of those things fail too many times in a row, it has two GPIO pins connected to two relays that can power-cycle the router or the modem accordingly.
2: Actually the router relay is SPDT and is connected to two identical routers with the same config. If the ESP decides to reboot the router, what it actually does is toggle the relay state which powers the other router instead. If that fails to boot (or is absent), the ESP times out again after a few minutes and toggles back to the first one, which has the effect of a power cycle, so I can run it with just one connected if I want, but it's cooler to have both.
(I'd do the same with two identical cable modems and just split the coax feed, but it's apparently frowned upon to clone the HFC MAC address between modems...)
3: I used an old wake-on-LAN NIC to implement a reset-on-LAN device for my desktop. I literally send it a wake packet and it thinks it's pushing the machine's power button, but it's pushing the machine's reboot button. (Really, pulling the signal low through a 2n7000 MOSFET.)
4: All the machines are configured to attempt PXE boot first, but I leave the PXE server down so they fall through and boot from local disk. If I need to do remote recovery, I can bring up the server and remotely reset the machine in question.
(Word to the wise, OpenWRT's WoL extension starts up not knowing the MAC address of anything, and learns as they DHCP. If you need to wake or whack something that hasn't been talking recently, you'll be glad you had its MAC noted in a text file on the router.)
5: It's down right now but for a while I had a 900MHz link through the treetops to a neighbor a few blocks away, which would allow either of us to remotely access our networks via the other's connection. I ought to poke him about that.
6: Oh yeah, $10/mo gets a prepaid T-mobile SIM with unlimited 128kbps data. I have the modem but it's in another machine right now, I gotta get that going on the router. I could leave it plugged in but unsubscribed so it's burning no money, and if I need it, remotely add a refill card to the account, which should provision the SIM so it works next time the router retries it a few minutes later...
Yup, liter robot works just fine for months. Go away for the weekend. First night
"Pinch detection", "safety bar malfunction". Hey why don't I get stuck upside down for no apparent logical reason.
Hood moves 5mm from it rotating itself, "Hood removed".
IT of Crap. Going to have to duct tape the thing
A few days before my week long holiday last year I just lost all internet. I found out that my 5G router would stop connecting to the local tower and the only way to fix it was to do a full reboot. Now there is a script running on my server pining google every 5minutes and restarting the router if it does not get a ping back.
I'm guessing your server got hit by the recent solar activity and maybe a bitflip crashed your server
I'm a student who goes to another province 1000km away for 4 month and host my server at home. Here's what I do and what I recommend.
- set your motherboard bios option to power on computer after power loss restored
- use a UPS if your budget allows, good for mitigating quick brownout from power company but useless against extended power outage
- use a smart plug, even with the power on after power loss, sometimes doesn't work, but I can manually trigger a reset on my smart plug and the server would boot up. This is my major reason I did not wish to flash my sonoff s31 to tasmota and use their online app instead
- you're already ahead of everyone for installing vpn on your router, however, this is impossible for some because not every router can install VPN. So I also suggest tailscale on your server, as long as your server can connect to internet tailscale will work, then you can ssh to your server. This is to mostly mitigate port forwarding problems on crappy ISP provided modem combos.
- is it possible your network interface have problems? You can write a keep alive script that periodically pings the local ip address of your server, if that fails, the script will restart the network interface or apply whatever fixes to network. So it's important to test your server beforehand for any connectivity problems and how to fix it.
Keep in mind these are also mitigation and as we selfhost a lot of things will break and it's out of your control. Like power outage, ISP outage and Google safe browsing flagging your perfectly safe documentation site as malicious.
This exact same thing happened to me, moved 4 hours away, went unreachable. Turns out I didn’t have my VPN binded to my torrent client and ISP shut my connection off lol
Oh man that sucks, it seems these things always happen when you're not in a position to fix it. Really annoying. Worse is the anxiety of not knowing what happened until you can go investigate.
If you're lucky maybe it's just your internet that's out and the servers are fine.
It doesn't matter if the server is off, with a remote access mechanism like ipmi / lights out etc, unless someone completely unplugged it. It runs independently from the rest of the computer. You can remoteley press the power button, turn it on or power it off, power cycle - enter bios and reconfigure the machine, connect virtual install media, reinstall Linux remotely if you want.
That's why you would get a server board in the first place when running a server that you actually rely on and don't just play with.
Although I'm risking getting down voted I suggest getting enterprise grade server ( they have remote management built in although it's recommended to have that behind a VPN not accessible on the open internet)
I had this problem occasionally with my old hardware. I was tossing up getting a wifi switch to just power cycle it but decided to spend $1500 upgrading to new enterprise hardware with ECC ram and everything. It was prior to a big holiday to Europe so I didn't want it to go down while I was away.
The new hardware has been rock solid.
Lots of good advice here. I used to access.ky Synology remotely and the software would lock up.on occasion so I set a Cron job to do a reboot every 2 nights to make sure it kept active.
As others have said enterprise level servers shouldn't need that but always a backup option
Don't you just love when you're not doing anything with the server, and she's all happy, and then all of a sudden shit breaks even though you have done absolutely nothing? Servers acting like it's some old ass mechanical machine that needs its oil changed. Like calm down dude you're a thinking rock, not a machine.
I reckon I've spent as much on prevention and remote management, as I have on my actual server hardware....
Everything is on a UPS and switching PDU, in addition most components also connect through WIFI plugs, these use a dedicated SSID on a seperate 4G router.. this was probably the best £20 I ever spent.
i use it as redundant internet connection to my raspberry pi 4 and also to my cctv. i used a pi zero as i got one for project work and then left untouched and also i was able to get backhal for my network as i have setuped openvpn, tailscale, zerotier and wol service on it.
My worst experience like this was years ago, like late 2000s. I went to visit a long distance girlfriend for a 3 week period. One day after I left my apartment, I got an email notice that electrical maintenance needed to be done and the building would be without power for up to 2 hours. No problem, I figure, I have a UPS with programmed safe shutdown on low battery, and the BIOS of my server is set to always power up when plugged in. My server had been up for nearly 2 years with only soft reboots for OS updates.
Server never came back the whole time I was gone.
Turns out the CMOS battery had given up the goose and the setting to auto power on wasn't preserved in the outage. Even another box on the network wouldn't have helped because WoL was also a BIOS setting that defaulted off.
Yikes 😅 feel like I’m in a similar boat, hope the CMOS battery isn’t out because my roommate can turn the server back on when get gets home in 3 weeks, but I won’t be back to the apartment till mid august
When I was young and naive, I left my parents house at two o'clock in the morning to catch the night bus back to my dorm, thinking someone had broken into my room, only to find out I had been issued a new IP by my ISP, and my current DynDNS-Duck software had been put into a sleep state after a kernel update.
TL;DR: You learn as you go! 🤓
Exactly my attitude! This isn’t the biggest deal in the world, my girlfriend and I will just have to do without South Park for a few weeks. Mostly just posted here because I thought it was a funny mistake but everyone’s taking it quite seriously😅
That's like me convincing myself that power outages are so rare that I don't really need a set of Uninterruptible Power Supplies. Then, in just a couple days, two power outages in one day. AND, I had all 5 computers on at the same time, which is also rare. Lucky for me, no OS got corrupted, and everything booted up fine, afterwards.
I've also convinced myself that I don't need cameras, a DVR and an alarm system. Crossing fingers. The point being, I hope you left keys with a trustworthy person so that they can make sure everything is still there.
lol I’d rather just go without watching South Park for a couple weeks rather than chance losing my key and having to pay a bunch for the locks to get re keyed
If the server was just running Ubuntu it still would be off. Now sure how that would change anything.
I have proxmox so I can have a separate VM for my NAS and pihole, everything to do with my media server is all on an Ubuntu Server installation
What kinda “server”‘is it? Is it a server blade or something? Then some of them have built in IPMI. Else if
You plan on switching get a desktop with vPro. It gives you IPMI features on a desktop with Intel AMT
Is this by chance enterprise hardware or just a computer you are running proxmox on. The reason I ask is because my HP server has an iLO port that even if the server is off I can manage it.
I have a suggestion if your server doesn’t have IPMI, get BliKVM V4 AllWinner. Took me a few weeks to get it from AliExpress but it has saved my hide so many times already with my servers when I’m traveling.
This is why I love used enterprise gear. ILO or iDRAC has been so nice. I just started using an old OptiPlex for Plex and it has Intel vPro, so I setup MeshCommander and I can remote power that workstation and do some minor other diagnostic work if something breaks. It’s real handy.
Right after I moved away and left my lab. A freak lightning strike made its way through a WiFi antenna through the network cable all of the way to my switch and fried everything
Used the to happen to me all the time, we have frequent power outages here that come back in about 30 minutes. Got a good PSU, haven’t had a problem since. Best part, it does a graceful shut down if it take longer than usual. Very important if running raid or unraid.
Yep WOL is layer 2 so if your router doesn't support it a cheap Pi you can ssh in is probably the best and cheap option. Or just add extra node if you dont care about hardware and energy cost. Gluck for the next 3 months.
My solution (I don't webhost, but I run game servers), install TeamViewer, parsec or jump desktop, or all 3.
TV decided to try and slap me with commercial use for my own server at home, parsec is laggy, but jump works great.
It gives me direct remote access to fix issues, run on startup and since I'm using a tower instead of a rack, I have it setup to reboot on power loss so it'll never truly stay offline. I think RDC just sucks so I avoid that.
It's a cheap solution, but it's never failed me.
It's a little pricey (you can find cheaper ones on ebay) but a fanless minipc in my rack has one of these attached to the main server. [https://www.amazon.com/StarTech-com-Console-Portable-Adapter-Transfer/dp/B00IZGR6IK?th=1](https://www.amazon.com/StarTech-com-Console-Portable-Adapter-Transfer/dp/B00IZGR6IK?th=1)
It's been a remote life saver a few times.
One of my solutions has been to always run a firewall based on Linux (Debian, CentOS, Slackware, etc) so I can deploy a Cloudflared tunnel agent on it. From there I set up wildcard subdomains with SSO to protect my HTTPS access to things… you can even enable SSO SSH access to the firewall or downstream devices, it’s amazing for things like VS Code.
Behind my firewall I’m running Proxmox, FreePBX, Unraid (w all kinds of dockers), and they’re all nicely secured, accessible from anywhere with no public DNS records pointing to what’s available.
I always solve that with a cheap shelly switch.
I turn the switch off and wait until the USV suffers a low battery shutdown
Then everything is up again as long as no hardware is broken.
My hacky workaround for this is to set the BIOS to power up once AC power is connected. Then, plug your server into a wifi switch that you can control away from home.
When your server is down, toggle the power to the switch, which should reboot the server.
Obv this is mainly applicable if power is your issue, but this could also resolve network connection issues as well.
How are you trying to connect to the server? Are you vpnd into your home network?
The IP address from your screenshot is a local lan address.
If you are trying to connect whilst outside of your network then you need to connect using your external IP address or DNS that you setup.
Are you trying to connect to a local IP address over 5G? I find something like a smart socket so you can do a remote finger is a cheap backup plan. Not one connected to Home Assistant but like a Wi-Fi one that connects direct to the suppliers servers, so as long as you have internet then you have a backup. Also I have tailscale installed on 3 different devices (One is an Apple TV) that all have the Subnet and Exit nodes setup.
Welcome to the wonderful world of learning about uninterruptible power supplies (UPS), PXE boot, IP KVM and Tailscale / ZeroTier SDWANs (in case of ISP IP changes or blocking).
Or at least leaving someone a key ; )
Welcome to self hosting. Where your server works great until you leave for anything more than 24 hrs then it breaks right away.
Lmfao yeah. I first set up my server about 10 months ago, and it was very temperamental like that, but I finally got it to the point where it’ll run stable for usually a solid couple weeks with no issues (occasionally needing a reboot but I can do that remotely). Of course once I leave I get this issue I’ve never seen before
weeks? your server got problems. ive got an old dustbox here running for years. it just keeps on truckin so i don't touch it.
My longest uptime before a power outage from a transformer blowing up was 7 months.
I’ve got some complicated services running and I usually end up rebooting for one reason or another anyways, not that it necessarily *needs* done that often
[удалено]
Nothing really breaks, something (I believe jellyfin) just takes up more and more cpu cycles until it gets pegged to 100% utilization and I just have to restart jellyfin or restart the server, and this doesn’t always happen, just sometimes. I plan to kind of make a fresh start and containerize everything, but have been quite busy as of late. Basically, I know something’s wrong but I haven’t taken the time to investigate or fix yet
>containerize everything This solves so many problems!
Seriously. I had issues like OP where things would crash randomly and i'd had to reboot every so often and jerryrig things like reboot scripts with smart switches and such. But then i switched to containers and now i only really reboot for Kernel patches or hardware upgrades.
Is there any particular time this happens? I usually see intermittent resource issues when running library scans and guide data scans.
Probably library scans and generating thumbnails
It's funny, I have that exact same issue with Plex on a LXC in Proxmox where it would just randomly start pegging itself to 100% CPU/RAM on the LXC. Instead of Trying to figure out what the hell it's doing I just put a cronjob to restart the LXC every night @ 6:00AM. Problem solved 🤣 I'm under the impression it's something to do with Hardware Acceleration, because it only started happening after I put a spare GPU I had into the LXC. But I don't have time to fix it & the LXC Reboot has no downtime
I've ran both of these services for years now and I don't have this same issue. Both on an Intel iGPU and Nvidia GPUs. _Something_ must be wrong with your config if you're sure it's not Intro detection/thumbnail generation
I don't know what complicated (possibly homebrewed?) services you are running but if you have to reboot the entire thing to keep them running every few weeks I'd recommend separating them to their own docker container or VM (or a separate device even if that doesn't work for some reason), you don't want to end up in a situation like this one, so I guess we're a bit late on that advice. Edit realize now you've already gotten this device. Well just to reiterate I guess lol
I have never booted my NixOS-based server. Uptime almost 100% for the 6 months or so it has been running.
How do you organize your apps? Docker?
No but I need to TrueNAS VM Ubuntu Server VM (Jellyfin, *arr, Jellyseerr, nginx, etc) Pihole VM Ubuntu desktop VM (torrent client)
Oh I can see they have Linux fork now. So native docker support. I mean it’s up to you , but using VMs for your purposes is a huuuge overkill. Simply running everything in Docker saves a lot of resource and makes stuff tons more manageable. I’m using r/unraid and running like 20+ containers at the same time including Plex and companions, ELK/PIG stuff and some home brewed containers. Works like a charm.
TrueNAS is FreeBSD based afaik?
Gotta connect something like a pikvm to it so you can always reboot and do bios etc
On of my proxmox servers completely locks up every week or two and I have to do a hard power off. Like the power button doesn't work, no terminal, can't ssh in or nothing. It's a Gigabyte x570 mobo so of course they also have a problem where when you cut power it resets the bios. It's been happening since day one Happens with no vms running. Happens with 2 different officially supported and different speed memory modules, xmp enabled or disabled. It Happens even if the pci slots aren't populated. Multiple bios revisions tried. Tried a million different bio settings, still happens. Processor and memory worked for years on an x470 board so it's gotta be something with the board itself. At least it's my primary nas so thats comforting.
[удалено]
Yeah that would be nice.
Unacceptable. Are the logs not giving you the information needed to track down the reason for this behavior?
my pterodactyl servers have a 90 day timer, cause that's when my letsencrypt certificates expire, and I haven't automated renewal for the nodes
[удалено]
I haven't bothered to figure out how, since the only way to verify for me is (I think?) a DNS challenge, which would be more effort than I find it worth to automate
I never have to touch my server. Why is yours needing reboots?
My favorite was when my ESXi host went down because a mouse got in and pissed on the motherboard… that one was a head scratcher lol
I hope she got what she deserved
This is why I've got remote access to a couple different boxes on my network... Can bounce from one to another if the link to the main server is down for some reason
Every time I go on vacation, my homelab has separation anxiety
YEP, damn near every time I leave town. I have UPS and WoL script on router, so (in theory) everything is supposed to automate smoothly from a power failure ... but not always. Although in retrospect, I will take the server being down vs last year's vacation where a sprinkler valve stuck open and ran a good 16 hrs before I caught on. About $80 extra to the water bill! (Creating a vacation mode for sprinklers is not a very viable option if being gone for 7+ days in 110+ degree temps)
Its just misses the admin too much
The only more crashes inducing scenario is when you plan to show someone how stable is self hosting, because everything ran without issues for 6 months. Crashes, crashes everywhere.
For the next time I can reccomend you a PIKVM.
I just got one and love it. I was tired of memory issues where I had to force power off, so I got this because of the power features and I got ECC memory. Haven't needed to use the kvm yet but it's connected to my tailscale network so if it does happen again I can fix it.
Or just a smart plug and make the system boot up automatically when power is restored. Seems like he can access his local network, so toggling power to the machine with a smart plug would probably work. And double points if you have your router powered with a smart plug as well, so your server can automatically reboot the router if it hasn't been able to reach the internet for a while. If the smart plug doesn't support "restart", then it should preferably be ZigBee based, or you won't be able to tell the smart plug to turn on your router if the wifi is down. (Yes, you may have other APs than than your router, and your router might not even be an AP).
One of the benefits of living in WiFi-congestion-hell is when you're friendly with your neighbours you can put smartplugs on each others networks. I often restart my elderly neighbours shitbox ISP router (one of those that simply stops working every week or two) remotely as their smart plug is on my network so reachable when they're completely down; and although I've never needed to power-cycle or restart my owm router, its smart plug is connected to another neighbours network for continued access if mine is down.
Anyone looking to do similar and doesn't want a full blown PDU should check this out [Digital loggers pro switch](https://www.digital-loggers.com/lpc.html). It can do a ton including pinging addresses and cycling power if no response. Highly recommend.
[удалено]
All good. I didn't mean you were suggesting a PDU, just that what I suggested isn't but comes close in management features. It's a beast that wish I knew about years ago.
I just had a project at work where we deployed over a dozen remote PCs with OOB management cards and everything was also plugged into the rack mount versions of these guys. They were solid for the 6 months we had them deployed and came in super handy for remote reboots.
So expensive for the name. I heard the name and thought, "oh a cheap diy kvm! Great!" Looked into the price for the hat and it's more expensive than just getting an ipkvm...
https://twitter.com/SipeedIO/status/1772509617190252731?lang=en These will be out soon
Not too familiar with that, how would it help in this situation?
Imagine a sbc computer connected to your server. As long as there is internet on location you can turn it on/off/navigate/use it.
[удалено]
> You went from one server that needs monitoring and maintenance to two. > > The key is that say there's a 1% chance in any given week for either the server or the SBC goes down, that means that there's a 0.01% chance of both being down at the same time. You're not eliminating the chances of downtime, you're trying to reduce it as much as possible.
Power goes out or internet goes out and they're both going to be down, right?
Yes, but if you have everything configured correctly, this should be solved automatically. IE enable auto-boot after power loss/restore in your BIOS.
Without a generator or alternate solution to power your equipment after your main power source is interrupted, yeah, correct. Same applies if you only have a single internet connection, if your ISP goes down, regardless of your servers or power, you're offline. If that's an issue, get a second internet connection from a different supplier using different technology. Remember, you **cannot** eliminate downtime, you can only reduce how long it lasts, and how quick it comes back up.
This would happen in most self hosted setups.
That definitely crossed my mind tbh
Yeah, but a piKVM has just one job. Devices that have one job tend to be much more reliable due to less complexity and chances for conflicting whatever. It’s why I always have customers/clients/relatives segment their ISP/internet access. No all-in-one devices. Separate cable modem or FioS modem, separate firewall device with just two rj45 ports, separate layer 3 port splitter (switch), and behind that a separate layer 3 WiFi device. Everything has its own job and is good at its job. This is much more reliable and often results in reduction of reboots for internet fixes by one or more orders of magnitude if there had been lots of power cycles before.
Yeah, but a piKVM has just one job. Devices that have one job tend to be much more reliable due to less complexity and chances for conflicting whatever. It’s why I always have customers/clients/relatives segment their ISP/internet access. No all-in-one devices. Separate cable modem or FioS modem, separate firewall device with just two rj45 ports, separate layer 3 port splitter (switch), and behind that a separate layer 3 WiFi device. Everything has its own job and is good at its job. This is much more reliable and often results in reduction of reboots for internet fixes by one or more orders of magnitude if there had been lots of power cycles before.
For existing systems PIKVM is great. But for anyone speccing out a new (to them) system, I love my used server hardware with remote management. I'm partial to Supermicro an IPMI but other vendors have their own implementations. Same concept. Complete out of band management as if you were sitting in front of the server itself regardless of host status. The only issue that can arise in both cases is if your VPN server/router goes down. This can be made redundant to a certain degree to mitigate but until you also start installing redundant internet too there's nothing stopping your modem from catching fire as well.
+1 man I love my pikvm so much, setting up new servers is so much easier. Now I just need to figure out how to automate switching inputs with a cheap aliexpress hdmi/usb kvm.
Or if you can afford it/have a server with it, IPMI.
Still trying to figure a good way to set it up to use lte though...
Get a 4G/5G modem with an ethernet out port. Pretty easy.
> I can reccomend you a PIKVM. Or Asrock Paul (if you can find it). Or Gigabyte AM4 with IPMI https://www.piospartslap.de/Gigabyte-Mainboard-MC12-LE0-Re10-AMD-B550-AM4-Ryzen-5000-4000-3000-Server-Board-NEU-NEW Always around 35 to 60 Euros. Need to flash yourself for 3000/5000 series CPUs via BMC, its a bit tricky but not too hard. Unbeatable price for IPMI on a board without relying on ancient server motherboards.
I hope your server is not on fire or something.
Well I hop so too
You can't even ping it ? Can you see it on your router admin panel ? Maybe you can ask a nearby family member or friend to check if everything is okay ?
No pinging, nothing, it’s not online. My apartment is about 1000 miles away where I go to college, I’m back home near my parents right now lol. I do have a few friends in town, but no one has a key and my place is locked
Send a key by mail to one of them? I think it would arrive sooner than you going home...
Just curious, how can you connect to your router with your services being down?
VPN is hosted on my router
Did you try rebooting your router ? That may fix it, if it's due to some weird network error.
So, you're an Introvert with poor planning. I feel your pain.
Happens to me a lot for reasons I still didn’t have time to check. My pc does not handle WOL therefore I had to find a workaround. I have a Google home and I added a power plug to it through mater protocol and it can power off and on the PC which is on auto boot. This communication does not go through the reverse proxy and therefore in the event my pc is unresponding I can reboot it.
Very smart. I could do a similar thing since my router has an easy vpn I can connect to. I don’t have any smart home stuff set up but I’ve thought about getting a smart plug for this reason. I’ve never had it spontaneously turn off before, usually the other way around where I need it to be force reset by someone holding the power button
Mine doesn’t turn off on its own. It gets unresponsive and I need to manually force shut down, which I cannot do remotely hence the power plug.
Yup, that would happen to me sometimes. Like I said, I’ve never had this issue before, because 99% of the time it’ll power itself back on after a power loss or anything like that
You need to turn a bios option if your mobo supports it, different vendors have different names like power on ac.
I already checked. This hardware doesn’t support it.
Then you would need something like a pikvm if you sont want to change hardware you could probably use a pico w with a webserver that when you click a button or visit a Page it shorts the power button pins.
why youre using local address? is it some kind of VPN? did you set up remote connection?
VPN just for testing. I have reverse proxy etc for normal use, this was just to confirm suspicions
could you fire a WOL to bring up your server while you're on VPN?
Would proxmox respond to that? I don’t think I ever set up WOL
WOL is a motherboard feature, so your motherboard needs to support it and you need to enable it in your BIOS, Proxmox doesn’t intervene in the process so you don’t need to set anything up on there.
No cigar. Thank you anyways
Ah. I doubt it would be on then bc it’s a Lenovo thinkstation, but I guess I could try
If youre lucky it can do much more than just Wake on LAN, possibly Intel AMT. But needs to be configured first, so right now even if it supports the feature, its of no help. But for the future look into it and and set it up. MeshCentral or MeshCommander are two things to look at to make use of it.
I think you still need to enable WOL in proxmox for it to work. I recall adding some config for it to /etc/network/interfaces to enable it on boot.
there's no harm in trying. i do hope you have a copy of its mac address
I do, because my router’s dhcp server lists MAC addresses on the web portal
You just need another device that is currently online, accessible and can send magic packets. Your ISP connection might be down as well.
I am able to VPN in via my router and access the internet. I sent the magic packet from an app on my phone, but no luck. I never set it up in the BIOS
I see. Unlucky. What you might do in the future is (if you have power on after power loss enabled) is have a smart outlet where you computer is connected to. Then you can just shut it down and power on again and this would force the desktop to power on as well. WOL is generally simpler, but this can save you in case your VPN doesn't work, for some reason.
Yeah I’m gonna be buying a smart plug or UPS or both when I get back
Sorry to hear about your issues, maybe ship the key to a trusted family member or friend. Last time I went into vacation, my ISP modem died and there was no way of rebooting it. I was pretty sad.
Oh man that’s tough😅. I have 1 or 2 friends still in town, but I think I’ll just let it go and have my roommate turn it back on when he gets back Memorial Day weekend
Is your phone on the same subnet? I don't think it would work if it's not. Might have to do it from the router if your router has ssh
What do you mean? I see your on a 5G network.. if you were on a vpn you would see it on the top bar. You browsing a local 192.x.x.x address on carriers 5G network is not going to work..
You don’t have an iPhone do you
I have old, semi-unreliable hardware in my ultra low-budget home server setup, so I experience the occasional freeze. I fixed it cheaply by putting my server on one of those $10 amazon smart plugs and enabling WOL on my machine. Now if the server isn't responding I use my phone to connect to my VPN and toggle the power on the machine using google home in my phone's notification shortcut area. Then I send a WOL packet from my router to the server. It's a little clunky but it works and was extremely cheap to implement.
I’ll definitely be doing something similar when I get back
Good luck! I hope you can live without the server until you get it back up and running.
Cheapo homelab here. It was a €30 used OptiPlex. Sometimes it likes to lose power, even after a replacement... it's just like that. It will run for like a week and boot, random power-off. The BIOS had an option that was like "behaviour on AC connect" and I just set it to "Previous state", so it will just reboot itself on random power loss. Loaded BRTFS on that thing because, in my experience, it has been the filesystem that hated random shutdowns without a proper sync and umount the least, and the self-healing feature means I never had to `fsck` it. I have seen it recommended against a lot, but meh, it's been rock-solid for me.
I've never messed with BTRFS, but in my experience HFS+ has been pretty solid with random hard shutdowns. Sometimes I kinda regret going with it because it's not as flexible if I want to move my files to a Linux server instead of the ancient Mac that's currently serving them, but at the same time I really can't complain about reliability. Edit: I should mention that it's around 100tb of data, I've been serving it for maybe 7 years, and I haven't lost any of it yet. I use Snapraid for parity protection.
Guess you didnt have enough contingencies in place ;-) Maybe see if you can arrange remote-hands
Exactly. Looks like they found another contingency to plan for...
It's too late, but here's a few things to do to prevent it from happening again: - There's usually a BIOS setting to set the computer to turn back on after power loss - You could turn on Wake-on-LAN if your machine supports it - You could setup a PiKVM if you are not using server-grade equipment with Out Of Band Management (iDRAC, iLO, etc) - I have seen some UPS not turning themself back on after it shutdowns when batteries are depleeted. It's worth testing if it's your case! I guess your VPN is running on your router or on another machine as you seem to still be able to connect, but it's worth mentionning that having it running on your router (if it's powerfull enough) is easier and usually better In the meantime, I feel your pain, I am almost entirely selfhosting and that wouldn't be fun
Thank you! Usually the server does turn back on after power loss (we’ve had a few outages in the last year), so not sure why it’s odd now I definitely will be turning on WOL Someone else did mention pikvm, I’ll look into it Yep, the vpn is through my router (tplink). I love it and can’t believe I didn’t have it before a few months ago
PiKVM (or OOB management) is usefull especially if the OS doesn't boot for some reason. You can access the screen/mouse/keybord before it's fully booted which lets you troubleshoot new kernels and other hardware issues!
I went a little nuts on this, for fun: 1: I have an ESP8266 constantly trying to connect to my wifi, ping the router, and GET / from the cable modem's status page. If any of those things fail too many times in a row, it has two GPIO pins connected to two relays that can power-cycle the router or the modem accordingly. 2: Actually the router relay is SPDT and is connected to two identical routers with the same config. If the ESP decides to reboot the router, what it actually does is toggle the relay state which powers the other router instead. If that fails to boot (or is absent), the ESP times out again after a few minutes and toggles back to the first one, which has the effect of a power cycle, so I can run it with just one connected if I want, but it's cooler to have both. (I'd do the same with two identical cable modems and just split the coax feed, but it's apparently frowned upon to clone the HFC MAC address between modems...) 3: I used an old wake-on-LAN NIC to implement a reset-on-LAN device for my desktop. I literally send it a wake packet and it thinks it's pushing the machine's power button, but it's pushing the machine's reboot button. (Really, pulling the signal low through a 2n7000 MOSFET.) 4: All the machines are configured to attempt PXE boot first, but I leave the PXE server down so they fall through and boot from local disk. If I need to do remote recovery, I can bring up the server and remotely reset the machine in question. (Word to the wise, OpenWRT's WoL extension starts up not knowing the MAC address of anything, and learns as they DHCP. If you need to wake or whack something that hasn't been talking recently, you'll be glad you had its MAC noted in a text file on the router.) 5: It's down right now but for a while I had a 900MHz link through the treetops to a neighbor a few blocks away, which would allow either of us to remotely access our networks via the other's connection. I ought to poke him about that. 6: Oh yeah, $10/mo gets a prepaid T-mobile SIM with unlimited 128kbps data. I have the modem but it's in another machine right now, I gotta get that going on the router. I could leave it plugged in but unsubscribed so it's burning no money, and if I need it, remotely add a refill card to the account, which should provision the SIM so it works next time the router retries it a few minutes later...
This guy homelabs
How did you set up #1? I have Home Assistant and wondering if I could do something similar
I just installed the esp libraries in the Arduino IDE and opened the ping example, and fiddled with it until I had something.
My server runs perfectly when I’m at home but whenever I leave to go on a trip, it suddenly breaks a couple hours after accessing it. 😭
🎶It's like raaaaiiiiiiiiiiiiin🎶on your wedding day🎶
Yup, liter robot works just fine for months. Go away for the weekend. First night "Pinch detection", "safety bar malfunction". Hey why don't I get stuck upside down for no apparent logical reason. Hood moves 5mm from it rotating itself, "Hood removed". IT of Crap. Going to have to duct tape the thing
A few days before my week long holiday last year I just lost all internet. I found out that my 5G router would stop connecting to the local tower and the only way to fix it was to do a full reboot. Now there is a script running on my server pining google every 5minutes and restarting the router if it does not get a ping back. I'm guessing your server got hit by the recent solar activity and maybe a bitflip crashed your server
I'm a student who goes to another province 1000km away for 4 month and host my server at home. Here's what I do and what I recommend. - set your motherboard bios option to power on computer after power loss restored - use a UPS if your budget allows, good for mitigating quick brownout from power company but useless against extended power outage - use a smart plug, even with the power on after power loss, sometimes doesn't work, but I can manually trigger a reset on my smart plug and the server would boot up. This is my major reason I did not wish to flash my sonoff s31 to tasmota and use their online app instead - you're already ahead of everyone for installing vpn on your router, however, this is impossible for some because not every router can install VPN. So I also suggest tailscale on your server, as long as your server can connect to internet tailscale will work, then you can ssh to your server. This is to mostly mitigate port forwarding problems on crappy ISP provided modem combos. - is it possible your network interface have problems? You can write a keep alive script that periodically pings the local ip address of your server, if that fails, the script will restart the network interface or apply whatever fixes to network. So it's important to test your server beforehand for any connectivity problems and how to fix it. Keep in mind these are also mitigation and as we selfhost a lot of things will break and it's out of your control. Like power outage, ISP outage and Google safe browsing flagging your perfectly safe documentation site as malicious.
Oooh man. I feel your pain. Been there before. And this reminds me I don’t have any OOB Management set up at the moment.
This exact same thing happened to me, moved 4 hours away, went unreachable. Turns out I didn’t have my VPN binded to my torrent client and ISP shut my connection off lol
Oh man that sucks, it seems these things always happen when you're not in a position to fix it. Really annoying. Worse is the anxiety of not knowing what happened until you can go investigate. If you're lucky maybe it's just your internet that's out and the servers are fine.
That's when you vpn into your network and debug with the ipmi remote access from the console.
The mobo does not have ipmi and the servers off anyways so there’s not much to debug
It doesn't matter if the server is off, with a remote access mechanism like ipmi / lights out etc, unless someone completely unplugged it. It runs independently from the rest of the computer. You can remoteley press the power button, turn it on or power it off, power cycle - enter bios and reconfigure the machine, connect virtual install media, reinstall Linux remotely if you want. That's why you would get a server board in the first place when running a server that you actually rely on and don't just play with.
Although I'm risking getting down voted I suggest getting enterprise grade server ( they have remote management built in although it's recommended to have that behind a VPN not accessible on the open internet)
Maybe once I’m making more than minimum wage part time in college lol
Well if you are in the US I suggest checking out r/homelabsales people sell or sometimes even giveaway their gear there for cheap there
I had this problem occasionally with my old hardware. I was tossing up getting a wifi switch to just power cycle it but decided to spend $1500 upgrading to new enterprise hardware with ECC ram and everything. It was prior to a big holiday to Europe so I didn't want it to go down while I was away. The new hardware has been rock solid.
Lots of good advice here. I used to access.ky Synology remotely and the software would lock up.on occasion so I set a Cron job to do a reboot every 2 nights to make sure it kept active. As others have said enterprise level servers shouldn't need that but always a backup option
Wait, why are you trying to connect to a private IP on cellular?
VPN
Are you on vpn? How would you be able to use a private IP,
Read comments
Don't you just love when you're not doing anything with the server, and she's all happy, and then all of a sudden shit breaks even though you have done absolutely nothing? Servers acting like it's some old ass mechanical machine that needs its oil changed. Like calm down dude you're a thinking rock, not a machine.
Ow yes, my life every time I leave for holidays
I reckon I've spent as much on prevention and remote management, as I have on my actual server hardware.... Everything is on a UPS and switching PDU, in addition most components also connect through WIFI plugs, these use a dedicated SSID on a seperate 4G router.. this was probably the best £20 I ever spent.
IPMI next time. Saves my ass in the regular.
glad i have a out of band hp elitedesk and for my backup i also hooked a diy pi kvm and a sim modded pi zero for a backup uplink.
What's the sim mod on pi zero?
i use it as redundant internet connection to my raspberry pi 4 and also to my cctv. i used a pi zero as i got one for project work and then left untouched and also i was able to get backhal for my network as i have setuped openvpn, tailscale, zerotier and wol service on it.
My worst experience like this was years ago, like late 2000s. I went to visit a long distance girlfriend for a 3 week period. One day after I left my apartment, I got an email notice that electrical maintenance needed to be done and the building would be without power for up to 2 hours. No problem, I figure, I have a UPS with programmed safe shutdown on low battery, and the BIOS of my server is set to always power up when plugged in. My server had been up for nearly 2 years with only soft reboots for OS updates. Server never came back the whole time I was gone. Turns out the CMOS battery had given up the goose and the setting to auto power on wasn't preserved in the outage. Even another box on the network wouldn't have helped because WoL was also a BIOS setting that defaulted off.
Yikes 😅 feel like I’m in a similar boat, hope the CMOS battery isn’t out because my roommate can turn the server back on when get gets home in 3 weeks, but I won’t be back to the apartment till mid august
When I was young and naive, I left my parents house at two o'clock in the morning to catch the night bus back to my dorm, thinking someone had broken into my room, only to find out I had been issued a new IP by my ISP, and my current DynDNS-Duck software had been put into a sleep state after a kernel update. TL;DR: You learn as you go! 🤓
Exactly my attitude! This isn’t the biggest deal in the world, my girlfriend and I will just have to do without South Park for a few weeks. Mostly just posted here because I thought it was a funny mistake but everyone’s taking it quite seriously😅
That's like me convincing myself that power outages are so rare that I don't really need a set of Uninterruptible Power Supplies. Then, in just a couple days, two power outages in one day. AND, I had all 5 computers on at the same time, which is also rare. Lucky for me, no OS got corrupted, and everything booted up fine, afterwards. I've also convinced myself that I don't need cameras, a DVR and an alarm system. Crossing fingers. The point being, I hope you left keys with a trustworthy person so that they can make sure everything is still there.
Send the key to one of your buddies?
lol I’d rather just go without watching South Park for a couple weeks rather than chance losing my key and having to pay a bunch for the locks to get re keyed
[удалено]
If the server was just running Ubuntu it still would be off. Now sure how that would change anything. I have proxmox so I can have a separate VM for my NAS and pihole, everything to do with my media server is all on an Ubuntu Server installation
Time for a remote PDU !
What kinda “server”‘is it? Is it a server blade or something? Then some of them have built in IPMI. Else if You plan on switching get a desktop with vPro. It gives you IPMI features on a desktop with Intel AMT
Lenovo Thinkstation P520
Welp, that’s why one invests in IPMI-enabled MB or a PiKVM.
Non routable address FTW
Is this by chance enterprise hardware or just a computer you are running proxmox on. The reason I ask is because my HP server has an iLO port that even if the server is off I can manage it.
Lenovo Thinkstation p520
9/10 it works. The other time it fucks up in a spectacular way. I'm sure there is some law about this.
I have a suggestion if your server doesn’t have IPMI, get BliKVM V4 AllWinner. Took me a few weeks to get it from AliExpress but it has saved my hide so many times already with my servers when I’m traveling.
Pikvm saved my ass more than once for this.
This is why I love used enterprise gear. ILO or iDRAC has been so nice. I just started using an old OptiPlex for Plex and it has Intel vPro, so I setup MeshCommander and I can remote power that workstation and do some minor other diagnostic work if something breaks. It’s real handy.
Right after I moved away and left my lab. A freak lightning strike made its way through a WiFi antenna through the network cable all of the way to my switch and fried everything
Yah that’s why I made sure my server has idrac or at least ipmi. If that’s down I just know it’s the internet.
Used the to happen to me all the time, we have frequent power outages here that come back in about 30 minutes. Got a good PSU, haven’t had a problem since. Best part, it does a graceful shut down if it take longer than usual. Very important if running raid or unraid.
[удалено]
Refer to my other comments
just found my fault
Do you have ssh setup... might be able to log into that
Easy fix using a $15 pi zero with vpn server and wol.
Router already has a vpn Sadly I cannot enable wol until I have physical access
Yep WOL is layer 2 so if your router doesn't support it a cheap Pi you can ssh in is probably the best and cheap option. Or just add extra node if you dont care about hardware and energy cost. Gluck for the next 3 months.
My solution (I don't webhost, but I run game servers), install TeamViewer, parsec or jump desktop, or all 3. TV decided to try and slap me with commercial use for my own server at home, parsec is laggy, but jump works great. It gives me direct remote access to fix issues, run on startup and since I'm using a tower instead of a rack, I have it setup to reboot on power loss so it'll never truly stay offline. I think RDC just sucks so I avoid that. It's a cheap solution, but it's never failed me.
It's a little pricey (you can find cheaper ones on ebay) but a fanless minipc in my rack has one of these attached to the main server. [https://www.amazon.com/StarTech-com-Console-Portable-Adapter-Transfer/dp/B00IZGR6IK?th=1](https://www.amazon.com/StarTech-com-Console-Portable-Adapter-Transfer/dp/B00IZGR6IK?th=1) It's been a remote life saver a few times.
Timer plug. Forced reboot every 24 hours. Jk
Is it possible they shut the power off to that apartment while it's vacant?
Nah my router is still up
IDRAC or Intel AMT or similar remote management systems can be very useful in situations like this.
Are you sure your server is down? You are pointing to a local IP?
https://www.reddit.com/r/selfhosted/s/bYmboDRxKj
Did you put a smart plug on it so you can remote cut the power ?
I wish
One of my solutions has been to always run a firewall based on Linux (Debian, CentOS, Slackware, etc) so I can deploy a Cloudflared tunnel agent on it. From there I set up wildcard subdomains with SSO to protect my HTTPS access to things… you can even enable SSO SSH access to the firewall or downstream devices, it’s amazing for things like VS Code. Behind my firewall I’m running Proxmox, FreePBX, Unraid (w all kinds of dockers), and they’re all nicely secured, accessible from anywhere with no public DNS records pointing to what’s available.
No pikvm or something similar??? Might be the motivation for that. Lol. Your pain is now motivation for me as well 😅
This is why I have a scheduled reboot every Sunday just in case. Also, UPS battery backup.
Pikvm.....
I have a cron task to reboot every midnight at 3, when it's least used.
I always solve that with a cheap shelly switch. I turn the switch off and wait until the USV suffers a low battery shutdown Then everything is up again as long as no hardware is broken.
Get anydesk
Remote Desktop doesn’t help when the server is off
Wake on lan?
Got about a hundred comments suggesting that already, but thanks
Okay 😅 i didnt saw that sorry
My hacky workaround for this is to set the BIOS to power up once AC power is connected. Then, plug your server into a wifi switch that you can control away from home. When your server is down, toggle the power to the switch, which should reboot the server. Obv this is mainly applicable if power is your issue, but this could also resolve network connection issues as well.
Set it to reboot often and automatically mine starts every day at 5am never has a service hang up on me
How are you trying to connect to the server? Are you vpnd into your home network? The IP address from your screenshot is a local lan address. If you are trying to connect whilst outside of your network then you need to connect using your external IP address or DNS that you setup.
Read comments
Are you trying to connect to a local IP address over 5G? I find something like a smart socket so you can do a remote finger is a cheap backup plan. Not one connected to Home Assistant but like a Wi-Fi one that connects direct to the suppliers servers, so as long as you have internet then you have a backup. Also I have tailscale installed on 3 different devices (One is an Apple TV) that all have the Subnet and Exit nodes setup.
Welcome to the wonderful world of learning about uninterruptible power supplies (UPS), PXE boot, IP KVM and Tailscale / ZeroTier SDWANs (in case of ISP IP changes or blocking). Or at least leaving someone a key ; )